WO2024108552A1 - 虚拟模型的面部驱动方法、装置、设备及存储介质 - Google Patents

虚拟模型的面部驱动方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2024108552A1
WO2024108552A1 PCT/CN2022/134372 CN2022134372W WO2024108552A1 WO 2024108552 A1 WO2024108552 A1 WO 2024108552A1 CN 2022134372 W CN2022134372 W CN 2022134372W WO 2024108552 A1 WO2024108552 A1 WO 2024108552A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
transformation parameters
facial
key points
expression
Prior art date
Application number
PCT/CN2022/134372
Other languages
English (en)
French (fr)
Inventor
章腾飞
张树业
蔡海军
Original Assignee
广州酷狗计算机科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州酷狗计算机科技有限公司 filed Critical 广州酷狗计算机科技有限公司
Priority to PCT/CN2022/134372 priority Critical patent/WO2024108552A1/zh
Publication of WO2024108552A1 publication Critical patent/WO2024108552A1/zh

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00Animation
    • G06T13/203D [Three Dimensional] animation
    • G06T13/403D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings

Definitions

  • the present application relates to the fields of Internet and computers, and in particular to a facial driving method, device, equipment and storage medium for a virtual model.
  • a corresponding three-dimensional virtual model can be constructed based on the face image, and the three-dimensional virtual model can be adjusted according to the changes in facial expressions so that the three-dimensional virtual model and the face show the same expression changes.
  • a facial image is collected by a depth camera during live broadcast, and then the position information of the facial key points is obtained based on the facial image, and the depth information of the face is obtained based on the data recorded by the depth camera. Thereafter, based on the position information of the facial key points and the depth information of the face, the three-dimensional virtual model corresponding to the face is adjusted so that the three-dimensional virtual model presents the same expression changes as the face.
  • a depth camera needs to be used to collect face images to obtain depth information, and the collection of face images is inflexible.
  • the embodiment of the present application provides a virtual model face driving method, device, equipment and storage medium, which can improve the flexibility of two-dimensional face image acquisition and make the driving scheme of three-dimensional virtual model more versatile.
  • the technical solution is as follows:
  • an embodiment of the present application provides a facial driving method of a virtual model, the method comprising:
  • the two-dimensional face key points and the three-dimensional face key points in the three-dimensional face model iteratively update the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain final two-dimensional rigid transformation parameters and final expression transformation parameters; wherein the two-dimensional rigid transformation parameters are used to transform the facial posture of the three-dimensional face model, and the expression transformation parameters are used to transform the facial expression of the three-dimensional face model;
  • target rigid transformation parameters and target expression transformation parameters are determined; wherein the target rigid transformation parameters and target expression transformation parameters are used to drive the facial posture and facial expression of the three-dimensional virtual model so that the facial posture and facial expression of the three-dimensional virtual model match the facial posture and facial expression in the two-dimensional face image.
  • an embodiment of the present application provides a facial driving device for a virtual model, the device comprising:
  • a key point acquisition module is used to detect and obtain two-dimensional face key points from a two-dimensional face image
  • a parameter iteration module used for iteratively updating the two-dimensional rigid transformation parameters and the expression transformation parameters based on the two-dimensional face key points and the three-dimensional face key points in the three-dimensional face model, to obtain final two-dimensional rigid transformation parameters and final expression transformation parameters; wherein the two-dimensional rigid transformation parameters are used to transform the facial posture of the three-dimensional face model, and the expression transformation parameters are used to transform the facial expression of the three-dimensional face model;
  • a parameter determination module is used to determine target rigid transformation parameters and target expression transformation parameters based on the final two-dimensional rigid transformation parameters and the final expression transformation parameters; wherein the target rigid transformation parameters and the target expression transformation parameters are used to drive the facial posture and facial expression of the three-dimensional virtual model so that the facial posture and facial expression of the three-dimensional virtual model match the facial posture and facial expression in the two-dimensional face image.
  • an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the facial driving method of the virtual model.
  • an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored.
  • the computer program is loaded and executed by the processor to implement the facial driving method of the virtual model.
  • a computer program product is provided.
  • the computer device is enabled to execute the above-mentioned facial driving method of the virtual model.
  • the target rigid transformation parameters and the target expression transformation parameters for driving the facial posture and facial expression of the three-dimensional virtual model are determined.
  • the depth information in the two-dimensional face image that is, any image acquisition device can be used to acquire the two-dimensional face image, thereby improving the flexibility of two-dimensional face image acquisition.
  • it can be applied to the driving of the three-dimensional virtual model in different scenes, making the driving scheme of the three-dimensional virtual model highly universal.
  • iterative calculation is continuously performed to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters, and then the target rigid transformation parameters and the target expression transformation parameters are determined, thereby improving the accuracy of the target rigid transformation parameters and the target expression transformation parameters.
  • FIG1 is a schematic diagram of a facial driving system of a three-dimensional virtual model provided by an embodiment of the present application
  • FIG2 exemplarily shows a schematic diagram of a facial driving system of a three-dimensional virtual model
  • FIG3 is a flow chart of a facial driving method of a virtual model provided by an embodiment of the present application.
  • FIG4 is a flow chart of a facial driving method of a virtual model provided by another embodiment of the present application.
  • FIG5 is a block diagram of a facial driving device of a virtual model provided by one embodiment of the present application.
  • FIG6 is a block diagram of a facial driving device of a virtual model provided by another embodiment of the present application.
  • FIG. 7 is a structural block diagram of a computer device provided in one embodiment of the present application.
  • FIG1 shows a schematic diagram of a facial driving system of a three-dimensional virtual model provided by an embodiment of the present application.
  • the facial driving system of the three-dimensional virtual model includes: a terminal device 10 and a server 20 .
  • the terminal device 10 may be an electronic device such as a mobile phone, a tablet computer, a PC (Personal Computer), etc., which is not limited in the embodiments of the present application.
  • the terminal device 10 includes any application with a three-dimensional virtual model display function, such as a live broadcast application, a video application, a game application, etc., which is not limited in the embodiments of the present application.
  • the above-mentioned application may be an application that needs to be installed and downloaded, or it may be a point-and-click application, which is not limited in the embodiments of the present application.
  • Server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.
  • cloud services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.
  • the terminal device 10 and the server 20 communicate with each other via a network.
  • the server 20 is used to provide a facial driving service for a three-dimensional virtual model for the terminal device 10.
  • the terminal device 10 obtains a two-dimensional face image through an image acquisition device and sends the two-dimensional face image to the server 20.
  • the server 20 obtains two-dimensional face key points from the two-dimensional image, and divides the two-dimensional face key points into a first two-dimensional face key point set for rigid transformation parameter calculation, and a second two-dimensional face key point set for expression transformation parameter calculation.
  • the server 20 determines the two-dimensional rigid transformation parameters based on the first two-dimensional face key points and the first three-dimensional face key point set, and determines the expression transformation parameters based on the second two-dimensional face key points, the second three-dimensional face key point set and the two-dimensional rigid transformation parameters. Afterwards, when the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iteration stop condition, the server 20 determines the target rigid transformation parameters based on the final two-dimensional rigid transformation parameters obtained in the last iteration, determines the target expression transformation parameters based on the final expression transformation parameters obtained in the last iteration, and sends the target rigid transformation parameters and the target expression transformation parameters to the terminal device 10.
  • the terminal device 10 drives the facial posture of the three-dimensional virtual model according to the target rigid transformation parameters, and drives the facial expression of the three-dimensional virtual model according to the target expression transformation parameters, so that the facial posture and facial expression of the three-dimensional virtual model match the facial posture and facial expression in the two-dimensional face image.
  • the above-mentioned image acquisition device can be any device with an image acquisition function, such as a camera built into the terminal device 10.
  • the above description of the terminal device 10 and the server 20 is only exemplary and explanatory.
  • the functions of the terminal device 10 and the server 20 can be flexibly set and adjusted according to the actual situation.
  • the server 20 drives the facial posture of the three-dimensional virtual model according to the target rigid transformation parameters, and drives the facial expression of the three-dimensional virtual model according to the target expression transformation parameters to generate a three-dimensional virtual model image corresponding to the two-dimensional face image, and then the terminal device 10 directly displays the three-dimensional virtual model image.
  • FIG3 shows a flowchart of a facial driving method of a virtual model provided by an embodiment of the present application.
  • the method can be applied to a facial driving system of a three-dimensional virtual model, and the execution subject of each step can be the terminal device 10 and/or the server 20 (hereinafter collectively referred to as a computer device) in the embodiment of FIG1.
  • the method can include at least one of the following steps (301-303):
  • Step 301 Detect and obtain two-dimensional facial key points from a two-dimensional facial image.
  • a two-dimensional face image refers to an image containing a face by an image acquisition device.
  • the image acquisition device can be any device with an image acquisition function, and the embodiment of the present application does not limit this.
  • the image acquisition device is a camera built into the computer device, or the image acquisition device is other devices that communicate with the computer device through a network.
  • the above-mentioned two-dimensional face image is an image extracted from a video stream.
  • the computer device obtains a video stream through an image acquisition device, and then performs frame processing on the video stream to obtain at least one first candidate image, and performs face detection on each first candidate image respectively, and determines the image containing the face as the second candidate image, and then performs face posture detection on each second candidate image respectively, and determines the second candidate image whose face rotation angle meets the target condition as a two-dimensional face image.
  • the above-mentioned preset condition is that the rotation angle is less than a threshold value
  • the threshold value can be any value.
  • the threshold value can be flexibly set and adjusted according to actual conditions, and the embodiment of the present application does not limit this.
  • the computer device after acquiring the above-mentioned two-dimensional face image, the computer device detects and obtains the two-dimensional face key points from the two-dimensional face image.
  • the computer device can obtain the two-dimensional face key points through a pre-trained deep learning model, or can obtain the two-dimensional face key points through edge detection.
  • the computer device obtains the two-dimensional facial key points through a pre-trained deep learning model.
  • the computer device inputs the two-dimensional facial image into a pre-trained deep learning model to obtain the two-dimensional facial key points output by the deep learning model.
  • the computer device obtains the two-dimensional face key points through edge detection.
  • the computer device after obtaining the above-mentioned two-dimensional face image, the computer device obtains the region of interest from the two-dimensional face image, and performs grayscale processing, filtering processing, and edge detection processing on the region of interest to obtain an edge image corresponding to the two-dimensional face image, and then extracts the above-mentioned two-dimensional face key points from the edge image.
  • Step 302 based on the two-dimensional face key points and the three-dimensional face key points in the three-dimensional face model, iteratively update the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters.
  • a three-dimensional face model refers to a face model established in a three-dimensional coordinate system.
  • the three-dimensional face model includes an initial face model and a difference matrix.
  • the difference matrix is used to record the difference coordinates of other face models transformed based on the initial face model.
  • each column of data in the difference matrix is used to store the difference coordinates for the same face model, and each row of data in the difference matrix is used to store the difference coordinates for the same face key point.
  • an expressionless three-dimensional face model and at least two expressive three-dimensional face models are obtained, and the three-dimensional coordinates of the face key points in the expressionless three-dimensional face model and the three-dimensional coordinates of the face key points in each expressive three-dimensional face model are obtained. Further, the coordinate difference between the three-dimensional coordinates corresponding to each expressive three-dimensional face model and the three-dimensional coordinates corresponding to the expressionless three-dimensional face model is obtained.
  • the expressionless 3D face model is determined as the initial face model of the 3D face model, and the coordinate difference between the 3D coordinates corresponding to the expressive 3D face model and the 3D coordinates corresponding to the expressionless 3D face model is dimensionally expanded to obtain a one-dimensional difference matrix of the target expressive 3D face model, and then the one-dimensional difference matrix of the expressive 3D face model is used as a column of data, and a difference matrix of the 3D face model is constructed according to the one-dimensional difference matrices of each expressive 3D face model.
  • the above 3D face model is an ARkit model.
  • the computer device before calculating the target rigid transformation parameters and the target expression transformation parameters, obtains a three-dimensional face model and obtains three-dimensional face key points from the three-dimensional face model.
  • the three-dimensional face key points are face key points pre-marked in the three-dimensional face model.
  • the computer device after obtaining the two-dimensional face key points, iteratively updates the two-dimensional rigid transformation parameters and the expression transformation parameters based on the two-dimensional face key points and the three-dimensional face key points in the three-dimensional face model to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters.
  • the two-dimensional rigid transformation parameters are used to transform the facial posture of the three-dimensional face model
  • the expression transformation parameters are used to transform the facial expression of the three-dimensional face model.
  • the number of three-dimensional face key points is the same as the number of the two-dimensional face key points
  • the positions of the three-dimensional face key points in the three-dimensional face model correspond one-to-one to the positions of the two-dimensional face key points in the two-dimensional face image.
  • the computer device maps the two-dimensional face key points and the three-dimensional face key points to the same two-dimensional coordinate system, and then determines the two-dimensional rigid transformation parameters according to the difference between the two-dimensional coordinates of the two-dimensional face key points and the two-dimensional coordinates of the three-dimensional face key points, and then transforms the two-dimensional coordinates of the three-dimensional face key points according to the two-dimensional rigid transformation parameters to obtain the two-dimensional coordinates of the transformed three-dimensional face key points, and determines the expression transformation parameters according to the difference between the two-dimensional coordinates of the two-dimensional face key points and the two-dimensional coordinates of the transformed three-dimensional face key points.
  • the above-mentioned two-dimensional coordinate system can be the image coordinate system of the above-mentioned two-dimensional face image, or it can be a standard two-dimensional coordinate system constructed based on the above-mentioned three-dimensional coordinate system, which is not limited in the embodiments of the present application.
  • the above-mentioned two-dimensional coordinate system can also be called a two-dimensional normalized coordinate system.
  • the computer device determines whether the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update conditions. If the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update conditions, the two-dimensional rigid transformation parameters obtained in the last iteration are determined as the final two-dimensional rigid transformation parameters, and the expression transformation parameters obtained in the last iteration are determined as the final expression transformation parameters; if the two-dimensional rigid transformation parameters and the expression transformation parameters do not meet the iterative update conditions, the two-dimensional coordinates of the three-dimensional face key points are transformed based on the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain new two-dimensional coordinates of the three-dimensional face key points, and iterative calculations are continued based on the new two-dimensional coordinates of the three-dimensional face key points in combination with the two-dimensional coordinates of the above-mentioned two-dimensional face key points until the two-dimensional rigid transformation parameters
  • the above-mentioned iterative stopping condition includes at least one of the following: the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold, and the three-dimensional facial key points obtained based on the transformation of the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional facial key points in the same two-dimensional coordinate system.
  • the iteration stop condition includes: the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold.
  • the computer device after acquiring the two-dimensional rigid transformation parameters and the expression transformation parameters, the computer device records the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters, and then determines that the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update condition when the number of iterative updates reaches the threshold.
  • the above threshold can be any value, such as 3, 5, 9, etc., and the threshold can be flexibly set and adjusted according to actual conditions, and the embodiments of the present application do not limit this.
  • the iteration stop condition includes: the three-dimensional face key points obtained by transformation based on the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional face key points in the same two-dimensional coordinate system.
  • the computer device transforms the two-dimensional coordinates of the three-dimensional face key points based on the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain new two-dimensional coordinates of the three-dimensional face key points.
  • the two-dimensional coordinates of the new three-dimensional face key points are aligned with the two-dimensional coordinates of the two-dimensional face key points, it is determined that the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iteration update condition.
  • the iteration stop condition includes: the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold, or the three-dimensional face key points obtained based on the transformation of the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional face key points in the same two-dimensional coordinate system.
  • the computer device after acquiring the two-dimensional rigid transformation parameters and the expression transformation parameters, the computer device records the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters.
  • the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update condition; when the number of iterative updates does not reach the threshold, the two-dimensional coordinates of the three-dimensional face key points are transformed based on the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain new two-dimensional coordinates of the three-dimensional face key points.
  • the two-dimensional coordinates of the new three-dimensional face key points are aligned with the two-dimensional coordinates of the two-dimensional face key points, it is determined that the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update condition.
  • the iteration stop condition includes: the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold, and the three-dimensional face key points obtained based on the transformation of the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional face key points in the same two-dimensional coordinate system.
  • the computer device after obtaining the two-dimensional rigid transformation parameters and the expression transformation parameters, the computer device records the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters.
  • the two-dimensional coordinates of the three-dimensional face key points are transformed based on the two-dimensional rigid transformation parameters and the expression transformation parameters to obtain new two-dimensional coordinates of the three-dimensional face key points.
  • the two-dimensional coordinates of the new three-dimensional face key points are aligned with the two-dimensional coordinates of the two-dimensional face key points, it is determined that the two-dimensional rigid transformation parameters and the expression transformation parameters meet the iterative update condition.
  • Step 303 Determine target rigid transformation parameters and target expression transformation parameters based on the final two-dimensional rigid transformation parameters and the final expression transformation parameters.
  • the computer device obtains the above-mentioned final two-dimensional rigid transformation parameters and the above-mentioned final expression transformation parameters, and determines the target rigid transformation parameters and the target expression transformation parameters based on the final two-dimensional rigid transformation parameters and the final expression transformation parameters.
  • the above-mentioned target rigid transformation parameters and the above-mentioned target expression transformation parameters are used to drive the facial posture and facial expression of the three-dimensional virtual model, so that the facial posture and facial expression of the three-dimensional virtual model match the facial posture and facial expression in the two-dimensional face image.
  • the three-dimensional virtual model refers to a three-dimensional image corresponding to a face in a two-dimensional face image.
  • the three-dimensional virtual model can be a three-dimensional image constructed based on the facial features in the two-dimensional face image, or a pre-set three-dimensional image (such as a three-dimensional animal image, a three-dimensional cartoon image, etc.), which is not limited in the embodiments of the present application.
  • the computer device drives the three-dimensional virtual model for facial gestures based on the target rigid transformation parameters, and drives the three-dimensional virtual model for facial expressions based on the target expression transformation parameters, thereby displaying the three-dimensional virtual model that matches the facial gestures and facial expressions in the two-dimensional face image.
  • the server acquires the target rigid transformation parameters and the target expression transformation parameters
  • the terminal device drives the three-dimensional virtual model for facial gestures based on the target rigid transformation parameters, drives the three-dimensional virtual model for facial expressions based on the target expression transformation parameters, and displays the driven three-dimensional virtual model.
  • the server acquires the target rigid transformation parameters and the target expression transformation parameters, and drives the three-dimensional virtual model for facial gestures based on the target rigid transformation parameters, and drives the three-dimensional virtual model for facial expressions based on the target expression transformation parameters, so as to obtain a two-dimensional image corresponding to the driven three-dimensional virtual model, and then the terminal device displays the two-dimensional image corresponding to the three-dimensional virtual model.
  • the target rigid transformation parameters and the target expression transformation parameters for driving the facial posture and facial expression of the three-dimensional virtual model are determined through the two-dimensional face key points in the two-dimensional face image and the three-dimensional face key points in the three-dimensional face model.
  • any image acquisition device can be used to acquire the two-dimensional face image, thereby improving the flexibility of acquiring the two-dimensional face image.
  • the unified three-dimensional face model it can be applied to the driving of the three-dimensional virtual model in different scenes, so that the driving scheme of the three-dimensional virtual model has strong versatility.
  • iterative calculation is continuously performed to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters, and then the target rigid transformation parameters and the target expression transformation parameters are determined, thereby improving the accuracy of the target rigid transformation parameters and the target expression transformation parameters.
  • step 302 includes at least one of the following steps:
  • the two-dimensional rigid transformation parameters of the j+1th round are determined based on the position differences between the two-dimensional face key points and the three-dimensional face key points of the j-th round.
  • the computer device determines the two-dimensional rigid transformation parameters of the j+1th round based on the position difference between the two-dimensional face key points and the three-dimensional face key points of the j-th round.
  • j is an integer.
  • the computer device maps the two-dimensional face key points and the three-dimensional face key points of the j-th round to the same two-dimensional coordinate system, obtains the two-dimensional coordinates of the two-dimensional face key points and the two-dimensional coordinates of the three-dimensional face key points of the j-th round, and then determines the above-mentioned two-dimensional rigid transformation parameters of the j+1th round based on the coordinate difference (i.e., position difference) between the two-dimensional coordinates of the two-dimensional face key points and the two-dimensional coordinates of the three-dimensional face key points of the j-th round.
  • the coordinate difference i.e., position difference
  • the computer device when determining the two-dimensional rigid transformation parameters of the j+1th round, obtains the position information of the two-dimensional face key points and the three-dimensional face key points of the jth round in the same two-dimensional coordinate system, and obtains the position information of the two-dimensional face key points and the position information of the three-dimensional face key points of the jth round; further, the two-dimensional rigid transformation parameters of the j+1th round are calculated with the goal of minimizing the first objective function.
  • the first objective function is used to characterize the difference between the position information of the three-dimensional face key points of the jth round after the position information of the jth round is transformed using the two-dimensional rigid transformation parameters of the j+1th round and the position information of the two-dimensional face key points.
  • the first objective function is:
  • rt2D refers to the two-dimensional rigid transformation parameters of the j+1th round
  • kps3Drigid refers to the three-dimensional face key points in the jth round used to calculate the two-dimensional rigid transformation parameters
  • kps2Drigid refers to the two-dimensional face key points used to calculate the two-dimensional rigid transformation parameters.
  • the computer device determines the expression transformation parameters of the j+1th round based on the two-dimensional face key points, the three-dimensional face key points of the jth round, the two-dimensional rigid transformation parameters of the j+1th round, and the expression transformation parameters of the jth round.
  • the computer device after obtaining the two-dimensional rigid transformation parameters of the j+1th round, transforms the two-dimensional coordinates of the three-dimensional face key points of the jth round according to the two-dimensional rigid transformation parameters of the j+1th round, and then adjusts the expression transformation parameters of the jth round based on the coordinate difference (i.e., position difference) between the two-dimensional coordinates of the three-dimensional face key points of the jth round after the transformation and the two-dimensional face key points to determine the expression transformation parameters of the j+1th round.
  • the coordinate difference i.e., position difference
  • the computer device when determining the expression transformation parameters of the j+1th round, calculates the expression transformation parameters of the j+1th round with the goal of minimizing the second objective function.
  • the second objective function is used to characterize the difference between the position information of the three-dimensional face key points of the jth round after the position information is transformed using the two-dimensional rigid transformation parameters of the j+1th round and the expression transformation parameters of the j+1th round and the position information of the two-dimensional face key points, and the expression transformation parameters of the j+1th round are calculated based on the expression transformation parameters of the jth round.
  • the second objective function is:
  • rt2D refers to the two-dimensional rigid transformation parameters of the j+1th round
  • kps3Dexp refers to the three-dimensional face key points in the jth round used to calculate the expression transformation parameters
  • kps2Dexp refers to the two-dimensional face key points used to calculate the two-dimensional rigid transformation parameters
  • bs refers to the expression transformation parameters of the j+1th round.
  • the positions of the three-dimensional facial key points of the jth round are adjusted to obtain the three-dimensional facial key points of the j+1th round.
  • the computer device after obtaining the expression transformation parameters of the j+1th round, the computer device adjusts the positions of the three-dimensional facial key points of the jth round based on the expression transformation parameters of the j+1th round to obtain the three-dimensional facial key points of the j+1th round. In some embodiments, after obtaining the expression transformation parameters of the j+1th round, the computer device adjusts the two-dimensional coordinates of the three-dimensional facial key points of the jth round based on the expression transformation parameters of the j+1th round to determine the two-dimensional coordinates of the three-dimensional facial key points of the j+1th round.
  • the computer device determines whether the three-dimensional facial key points of the j+1th round meet the iteration stop condition.
  • the iteration stop condition includes at least one of the following: the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold, and the three-dimensional facial key points obtained by the transformation based on the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional facial key points in the same two-dimensional coordinate system.
  • the two-dimensional rigid transformation parameters of the j+1th round are determined as the final two-dimensional rigid transformation parameters, and the expression transformation parameters of the j+1th round are determined as the final expression transformation parameters.
  • the two-dimensional rigid transformation parameters and the expression transformation parameters are iteratively calculated through the two-dimensional face key points and the three-dimensional face key points until the three-dimensional face key points meet the iteration stop condition, thereby improving the accuracy of the final two-dimensional rigid transformation parameters and the final expression transformation parameters, thereby improving the driving accuracy of the three-dimensional virtual model, and improving the matching degree between the driven three-dimensional virtual model and the face in the two-dimensional face image.
  • the above step 303 includes at least one of the following:
  • the target rigid transformation parameters are obtained.
  • the final two-dimensional rigid transformation parameters include a first rotation parameter, a second rotation parameter, a first translation parameter, and a second translation parameter.
  • the first rotation parameter is used to characterize the rotation and scaling of the three-dimensional face model in the first direction of the three-dimensional coordinate system.
  • the first rotation parameter is the product of the component of the rotation matrix in the first direction and the scaling factor;
  • the second rotation parameter is used to characterize the rotation and scaling of the three-dimensional face model in the second direction of the three-dimensional coordinate system.
  • the second rotation parameter is the product of the component of the rotation matrix in the second direction and the scaling factor; the first translation parameter is used to characterize the translation of the three-dimensional face model in the first direction; the second translation parameter is used to characterize the translation of the three-dimensional face model in the second direction.
  • the final two-dimensional rigid transformation parameter rt2D is:
  • s refers to the scaling factor (scaling driving coefficient)
  • P refers to the orthogonal projection matrix
  • R refers to the rotation matrix (rotation driving coefficient)
  • t 2D refers to the translation amount of the three-dimensional face model in the two-dimensional coordinate system
  • X [x 1 , x 2 , x 3 ] refers to the first rotation parameter
  • t x refers to the first translation parameter
  • Y [y 1 , y 2 , y 3 ] refers to the second rotation parameter
  • t y refers to the second translation parameter.
  • the computer device determines the three-dimensional transformation parameters according to the final two-dimensional rigid transformation parameters.
  • the three-dimensional transformation parameters include a third rotation parameter and a third translation parameter.
  • the third rotation parameter is used to characterize the rotation amount and scaling amount of the three-dimensional face model in the third direction of the three-dimensional coordinate system
  • the third translation parameter is used to characterize the translation amount of the three-dimensional face model in the third direction.
  • the computer device determines the third rotation parameter based on the first rotation parameter and the second rotation parameter, and assigns the third translation parameter to zero.
  • the three-dimensional transformation parameters are:
  • Z refers to the third rotation parameter
  • t z refers to the third translation parameter
  • the computer device after obtaining the above-mentioned three-dimensional transformation parameters, obtains the target rigid transformation parameters based on the final two-dimensional transformation parameters and the three-dimensional transformation parameters.
  • the target rigid transformation parameters include a scaling driving coefficient, a translation driving coefficient, and a rotation driving coefficient.
  • the computer device determines the scaling driving coefficient of the three-dimensional face model based on the first rotation parameter and the second rotation parameter; determines the translation driving coefficient of the three-dimensional face model based on the first translation parameter, the second translation parameter, and the third translation parameter; determines the rotation driving coefficient of the three-dimensional face model based on the first rotation parameter, the second rotation parameter, and the third rotation parameter.
  • the target rigid transformation parameters are:
  • t refers to the translation driving coefficient of the 3D face model
  • Rt refers to the target rigid transformation parameter
  • the target rigid transformation parameters are calculated by matrix manifold projection, and the target rigid transformation parameters are directly determined based on the two-dimensional rigid transformation parameters in the two-dimensional coordinate system, thereby improving the calculation efficiency of the target rigid transformation parameters.
  • the above-mentioned two-dimensional facial key points include facial key points for iteratively updating the two-dimensional rigid transformation parameters, and facial key points for iteratively updating the expression transformation parameters.
  • FIG. 4 shows a flowchart of a facial driving method of a virtual model provided by another embodiment of the present application.
  • the method can be applied to a facial driving system of a three-dimensional virtual model, and the execution subject of each step can be the terminal device 10 and/or the server 20 (hereinafter collectively referred to as a computer device) in the embodiment of FIG. 1.
  • the method can include at least one of the following steps (401-406):
  • Step 401 determining a first two-dimensional face key point set and a second two-dimensional face key point set according to the two-dimensional face key points.
  • the computer device determines a first two-dimensional face key point set and a second two-dimensional face key point set according to the two-dimensional face key points.
  • the computer device divides the two-dimensional face key points into a first two-dimensional face key point set and a second two-dimensional face key point set based on the position of each two-dimensional face key point in the face.
  • the first two-dimensional face key point set includes key points that have a great influence on facial posture
  • the second two-dimensional face key point set includes key points that have a great influence on facial expression.
  • the two-dimensional face key points located in the facial contour of the face are determined as the first two-dimensional face key point set
  • the two-dimensional face key points located in the facial features of the face are determined as the second two-dimensional face key point set.
  • At least one of the same two-dimensional facial key points exists in the first two-dimensional facial key point set and the second two-dimensional facial key point set.
  • the two-dimensional facial key points in the nose have similar effects on facial posture and facial expression, the two-dimensional facial key points in the nose exist in both the first two-dimensional facial key point set and the second two-dimensional facial key point set.
  • Step 402 Determine the three-dimensional face key points in the three-dimensional face model that correspond to the first two-dimensional face key point set as the first three-dimensional face key point set.
  • the position of the two-dimensional face key points in the face corresponds to the position of the three-dimensional face key points in the three-dimensional face model.
  • the computer device determines the three-dimensional face key points corresponding to the first two-dimensional face key point set from the corresponding positions of the three-dimensional face model as the first three-dimensional face key point set based on the positions of the two-dimensional face key points in the face included in the first two-dimensional face key point set.
  • the first two-dimensional face key point set and the first three-dimensional face key point set are used to iteratively update the two-dimensional rigid transformation parameters.
  • Step 403 determine the three-dimensional face key points in the three-dimensional face model that correspond to the second two-dimensional face key point set as the second three-dimensional face key point set.
  • the computer device determines the three-dimensional face key points corresponding to the second two-dimensional face key point set from the corresponding positions of the three-dimensional face model as the second three-dimensional face key point set based on the positions of the two-dimensional face key points contained in the second two-dimensional face key point set in the face.
  • the second two-dimensional face key point set and the second three-dimensional face key point set are used to iteratively update the expression transformation parameters.
  • Step 404 Obtain two-dimensional rigid transformation parameters based on the first two-dimensional facial key points and the first three-dimensional facial key points.
  • the computer device after obtaining the first two-dimensional facial key points and the first three-dimensional facial key points, the computer device obtains two-dimensional rigid transformation parameters based on the position difference between the first two-dimensional facial key points and the first three-dimensional facial key points.
  • Step 405 obtaining expression transformation parameters based on the second two-dimensional facial key points, the second three-dimensional facial key points and the two-dimensional rigid transformation parameters.
  • the computer device after obtaining the above-mentioned two-dimensional rigid transformation parameters, transforms the three-dimensional facial key points according to the two-dimensional rigid transformation parameters, and then obtains the transformed second three-dimensional facial key points from the transformed three-dimensional facial key points, and then obtains the expression transformation parameters based on the position difference between the transformed second three-dimensional facial key points and the second two-dimensional facial key points.
  • Step 406 when the expression transformation parameters satisfy the iteration stop condition, the target rigid transformation parameters and the target expression transformation parameters are determined based on the two-dimensional rigid transformation parameters and the expression transformation parameters obtained in the last iteration.
  • the computer device transforms the second three-dimensional facial key points based on the expression transformation parameters to obtain updated three-dimensional facial key points, and then when the updated three-dimensional facial key points meet the iteration stop condition, it is determined that the expression transformation parameters meet the iteration stop condition, and then the two-dimensional rigid transformation parameters obtained in the last iteration are determined as the final two-dimensional rigid transformation parameters, and the expression transformation parameters obtained in the last iteration are determined as the final expression transformation parameters.
  • the target rigid transformation parameters are determined based on the final two-dimensional rigid transformation parameters, and the target expression transformation parameters are determined based on the final expression transformation parameters.
  • the updated three-dimensional facial key points do not meet the iteration stop condition, it is determined that the expression transformation parameters do not meet the iteration update condition, and then a new first three-dimensional facial key point set and a new second three-dimensional facial key point set are obtained from the updated three-dimensional facial key points, and the above steps are repeated until the updated three-dimensional facial key points meet the iteration stop condition.
  • Table 1 The overall acquisition process of target rigid transformation parameters and target expression transformation parameters
  • FIG5 shows a block diagram of a facial driving device for a virtual model provided by an embodiment of the present application.
  • the device has the function of implementing the facial driving method of the virtual model, and the function can be implemented by hardware or by hardware executing corresponding software.
  • the device can be a computer device or can be set in a computer device.
  • the device 500 can include: a key point acquisition module 510, a parameter iteration module 520 and a parameter determination module 530.
  • the key point acquisition module 510 is used to detect and obtain two-dimensional face key points from a two-dimensional face image.
  • the parameter iteration module 520 is used to iteratively update the two-dimensional rigid transformation parameters and the expression transformation parameters based on the two-dimensional facial key points and the three-dimensional facial key points in the three-dimensional facial model to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters; wherein the two-dimensional rigid transformation parameters are used to transform the facial posture of the three-dimensional facial model, and the expression transformation parameters are used to transform the facial expression of the three-dimensional facial model.
  • the parameter determination module 530 is used to determine the target rigid transformation parameters and the target expression transformation parameters based on the final two-dimensional rigid transformation parameters and the final expression transformation parameters; wherein the target rigid transformation parameters and the target expression transformation parameters are used to drive the facial posture and facial expression of the three-dimensional virtual model so that the facial posture and facial expression of the three-dimensional virtual model match the facial posture and facial expression in the two-dimensional face image.
  • the parameter iteration module 520 includes: a rigidity determination unit 521 , an expression determination unit 522 , a key point updating unit 523 , an iteration execution unit 524 and a parameter acquisition unit 525 .
  • the rigidity determination unit 521 is used to determine the two-dimensional rigid transformation parameters of the j+1th round based on the position difference between the two-dimensional face key points and the three-dimensional face key points of the jth round during the jth round of iteration, where j is an integer.
  • the expression determination unit 522 is used to determine the expression transformation parameters of the j+1th round based on the two-dimensional facial key points, the three-dimensional facial key points of the jth round, the two-dimensional rigid transformation parameters of the j+1th round and the expression transformation parameters of the jth round.
  • the key point updating unit 523 is used to adjust the positions of the three-dimensional facial key points of the j-th round based on the expression transformation parameters of the j+1-th round to obtain the three-dimensional facial key points of the j+1-th round.
  • the parameter acquisition unit 525 is used to determine the two-dimensional rigid transformation parameters of the j+1th round as the final two-dimensional rigid transformation parameters and the expression transformation parameters of the j+1th round as the final expression transformation parameters if the three-dimensional facial key points of the j+1th round meet the iteration stopping condition.
  • the rigidity determination unit 521 is further configured to:
  • the two-dimensional rigid transformation parameters of the j+1th round are calculated; wherein the first objective function is used to characterize the difference between the position information of the three-dimensional facial key points of the jth round after the position information is transformed by using the two-dimensional rigid transformation parameters of the j+1th round and the position information of the two-dimensional facial key points.
  • the expression determination unit 522 is further configured to:
  • the expression transformation parameters of the j+1th round are calculated; wherein the second objective function is used to characterize the difference between the position information of the three-dimensional facial key points of the jth round after the position information is transformed by using the two-dimensional rigid transformation parameters of the j+1th round and the expression transformation parameters of the j+1th round and the position information of the two-dimensional facial key points, and the expression transformation parameters of the j+1th round are calculated based on the expression transformation parameters of the jth round.
  • the iteration stop condition includes at least one of the following:
  • the number of iterative updates of the two-dimensional rigid transformation parameters and the expression transformation parameters reaches a threshold
  • the three-dimensional facial key points obtained based on the two-dimensional rigid transformation parameters and the expression transformation parameters are aligned with the two-dimensional facial key points in the same two-dimensional coordinate system.
  • the parameter determination module 530 is further configured to:
  • the final expression change parameters are determined as the target expression change parameters.
  • the final two-dimensional rigid transformation parameter includes a first rotation parameter, a second rotation parameter, a first translation parameter, and a second translation parameter, wherein the first rotation parameter is used to characterize the rotation amount and scaling amount of the three-dimensional face model in a first direction of the three-dimensional coordinate system, the second rotation parameter is used to characterize the rotation amount and scaling amount of the three-dimensional face model in a second direction of the three-dimensional coordinate system, the first translation parameter is used to characterize the translation amount of the three-dimensional face model in the first direction, and the second translation parameter is used to characterize the translation amount of the three-dimensional face model in the second direction; the parameter determination module 530 is further used to:
  • the target rigid transformation parameters include the scaling driving coefficient, the translation driving coefficient and the rotation driving coefficient.
  • the parameter determination module 530 is further configured to:
  • the three-dimensional transformation parameters include the third rotation parameter and the third translation parameter.
  • the key point acquisition module 510 is further used to:
  • the first two-dimensional facial key point set and the first three-dimensional facial key point set are used to iteratively update the two-dimensional rigid transformation parameters
  • the second two-dimensional facial key point set and the second three-dimensional facial key point set are used to iteratively update the expression transformation parameters.
  • the three-dimensional face model includes an initial face model and a difference matrix; wherein the difference matrix is used to record the difference coordinates transformed from the initial face model to other face models, each column of data in the difference matrix is used to store the difference coordinates for the same face model, and each row of data in the difference matrix is used to store the difference coordinates for the same face key point.
  • the target rigid transformation parameters and the target expression transformation parameters for driving the facial posture and facial expression of the three-dimensional virtual model are determined through the two-dimensional face key points in the two-dimensional face image and the three-dimensional face key points in the three-dimensional face model.
  • the depth information in the two-dimensional face image that is, any image acquisition device can be used to acquire the two-dimensional face image, thereby improving the flexibility of acquiring the two-dimensional face image.
  • the final two-dimensional rigid transformation parameters and the final expression transformation parameters are obtained by continuously iterating the calculation, and then the target rigid transformation parameters and the target expression transformation parameters are determined, thereby improving the accuracy of the target rigid transformation parameters and the target expression transformation parameters.
  • the device provided in the above embodiment when implementing its functions, only uses the division of the above functional modules as an example.
  • the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.
  • FIG. 7 shows a block diagram of a computer device 700 provided in one embodiment of the present application.
  • the computer device may be a vehicle-mounted terminal in a target vehicle, and the device may implement the facial driving method of the virtual model described above. Specifically:
  • the computer device 700 includes a processing unit (such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and an FPGA (Field Programmable Gate Array)) 701, a system memory 704 including a RAM (Random Access Memory) 702 and a ROM (Read Only Memory) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701.
  • the server 700 also includes a basic I/O system (Input/Output) 706 that helps transmit information between various devices in the computing server, and a large-capacity storage device 707 for storing an operating system 713, application programs 714 and other program modules 712.
  • the basic input/output system 706 includes a display 708 for displaying information and an input device 709 such as a mouse and a keyboard for user inputting information.
  • the display 708 and the input device 709 are connected to the central processing unit 701 through an input/output controller 710 connected to the system bus 705.
  • the basic input/output system 706 may also include an input/output controller 710 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus.
  • the input/output controller 710 also provides output to a display screen, a printer, or other types of output devices.
  • the mass storage device 707 is connected to the central processing unit 701 through a mass storage controller (not shown) connected to the system bus 705.
  • the mass storage device 707 and its associated computer readable medium provide non-volatile storage for the server 700. That is, the mass storage device 707 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.
  • the computer readable medium may include computer storage media and communication media.
  • Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage devices, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices.
  • RAM random access memory
  • ROM read only memory
  • EPROM Erasable Programmable Read Only Memory
  • EEPROM Electrical Erasable Programmable Read Only Memory
  • flash memory or other solid-state storage devices
  • CD-ROM Compact Disc
  • DVD Digital Video Disc
  • the server 700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 700 can be connected to the network 712 through the network interface unit 711 connected to the system bus 705, or the network interface unit 711 can be used to connect to other types of networks or remote computer systems (not shown).
  • the memory stores a computer program, which is loaded by the processor and implements the facial driving method of the virtual model.
  • a computer-readable storage medium in which a computer program is stored.
  • the computer program is executed by a processor, the facial driving method of the virtual model is implemented.
  • the computer readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or optical disk, etc.
  • the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).
  • a computer program product is also provided.
  • the computer program product is executed by a processor, it is used to implement the facial driving method of the virtual model.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请公开了一种虚拟模型的面部驱动方法、装置、设备及存储介质,属于互联网及计算机领域。所述方法包括:从二维人脸图像中检测得到二维人脸关键点;基于二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;基于最终的二维刚性变换参数和最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数。本申请实施例提供的技术方案中,提高了二维人脸图像获取的灵活性,使得三维虚拟模型的驱动方案具有较强的通用性,提高了目标刚性变换参数和目标表情变换参数的准确性。

Description

虚拟模型的面部驱动方法、装置、设备及存储介质 技术领域
本申请涉及互联网及计算机领域,特别涉及一种虚拟模型的面部驱动方法、装置、设备及存储介质。
背景技术
目前,在直播时可以根据人脸图像构建对应的三维虚拟模型,并根据人脸表情变换对三维虚拟模型进行调整,以使得三维虚拟模型与人脸呈现相同的表情变化。
在相关技术中,在直播时以深度摄像机收集的人脸图像,进而根据人脸图像获取人脸关键点的位置信息,并根据深度摄像机所记录的数据获取人脸的深度信息,之后,根据人脸关键点的位置信息和人脸的深度信息,对人脸对应的三维虚拟模型进行调整,以使得三维虚拟模型与该人脸呈现相同的表情变化。
然而,在上述相关技术中,需要采用深度摄像机收集人脸图像以获取深度信息,人脸图像的收集不灵活。
发明内容
本申请实施例提供了一种虚拟模型的面部驱动方法、装置、设备及存储介质,能够提高二维人脸图像获取的灵活性,使得三维虚拟模型的驱动方案具有较强的通用性。所述技术方案如下:
一方面,本申请实施例提供了一种虚拟模型的面部驱动方法,所述方法包括:
从二维人脸图像中检测得到二维人脸关键点;
基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;其中,所述二维刚性变换参数用于变换所述三维人脸模型的面部姿态,所述表情变换参数用于变换所述三维人脸模型的面部表情;
基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标 刚性变换参数和目标表情变换参数;其中,所述目标刚性变换参数和目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动,使得所述三维虚拟模型的面部姿态和面部表情与所述二维人脸图像中的面部姿态和面部表情相匹配。
另一方面,本申请实施例提供了一种虚拟模型的面部驱动装置,所述装置包括:
关键点获取模块,用于从二维人脸图像中检测得到二维人脸关键点;
参数迭代模块,用于基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;其中,所述二维刚性变换参数用于变换所述三维人脸模型的面部姿态,所述表情变换参数用于变换所述三维人脸模型的面部表情;
参数确定模块,用于基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数;其中,所述目标刚性变换参数和目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动,使得所述三维虚拟模型的面部姿态和面部表情与所述二维人脸图像中的面部姿态和面部表情相匹配。
再一方面,本申请实施例提供了一种计算机设备,所述计算机设备包括处理器和存储器,所述存储器中存储有计算机程序,所述计算机程序由所述处理器加载并执行以实现上述虚拟模型的面部驱动方法。
还一方面,本申请实施例提供了一种计算机可读存储介质,所述可读存储介质中存储有计算机程序,所述计算机程序由所述处理器加载并执行以实现上述虚拟模型的面部驱动方法。
还一方面,提供了一种计算机程序产品,当所述计算机程序产品在计算机设备上运行时,使得计算机设备执行上述虚拟模型的面部驱动方法。
本申请实施例提供的技术方案可以带来如下有益效果:
通过二维人脸图像中的二维人脸关键点和三维人脸模型中的三维人脸关键点,确定用于对三维虚拟模型进行面部姿态和面部表情进行驱动的目标刚性变换参数和目标表情变换参数,不需要记录二维人脸图像中的深度信息,即采集 任意图像采集设备获取二维人脸图像即可,提高二维人脸图像获取的灵活性,基于统一的三维人脸模型能够适用于不同场景中三维虚拟模型的驱动,使得三维虚拟模型的驱动方案具有较强的通用性;而且,不断迭代计算以获取最终的二维刚性变换参数和最终的表情变换参数,进而确定目标刚性变换参数和目标表情变换参数,提高了目标刚性变换参数和目标表情变换参数的准确性。
附图说明
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1是本申请一个实施例提供的三维虚拟模型的面部驱动系统的示意图;
图2示例性示出了一种三维虚拟模型的面部驱动系统的示意图;
图3是本申请一个实施例提供的虚拟模型的面部驱动方法的流程图;
图4是本申请另一个实施例提供的虚拟模型的面部驱动方法的流程图;
图5是本申请一个实施例提供的虚拟模型的面部驱动装置的框图;
图6是本申请另一个实施例提供的虚拟模型的面部驱动装置的框图;
图7是本申请一个实施例提供的计算机设备的结构框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚,下面将结合附图对本申请实施方式作进一步地详细描述。
请参考图1,其示出了本申请一个实施例提供的三维虚拟模型的面部驱动系统的示意图。该三维虚拟模型的面部驱动系统包括:终端设备10和服务器20。
终端设备10可以为诸如手机、平板电脑、PC(Personal Computer,个人计算机)等电子设备,本申请实施例对此不作限定。在一些实施例中,终端设备10中包括任意具有三维虚拟模型显示功能的应用程序,如直播应用程序、视频应用程序、游戏应用程序等,本申请实施例对此不作限定。示例性地,上述应用程序可以是需要安装下载的应用程序,也可以是即点即用的应用程序,本申 请实施例对此不作限定。
服务器20可以是独立的物理服务器,也可以是多个物理服务器构成的服务器集群或者分布式系统,还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN、以及大数据和人工智能平台等基础云计算服务的云服务器。
在一些实施例中,上述终端设备10和上述服务器20之间通过网络进行通信。
在本申请实施例中,服务器20用于为终端设备10提供针对三维虚拟模型的面部驱动服务。示例性地,如图2所示,终端设备10通过图像采集设备获取二维人脸图像并向服务器20发送该二维人脸图像,进一步地,由服务器20从二维图像中获取二维人脸关键点,并将该二维人脸关键点划分为用于刚性变换参数计算的第一二维人脸关键点集合,以及用于表情变换参数计算的第二二维人脸关键点集合。之后,服务器20基于第一二维人脸关键点和第一三维人脸关键点集合确定二维刚性变换参数,基于第二二维人脸关键点、第二三维人脸关键点集合和二维刚性变换参数确定表情变换参数。之后,服务器20在二维刚性变换参数和表情变换参数满足迭代停止条件的情况下,基于最后一次迭代获取的最终的二维刚性变换参数确定目标刚性变换参数,基于最后一次迭代获取的最终的表情变换参数确定目标表情变换参数,并向终端设备10发送该目标刚性变换参数和该目标表情变换参数。之后,终端设备10根据目标刚性变换参数对三维虚拟模型进行面部姿态驱动,并根据目标表情变换参数对三维虚拟模型进行面部表情驱动,使得三维虚拟模型的面部姿态和面部表情与二维人脸图像中的面部姿态和面部表情相匹配。其中,上述图像采集设备可以是任意具有图像采集功能的设备,如终端设备10中自带的摄像头。
需要说明的一点是,上述对终端设备10和服务器20的介绍只是示例性和解释性的,在示例性实施例中,可以根据实际情况对终端设备10和服务器20的功能进行灵活设置和调整。示例性地,由服务器20据目标刚性变换参数对三维虚拟模型进行面部姿态驱动,并根据目标表情变换参数对三维虚拟模型进行面部表情驱动,以生成二维人脸图像对应的三维虚拟模型图像,进而由终端设备10直接显示该三维虚拟模型图像。
请参考图3,其示出了本申请一个实施例提供的虚拟模型的面部驱动方法的流程图。该方法可应用于三维虚拟模型的面部驱动系统中,如各步骤的执行主体可以是图1实施例中的终端设备10和/或服务器20(以下统称为计算机设备)。该方法可以包括以下至少一个步骤(301~303):
步骤301,从二维人脸图像中检测得到二维人脸关键点。
二维人脸图像是指由图像采集设备包含人脸的图像。其中,图像采集设备可以为认为任意具有图像采集功能的设备,本申请实施例对此不作限定。示例性地,图像采集设备为计算机设备自带的摄像头,或者,图像采集设备为通过网络与计算机设备相互通信的其它设备。在一些实施例中,上述二维人脸图像是从视频流中提取得到的图像。示例性地,计算机设备通过图像采集设备获取视频流,进而对该视频流进行分帧处理,得到至少一个第一候选图像,并对各个第一候选图像分别进行人脸检测,将包含由人脸的图像确定为第二候选图像,之后,对各个第二候选图像分别进行人脸姿态检测,将人脸旋转角度满足目标条件的第二候选图像确定为二维人脸图像。示例性地,上述预设条件为旋转角度小于阈值,该阈值可以为任意数值,根据实际情况可以对该阈值进行灵活设置和调整,本申请实施例对此不作限定。
在本申请实施例中,计算机设备在获取上述二维人脸图像之后,从该二维人脸图像中检测得到二维人脸关键点。在一些实施例中,计算机设备可以通过预先训练得到的深度学习模型获取二维人脸关键点,也可以通过边缘检测获取二维人脸关键点。
在一种可能的实施方式中,计算机设备通过预先训练得到的深度学习模型获取二维人脸关键点。在一些实施例中,计算机设备在获取上述二维人脸图像之后,将该二维人脸图像输入至预先训练得到的深度学习模型,进而得到深度学习模型所输出的二维人脸关键点。
在另一种可能的实施方式中,计算机设备通过边缘检测获取二维人脸关键点。在一些实施例中,计算机设备在获取上述二维人脸图像之后,从该二维人脸图像中获取感兴趣区域,并将该感兴趣区域进行灰度处理、滤波处理、边缘检测处理,以获取二维人脸图像对应的边缘图像,进而从该边缘图像中提取得到上述二维人脸关键点。
步骤302,基于二维人脸关键点和三维人脸模型中的三维人脸关键点,对二 维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数。
三维人脸模型是指在三维坐标系中建立的人脸模型。在一些实施例中,三维人脸模型中包括初始人脸模型和差异矩阵。其中,差异矩阵用于记录以初始人脸模型为基准变换至其它人脸模型的差异坐标。在一些实施例中,为了提高目标刚性变换参数与目标表情变换参数的计算效率,差异矩阵中的每一列数据用于存储针对同一人脸模型的差异坐标,差异矩阵中的每一行数据用于存储针对同一人脸关键点的差异坐标。在一些实施例中,在构建上述三维人脸模型时,获取无表情三维人脸模型和至少两个有表情三维人脸模型,并获取无表情三维人脸模型中人脸关键点的三维坐标,以及各个有表情三维人脸模型中人脸关键点的三维坐标,进一步地,分别获取各个有表情三维人脸模型对应的三维坐标与无表情三维人脸模型对应的三维坐标之间的坐标差值。之后,将无表情三维人脸模型确定为三维人脸模型的初始人脸模型,对有表情三维人脸模型对应的三维坐标与无表情三维人脸模型对应的三维坐标之间的坐标差值进行维度展开,得到目标有表情三维人脸模型的一维差值矩阵,进而以有表情三维人脸模型的一维差值矩阵作为一列数据,根据各个有表情三维人脸模型的一维差值矩阵,构建三维人脸模型的差异矩阵。示例性地,上述三维人脸模型为ARkit模型。
在一些实施例中,计算机设备在对目标刚性变换参数和目标表情变换参数进行计算之前,获取三维人脸模型,并从三维人脸模型中获取三维人脸关键点。其中,上述三维人脸关键点是预先在三维人脸模型中标注的人脸关键点。在本申请实施例中,计算机设备在获取上述二维人脸关键点之后,基于二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数。其中,二维刚性变换参数用于变换三维人脸模型的面部姿态,表情变换参数用于变换三维人脸模型的面部表情。示例性地,三维人脸关键点的数量与上述二维人脸关键点的数量相同,且三维人脸关键点在三维人脸模型中的位置与二维人脸关键点在二维人脸图像中的位置一一对应。
在一些实施例中,计算机设备在获取上述二维人脸关键点和上述三维人脸关键点之后,将二维人脸关键点和三维人脸关键点映射至同一二维坐标系中, 进而根据二维人脸关键点的二维坐标和三维人脸关键点的二维坐标之间的差异,确定二维刚性变换参数,之后,根据二维刚性变换参数对三维人脸关键点的二维坐标进行变换,得到变换后的三维人脸关键点的二维坐标,并根据二维人脸关键点的二维坐标和变换后的三维人脸关键点的二维坐标之间的差异,确定表情变换参数。其中,上述二维坐标系可以为上述二维人脸图像的图像坐标系,也可以是基于上述三维坐标系所构建的标准二维坐标系,本申请实施例对此不作限定。示例性地,上述二维坐标系也可以称为二维归一化坐标系。
在一些实施例中,计算机设备在获取上述二维刚性变换参数和上述表情变换参数之后,判断二维刚性变换参数和表情变换参数是否满足迭代更新条件。在二维刚性变换参数和表情变换参数满足迭代更新条件的情况下,将最后一次迭代获取的二维刚性变换参数确定为最终的二维刚性变换参数,将最后一次迭代获取的表情变换参数确定为最终的表情变换参数;在二维刚性变换参数和表情变换参数未满足迭代更新条件的情况下,基于二维刚性变换参数和表情变换参数对三维人脸关键点的二维坐标进行变换,得到新的三维人脸关键点的二维坐标,并根据该新的三维人脸关键点的二维坐标,结合上述二维人脸关键点的二维坐标继续进行迭代计算,直至二维刚性变换参数和表情变换参数满足迭代更新条件。
在一些实施例中,上述迭代停止条件包括以下至少之一:二维刚性变换参数和表情变换参数的迭代更新次数达到阈值,基于二维刚性变换参数和表情变换参数变换得到的三维人脸关键点与二维人脸关键点在同一二维坐标系中对齐。
在一种可能的实施方式中,迭代停止条件包括:二维刚性变换参数和表情变换参数的迭代更新次数达到阈值。在一些实施例中,计算机设备在获取二维刚性变换参数和表情变换参数之后,记录本次二维刚性变换参数和表情变换参数的迭代更新次数,进而在迭代更新次数达到阈值的情况下,确定二维刚性变换参数和表情变换参数满足迭代更新条件。示例性地,上述阈值可以是任意数值,如3、5、9等,根据实际情况可以对该阈值进行灵活设置和调整,本申请实施例对此不作限定。
在另一种可能的实施方式中,迭代停止条件包括:基于二维刚性变换参数和表情变换参数变换得到的三维人脸关键点与二维人脸关键点在同一二维坐标 系中对齐。在一些实施例中,计算机设备在获取二维刚性变换参数和表情变换参数之后,基于二维刚性变换参数和表情变换参数对三维人脸关键点的二维坐标进行变换,得到新的三维人脸关键点的二维坐标,在该新的三维人脸关键点的二维坐标与二维人脸关键点的二维坐标对齐的情况下,确定二维刚性变换参数和表情变换参数满足迭代更新条件。
在再一种可能的实施方式中,迭代停止条件包括:二维刚性变换参数和表情变换参数的迭代更新次数达到阈值,或者,基于二维刚性变换参数和表情变换参数变换得到的三维人脸关键点与二维人脸关键点在同一二维坐标系中对齐。在一些实施例中,计算机设备在获取二维刚性变换参数和表情变换参数之后,记录本次二维刚性变换参数和表情变换参数的迭代更新次数。之后,在迭代更新次数达到阈值的情况下,确定二维刚性变换参数和表情变换参数满足迭代更新条件;在迭代更新次数未达到阈值的情况下,基于二维刚性变换参数和表情变换参数对三维人脸关键点的二维坐标进行变换,得到新的三维人脸关键点的二维坐标,在该新的三维人脸关键点的二维坐标与二维人脸关键点的二维坐标对齐的情况下,确定二维刚性变换参数和表情变换参数满足迭代更新条件。
当然,在其它可能的实施方式中,迭代停止条件包括:二维刚性变换参数和表情变换参数的迭代更新次数达到阈值,并且,基于二维刚性变换参数和表情变换参数变换得到的三维人脸关键点与二维人脸关键点在同一二维坐标系中对齐。在一些实施例中,计算机设备在获取二维刚性变换参数和表情变换参数之后,记录本次二维刚性变换参数和表情变换参数的迭代更新次数。之后,在迭代更新次数达到阈值的情况下,基于二维刚性变换参数和表情变换参数对三维人脸关键点的二维坐标进行变换,得到新的三维人脸关键点的二维坐标,在该新的三维人脸关键点的二维坐标与二维人脸关键点的二维坐标对齐的情况下,确定二维刚性变换参数和表情变换参数满足迭代更新条件。
步骤303,基于最终的二维刚性变换参数和最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数。
在本申请实施例中,计算机设备在获取上述最终的二维刚性变换参数和上述最终的表情变换参数,基于最终的二维刚性变换参数和最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数。其中,上述目标刚性变换参数和上述目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动, 使得三维虚拟模型的面部姿态和面部表情与二维人脸图像中的面部姿态和面部表情相匹配。
在一些实施例中,上述三维虚拟模型是指二维人脸图像中的人脸所对应的三维形象。示例性地,该三维虚拟模型可以是根据二维人脸图像中的人脸特征所构建的三维形象,也可以预先设定的三维形象(如动物三维形象、卡通三维形象等),本申请实施例对此不作限定。
在一些实施例中,计算机设备在获取目标刚性变换参数和目标表情变换参数之后,基于目标刚性变换参数对三维虚拟模型进行面部姿态驱动,基于目标表情变换参数对三维虚拟模型进行面部表情驱动,进而显示与二维人脸图像中的面部姿态和面部表情相匹配三维虚拟模型。在一种可能的实施方式中,由服务器获取目标刚性变换参数和目标表情变换参数,由终端设备基于目标刚性变换参数对三维虚拟模型进行面部姿态驱动,基于目标表情变换参数对三维虚拟模型进行面部表情驱动,并显示驱动后的三维虚拟模型。在另一种可能的实施方式中,由服务器获取目标刚性变换参数和目标表情变换参数,并基于目标刚性变换参数对三维虚拟模型进行面部姿态驱动,基于目标表情变换参数对三维虚拟模型进行面部表情驱动,以获取驱动后的三维虚拟模型对应的二维图像,进而由终端设备显示该三维虚拟模型对应的二维图像。
综上所述,本申请实施例提供的技术方案中,通过二维人脸图像中的二维人脸关键点和三维人脸模型中的三维人脸关键点,确定用于对三维虚拟模型进行面部姿态和面部表情进行驱动的目标刚性变换参数和目标表情变换参数,不需要记录二维人脸图像中的深度信息,即采集任意图像采集设备获取二维人脸图像即可,提高二维人脸图像获取的灵活性,基于统一的三维人脸模型能够适用于不同场景中三维虚拟模型的驱动,使得三维虚拟模型的驱动方案具有较强的通用性;而且,不断迭代计算以获取最终的二维刚性变换参数和最终的表情变换参数,进而确定目标刚性变换参数和目标表情变换参数,提高了目标刚性变换参数和目标表情变换参数的准确性。
下面,以第j轮迭代过程为例,对二维刚性变换参数和表情变换参数的迭代更新方式进行介绍。
在示例性实施例中,上述步骤302包括以下至少一个步骤:
1、在第j轮迭代过程中,基于二维人脸关键点和第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数。
在本申请实施例中,对于第j轮迭代过程,计算机设备基于二维人脸关键点和第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数。其中,j为整数。在一些实施例中,计算机设备在获取二维人脸关键点和第j轮的三维人脸关键点之后,将二维人脸关键点和第j轮的三维人脸关键点映射至同一二维坐标系,得到二维人脸关键点的二维坐标和第j轮的三维人脸关键点的二维坐标,进而基于二维人脸关键点的二维坐标和第j轮的三维人脸关键点的二维坐标之间的坐标差异(即位置差异),确定上述第j+1轮的二维刚性变换参数。
在一些实施例中,计算机设备在确定第j+1轮的二维刚性变换参数时,获取二维人脸关键点和第j轮的三维人脸关键点在同一个二维坐标系下的位置信息,得到二维人脸关键点的位置信息和第j轮的三维人脸关键点的位置信息;进一步地,以最小化第一目标函数为目标,计算得到第j+1轮的二维刚性变换参数。其中,第一目标函数用于表征采用第j+1轮的二维刚性变换参数对第j轮的三维人脸关键点的位置信息进行变换后,与二维人脸关键点的位置信息之间的差异。
示例性地,假设二维人脸关键点的数量为n,n为整数,上述第一目标函数为:
Figure PCTCN2022134372-appb-000001
其中,rt2D是指第j+1轮的二维刚性变换参数,kps3Drigid是指在用于计算二维刚性变换参数的第j轮三维人脸关键点,kps2Drigid是指用于计算二维刚性变换参数的二维人脸关键点。
2、基于二维人脸关键点、第j轮的三维人脸关键点、第j+1轮的二维刚性变换参数以及第j轮的表情变换参数,确定第j+1轮的表情变换参数。
在本申请实施例中,计算机设备在获取上述第j+1轮的二维刚性变换参数之后,基于二维人脸关键点、第j轮的三维人脸关键点、第j+1轮的二维刚性变换参数以及第j轮的表情变换参数,确定第j+1轮的表情变换参数。在一些实施例中,计算机设备在获取第j+1轮的二维刚性变换参数之后,根据该j+1轮的二维刚性变换参数对第j轮的三维人脸关键点的二维坐标进行变换,进而基于变换后的第j轮的三维人脸关键点的二维坐标与二维人脸关键之间的坐标差异(即位置差异),对第j轮的表情变换参数进行调整以确定第j+1轮的表情变换参数。
在一些实施例中,计算机设备在确定第j+1轮的表情变换参数时,以最小化第二目标函数为目标,计算得到第j+1轮的表情变换参数。其中,第二目标函数用于表征采用第j+1轮的二维刚性变换参数以及第j+1轮的表情变换参数对第j轮的三维人脸关键点的位置信息进行变换后,与二维人脸关键点的位置信息之间的差异,且第j+1轮的表情变换参数是基于第j轮的表情变换参数计算得到的。
示例性地,上述第二目标函数为:
Figure PCTCN2022134372-appb-000002
subjevt to 0≤bs[k]≤1,k=0,1,...,50
其中,rt2D是指第j+1轮的二维刚性变换参数,kps3Dexp是指在用于计算表情变换参数的第j轮三维人脸关键点,kps2Dexp是指用于计算二维刚性变换参数的二维人脸关键点,bs是指第j+1轮的表情变换参数。
3、基于第j+1轮的表情变换参数对第j轮的三维人脸关键点进行位置调整,得到第j+1轮的三维人脸关键点。
在本申请实施例中,计算机设备在获取上述第j+1轮的表情变换参数之后,基于第j+1轮的表情变换参数对第j轮的三维人脸关键点进行位置调整,得到第j+1轮的三维人脸关键点。在一些实施例中,计算机设备在获取上述第j+1轮的表情变换参数,基于该第j+1轮的表情变换参数对第j轮的三维人脸关键点的二维坐标进行调整,以确定第j+1轮的三维人脸关键点的二维坐标。
在一些实施例中,计算机设备在获取上述第j+1轮的三维人脸关键点之后,判断该第j+1轮的三维人脸关键点是否满足迭代停止条件。示例性地,迭代停止条件包括以下至少之一:二维刚性变换参数和表情变换参数的迭代更新次数达到阈值,基于二维刚性变换参数和表情变换参数变换得到的三维人脸关键点与二维人脸关键点在同一二维坐标系中对齐。
4、若第j+1轮的三维人脸关键点不满足迭代停止条件,则令j=j+1,并再次从基于二维人脸关键点和第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数的步骤开始执行。
5、若第j+1轮的三维人脸关键点满足迭代停止条件,则将第j+1轮的二维刚性变换参数确定为最终的二维刚性变换参数,将第j+1轮的表情变换参数确定为最终的表情变换参数。
综上所述,本申请实施例提供的技术方案中,通过二维人脸关键点和三维 人脸关键点对二维刚性变换参数和表情变换参数进行迭代计算,直至三维人脸关键点满足迭代停止条件,提高了最终的二维刚性变换参数和最终的表情变换参数的准确性,进而提高了三维虚拟模型的驱动准确性,提高了驱动后的三维虚拟模型与二维人脸图像中的人脸的匹配度。
下面,对目标刚性变换参数和目标表情变换参数的获取方式进行介绍。
在示例性实施例中,上述步骤303包括以下至少一项:
1、根据最终的二维刚性变换参数确定三维变换参数;
2、基于最终的二维变换参数和三维变换参数,得到目标刚性变换参数;
3、将最终的表情变换参数确定为目标表情变换参数。
在一些实施例中,最终的二维刚性变换参数包括第一旋转参数、第二旋转参数、第一平移参数和第二平移参数。其中,第一旋转参数用于表征三维人脸模型在三维坐标系的第一方向上的旋转量和缩放量,示例性地,第一旋转参数为旋转矩阵在第一方向上的分量与缩放系数的乘积;第二旋转参数用于表征三维人脸模型在三维坐标系的第二方向上的旋转量和缩放量,示例性地,第二旋转参数为旋转矩阵在第二方向上的分量与缩放系数的乘积;第一平移参数用于表征三维人脸模型在第一方向上的平移量;第二平移参数用于表征三维人脸模型在第二方向上的平移量。示例性地,最终的二维刚性变换参数rt2D为:
Figure PCTCN2022134372-appb-000003
其中,s是指缩放系数(缩放驱动系数),P是指正交投影矩阵,R是指旋转矩阵(旋转驱动系数),t 2D是指三维人脸模型在二维坐标系中的平移量,X=[x 1,x 2,x 3]是指第一旋转参数,t x是指第一平移参数,Y=[y 1,y 2,y 3]是指第二旋转参数,t y是指第二平移参数。
在本申请实施例中,计算机设备在获取上述最终的二维刚性变换参数之后,根据该最终的二维刚性变换参数确定三维变换参数。在一些实施例中,三维变换参数包括第三旋转参数和第三平移参数。其中,第三旋转参数用于表征三维人脸模型在三维坐标系的第三方向上的旋转量和缩放量,第三平移参数用于表征三维人脸模型在第三方向上的平移量。在一些实施例中,计算机设备根据第一旋转参数和第二旋转参数,确定第三旋转参数,并将第三平移参数赋值为零。示例性地,三维变换参数为:
Figure PCTCN2022134372-appb-000004
其中,Z是指第三旋转参数,t z是指第三平移参数。
在本申请实施例中,计算机设备在获取上述三维变换参数之后,基于最终的二维变换参数和三维变换参数,得到目标刚性变换参数。在一些实施例中,目标刚性变换参数中包括缩放驱动系数、平移驱动系数和旋转驱动系数。计算机设备根据第一旋转参数和第二旋转参数,确定三维人脸模型的缩放驱动系数;根据第一平移参数、第二平移参数和第三平移参数,确定三维人脸模型的平移驱动系数;根据第一旋转参数、第二旋转参数和第三旋转参数,确定三维人脸模型的旋转驱动系数。示例性地,目标刚性变换参数为:
Figure PCTCN2022134372-appb-000005
其中,t是指三维人脸模型的平移驱动系数,Rt是指目标刚性变换参数。
综上所述,本申请实施例提供的技术方案中,通过矩阵流形投影计算目标刚性变换参数,基于二维坐标系中的二维刚性变换参数直接确定目标刚性变换参数,提高目标刚性变换参数的计算效率。
在一些实施例中,为了提高目标刚性变换参数和目标表情变换参数的准确性,上述二维人脸关键点中包括用于对二维刚性变换参数进行迭代更新的人脸关键点,以及用于对表情变换参数进行迭代更新的人脸关键点。
请参考图4,其示出了本申请另一个实施例提供的虚拟模型的面部驱动方法的流程图。该方法可应用于三维虚拟模型的面部驱动系统中,如各步骤的执行主体可以是图1实施例中的终端设备10和/或服务器20(以下统称为计算机设备)。该方法可以包括以下至少一个步骤(401~406):
步骤401,根据二维人脸关键点确定第一二维人脸关键点集合和第二二维人脸关键点集合。
在本申请实施例中,计算机设备在获取上述二维人脸关键点之后,根据二维人脸关键点确定第一二维人脸关键点集合和第二二维人脸关键点集合。在一 些实施例中,计算机设备基于各个二维人脸关键点在人脸中的位置,将二维人脸关键点区分为第一二维人脸关键点集合和第二二维人脸关键点集合。其中,第一二维人脸关键点集合中包括对面部姿态影响大的关键点,第二二维人脸关键点集合中包括对面部表情影响大的关键点。示例性地,将位于人脸面部轮廓部位的二维人脸关键点确定为第一二维人脸关键点集合,将位于人脸面部五官部位的二维人脸关键点确定为第二二维人脸关键点集合。
在一些实施例中,第一二维人脸关键点集合和第二二维人脸关键点集合中,存在至少一个相同的二维人脸关键点。示例性地,由于鼻子部位的二维人脸关键点对面部姿态和面部表情的影响程度相似,因此,位于鼻子部位的二维人脸关键点既存在于第一二维人脸关键点集合中,又存在于第二二维人脸关键点集合中。
步骤402,将三维人脸模型中与第一二维人脸关键点集合相对应的三维人脸关键点,确定为第一三维人脸关键点集合。
在一些实施例中,二维人脸关键点在人脸中的位置与三维人脸关键点在三维人脸模型中的位置相对应。在本申请实施例中,计算机设备在获取上述第一二维人脸关键点之后,基于第一二维人脸关键点集合所包含的二维人脸关键点在人脸中的位置,从三维人脸模型相对应的位置处将与第一二维人脸关键点集合相对应的三维人脸关键点,确定为第一三维人脸关键点集合。其中,第一二维人脸关键点集合和第一三维人脸关键点集合用于对二维刚性变换参数进行迭代更新。
步骤403,将三维人脸模型中与第二二维人脸关键点集合相对应的三维人脸关键点,确定为第二三维人脸关键点集合。
在本申请实施例中,计算机设备在获取上述第二二维人脸关键点集合之后,基于第二二维人脸关键点集合所包含的二维人脸关键点在人脸中的位置,从三维人脸模型相对应的位置处将与第二二维人脸关键点集合相对应的三维人脸关键点,确定为第二三维人脸关键点集合。其中,第二二维人脸关键点集合和第二三维人脸关键点集合用于对表情变换参数进行迭代更新。
步骤404,基于第一二维人脸关键点和第一三维人脸关键点,得到二维刚性变换参数。
在一些实施例中,计算机设备在获取上述第一二维人脸关键点和上述第一 三维人脸关键点之后,基于第一二维人脸关键点和第一三维人脸关键点之间的位置差异,得到二维刚性变换参数。
步骤405,基于第二二维人脸关键点、第二三维人脸关键点和二维刚性变换参数,得到表情变换参数。
在一些实施例中,计算机设备在获取上述二维刚性变换参数之后,根据二维刚性变换参数对三维人脸关键点进行变换,进而从变换后的三维人脸关键点中获取变换后的第二三维人脸关键点,进而基于变换后的第二三维人脸关键点与第二二维人脸关键点之间的位置差异,得到表情变换参数。
步骤406,在表情变换参数满足迭代停止条件的情况下,基于最后一次迭代得到的二维刚性变换参数和表情变换参数,确定目标刚性变换参数和目标表情变换参数。
在一些实施例中,计算机设备在获取上述表情变换参数之后,基于该表情变换参数对第二三维人脸关键点进行变换,得到更新后的三维人脸关键点,进而在更新后的三维人脸关键点满足迭代停止条件时,确定表情变换参数满足迭代停止条件,进而将最后一次迭代得到的二维刚性变换参数确定为最终的二维刚性变换参数,将最后一次迭代得到的表情变换参数确定为最终的表情变换参数。之后,基于最终的二维刚性变换参数确定目标刚性变换参数,基于最终的表情变换参数确定目标表情变换参数。
在一些实施例中,若更新后的三维人脸关键点未满足迭代停止条件,则确定表情变换参数未满足迭代更新条件,进而从更新后的三维人脸关键点中获取新的第一三维人脸关键点集合和新的第二三维人脸关键点集合,并重复上述步骤直至更新后的三维人脸关键点满足迭代停止条件。
另外,目标刚性变换参数和目标表情变换参数的整体获取流程如下表一所示:
表一目标刚性变换参数和目标表情变换参数的整体获取流程
Figure PCTCN2022134372-appb-000006
Figure PCTCN2022134372-appb-000007
下述为本申请装置实施例,可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节,请参照本申请方法实施例。
请参考图5,其示出了本申请一个实施例提供的虚拟模型的面部驱动装置的框图。该装置具有实现上述虚拟模型的面部驱动方法的功能,所述功能可以由硬件实现,也可以由硬件执行相应的软件实现。该装置可以是计算机设备,也可以设置在计算机设备中。该装置500可以包括:关键点获取模块510、参数迭代模块520和参数确定模块530。
关键点获取模块510,用于从二维人脸图像中检测得到二维人脸关键点。
参数迭代模块520,用于基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;其中,所述二维刚性变换参数用于变换所述三维人脸模型的面部姿态,所述表情变换参数用于变换所述三维人脸模型的面部表情。
参数确定模块530,用于基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数;其中,所述目标刚性变换参数和目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动,使得所述三维虚拟模型的面部姿态和面部表情与所述二维人脸图像中的面部姿态和面部表情相匹配。
在示例性实施例中,如图6所示,所述参数迭代模块520,包括:刚性确定单元521、表情确定单元522、关键点更新单元523、迭代执行单元524和参数获取单元525。
刚性确定单元521,用于在第j轮迭代过程中,基于所述二维人脸关键点和所述第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数,j为整数。
表情确定单元522,用于基于所述二维人脸关键点、所述第j轮的三维人脸关键点、所述第j+1轮的二维刚性变换参数以及第j轮的表情变换参数,确定第j+1轮的表情变换参数。
关键点更新单元523,用于基于所述第j+1轮的表情变换参数对所述第j轮的三维人脸关键点进行位置调整,得到第j+1轮的三维人脸关键点。
迭代执行单元524,用于若所述第j+1轮的三维人脸关键点不满足迭代停止条件,则令j=j+1,并再次从所述基于所述二维人脸关键点和所述第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数的步骤开始执行。
参数获取单元525,用于若所述第j+1轮的三维人脸关键点满足所述迭代停止条件,则将所述第j+1轮的二维刚性变换参数确定为所述最终的二维刚性变换参数,将所述第j+1轮的表情变换参数确定为所述最终的表情变换参数。
在示例性实施例中,所述刚性确定单元521,还用于:
获取所述二维人脸关键点和所述第j轮的三维人脸关键点在同一个二维坐标系下的位置信息,得到所述二维人脸关键点的位置信息和所述第j轮的三维人脸关键点的位置信息;
以最小化第一目标函数为目标,计算得到所述第j+1轮的二维刚性变换参数;其中,所述第一目标函数用于表征采用所述第j+1轮的二维刚性变换参数对所述第j轮的三维人脸关键点的位置信息进行变换后,与所述二维人脸关键点的位置信息之间的差异。
在示例性实施例中,所述表情确定单元522,还用于:
以最小化第二目标函数为目标,计算得到所述第j+1轮的表情变换参数;其中,所述第二目标函数用于表征采用所述第j+1轮的二维刚性变换参数以及所述第j+1轮的表情变换参数对所述第j轮的三维人脸关键点的位置信息进行变换后,与所述二维人脸关键点的位置信息之间的差异,且所述第j+1轮的表情变换参数是基于所述第j轮的表情变换参数计算得到的。
在示例性实施例中,所述迭代停止条件包括以下至少之一:
所述二维刚性变换参数和所述表情变换参数的迭代更新次数达到阈值;
基于所述二维刚性变换参数和所述表情变换参数变换得到的三维人脸关键点与所述二维人脸关键点在同一二维坐标系中对齐。
在示例性实施例中,所述参数确定模块530,还用于:
根据所述最终的二维刚性变换参数确定三维变换参数;
基于所述最终的二维变换参数和所述三维变换参数,得到所述目标刚性变换参数;
将所述最终的表情变换参数确定为所述目标表情变换参数。
在示例性实施例中,所述最终的二维刚性变换参数包括第一旋转参数、第二旋转参数、第一平移参数和第二平移参数,所述第一旋转参数用于表征所述三维人脸模型在三维坐标系的第一方向上的旋转量和缩放量,所述第二旋转参数用于表征所述三维人脸模型在所述三维坐标系的第二方向上的旋转量和缩放量,所述第一平移参数用于表征所述三维人脸模型在所述第一方向上的平移量,所述第二平移参数用于表征所述三维人脸模型在所述第二方向上的平移量;所述参数确定模块530,还用于:
根据所述第一旋转参数和所述第二旋转参数,确定所述三维人脸模型的缩放驱动系数;
根据所述第一平移参数、第二平移参数和第三平移参数,确定所述三维人脸模型的平移驱动系数;
根据所述第一旋转参数、所述第二旋转参数和所述第三旋转参数,确定所述三维人脸模型的旋转驱动系数;
其中,所述目标刚性变换参数中包括所述缩放驱动系数、所述平移驱动系数和所述旋转驱动系数。
在示例性实施例中,所述参数确定模块530,还用于:
根据所述第一旋转参数和所述第二旋转参数,确定第三旋转参数,所述第三旋转参数用于表征所述三维人脸模型在所述三维坐标系的第三方向上的旋转量和缩放量;
将第三平移参数赋值为零,所述第三平移参数用于表征所述三维人脸模型在所述第三方向上的平移量;
其中,所述三维变换参数包括所述第三旋转参数和所述第三平移参数。
在示例性实施例中,所述关键点获取模块510,还用于:
根据所述二维人脸关键点确定第一二维人脸关键点集合和第二二维人脸关键点集合;其中,所述第一二维人脸关键点集合和所述第二二维人脸关键点集合中,存在至少一个相同的二维人脸关键点;
将所述三维人脸模型中与所述第一二维人脸关键点集合相对应的三维人脸关键点,确定为第一三维人脸关键点集合;
将所述三维人脸模型中与所述第二二维人脸关键点集合相对应的三维人脸关键点,确定为第二三维人脸关键点集合;
其中,所述第一二维人脸关键点集合和所述第一三维人脸关键点集合用于对所述二维刚性变换参数进行迭代更新,所述第二二维人脸关键点集合和所述第二三维人脸关键点集合用于对所述表情变换参数进行迭代更新。
在示例性实施例中,所述三维人脸模型中包括初始人脸模型和差异矩阵;其中,所述差异矩阵用于记录以所述初始人脸模型为基准变换至其它人脸模型的差异坐标,所述差异矩阵中的每一列数据用于存储针对同一人脸模型的差异坐标,所述差异矩阵中的每一行数据用于存储针对同一人脸关键点的差异坐标。
综上所述,本申请实施例提供的技术方案中,通过二维人脸图像中的二维人脸关键点和三维人脸模型中的三维人脸关键点,确定用于对三维虚拟模型进行面部姿态和面部表情进行驱动的目标刚性变换参数和目标表情变换参数,不需要记录二维人脸图像中的深度信息,即采集任意图像采集设备获取二维人脸图像即可,提高二维人脸图像获取的灵活性,基于统一的三维人脸模型能够适用于不同场景中三维虚拟模型的驱动,使得三维虚拟模型的驱动方案具有较强的通用性;而且,不断迭代计算以获取最终的二维刚性变换参数和最终的表情变换参数,进而确定目标刚性变换参数和目标表情变换参数,提高了目标刚性 变换参数和目标表情变换参数的准确性。
需要说明的是,上述实施例提供的装置,在实现其功能时,仅以上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将设备的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。另外,上述实施例提供的装置与方法实施例属于同一构思,其具体实现过程详见方法实施例,这里不再赘述。
请参考图7,其示出了本申请一个实施例提供的计算机设备700的结构框图。该计算机设备可以是目标车辆中的车载终端,该设备可以实现上述虚拟模型的面部驱动方法。具体来讲:
该计算机设备700包括处理单元(如CPU(Central Processing Unit,中央处理器)、GPU(Graphics Processing Unit,图形处理器)和FPGA(Field Programmable Gate Array,现场可编程逻辑门阵列)等)701、包括RAM(Random Access Memory,随机存取存储器)702和ROM(Read Only Memory,只读存储器)703的系统存储器704,以及连接系统存储器704和中央处理单元701的系统总线705。该服务器700还包括帮助计算服务器内的各个器件之间传输信息的基本I/O系统(Input/Output,输入/输出)706,和用于存储操作系统713、应用程序714和其他程序模块712的大容量存储设备707。
该基本输入/输出系统706包括有用于显示信息的显示器708和用于用户输入信息的诸如鼠标、键盘之类的输入设备709。其中,该显示器708和输入设备709都通过连接到系统总线705的输入输出控制器710连接到中央处理单元701。该基本输入/输出系统706还可以包括输入输出控制器710以用于接收和处理来自键盘、鼠标、或电子触控笔等多个其他设备的输入。类似地,输入输出控制器710还提供输出到显示屏、打印机或其他类型的输出设备。
该大容量存储设备707通过连接到系统总线705的大容量存储控制器(未示出)连接到中央处理单元701。该大容量存储设备707及其相关联的计算机可读介质为服务器700提供非易失性存储。也就是说,该大容量存储设备707可以包括诸如硬盘或者CD-ROM(Compact Disc Read-Only Memory,只读光盘)驱动器之类的计算机可读介质(未示出)。
不失一般性,该计算机可读介质可以包括计算机存储介质和通信介质。计算机存储介质包括以用于存储诸如计算机可读指令、数据结构、程序模块或其他数据等信息的任何方法或技术实现的易失性和非易失性、可移动和不可移动介质。计算机存储介质包括RAM、ROM、EPROM(Erasable Programmable Read Only Memory,可擦除可编程只读存储器)、EEPROM(Electrically Erasable Programmable Read Only Memory,电可擦可编程只读存储器)、闪存或其他固态存储设备,CD-ROM、DVD(Digital Video Disc,高密度数字视频光盘)或其他光学存储、磁带盒、磁带、磁盘存储或其他磁性存储设备。当然,本领域技术人员可知该计算机存储介质不局限于上述几种。上述的系统存储器704和大容量存储设备707可以统称为存储器。
根据本申请实施例,该服务器700还可以通过诸如因特网等网络连接到网络上的远程计算机运行。也即服务器700可以通过连接在该系统总线705上的网络接口单元711连接到网络712,或者说,也可以使用网络接口单元711来连接到其他类型的网络或远程计算机系统(未示出)。
该存储器中存储有计算机程序,该计算机程序由处理器加载并实现上述虚拟模型的面部驱动方法。
在示例性实施例中,还提供了一种计算机可读存储介质,所述存储介质中存储有计算机程序,所述计算机程序在被处理器执行时以实现上述虚拟模型的面部驱动方法。
可选地,该计算机可读存储介质可以包括:ROM(Read Only Memory,只读存储器)、RAM(Random Access Memory,随机存取记忆体)、SSD(Solid State Drives,固态硬盘)或光盘等。其中,随机存取记忆体可以包括ReRAM(Resistance Random Access Memory,电阻式随机存取记忆体)和DRAM(Dynamic Random Access Memory,动态随机存取存储器)。
在示例性实施例中,还提供一种计算机程序产品,所述计算机程序产品被处理器执行时,用于实现上述虚拟模型的面部驱动方法。
应当理解的是,在本文中提及的“多个”是指两个或两个以上。“和/或”,描 述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。另外,本文中描述的步骤编号,仅示例性示出了步骤间的一种可能的执行先后顺序,在一些其它实施例中,上述步骤也可以不按照编号顺序来执行,如两个不同编号的步骤同时执行,或者两个不同编号的步骤按照与图示相反的顺序执行,本申请实施例对此不作限定。
以上所述仅为本申请的示例性实施例,并不用以限制本申请,凡在本申请的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本申请的保护范围之内。

Claims (13)

  1. 一种三维虚拟模型的面部驱动方法,其特征在于,所述方法包括:
    从二维人脸图像中检测得到二维人脸关键点;
    基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;其中,所述二维刚性变换参数用于变换所述三维人脸模型的面部姿态,所述表情变换参数用于变换所述三维人脸模型的面部表情;
    基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数;其中,所述目标刚性变换参数和目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动,使得所述三维虚拟模型的面部姿态和面部表情与所述二维人脸图像中的面部姿态和面部表情相匹配。
  2. 根据权利要求1所述的方法,其特征在于,所述基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数,包括:
    在第j轮迭代过程中,基于所述二维人脸关键点和所述第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数,j为整数;
    基于所述二维人脸关键点、所述第j轮的三维人脸关键点、所述第j+1轮的二维刚性变换参数以及第j轮的表情变换参数,确定第j+1轮的表情变换参数;
    基于所述第j+1轮的表情变换参数对所述第j轮的三维人脸关键点进行位置调整,得到第j+1轮的三维人脸关键点;
    若所述第j+1轮的三维人脸关键点不满足迭代停止条件,则令j=j+1,并再次从所述基于所述二维人脸关键点和所述第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数的步骤开始执行;
    若所述第j+1轮的三维人脸关键点满足所述迭代停止条件,则将所述第j+1轮的二维刚性变换参数确定为所述最终的二维刚性变换参数,将所述第j+1轮的表情变换参数确定为所述最终的表情变换参数。
  3. 根据权利要求2所述的方法,其特征在于,所述基于所述二维人脸关键点 和所述第j轮的三维人脸关键点的位置差异,确定第j+1轮的二维刚性变换参数,包括:
    获取所述二维人脸关键点和所述第j轮的三维人脸关键点在同一个二维坐标系下的位置信息,得到所述二维人脸关键点的位置信息和所述第j轮的三维人脸关键点的位置信息;
    以最小化第一目标函数为目标,计算得到所述第j+1轮的二维刚性变换参数;其中,所述第一目标函数用于表征采用所述第j+1轮的二维刚性变换参数对所述第j轮的三维人脸关键点的位置信息进行变换后,与所述二维人脸关键点的位置信息之间的差异。
  4. 根据权利要求2所述的方法,其特征在于,所述基于所述二维人脸关键点、所述第j轮的三维人脸关键点、所述第j+1轮的二维刚性变换参数以及第j轮的表情变换参数,确定第j+1轮的表情变换参数,包括:
    以最小化第二目标函数为目标,计算得到所述第j+1轮的表情变换参数;其中,所述第二目标函数用于表征采用所述第j+1轮的二维刚性变换参数以及所述第j+1轮的表情变换参数对所述第j轮的三维人脸关键点的位置信息进行变换后,与所述二维人脸关键点的位置信息之间的差异,且所述第j+1轮的表情变换参数是基于所述第j轮的表情变换参数计算得到的。
  5. 根据权利要求2所述的方法,其特征在于,所述迭代停止条件包括以下至少之一:
    所述二维刚性变换参数和所述表情变换参数的迭代更新次数达到阈值;
    基于所述二维刚性变换参数和所述表情变换参数变换得到的三维人脸关键点与所述二维人脸关键点在同一二维坐标系中对齐。
  6. 根据权利要求1所述的方法,其特征在于,所述基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数,包括:
    根据所述最终的二维刚性变换参数确定三维变换参数;
    基于所述最终的二维变换参数和所述三维变换参数,得到所述目标刚性变 换参数;
    将所述最终的表情变换参数确定为所述目标表情变换参数。
  7. 根据权利要求6所述的方法,其特征在于,所述最终的二维刚性变换参数包括第一旋转参数、第二旋转参数、第一平移参数和第二平移参数,所述第一旋转参数用于表征所述三维人脸模型在三维坐标系的第一方向上的旋转量和缩放量,所述第二旋转参数用于表征所述三维人脸模型在所述三维坐标系的第二方向上的旋转量和缩放量,所述第一平移参数用于表征所述三维人脸模型在所述第一方向上的平移量,所述第二平移参数用于表征所述三维人脸模型在所述第二方向上的平移量;
    所述对所述最终的二维刚性变换参数进行三维转换,得到所述目标刚性变换参数,包括:
    根据所述第一旋转参数和所述第二旋转参数,确定所述三维人脸模型的缩放驱动系数;
    根据所述第一平移参数、第二平移参数和第三平移参数,确定所述三维人脸模型的平移驱动系数;
    根据所述第一旋转参数、所述第二旋转参数和所述第三旋转参数,确定所述三维人脸模型的旋转驱动系数;
    其中,所述目标刚性变换参数中包括所述缩放驱动系数、所述平移驱动系数和所述旋转驱动系数。
  8. 根据权利要求7所述的方法,其特征在于,所述根据所述最终的二维刚性变换参数确定三维变换参数,包括:
    根据所述第一旋转参数和所述第二旋转参数,确定第三旋转参数,所述第三旋转参数用于表征所述三维人脸模型在所述三维坐标系的第三方向上的旋转量和缩放量;
    将第三平移参数赋值为零,所述第三平移参数用于表征所述三维人脸模型在所述第三方向上的平移量;
    其中,所述三维变换参数包括所述第三旋转参数和所述第三平移参数。
  9. 根据权利要求1所述的方法,其特征在于,所述方法还包括:
    根据所述二维人脸关键点确定第一二维人脸关键点集合和第二二维人脸关键点集合;其中,所述第一二维人脸关键点集合和所述第二二维人脸关键点集合中,存在至少一个相同的二维人脸关键点;
    将所述三维人脸模型中与所述第一二维人脸关键点集合相对应的三维人脸关键点,确定为第一三维人脸关键点集合;
    将所述三维人脸模型中与所述第二二维人脸关键点集合相对应的三维人脸关键点,确定为第二三维人脸关键点集合;
    其中,所述第一二维人脸关键点集合和所述第一三维人脸关键点集合用于对所述二维刚性变换参数进行迭代更新,所述第二二维人脸关键点集合和所述第二三维人脸关键点集合用于对所述表情变换参数进行迭代更新。
  10. 根据权利要求1至9任一项所述的方法,其特征在于,所述三维人脸模型中包括初始人脸模型和差异矩阵;其中,所述差异矩阵用于记录以所述初始人脸模型为基准变换至其它人脸模型的差异坐标,所述差异矩阵中的每一列数据用于存储针对同一人脸模型的差异坐标,所述差异矩阵中的每一行数据用于存储针对同一人脸关键点的差异坐标。
  11. 一种虚拟模型的面部驱动装置,其特征在于,所述装置包括:
    关键点获取模块,用于从二维人脸图像中检测得到二维人脸关键点;
    参数迭代模块,用于基于所述二维人脸关键点和三维人脸模型中的三维人脸关键点,对二维刚性变换参数和表情变换参数进行迭代更新,得到最终的二维刚性变换参数和最终的表情变换参数;其中,所述二维刚性变换参数用于变换所述三维人脸模型的面部姿态,所述表情变换参数用于变换所述三维人脸模型的面部表情;
    参数确定模块,用于基于所述最终的二维刚性变换参数和所述最终的表情变换参数,确定目标刚性变换参数和目标表情变换参数;其中,所述目标刚性变换参数和目标表情变换参数用于对三维虚拟模型的面部姿态和面部表情进行驱动,使得所述三维虚拟模型的面部姿态和面部表情与所述二维人脸图像中的面部姿态和面部表情相匹配。
  12. 一种计算机设备,其特征在于,所述计算机设备包括处理器和存储器,所述存储器中存储有计算机程序,所述计算机程序由所述处理器加载并执行以实现如权利要求1至10任一项所述的虚拟模型的面部驱动方法。
  13. 一种计算机可读存储介质,其特征在于,所述存储介质中存储有计算机程序,所述计算机程序由处理器加载并执行以实现如权利要求1至10任一项所述的虚拟模型的面部驱动方法。
PCT/CN2022/134372 2022-11-25 2022-11-25 虚拟模型的面部驱动方法、装置、设备及存储介质 WO2024108552A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/134372 WO2024108552A1 (zh) 2022-11-25 2022-11-25 虚拟模型的面部驱动方法、装置、设备及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/134372 WO2024108552A1 (zh) 2022-11-25 2022-11-25 虚拟模型的面部驱动方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2024108552A1 true WO2024108552A1 (zh) 2024-05-30

Family

ID=91195004

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/134372 WO2024108552A1 (zh) 2022-11-25 2022-11-25 虚拟模型的面部驱动方法、装置、设备及存储介质

Country Status (1)

Country Link
WO (1) WO2024108552A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130121526A1 (en) * 2011-11-11 2013-05-16 Microsoft Corporation Computing 3d shape parameters for face animation
CN112906494A (zh) * 2021-01-27 2021-06-04 浙江大学 一种面部捕捉方法、装置、电子设备及存储介质
WO2022147736A1 (zh) * 2021-01-07 2022-07-14 广州视源电子科技股份有限公司 虚拟图像构建方法、装置、设备及存储介质
CN115330980A (zh) * 2022-08-16 2022-11-11 网易(杭州)网络有限公司 表情迁移方法、装置、电子设备及存储介质
CN115330947A (zh) * 2022-08-12 2022-11-11 百果园技术(新加坡)有限公司 三维人脸重建方法及其装置、设备、介质、产品

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130121526A1 (en) * 2011-11-11 2013-05-16 Microsoft Corporation Computing 3d shape parameters for face animation
WO2022147736A1 (zh) * 2021-01-07 2022-07-14 广州视源电子科技股份有限公司 虚拟图像构建方法、装置、设备及存储介质
CN112906494A (zh) * 2021-01-27 2021-06-04 浙江大学 一种面部捕捉方法、装置、电子设备及存储介质
CN115330947A (zh) * 2022-08-12 2022-11-11 百果园技术(新加坡)有限公司 三维人脸重建方法及其装置、设备、介质、产品
CN115330980A (zh) * 2022-08-16 2022-11-11 网易(杭州)网络有限公司 表情迁移方法、装置、电子设备及存储介质

Similar Documents

Publication Publication Date Title
US20160292900A1 (en) Image group processing and visualization
WO2023016271A1 (zh) 位姿确定方法、电子设备及可读存储介质
CN111667001B (zh) 目标重识别方法、装置、计算机设备和存储介质
CN111291768B (zh) 图像特征匹配方法及装置、设备、存储介质
JP2018504710A (ja) 位置特定およびマッピングの方法
CN112328715B (zh) 视觉定位方法及相关模型的训练方法及相关装置、设备
WO2022033111A1 (zh) 图像信息提取方法、训练方法及装置、介质和电子设备
CN109325996B (zh) 用于生成信息的方法和装置
CN110648363A (zh) 相机姿态确定方法、装置、存储介质及电子设备
US20170323149A1 (en) Rotation invariant object detection
CN110956131B (zh) 单目标追踪方法、装置及系统
CN111639147B (zh) 一种地图压缩方法、系统及计算机可读存储介质
CN111784776A (zh) 视觉定位方法及装置、计算机可读介质和电子设备
WO2023016182A1 (zh) 位姿确定方法、装置、电子设备和可读存储介质
CN116563493A (zh) 基于三维重建的模型训练方法、三维重建方法及装置
CN114241141B (zh) 光滑物体三维重建方法、装置、计算机设备和存储介质
US20220375164A1 (en) Method and apparatus for three dimensional reconstruction, electronic device and storage medium
CN113516750B (zh) 三维点云地图构建方法、系统、电子设备及存储介质
CN111368860B (zh) 重定位方法及终端设备
CN110851639A (zh) 一种以图搜图的方法及设备
WO2024108552A1 (zh) 虚拟模型的面部驱动方法、装置、设备及存储介质
CN114742967B (zh) 一种基于建筑数字孪生语义图的视觉定位方法及装置
CN118318252A (zh) 虚拟模型的面部驱动方法、装置、设备及存储介质
Yang et al. Simultaneous pose and correspondence estimation based on genetic algorithm
CN115223110B (zh) 目标检测方法、装置、计算机设备和存储介质