WO2022135518A1

WO2022135518A1 - Eyeball registration method and apparatus based on three-dimensional cartoon model, and server and medium

Info

Publication number: WO2022135518A1
Application number: PCT/CN2021/140798
Authority: WO
Inventors: 章菲倩
Original assignee: 百果园技术(新加坡)有限公司; 章菲倩
Priority date: 2020-12-25
Filing date: 2021-12-23
Publication date: 2022-06-30
Also published as: CN112581518A

Abstract

Provided are an eyeball registration method and apparatus based on a three-dimensional cartoon model, and a server and a medium. The eyeball registration method based on a three-dimensional cartoon model comprises: performing transformation on a three-dimensional cartoon grid template according to a facial image to be customized, so as to obtain a customized three-dimensional facial cartoon model; calculating a scaling registration parameter of a preset three-dimensional eyeball template in the three-dimensional facial cartoon model according to eye socket transformation information in the three-dimensional facial cartoon model; and with the aim of there being no interpenetration between a three-dimensional eyeball template scaled according to the scaling registration parameter and an eyelid grid in the three-dimensional facial cartoon model, calculating a translation registration parameter of the three-dimensional eyeball template in the three-dimensional facial cartoon model.

Description

Eyeball registration method, device, server and medium based on 3D cartoon model

This application claims the priority of the Chinese Patent Application No. 202011565476.1 filed with the China Patent Office on December 25, 2020, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the technical field of three-dimensional model reconstruction, for example, to an eye registration method, device, server and medium based on a three-dimensional cartoon model.

Background technique

With the rapid development of computer technology, Augmented Reality (AR) products have been widely used in people's daily life. The virtual content in AR products needs to be seamlessly embedded in real images, so as to give users a virtual The feeling of the real existence of the content; at this time, in order to realize the diversified interaction of users in AR products, a solution will be specially designed to customize the corresponding 3D cartoon image according to the actual image of the user.

However, when customizing the 3D cartoon image for different users, it is necessary to adjust the size and position of the eyeballs in the 3D cartoon model according to the deformation of the model during the process of customizing the 3D cartoon model. A large number of model points are sampled in the 3D mesh model, and then by calculating the closest point among multiple model points, the 3D cartoon image and the specific deformation of the eyeball in the model are analyzed synchronously, so as to realize the synchronization of the eyeball during the customization process of the 3D cartoon model. At this time, when using a general three-dimensional mesh model to analyze the specific deformation of the eyeball, the deformation of the model will also be considered, which makes the eyeball registration algorithm in the related art more complicated, and there is a huge computational cost, which affects the three-dimensional cartoon. Eye registration efficiency in the model.

SUMMARY OF THE INVENTION

The present application provides an eyeball registration method, device, server and medium based on a three-dimensional cartoon model, which reduces the computational cost of eyeball registration in the three-dimensional cartoon model, reduces the complexity of eyeball registration in the three-dimensional cartoon model, and improves the three-dimensional cartoon model. Efficiency of mid-eye registration.

Provided is an eyeball registration method based on a three-dimensional cartoon model, the method comprising:

Deform the 3D cartoon grid template according to the face image to be customized to obtain a customized 3D face cartoon model;

According to the orbital deformation information in the 3D face cartoon model, calculate the scaling and registration parameters of the preset 3D eyeball template in the 3D face cartoon model;

With the goal that there is no interspersed between the 3D eyeball template scaled according to the scaling and registration parameters and the eyelid grid in the 3D face cartoon model, calculate the 3D eyeball template in the 3D face cartoon model. Translational registration parameters within the model.

Also provided is an eyeball registration device based on a three-dimensional cartoon model, the device comprising:

The cartoon model customization module is set to deform the 3D cartoon grid template according to the face image to be customized to obtain a customized 3D face cartoon model;

The scaling and registration determining module is configured to calculate the scaling and registration parameters of the preset 3D eyeball template in the 3D face cartoon model according to the orbital deformation information in the 3D face cartoon model;

The translation registration determination module is set to make the three-dimensional eyeball template scaled according to the scaling and registration parameters, and the eyelid grid in the three-dimensional face cartoon model not interspersed as the target, and calculate the three-dimensional eyeball The translation registration parameters of the template in the three-dimensional face cartoon model.

Also provided is a server that includes:

one or more processors;

storage means arranged to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional cartoon model-based eye registration method described in any embodiment of the present application.

A computer-readable storage medium is also provided, which stores a computer program, and when the computer program is executed by a processor, implements the three-dimensional cartoon model-based eyeball registration method described in any embodiment of the present application.

Description of drawings

1A is a flowchart of a method for eyeball registration based on a three-dimensional cartoon model provided by Embodiment 1 of the present application;

1B is a schematic diagram of the principle of an eyeball registration process based on a three-dimensional cartoon model provided in Embodiment 1 of the present application;

2A is a flowchart of a method for eyeball registration based on a three-dimensional cartoon model according to Embodiment 2 of the present application;

FIG. 2B is a schematic diagram of the principle of an eyeball registration process based on a three-dimensional cartoon model according to Embodiment 2 of the present application;

Fig. 2C is the schematic diagram of the eyelid edge feature point in the three-dimensional face cartoon model in the method that the second embodiment of the application provides;

2D is a schematic diagram of multiple feature points in the eyelid grid in the three-dimensional face cartoon model in the method provided in Embodiment 2 of the present application;

3A is a flowchart of a method for eyeball registration based on a three-dimensional cartoon model provided in Embodiment 3 of the present application;

3B is a schematic diagram of the center point of the pupil in the three-dimensional eyeball template in the method provided in Embodiment 3 of the present application;

3C is a schematic diagram of the closest feature point of the pupil center point in the three-dimensional eyeball template on the three-dimensional cartoon grid template in the method provided by the third embodiment of the present application;

3D is a schematic diagram of the principle of the translation registration process of the three-dimensional eyeball template in the method provided in Embodiment 3 of the present application;

4 is a schematic structural diagram of an eyeball registration device based on a three-dimensional cartoon model according to Embodiment 4 of the present application;

FIG. 5 is a schematic structural diagram of a server according to Embodiment 5 of the present application.

Detailed ways

The present application will be described below with reference to the accompanying drawings and embodiments. The embodiments described here are only used to explain the present application, but not to limit the present application. The drawings show only some but not all structures related to the present application. Furthermore, the embodiments in this application and features in the embodiments may be combined with each other without conflict.

Example 1

1A is a flowchart of a method for eyeball registration based on a 3D cartoon model provided in Embodiment 1 of the present application. size and location are registered. The method for eyeball registration based on a 3D cartoon model provided in this embodiment may be performed by the device for eyeball registration based on a 3D cartoon model provided in this embodiment of the present application, which may be implemented by software and/or hardware, and It is integrated in a server that executes the method, and the server can be a background server with the ability of three-dimensional image reconstruction.

Referring to Figure 1A, the method may include the following steps.

S110, deform the three-dimensional cartoon grid template according to the face image to be customized to obtain a customized three-dimensional face cartoon model.

In order to increase users' diversified interactive gameplay in AR products, a function to customize the corresponding 3D cartoon image according to the user's actual image will be specially designed in AR products. With a large number of model points, when the eyeball registration is realized synchronously during the customization of the 3D cartoon model, there are problems of excessive computational overhead and complex customization algorithms. At the same time, considering the eyeballs that need to be registered in the customization process of the 3D cartoon image, it belongs to a kind of Relative to a regular three-dimensional model, the present embodiment separates the three-dimensional cartoon image customization process and the eyeball registration process in the face image and implements them in sequence. The three-dimensional cartoon image is reconstructed, and the eyeballs are registered in the reconstructed three-dimensional face cartoon model, so as to obtain the final three-dimensional face cartoon image.

Among them, the face image is the collected image data including the face image. Since this embodiment is mainly used to customize the 3D cartoon image of the face image collected in the AR product, the camera and the like configured in the AR product can be used. Image collector to collect the current face image. At the same time, in order to ensure the fidelity of the three-dimensional face cartoon image, this embodiment will preset a three-dimensional cartoon grid template with a standard cartoon face image, and the three-dimensional cartoon grid template is pre-created according to the standard face cartoon image 3D cartoon model with grid distribution indication added; during the creation of 3D cartoon grid template, 3D cartoon models with different precisions or different grid shapes can be created according to the pixel resolution of the selected model and the different grid shapes. The grid template means that the density of grid distribution or the grid distribution shape in the created 3D cartoon grid template is different, for example, it can be composed of quadrilateral grids, and it can also be composed of triangular grids. In this embodiment, before customizing a corresponding 3D cartoon image according to a face image, a created 3D cartoon grid template will be pre-stored, and the same 3D cartoon grid template can be subsequently matched with different face images by referring to the same 3D cartoon grid template. , to customize the 3D cartoon image of multiple face images, so as to avoid the restriction of face pose or image in different face images.

Optionally, in this embodiment, before registering the eyeballs in the 3D cartoon image, AR products are used to collect the face images that need to be referenced when customizing the 3D cartoon image, and the pre-stored images that support the accuracy of multiple users are collected. Customize the 3D cartoon grid template of the 3D cartoon image, and then perform corresponding deformation processing on the same feature point in the 3D cartoon grid template by analyzing the posture position and other information of each feature point in the face image, so as to make the deformation The cartoon pose in the resulting three-dimensional cartoon grid template can be approximately the same as the face pose in the face image, and the deformed three-dimensional cartoon grid template is used as the three-dimensional face customized according to the face image in this embodiment. Cartoon model, at this time, for different face images, you can refer to the 3D cartoon grid template for corresponding deformation, so as to obtain a customized 3D face cartoon model under each face image. At this time, the customized 3D face cartoon The model and the 3D cartoon mesh template have the same mesh vertices, so that the registration of the eyeball can be accurately analyzed according to the deformation of specific vertices on the eyelid.

S120, according to the orbital deformation information in the three-dimensional face cartoon model, calculate the scaling and registration parameters of the preset three-dimensional eyeball template in the three-dimensional face cartoon model.

Optionally, considering that the eyeballs that need to be registered in the customizing process of the three-dimensional cartoon image belong to a relatively regular three-dimensional model, this embodiment separates the three-dimensional cartoon image customizing process and the eyeball registration process in the face image. In this case, for the accurate registration of the eyeballs in the three-dimensional face cartoon model, this embodiment will preset a three-dimensional eyeball template with standard eyeball specifications, and customize the three-dimensional face according to the face posture in the face image. After the cartoon model, the three-dimensional eyeball template can be adjusted adaptively, so that the adjusted three-dimensional eyeball template can completely fit the eye region in the customized three-dimensional face cartoon model.

Among them, when the 3D face cartoon model is customized by deforming the 3D cartoon grid template according to the face image, the size and depth of the eye sockets in the customized 3D face cartoon model will be changed accordingly, so that the size of the eyeball and the The position will also change accordingly. At this time, the change of the eyeball size can be represented by the zoom registration parameter of the eyeball, and the change of the eyeball position can be represented by the translation registration parameter of the eyeball.

In this embodiment, since the preset 3D eyeball template can be completely adapted to the orbit in the preset 3D cartoon grid template, the 3D eyeball model after eyeball registration is also required to be completely adapted to the customized 3D eyeball model. In the orbit of the 3D face cartoon model, as shown in Figure 1B, for the eye registration under either eye of the left eye and the right eye in the 3D cartoon grid template, it can be analyzed by analyzing the 3D cartoon grid template. The difference between the original size of the orbit under the eye and the actual size of the orbit under the eye in the customized 3D face cartoon model after deformation is used to calculate the scaling of the 3D eye template under each eye in the 3D face cartoon model. Then use the scaling and registration parameters to perform corresponding scaling operations on the 3D eyeball template under different eyes, so that the 3D eyeball model obtained after scaling can be completely adapted to the orbits of the corresponding eyes in the 3D face cartoon model In order to ensure the accuracy of eyeball size registration in the customized 3D face cartoon model.

S130, calculating the translation registration of the 3D eyeball template in the 3D face cartoon model with the goal of making the 3D eyeball template scaled according to the scaling and registration parameters not interspersed with the eyelid grid in the 3D face cartoon model parameter.

Optionally, when customizing the 3D face cartoon model, the depth of the eye socket in the 3D cartoon mesh model may be deformed according to the eye pose of the face image. The scaled 3D eyeball template protrudes from the orbit in the 3D cartoon mesh model. Therefore, in order to ensure the adaptability between the scaled 3D eyeball template and the depth of the orbit in the 3D cartoon mesh model, this embodiment also requires the scaled 3D eyeball template. The position of the 3D eyeball template is translated, so that the eyelids in the customized 3D face cartoon model can completely wrap the 3D eyeball model obtained after registration, so as to ensure the fidelity of eyeball registration in the 3D face cartoon model.

In this embodiment, for the eyeball registration in each eye in the 3D face cartoon model, it is possible to analyze whether there is interspersed between the scaled 3D eyeball template and the eyelid grid under the eye in the 3D face cartoon model , to judge whether the registered 3D eyeball model and the orbital depth in the 3D cartoon mesh model are completely suitable. This kind of eyelid grid is customized when eyes are open or closed. At this time, if the 3D eye template is zoomed according to the scaling and registration parameters of the 3D eye template in the 3D face cartoon model, it can be analyzed by analyzing the scaled Whether there is interspersed between the 3D eyeball template and each eyelid grid in the 3D face cartoon model, and the specific interleaving situation, so as to determine the ability to make the scaled 3D eyeball template and the eyelid grid in the 3D face cartoon model. There are no interspersed translational registration parameters.

In addition, in this embodiment, after the scaling registration parameters and translation registration parameters of the 3D eyeball template in the 3D face cartoon model are determined, the scaling registration parameters and the translation registration parameters are also used in the 3D face cartoon model. The three-dimensional eyeball template is registered in sequence; for the eyeball registration under each eye in the three-dimensional face cartoon model, the scaling and registration parameters calculated for the three-dimensional eyeball template under the eye in the three-dimensional face cartoon model are used. The three-dimensional eyeball template is scaled in the eye of the cartoon face model, and the translation registration parameters under the eye are used again to continue to perform corresponding translation on the scaled three-dimensional eyeball template in the eye of the three-dimensional face cartoon model, thereby realizing The eyeballs in each eye of the 3D face cartoon model are accurately registered to avoid the situation of interspersed between the registered 3D eyeball model and the eyelid grid in the 3D face cartoon model.

In the technical solution provided by this embodiment, the uniformly set 3D cartoon grid template is deformed according to the face image to be customized to obtain a customized 3D face cartoon model, and according to the orbital deformation information in the 3D face cartoon model, To calculate the preset 3D eyeball template in the 3D face cartoon model to adapt to the orbital deformation under the scaling registration parameters, and then make the 3D eyeball template scaled according to the scaling registration parameters, and the 3D face cartoon model. There is no interspersed between the eyelid grids as the target, and the translation registration parameters of the 3D eye template in the 3D face cartoon model are calculated, so as to avoid the situation that the eyeballs in the 3D face cartoon model do not fit, and there is no need to calculate the general 3D face model. The deformation of a large number of model points in the mesh model greatly reduces the computational overhead of eyeball registration in the 3D cartoon model. The 3D face can be realized directly by scaling and panning the 3D eye template in the 3D face cartoon model. The eyeball registration in the cartoon model reduces the complexity of the eyeball registration in the three-dimensional cartoon model and improves the efficiency of the eyeball registration in the three-dimensional cartoon model.

Embodiment 2

2A is a flowchart of a method for eyeball registration based on a 3D cartoon model according to Embodiment 2 of the present application, and FIG. 2B is a schematic schematic diagram of the principle of an eyeball registration process based on a 3D cartoon model according to Embodiment 2 of the present application. This embodiment is described on the basis of the above-mentioned embodiment. As shown in Figure 2A, in this embodiment, the deformation process of the three-dimensional cartoon grid template and the calculation process of the registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model are mainly explained.

Optionally, as shown in FIG. 2A , this embodiment may include the following steps.

S210 , transforming the three-dimensional cartoon grid template in turn according to the face shape and face expression in the face image to be customized, to obtain a customized three-dimensional face cartoon model.

Optionally, when customizing the corresponding three-dimensional cartoon image according to the face image, it is required that the cartoon image of the face shape and the face expression can be matched with the face image. In this embodiment, in order to ensure the fidelity of the three-dimensional face cartoon model, As shown in FIG. 2B , the three-dimensional cartoon mesh template can be processed according to the facial shape and posture by analyzing the postures of multiple feature points representing the shape of the human face and the postures of the multiple feature points representing the facial expression in the face image. Shape deformation to ensure the matching between the cartoon shape in the 3D face cartoon model and the face image, and the deformed 3D cartoon mesh template is adjusted again under the facial expression according to the facial expression posture, so as to obtain a customized 3D model. The face cartoon model ensures the fidelity of the face shape and facial expression in the three-dimensional face cartoon model.

Exemplarily, in this embodiment, the deformation between multiple feature points in the face image can be used to carry out three-dimensional cartoon modeling of the face shape, and in order to ensure the fidelity of the facial expressions in the three-dimensional face cartoon model, a three-dimensional cartoon model can be used. Blend shape model realization, including smile blendshape, left eye closed blendshape (eyeblink_L) and right eye closed blendshape (eyeblink), etc. At this time, each blendshape grid is deformed correspondingly according to the facial expression, so as to obtain a face conforming to the human face. 3D face cartoon model of actual image.

S220, extracting the eyelid edge feature points in the three-dimensional face cartoon model, and calculating the corresponding orbital deformation information according to the deformation information of each eyelid edge feature point.

In this embodiment, the orbital deformation information referred to when calculating the scaling and registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model is mainly used to determine the size of the orbit in each eye of the customized three-dimensional face cartoon model, In this embodiment, a plurality of eyelid edge feature points that can accurately represent the position of the orbit can be extracted from the eye region where the left eye or the right eye is located in the three-dimensional face cartoon model, and the three-dimensional cartoon image of each eyelid edge feature point before deformation can be analyzed. The initial position in the grid template and the deformed position in the deformed 3D face cartoon model, and each eyelid edge feature is determined according to the deformation between the initial position of each eyelid edge feature point and the deformed position The deformation information of the point is calculated to calculate the corresponding orbital deformation information.

Exemplarily, as shown in FIG. 2C , if there are four eyelid edge feature points v _l,i , v _r,i , v _u,i and v _b,i in an eyelid part of the three-dimensional face cartoon model, and each The position coordinates of the eyelid edge feature points before deformation are represented by P, and the position coordinates after deformation are represented by Q, where i represents the left eye or right eye in the 3D face cartoon model, then the 3D eyeball template is in the 3D face cartoon model. The scaling registration parameters of can be:

S230, according to the orbital deformation information in the three-dimensional face cartoon model, calculate the scaling and registration parameters of the preset three-dimensional eyeball template in the three-dimensional face cartoon model.

S240: Determine eyelid grids in different eyelid states in the three-dimensional face cartoon model, and extract feature points in each eyelid grid.

Optionally, during the customization process of the 3D face cartoon model, in order to meet the fidelity of eyelid opening and closing under various eye expressions, the corresponding eyelid mesh will be specially customized for the eyelid states under various eye expressions. In order to ensure the fidelity of the 3D face cartoon model under various eye expressions; at the same time, after scaling the 3D eyeball template according to the scaling and registration parameters, you can analyze the center point of the eyeball in the scaled 3D eyeball template and the Whether the distance between each feature point in the eyelid grid exceeds the zoomed eyeball radius to determine whether there is an interspersed between the zoomed 3D eyeball template and the eyelid grid in the 3D face cartoon model, as shown in Figure 2D As shown, a large number of mesh vertices need to be extracted from each eyelid mesh under different eyelid states in the 3D face cartoon model, as the feature points in the eyelid mesh, and then the scaled 3D eyeball template will be analyzed subsequently. Whether there is interspersed with the eyelid mesh in the 3D face cartoon model.

S250: Calculate the center point of the eyeball and the radius of the eyeball after the three-dimensional eyeball template is scaled according to the scaling and registration parameters.

In this embodiment, in order to accurately analyze whether there is interspersed between the scaled 3D eyeball template and the eyelid grid in the 3D face cartoon model, it is necessary to perform a scaling operation on the 3D eyeball template according to the scaling and registration parameters. , correspondingly calculate the eyeball radius after scaling of the 3D eyeball template; and analyze the associated feature points between the 3D eyeball template and the 3D cartoon grid template before deformation, obtain the deformation position after the corresponding deformation, and then according to the associated feature point after deformation The position corresponds to the eyeball radius after the scaled 3D eyeball template, and the center point of the eyeball relative to the 3D face cartoon model after the 3D eyeball template is scaled is calculated correspondingly, and then each feature point in the 3D face cartoon model and the center point of the eyeball are analyzed. Whether the distance between them exceeds the zoomed eyeball radius is used to determine whether there is interspersed between the zoomed three-dimensional eyeball template and the eyelid grid in the three-dimensional face cartoon model.

S260, for each feature point on the eyelid grid in the three-dimensional face cartoon model, according to the difference between the distance between the feature point and the center point of the eyeball, and the radius of the eyeball, calculate the sub-translation parameter corresponding to the feature point.

After extracting multiple feature points from each eyelid grid in the 3D face cartoon model, for each feature point, the distance between the feature point and the center point of the eyeball scaled by the 3D eyeball template can be calculated separately. If the distance is smaller than the scaled eyeball radius of the 3D eyeball template, it means that the feature point in the eyelid grid in the 3D face cartoon model is interspersed in the scaled 3D eyeball template, so that the scaled 3D eyeball template is in the 3D face cartoon The eye socket of the model is highlighted. At this time, it is necessary to translate the scaled 3D eyeball template to the inside of the eye frame of the 3D face cartoon model, so that the feature point on the eyelid grid in the 3D face cartoon model will not be interspersed. In the 3D eyeball model, at this time, according to the distance between the feature point and the center point of the eyeball, relative to the distance difference between the zoomed eyeball radii, the corresponding feature point can be calculated when it meets the requirements of not being interspersed in the 3D eyeball model. Sub-translation parameters; at this time, according to the above steps, the sub-translation parameters corresponding to each feature point in each eyelid grid can be calculated, and then one can be selected from the sub-translation parameters corresponding to multiple feature points to make all the feature points There are no translational registration parameters interspersed in the 3D eyeball model.

S270, compare the sub-translation parameters corresponding to each feature point on the eyelid grid in the three-dimensional face cartoon model, and determine the translation registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model.

Optionally, after calculating the sub-translation parameter corresponding to each feature point in each eyelid grid in the 3D face cartoon model, since each sub-translation parameter represents the translation distance of the 3D eyeball template, it can be compared by comparing each sub-translation parameter. A sub-translation parameter corresponding to a feature point is screened out with the largest sub-translation parameter, so that all the feature points are not interspersed in the three-dimensional eyeball model, and then the screened out maximum sub-translation parameter is used as the three-dimensional eyeball template in this embodiment. Translation registration parameters in the 3D face cartoon model to ensure the fidelity of eyeball registration in the 3D face cartoon model.

The technical solution provided by this embodiment uses the distance between each feature point on the eyelid grid in the three-dimensional face cartoon model and the center point of the eyeball after scaling, and the distance difference between the radius of the eyeball after scaling, to calculate each feature point. Sub-translation parameters when a feature point satisfies the requirements of not interspersed in the registered 3D eyeball model, compare the sub-translation parameters corresponding to multiple feature points, and determine the translation registration parameters of the 3D eye template in the 3D face cartoon model , to ensure the translation accuracy of the 3D eyeball template in the 3D face cartoon model, and improve the fidelity of eyeball registration in the 3D face cartoon model.

Embodiment 3

FIG. 3A is a flowchart of an eyeball registration method based on a three-dimensional cartoon model according to Embodiment 3 of the present application. This embodiment is described on the basis of the above-mentioned embodiment. As shown in FIG. 3A , in this embodiment, the process of calculating the center point of the eyeball and the radius of the eyeball after the three-dimensional eyeball template is scaled and the process of translation registration are mainly explained.

Optionally, as shown in FIG. 3A , this embodiment may include the following steps.

S310 , deform the three-dimensional cartoon grid template according to the face image to be customized to obtain a customized three-dimensional face cartoon model.

S320, according to the orbital deformation information in the three-dimensional face cartoon model, calculate the scaling and registration parameters of the preset three-dimensional eyeball template in the three-dimensional face cartoon model.

S330 , according to the original eyeball radius of the three-dimensional eyeball template and the scaling registration parameter, calculate the eyeball radius after the three-dimensional eyeball template is scaled according to the scaling registration parameter.

Optionally, since the 3D eyeball template is scaled according to the scaling and registration parameters, the eyeball radius will also be scaled correspondingly. At this time, the corresponding pupil center point can be extracted from the preset 3D eyeball template, as shown in FIG. 3B . , since the center point of the eyeball under the three-dimensional eyeball template can be the origin of coordinates, the original eyeball radius of the three-dimensional eyeball template can be calculated by analyzing the position of the pupil center point, and the 3D eyeball template is calculated according to the scaling registration parameters. The original eyeball radius is correspondingly scaled, and the eyeball radius after the three-dimensional eyeball template is scaled according to the scaling and registration parameters can be obtained.

Exemplarily, if the scaling registration parameter is s _i , the eyeball radius after the three-dimensional eyeball template is scaled according to the scaling registration parameter may be:

in,

is the original eyeball radius of the three-dimensional eyeball template, E _i is the coordinate system of the three-dimensional eyeball template, and c _i is the pupil center point of the three-dimensional eyeball template.

S340: Calculate the center point of the eyeball after the three-dimensional eyeball template is scaled according to the scaling registration parameter according to the position of the nearest feature point in the three-dimensional face cartoon model and the eyeball radius of the pupil center point in the three-dimensional eyeball template.

Optionally, since the translation of the 3D eyeball template is set for the 3D face cartoon model, when analyzing the distance between each feature point in the eyelid grid in the 3D face cartoon model and the zoomed eyeball center point, It is also necessary to calculate the position of the center point of the eyeball relative to the three-dimensional face cartoon model, so this embodiment can be approximated by analyzing the deformation position of the nearest feature point of the pupil center point of the three-dimensional eyeball template in the three-dimensional face cartoon model. Analyze the actual position of the pupil center point relative to the 3D face cartoon model, and then calculate the 3D eyeball template according to the distance difference between the closest feature point position of the pupil center point in the 3D face cartoon model and the radius of the eyeball. The center point of the eyeball relative to the 3D face cartoon model after quasi-parameter scaling.

Exemplarily, the calculation method for the position of the closest feature point of the pupil center point in the 3D face cartoon model may be: determine the original closest feature point of the pupil center point in the 3D eyeball template in the 3D cartoon mesh template; The deformation position of the nearest feature point in the 3D face cartoon model is to determine the position of the nearest feature point of the pupil center point in the 3D face cartoon model.

As shown in Figure 3C, according to the left eye and right eye in the 3D cartoon grid template, the position of the original closest feature point under any eyelid in the 3D cartoon grid template before deformation of the pupil center point in the 3D eyeball template can be calculated is: b _i ={barycentric, triangle}; wherein, barycentric is the barycentric coordinate of the original closest feature point, triangle is a plurality of grids in a grid where the original closest feature point is located in the 3D cartoon grid template Vertex, after deforming the 3D cartoon mesh template to obtain a customized 3D face cartoon model, the position of the deformed closest feature point in the deformed 3D face cartoon model can be calculated as t _i '=t. barycentricgQ(t.triangle); where t.barycentric is the barycentric coordinate of the original closest feature point in the 3D cartoon grid template of the pupil center point before deformation, and t.triangle is the 3D cartoon grid of the pupil center point before deformation The grid vertex in the grid where the original closest feature point in the template is located, and Q represents the specific deformation coordinates of the three-dimensional cartoon grid template. The closest feature point of the pupil center point in the 3D face cartoon model can be approximated as the pupil center point, and by calculating the distance between the position of the closest feature point and the eyeball radius, the 3D eyeball template can be calculated according to the scaling parameters. After scaling, the center point of the eyeball relative to the 3D face cartoon model. At this time, the center point of the eyeball after the 3D eyeball template is scaled according to the scaling and registration parameters can be

S350, for each feature point on the eyelid grid in the three-dimensional face cartoon model, the distance between the center point of the eyeball after the translation of the three-dimensional eyeball template and the feature point is equal to the radius of the eyeball as the target, and calculate the corresponding feature point. Sub translation parameter.

Exemplarily, for each feature point p ⁱ on each eyelid grid in the 3D face cartoon model, if it is required that the feature point is not interspersed in the scaled 3D eyeball template, the scaled 3D eyeball can be The template is exactly tangent to the feature point, that is, as shown in Figure 3D, the distance between the center point of the eyeball and the feature point obtained by translating the scaled 3D eyeball template by a certain distance is exactly equal to the scaled eyeball radius , at this time, if the sub-translation parameter corresponding to the feature point is set to be x, then the center point of the eyeball after the translation of the scaled 3D eyeball template is

At this point it can be determined

that is

Arrange to get:

The sub-translation parameter x corresponding to the feature point can be calculated; the sub-translation parameter corresponding to each feature point can be obtained according to the above steps, and the maximum sub-translation parameter can be obtained by comparing multiple sub-translation parameters

Thus, the translation registration parameters of the 3D eyeball template in the 3D face cartoon model are calculated as

The scaling registration parameter is s _i , and the 3D eyeball template is directly scaled and translated in sequence according to the scaling registration parameter and the translation registration parameter, so as to achieve accurate registration of the eyeballs in the 3D face cartoon model.

S360, compare the sub-translation parameters corresponding to each feature point on the eyelid grid in the three-dimensional face cartoon model, and determine the translation registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model.

Embodiment 4

FIG. 4 is a schematic structural diagram of an eyeball registration device based on a three-dimensional cartoon model according to Embodiment 4 of the present application. As shown in FIG. 4 , the device may include the following modules.

The cartoon model customization module 410 is configured to deform the three-dimensional cartoon grid template according to the face image to be customized to obtain a customized three-dimensional face cartoon model;

The scaling and registration determining module 420 is configured to calculate the scaling and registration parameters of the preset 3D eyeball template in the 3D face cartoon model according to the orbital deformation information in the 3D face cartoon model;

The translation registration determination module 430 is set to make the three-dimensional eyeball template scaled according to the scaling registration parameters and the eyelid meshes in the three-dimensional face cartoon model not interspersed as the goal, and calculate the three-dimensional The translation registration parameters of the eyeball template in the three-dimensional face cartoon model.

The apparatus for eyeball registration based on a 3D cartoon model provided in this embodiment can be adapted to execute the method for eyeball registration based on a 3D cartoon model provided in any of the above embodiments, and has corresponding functions and effects.

Embodiment 5

FIG. 5 is a schematic structural diagram of a server according to Embodiment 5 of the present application. As shown in FIG. 5 , the server includes a processor 50, a storage device 51 and a communication device 52; the number of processors 50 in the server may be one or more One processor 50 is taken as an example in FIG. 5 ; the processor 50 , the storage device 51 and the communication device 52 in the server may be connected through a bus or other means, and the connection through a bus is taken as an example in FIG. 5 .

The server provided in this embodiment can be used to execute the eye registration method based on the three-dimensional cartoon model provided by any of the above embodiments, and has corresponding functions and effects.

Embodiment 6

Embodiment 6 of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for eyeball registration based on a three-dimensional cartoon model in any of the foregoing embodiments can be implemented. The method can include:

A storage medium containing computer-executable instructions provided by an embodiment of the present application, the computer-executable instructions of the computer-executable instructions are not limited to the above-mentioned method operations, and can also perform the eyeball matching provided by any embodiment of the present application based on a three-dimensional cartoon model. related operations in the quasi-method.

The present application can be implemented by means of software and general hardware, and can also be implemented by hardware. The present application can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a floppy disk of a computer, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory) , RAM), flash memory (FLASH), hard disk or optical disk, etc., including multiple instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments of the present application. The storage medium may be a non-transitory storage medium.

In the above-mentioned embodiment of the eye registration device based on the three-dimensional cartoon model, the multiple units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; , the names of multiple functional units are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application.

Claims

An eye registration method based on a three-dimensional cartoon model, comprising:

Deform the 3D cartoon grid template according to the face image to be customized to obtain a customized 3D face cartoon model;

According to the orbital deformation information in the 3D face cartoon model, calculate the scaling and registration parameters of the preset 3D eyeball template in the 3D face cartoon model;

With the goal that there is no interspersed between the 3D eyeball template scaled according to the scaling and registration parameters and the eyelid grid in the 3D face cartoon model, calculate the 3D eyeball template in the 3D face cartoon model. Translational registration parameters within the model.
The method according to claim 1, wherein the goal is to make the three-dimensional eyeball template scaled according to the scaling registration parameters and the eyelid grid in the three-dimensional face cartoon model not interspersed as the goal, Calculate the translation registration parameters of the 3D eyeball template in the 3D face cartoon model, including:

calculating the center point of the eyeball and the radius of the eyeball after the three-dimensional eyeball template is scaled according to the scaling and registration parameters;

For each feature point on the eyelid grid in the three-dimensional face cartoon model, according to the distance between the feature point and the center point of the eyeball, and the difference between the radius of the eyeball, the corresponding feature point is calculated. subtranslation parameter;

The sub-translation parameters corresponding to each feature point on the eyelid grid in the three-dimensional face cartoon model are compared, and the translation registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model are determined.
The method according to claim 2, wherein the calculating the center point of the eyeball and the eyeball radius after the three-dimensional eyeball template is scaled according to the scaling and registration parameters, comprises:

According to the original eyeball radius of the three-dimensional eyeball template and the scaling registration parameter, calculate the eyeball radius after the three-dimensional eyeball template is scaled according to the scaling registration parameter;

Calculate the center point of the eyeball after the three-dimensional eyeball template is scaled according to the scaling and registration parameters according to the position of the nearest feature point in the three-dimensional face cartoon model and the eyeball radius of the pupil center point in the three-dimensional eyeball template .
The method according to claim 3, wherein, calculating the three-dimensional eyeball according to the position of the nearest feature point in the three-dimensional face cartoon model and the eyeball radius according to the pupil center point in the three-dimensional eyeball template The template further includes:

determining the original closest feature point of the pupil center point in the three-dimensional eyeball template in the three-dimensional cartoon grid template;

Based on the deformation position of the original nearest feature point in the three-dimensional face cartoon model, determine the nearest feature point position of the pupil center point in the three-dimensional face cartoon model.
The method according to claim 2, wherein, after the three-dimensional cartoon grid template is deformed according to the face image to be customized to obtain the customized three-dimensional face cartoon model, the method further comprises:

The eyelid grids under different eyelid states in the three-dimensional face cartoon model are determined, and the feature points in each eyelid grid are extracted.
The method according to claim 2, wherein, for each feature point on the eyelid grid in the three-dimensional face cartoon model, according to the distance between the feature point and the center point of the eyeball, and the eyeball Calculate the difference between the radii, and calculate the sub-translation parameters corresponding to the feature points, including:

For each feature point on the eyelid grid in the three-dimensional face cartoon model, the distance between the center point of the eyeball after the translation of the three-dimensional eyeball template and the feature point is equal to the radius of the eyeball, and calculating Sub-translation parameters corresponding to the feature points.
The method according to any one of claims 1-6, wherein, in the calculation according to the orbital deformation information in the three-dimensional face cartoon model, a preset three-dimensional eyeball template is calculated in the three-dimensional face cartoon model Before the scaling registration parameters, also include:

The eyelid edge feature points in the three-dimensional face cartoon model are extracted, and the corresponding orbital deformation information is calculated according to the deformation information of each eyelid edge feature point.
The method according to any one of claims 1-6, wherein the three-dimensional cartoon grid template is deformed according to the face image to be customized to obtain a customized three-dimensional face cartoon model, comprising:

The three-dimensional cartoon grid template is sequentially deformed according to the face shape and face expression in the face image to be customized to obtain the customized three-dimensional face cartoon model.
The method according to any one of claims 1-6, wherein after calculating the translation registration parameters of the three-dimensional eyeball template in the three-dimensional face cartoon model, the method further comprises:

In the three-dimensional face cartoon model, the three-dimensional eyeball template is registered sequentially by using the zoom registration parameter and the translation registration parameter.
An eye registration device based on a three-dimensional cartoon model, comprising:

The cartoon model customization module is set to deform the 3D cartoon grid template according to the face image to be customized to obtain a customized 3D face cartoon model;

The scaling and registration determining module is configured to calculate the scaling and registration parameters of the preset 3D eyeball template in the 3D face cartoon model according to the orbital deformation information in the 3D face cartoon model;

The translation registration determination module is set to make the three-dimensional eyeball template scaled according to the scaling and registration parameters, and the eyelid grid in the three-dimensional face cartoon model not interspersed as the target, and calculate the three-dimensional eyeball The translation registration parameters of the template in the three-dimensional face cartoon model.
A server that includes:

one or more processors;

storage means arranged to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional cartoon model-based eye registration according to any one of claims 1-9 method.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for eyeball registration based on a three-dimensional cartoon model according to any one of claims 1-9 is implemented.