CN112819971A

CN112819971A - Method, device, equipment and medium for generating virtual image

Info

Publication number: CN112819971A
Application number: CN202110111064.9A
Authority: CN
Inventors: 彭昊天
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-01-26
Filing date: 2021-01-26
Publication date: 2021-05-18
Anticipated expiration: 2041-01-26
Also published as: CN112819971B

Abstract

The present disclosure discloses a method, an apparatus, a device and a medium for generating a virtual image, which relate to the technical field of computers, and in particular to the technical fields of augmented reality, computer graphics and the like. The method for generating the virtual image comprises the following steps: acquiring a grid model of an image and a preset base skin skeleton model, wherein the base skin skeleton model comprises at least one skeleton node, and the grid model and the base skin skeleton model have the same topological structure; determining a current bone drive coefficient of each of the at least one bone node from the mesh model and the base skin bone model; and sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient. The method and the device can automatically determine the bone driving coefficient, and improve the automation degree of virtual image generation.

Description

Method, device, equipment and medium for generating virtual image

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for generating an avatar, and to the field of augmented reality and computer graphics.

Background

The Augmented Reality (AR) technology is a technology that skillfully fuses virtual information and the real world, and a plurality of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and the like are widely applied, and virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer is applied to the real world after being simulated, and the two kinds of information complement each other, so that the real world is enhanced.

The Photo-To-three-dimensional Avatar (PTA) is To generate a personalized three-dimensional Avatar based on a user picture, for example, a personalized three-dimensional head portrait based on a face image.

In the related art, the bone driving coefficients can be configured manually to generate a corresponding personalized three-dimensional avatar.

Disclosure of Invention

The present disclosure provides a method, apparatus, device and medium for generating an avatar.

According to an aspect of the present disclosure, there is provided a method of generating an avatar, including: acquiring a grid model of an image and a preset base skin skeleton model, wherein the base skin skeleton model comprises at least one skeleton node, and the grid model and the base skin skeleton model have the same topological structure; determining a current bone drive coefficient of each of the at least one bone node from the mesh model and the base skin bone model; and sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

According to another aspect of the present disclosure, there is provided an avatar generation apparatus including: the system comprises an acquisition unit, a calculation unit and a display unit, wherein the acquisition unit is used for acquiring a mesh model of an image and a preset substrate skin skeleton model, the mesh model comprises at least one first vertex, the at least one first vertex forms a first topological network, the substrate skin skeleton model comprises at least one second vertex and at least one skeleton node, the at least one second vertex forms a second topological network, and the first topological network and the second topological network have the same topological structure; a determining unit, configured to determine, according to the mesh model and the base skin bone model, a current bone driving coefficient of each bone node of the at least one bone node; and the sending unit is used for sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the above aspects.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method according to any one of the above aspects.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of the above aspects.

According to the technical scheme disclosed by the invention, the bone driving coefficient can be automatically determined, and the automation degree of virtual image generation is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

FIG. 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

FIG. 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

FIG. 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

FIG. 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

FIG. 10 is a schematic diagram according to a tenth embodiment of the present disclosure;

fig. 11 is a schematic view of an electronic device for implementing any one of the methods of generating an avatar according to the embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

For a better understanding and description of the embodiments of the present disclosure, some technical terms involved in the embodiments of the present disclosure are described as follows:

the avatar corresponds to the image, for example, the image is a human face image, and the avatar refers to a three-dimensional avatar (avatar) corresponding to the human face image.

The mesh model refers to a three-dimensional model corresponding to an image, and the three-dimensional model is a personalized model, for example, corresponding to different face images, and the corresponding mesh models are different. The mesh model is composed of a plurality of meshes (skin mesh), each of which may include three vertices (vertex). The mesh model is used to characterize the appearance of the avatar to be generated.

The skin skeleton model is a three-dimensional model containing skeleton information corresponding to the virtual image, and skeleton nodes (nodes) are added on the basis of the grid model, and can also be called joints. The skin skeleton model is used for representing the appearance and the skeleton of the virtual image, and the skeleton is formed by linking skeleton segments by skeleton nodes.

Further, the skin skeleton model can be divided into a base skin skeleton model and a current skin skeleton model, the base skin skeleton model is preset and universal, for example, the base skin skeleton models corresponding to different virtual images are the same and can be designed by designers; the current skin skeleton model is personalized, different avatars correspond to different current skin skeleton models, for example, an avatar corresponding to a first face image is called a first avatar, an avatar corresponding to a second face image is called a second avatar, when the first avatar is generated, a first current skin skeleton model is generated based on the first face image and the base skin skeleton model, and then the first current skin skeleton model is used as the first avatar; similarly, in generating the second avatar, a second current skinned bone model is generated based on the second face image and the base skinned bone model, and then the second current skinned bone model is used as the second avatar.

And each bone node in the skinned bone model has a bone driving coefficient which is a rigid transformation coefficient, wherein the rigid transformation comprises translation (Translate), Rotation (Rotation) and scaling (Scale). Accordingly, the bone driving coefficient may include a translation coefficient, a rotation coefficient, and a scaling coefficient, the translation coefficient refers to a translation value corresponding to three coordinate axes (x, y, z), the rotation coefficient refers to an euler angle rotation value corresponding to three coordinate axes, and the scaling coefficient refers to a scaling value corresponding to three coordinate axes, so that the bone driving coefficient may be represented by 9 numerical values. For a certain skeleton node, the skeleton node acts according to the corresponding TRS coefficient, so that the shape of the virtual image is deformed. Different bone driving coefficients can be generated corresponding to different face images, and if the face images correspond to a certain face image, the corresponding bone driving coefficients can be called current bone driving coefficients, and the current skinned bone model is generated based on the current bone driving coefficients and the base skinned bone model.

The local pose matrix is a 4x4 matrix and represents a three-dimensional state, and the bone driving coefficients and the local pose matrix can be mutually transformed in a lossless mode.

Assuming a bone node index of j, the bone drive coefficient of the bone node can be represented by localTRS_(j)The local pose matrix of the skeleton node can be expressed by matTRS_(j)And (4) showing. The relationship of the local pose matrix to the bone drive coefficients is as follows:

the bone driving coefficients comprise 3 translation coefficients, 3 rotation coefficients and 3 scaling coefficients, wherein the 3 translation coefficients are in a local pose matrix (T)_x,T_y,T_z) (ii) a The 3 rotation coefficients can be represented by (α, β, γ), where α ═ atan2 (-m)₂₀,m₀₀)，β＝asin(m₁₀)，γ＝atan2(-m₁₂,m₁₁) (ii) a The 3 scaling coefficients are respectively the modulo lengths of the first 3 columns in the local pose matrix.

An Inverse Kinematics (IK) solver, which is an IK solution applied in the fields of robots and computer animation, is a technique of inversing parent node transformation with child node transformation, unlike forward kinematics. In the related art, the IK solver is generally applied to solving, and in the embodiment of the present disclosure, it is applied to shape solving.

And the bone solver is used for solving the bone driving coefficient based on the grid model and the base skin bone model.

A bone driver to generate a current skinned bone model based on the bone drive coefficients and the base skinned bone model.

Fig. 1 is a schematic diagram according to a first embodiment of the present disclosure. The embodiment provides a method for generating an avatar, comprising the following steps:

101. the method comprises the steps of obtaining a grid model of an image and a preset base skin skeleton model, wherein the base skin skeleton model comprises at least one skeleton node, and the grid model and the base skin skeleton model have the same topological structure.

102. Determining a current bone drive coefficient for each of the at least one bone node from the mesh model and the base skin bone model.

103. And sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

As shown in fig. 2, a schematic diagram of a mesh model and a skinned bone model is shown, where the left side is the skinned bone model and the right side is the mesh model. Referring to the mesh model on the right, each triangle represents a mesh, and the three vertices of each triangle are respectively the vertices of the mesh model, and for the sake of distinction, the vertices of the mesh model may be referred to as first vertices and the vertices of the skinned bone model may be referred to as second vertices. Referring to the left skinned bone model, in addition to the vertex (i.e., the second vertex, not shown), there is a bone node, which is indicated by a circle plus a sign.

The vertices (first and second vertices) are on the data representation and may include: the position coordinates of the vertices and the vertex indices.

The bone nodes (which may be referred to simply as nodes) of the skinned bone model may form a tree structure, i.e., each node has a parent node in addition to the root node, and each node has 0 to a plurality of child nodes.

As shown in fig. 3, the nodes (indicated by the circle plus signs) and the vertices (indicated by the circles) may be in a many-to-many relationship, the skinned region affected by the nodes includes multiple vertices, each node has a skinning weight for each vertex in the region, the skinning weight is between 0 and 1, each vertex is controlled by one or more nodes, and the total skinning weight is 1.

The grid model for obtaining the image can be realized by various related technologies, for example, inputting the face image into a PTA solver, calculating a plurality of BlendShape coefficients and a plurality of BlendShape models corresponding to the face image by the PTA solver, and performing linear weighted synthesis on the plurality of BlendShape coefficients and the corresponding plurality of BlendShape models to obtain the grid model.

The base skin bone model is pre-set, i.e. the structure of the base skin bone model, which includes the mesh structure composed of the second vertices and the tree structure composed of the nodes, may be pre-configured, wherein the base skin bone model and the mesh model have the same topology. Wherein the topology may be predefined such that both have the same topology. The skin skeleton model and the grid model have the same appearance structure through the same topological structure of the skin skeleton model and the grid model, and the accuracy of the generated virtual image is improved.

The base skin skeleton model and the mesh model have the same topological structure, namely that the two models have the same number of vertexes, the same relative positions of the vertexes, the same vertex connection sequence and the same mesh connection sequence.

After the mesh model and the basal skinning bone model are obtained, an IK solver may be employed to determine the current bone drive coefficients for each bone node. The specific procedure for determining the current bone drive coefficient can be seen in the following description.

In this embodiment, the current bone driving coefficient is determined according to the mesh model and the base skin bone model, so that the current bone driving coefficient can be automatically determined, and an avatar can be generated based on the current bone driving coefficient, thereby improving the automation degree of avatar generation. Further, the data size of the bone drive coefficient is small, and is generally 9 values as analyzed above, so that the data size can be reduced, light deployment can be realized, and the bone drive coefficient is convenient to implement on a user terminal.

Fig. 4 is a schematic diagram of a fourth embodiment of the present disclosure. The embodiment provides a method for generating an avatar, comprising the following steps:

401. the client collects the face image.

The client may be located on a user terminal, such as a mobile terminal, further such as a tablet, personal digital assistant, cellular telephone, smart phone, wearable device, and other similar computing devices.

The user can input the face images in the picture library into the client, or the user can call the camera through the client to shoot to obtain the face images.

402. And the client sends the face image to the server.

The client can send the face image to the server through a communication network, and the communication network can include a local area network, a wide area network, the internet and the like.

403. And the PTA solver in the server side acquires a corresponding grid model according to the face image.

404. And a skeleton solver in the server side is used for solving the current skeleton driving coefficient of each skeleton node based on the grid model and a preset substrate skin skeleton model.

The specific solving process can be referred to the embodiment shown in fig. 5.

405. And the server side selects the current bone driving coefficient of the bone node meeting the preset selection condition from the current bone driving coefficients of all the bone nodes to obtain the selected current bone driving coefficient, and sends the selected current bone driving coefficient to the client side.

According to the skeleton specification, some skeleton nodes in the skin skeleton model cannot be subjected to rigid transformation, so that the current skeleton driving coefficients of the skeleton nodes which cannot be subjected to rigid transformation do not need to be sent to a client. Therefore, a bone node satisfying a preset selection condition may refer to: the rigid transformation can occur, and the skeleton node on one side (left side or right side) in the bilateral symmetry skeleton nodes.

Generally speaking, it is necessary to transmit current bone drive coefficients of several tens of bone nodes, each current bone drive coefficient including 9 element values.

The server may send the current bone driving coefficient to the client via the communication network.

406. And the bone driver in the client generates a current skin bone model based on the received selected current bone driving coefficient and a preset substrate skin bone model, and displays the current skin bone model as a virtual image corresponding to the face image to the user.

After receiving the selected current skeleton driving coefficient, the client determines the skeleton driving coefficient of the skeleton node on the other side according to the symmetric relation, so that the current skeleton driving coefficients of all skeleton nodes capable of generating rigid transformation can be obtained, and further, based on the current skeleton driving coefficient, the vertex positions corresponding to the skeleton nodes can be driven to change.

In this embodiment, the current bone drive coefficient is screened by the server, so that the data volume can be further reduced, the network flow overhead is further reduced, the resource occupation of the mobile terminal is reduced, and the mobile terminal is convenient to deploy.

Fig. 5 is a schematic diagram of a fifth embodiment of the present disclosure. The embodiment provides a method for solving a current bone drive coefficient, where an execution subject of the embodiment may be a server, and the method includes:

501. a mesh model and a base skin bone model are obtained.

The mesh model is obtained by adopting a blendshape mode according to the face image.

The base skin skeleton model is designed by a designer and is preset at a server side.

The mesh model comprises a first vertex whose position coordinate may be referred to as first position coordinate, a first vertex A with index i_iThe position coordinates of (A) can be defined by VertexA_iAnd (4) showing. The base skinned bone model includes a second vertex, whose position coordinate may be referred to as a second position coordinate, second vertex B, indexed i_iThe position coordinates of (A) can be expressed by VertexB_iExpressing, adopting the bone driving coefficient of each bone node of the base skin bone model, and carrying out coordinate transformation on the second vertex of the base skin bone model to obtainThe transformed position coordinates may be referred to as third position coordinates, which may be VertexB'_iAnd (4) showing.

502. And setting a return buffer matrix.

Initially, the return buffer matrix may be set to all 0 s. The matrix may then be continuously updated according to the execution of the flow.

503. And traversing the nodes in any order, and determining the currently traversed nodes as the current nodes.

504. And calculating the optimal local pose matrix of the current node according to the grid model and the basal skin skeleton model.

In some embodiments, the mesh model includes at least one first vertex and the base skin bone model includes at least one second vertex, and the calculation of the optimal local pose matrix may include:

acquiring a first group of point clouds and a second group of point clouds, wherein the second group of point clouds comprises a group of second vertexes, the group of second vertexes are second vertexes included in a covering region acted by the current node in the base covering skeleton model, the first group of point clouds comprises a group of first vertexes, and the group of first vertexes are first vertexes included in a region corresponding to the covering region in the mesh model; solving an optimal rigid transformation matrix to ensure that the error between the point cloud after the second group of point clouds is transformed by the optimal rigid transformation matrix and the first group of point clouds is minimum; and taking the optimal rigid transformation matrix as the optimal local pose matrix.

When a designer designs a base skin skeleton model, skin areas acted by all skeleton nodes can be designed and recorded in attribute data corresponding to the base skin skeleton model, so that the skin areas acted by current nodes can be determined based on the attribute data, a second group of point clouds consisting of second vertexes included in the skin areas can be obtained, and since a first topological network consisting of the first vertexes and a second topological network consisting of the second vertexes have the same topological structure, areas corresponding to the skin areas in the grid model can be determined, and a first group of point clouds consisting of the first vertexes in the corresponding areas can be obtained.

Assuming that the first group of point clouds is represented by a matrix A, the second group of point clouds is represented by a matrix B, and the rigid transformation matrix is represented by x, solving the optimal solution of A x B, namely the optimal rigid transformation matrix, and taking the optimal rigid transformation matrix as the optimal local pose matrix. The optimal solution of a x B can be implemented by using various related techniques, such as least square method, orthogonal probuk method, and pri's analysis, which will not be described in detail in this embodiment.

And solving through the optimal rigid transformation, and calculating to obtain an optimal local pose matrix.

505. And accumulating the optimal local pose matrix into a preset return cache matrix to obtain an accumulated return cache matrix, and taking the accumulated return cache matrix as a return cache matrix before updating.

Wherein the return buffer matrix of the current node with index j is represented as

The optimal local pose matrix is expressed as

The accumulated return buffer matrix is denoted as buff_(j)The calculation formula of the accumulated backhaul buffer matrix, i.e. the backhaul buffer matrix before updating, may be:

by setting and updating the back-transmission cache matrix, the optimization result of the child node can be applied to the parent node, and the calculation accuracy of the bone driving coefficient of each node is improved.

506. And calculating an IK pose matrix of each level of father nodes on an IK chain where the current node is positioned by adopting an IK resolver, and updating the post-update post-return cache matrix according to the IK pose matrix of the father nodes so as to obtain the post-update post-return cache matrix of each skeleton node.

The IK solver can calculate the position and orientation matrixes of all levels of father nodes, from father nodes, grandfather nodes and the like to the root node on the IK chain where the child nodes are located according to the position and orientation matrixes of the child nodes. As shown in FIG. 6, the IK solver ensures that the lengths from the parent node to the child node are not changed, and the calculation result mainly influences the rotation attribute of the parent-ancestor node. The computational principles of the IK solver can be implemented using various related techniques, which are not described in detail herein.

Let the IK pose matrix of parent node with index k of current node with index j be denoted as IktRS_(j,k)。

By adopting the IK solver, the IK application can be extended from action drive to profile drive, achieving the purpose of fully utilizing existing resources to solve the profile drive problem of the embodiments of the present disclosure.

In some embodiments, the updating the pre-update backhaul cache matrix according to the IK pose matrix of the parent node includes: calculating the skin total weight ratio of the father node, wherein the ratio is the ratio of a first skin total weight to a second skin total weight, the first skin total weight is the skin total weight of the current node, and the second skin total weight is the skin total weight of the father node on the IK chain; calculating a weighted pose matrix according to the skin total weight ratio, a preset global return coefficient and the IK pose matrix of the father node; and accumulating the weighted pose matrix into the return cache matrix before updating.

For example, the skinning weights corresponding to the nodes may be designed by a designer and recorded in the attribute data, so that the skinning weights corresponding to the nodes may be obtained according to the attribute data of the base skinning skeleton model, and the skinning weights corresponding to each node are added to obtain the first skinning total weight sum and the second skinning total weight, and the first skinning total weight is divided by the second skinning total weight to obtain the total weight ratio.

Suppose that the total weight ratio of the skin is q_(j,k)Where j is the index of the current node and k is the index of the parent node of the current node. The preset global return coefficient can be represented by r, and then the weighted pose matrix can be representedComprises the following steps: q. q.s_(j,k)*r*ikTRS_(j,k)(ii) a The updated backhaul buffer matrix is denoted as buff'_(j)Then, the calculation formula of the updated backhaul buffer matrix may be: buff'_(j)＝q_(j,k)*r*ikTRS_(j,k)+buff_(j)。

By applying the skin total weight ratio to the weights of the IK solver, the accuracy of the solution results can be improved.

507. And normalizing the updated back-transmission cache matrix of each bone node to obtain the normalized back-transmission cache matrix of each bone node.

During normalization, the lower right corner element of the normalized return buffer matrix of each bone node is 1. In addition, based on the above calculation process, it can be seen that the feedback cache matrix is actually a local pose matrix, and therefore, the normalized feedback cache matrix of the bone node with index j can be represented as a corresponding local pose matrix, i.e., matTRS_(j)。

508. And updating the bone driving coefficient of each bone node according to the normalized return cache matrix of each bone node.

In some embodiments, the step of updating the bone driving coefficient of each bone node according to the normalized return buffer matrix of each bone node includes: acquiring a source value corresponding to a bone driving coefficient before updating of each bone node, wherein an initial value of the bone driving coefficient before updating is a bone driving coefficient corresponding to the base skin bone model; determining a target value of each skeleton node according to the local pose matrix of each skeleton node; calculating the difference value between the target value and the source value corresponding to each bone node, multiplying the difference value by a preset step length to obtain a product value, and adding the source value to the product value to obtain a result value of each bone node; and taking the bone driving coefficient corresponding to the result value of each bone node as the updated bone driving coefficient of each bone node.

Assuming that the source value is represented as source, the target value is represented as target, and the result value is represented as result, the calculation formula may be: result ═ target-source ═ step + source.

Specifically, for the translation coefficient and the scaling coefficient, the source value and the result value are the translation coefficient and the scaling coefficient before and after updating, and the target value is the translation coefficient and the scaling coefficient corresponding to the normalized pass-back cache matrix. The quadruple (w, x, y, z) is related to the rotation coefficient (α, β, γ) as follows:

w＝c1*c2*c3-s1*s2*s3

x＝s1*s2*c3+c1*c2*s3

y＝s1*c2*c3+c1*s2*s3

z＝c1*s2*c3-s1*c2*s3

wherein c1 ═ cos (α/2), c2 ═ cos (β/2), c3 ═ cos (γ/2), s1 ═ sin (α/2), s2 ═ sin (β/2), and s1 ═ sin (γ/2).

Therefore, conversion between the quadruple and the rotation coefficient can be performed in accordance with the above-described relationship.

By updating the bone drive coefficients based on the source values and the target values, the accuracy of the calculation of the bone drive coefficients may be improved.

509. And updating the second position coordinate according to the updated bone driving coefficient to obtain a third position coordinate.

That is, based on the updated bone drive coefficients, the skinned bone model is driven forward to drive the position coordinates of the second vertices from the second position coordinates VertexB_iUpdated to third position coordinate VertexB'_i。

As shown in fig. 7, which is a schematic diagram of forward driving a skinned bone model based on a bone driving coefficient, as shown in fig. 7, a process of forward driving the skinned bone model may include:

firstly, obtaining a bone driving coefficient of each bone node.

For a bone driver, the bone drive coefficient is a known value, and for a bone solver, the bone drive coefficient is a value to be solved.

Secondly, calculating a local pose matrix of each skeleton node based on the skeleton driving coefficient of each skeleton node, and calculating a global pose matrix of each skeleton node based on the local pose matrix of each skeleton node.

According to the corresponding relation between the bone driving coefficient and the local pose matrix, the bone driving coefficient localTRS can be used for determining the local pose matrix based on the bone driving coefficient localTRS_(j)Calculating corresponding local pose matrix matTRS_(j)。

The calculation formula for calculating the global pose matrix based on the local pose matrix is as follows:

if the bone node j is a root node, then: CurrentNode_(j)Global＝matTRS_(j)；

If the bone node j is a root node, then:

currentNode_(j)Global＝parentNode_(j)Global*matTRS_(j)；

wherein, currentNode_(j)Global is the Global position matrix of the skeleton node j, parentNode_(j)Global is the Global pose matrix, matTRS, of the parent node of skeleton node j_(j)Is the local pose matrix of the bone node j.

That is, for the root node, the global pose matrix and the local pose matrix of the root node are the same; and for the non-root node, calculating to obtain a global position and orientation matrix of the non-root node based on the local position and orientation matrix of the non-root node and the global position and orientation matrix of the father node of the non-root node.

And thirdly, calculating a global transformation matrix of each skeleton node based on the global pose matrix of each skeleton node.

As shown in fig. 8, the calculation formula is:

Derform_(j)＝currentNode_(j)Global*initNode_(j)Global^-1；

wherein, Derform_(j)Is the global transformation matrix of the skeleton node j, initNode_(j)Global is an initial Global pose matrix of the skeleton node j, i.e. a Global pose matrix when the skinned skeleton model is made.

And fourthly, transforming the position coordinates of each vertex of the skinned skeleton model based on the global transformation matrix of each skeleton node.

The calculation formula is as follows:

wherein, Vertex_iIs the position coordinate before transformation of vertex i; vertex'_iIs the transformed position coordinate of vertex i; alpha is alpha_(i,j)Is the weight of the action of the bone node j on the vertex i, i.e. the skinning weight, and m is the number of bone nodes having an action on the vertex i.

Thus, VertexB can be combined with the schematic shown in FIG. 7_iThe position coordinates before updating are calculated to obtain updated position coordinates VertexB'_iObtaining a third position coordinate VertexB 'after calculation'_i。

510. An error value is calculated based on the first position coordinate and the third position coordinate.

The error value is, for example, the first position coordinate VertexA_iAnd third position coordinate VertexB'_iThe mean square error value between.

511. And judging whether the error value is converged, if so, executing 512, otherwise, repeatedly executing 503 and the subsequent steps.

512. And taking the bone driving coefficient at the moment as the current bone driving coefficient.

The above is an example of determining the ending condition based on the error value, and may also be based on the update number, and is not described in detail here.

By judging the ending condition, the current bone driving coefficient can be obtained in time.

In the embodiment, the optimal pose matrix can be obtained by calculation through solving the optimal rigid transformation; by setting and updating the back-transmission cache matrix, the optimization result of the child node can be applied to the parent node, and the calculation accuracy of the bone driving coefficient of each node is improved; the accuracy of the calculation result can be improved by applying the skin total weight ratio to the weight of the IK calculator; by updating the bone drive coefficients based on the source values and the target values, the calculation accuracy of the bone drive coefficients can be improved; by judging the ending condition, the current bone driving coefficient can be obtained in time.

Fig. 9 is a schematic diagram according to a ninth embodiment of the present disclosure. The present embodiment provides an avatar generation apparatus, as shown in fig. 9, the apparatus 900 includes: an acquisition unit 901, a determination unit 902 and a transmission unit 903.

The obtaining unit 901 is configured to obtain a mesh model of an image and a preset base skin skeleton model, where the base skin skeleton model includes at least one skeleton node, and the mesh model and the base skin skeleton model have the same topological structure; a determining unit 902 is configured to determine, according to the mesh model and the base skin bone model, a current bone driving coefficient of each of the at least one bone node; the sending unit 903 is configured to send the current bone driving coefficient to a client, so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

In some embodiments, referring to fig. 10, another avatar generation apparatus 1000 is provided, the apparatus 1000 including an acquisition unit 1001, a determination unit 1002, and a transmission unit 1003.

The determining unit 1002 includes: a calculation module 10021, a cache update module 10022, a normalization module 10023, a coefficient update module 10024, and a determination module 10025.

The calculating module 10021 is configured to use each bone node as a current node, and calculate a return cache matrix before updating of the current node according to the grid model and the base skin bone model; the cache updating module 10022 is configured to calculate an IK pose matrix of each level of parent nodes on the IK chain where the current node is located by using an IK resolver, and update the post-update return cache matrix according to the IK pose matrix of the parent nodes, so as to obtain an updated return cache matrix of each bone node; the normalization module 10023 is configured to normalize the updated backhaul cache matrix of each bone node to obtain a normalized backhaul cache matrix of each bone node; the coefficient updating module 10024 is configured to update the bone driving coefficient of each bone node according to the normalized feedback cache matrix of each bone node until a preset end condition is reached; the determining module 10025 is configured to determine, as the bone driving coefficient to be adopted, a bone driving coefficient when the end condition is reached.

In some embodiments, the calculating module 10021 is specifically configured to: calculating an optimal local pose matrix of the current node according to the grid model and the base skin skeleton model; and accumulating the optimal local pose matrix into a preset return cache matrix to obtain an accumulated return cache matrix, and taking the accumulated return cache matrix as the return cache matrix before updating.

In some embodiments, the cache update module 10022 is specifically configured to: calculating the skin total weight ratio of the father node, wherein the ratio is the ratio of a first skin total weight to a second skin total weight, the first skin total weight is the skin total weight of the current node, and the second skin total weight is the skin total weight of the father node on the IK chain; calculating a weighted pose matrix according to the skin total weight ratio, a preset global return coefficient and the IK pose matrix of the father node; and accumulating the weighted pose matrix into the return cache matrix before updating.

In some embodiments, the calculation module 10021 is further specifically configured to: acquiring a first group of point clouds and a second group of point clouds, wherein the second group of point clouds comprises a group of second vertexes, the group of second vertexes are second vertexes included in a covering region acted by the current node in the base covering skeleton model, the first group of point clouds comprises a group of first vertexes, and the group of first vertexes are first vertexes included in a region corresponding to the covering region in the mesh model; solving an optimal rigid transformation matrix to ensure that the error between the point cloud after the second group of point clouds is transformed by the optimal rigid transformation matrix and the first group of point clouds is minimum; and taking the optimal rigid transformation matrix as the optimal local pose matrix.

In some embodiments, the normalized feedback cache matrix is a local pose matrix, and the coefficient updating module 10025 is specifically configured to: acquiring a source value corresponding to a bone driving coefficient before updating of each bone node, wherein an initial value of the bone driving coefficient before updating is a bone driving coefficient corresponding to the base skin bone model; determining a target value of each skeleton node according to the local pose matrix of each skeleton node; calculating the difference value between the target value and the source value corresponding to each bone node, multiplying the difference value by a preset step length to obtain a product value, and adding the source value to the product value to obtain a result value of each bone node; and taking the bone driving coefficient corresponding to the result value of each bone node as the updated bone driving coefficient of each bone node.

In some embodiments, the determining module 10026 is further configured to: determining that the ending condition is reached when the updating times of the bone driving coefficient reach preset times; and/or updating the second position coordinate according to the updated bone driving coefficient to obtain a third position coordinate; calculating an error value according to the first position coordinate and the third position coordinate, and determining that the ending condition is reached when the error value is converged; the first position coordinate is a position coordinate of the first vertex, and the second position coordinate is a position coordinate of the second vertex.

In some embodiments, the sending unit 1003 is specifically configured to: selecting the current bone driving coefficient of the bone node meeting a preset selection condition from the current bone driving coefficients of all the bone nodes to obtain the selected current bone driving coefficient; and sending the selected current skeleton driving coefficient to a client so that the client generates a current skin skeleton driving model according to the selected current skeleton driving coefficient, and taking the current skin skeleton driving model as the virtual image.

In this embodiment, the current bone drive coefficient is screened by the server, so that the data volume can be further reduced, the network flow overhead is further reduced, the resource occupation of the mobile terminal is reduced, and the mobile terminal is convenient to deploy. The optimal pose matrix can be obtained through calculation by solving the optimal rigid transformation; by setting and updating the back-transmission cache matrix, the optimization result of the child node can be applied to the parent node, and the calculation accuracy of the bone driving coefficient of each node is improved; the accuracy of the calculation result can be improved by applying the skin total weight ratio to the weight of the IK calculator; by updating the bone drive coefficients based on the source values and the target values, the calculation accuracy of the bone drive coefficients can be improved; by judging the ending condition, the current bone driving coefficient can be obtained in time.

It is understood that the same or corresponding contents in different embodiments of the present disclosure may be mutually referred, and the contents not described in detail in the embodiments may be referred to the related contents in other embodiments.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 11 shows a schematic block diagram of an example electronic device 1100 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 11, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the electronic device 1100 may also be stored. The calculation unit 1101, the ROM 1102, and the RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

A number of components in electronic device 1100 connect to I/O interface 1105, including: an input unit 1106 such as a keyboard, a mouse, and the like; an output unit 1107 such as various types of displays, speakers, and the like; a storage unit 1108 such as a magnetic disk, optical disk, or the like; and a communication unit 1109 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 1109 allows the electronic device 1100 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 1101 can be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The calculation unit 1101 performs the respective methods and processes described above, such as the avatar generation method. For example, in some embodiments, the avatar generation method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 1100 via the ROM 1102 and/or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the avatar generation method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the avatar generation method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server can be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method of generating an avatar, comprising:

acquiring a grid model of an image and a preset base skin skeleton model, wherein the base skin skeleton model comprises at least one skeleton node, and the grid model and the base skin skeleton model have the same topological structure;

determining a current bone drive coefficient of each of the at least one bone node from the mesh model and the base skin bone model;

and sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

2. The method of claim 1, wherein said determining a current bone drive coefficient for each of said at least one bone node from said mesh model and said base skin bone model comprises:

taking each skeleton node as a current node, and calculating a return cache matrix of the current node before updating according to the grid model and the base skin skeleton model;

calculating an IK pose matrix of each level of father nodes on an IK chain where the current node is located by adopting a reverse motion IK resolver, and updating the post-update post-return cache matrix according to the IK pose matrix of the father nodes to obtain the post-update post-return cache matrix of each skeleton node;

normalizing the updated back-transmission cache matrix of each bone node to obtain the normalized back-transmission cache matrix of each bone node;

updating the bone driving coefficient of each bone node according to the normalized return cache matrix of each bone node until a preset end condition is reached;

and determining the bone driving coefficient when the end condition is reached as the bone driving coefficient to be adopted.

3. The method according to claim 2, wherein said computing a pre-update return cache matrix for the current node from the mesh model and the base skinning skeleton model comprises:

calculating an optimal local pose matrix of the current node according to the grid model and the base skin skeleton model;

and accumulating the optimal local pose matrix into a preset return cache matrix to obtain an accumulated return cache matrix, and taking the accumulated return cache matrix as the return cache matrix before updating.

4. The method according to claim 2, wherein said updating the pre-update backhaul cache matrix according to the IK pose matrix of the parent node comprises:

calculating the skin total weight ratio of the father node, wherein the ratio is the ratio of a first skin total weight to a second skin total weight, the first skin total weight is the skin total weight of the current node, and the second skin total weight is the skin total weight of the father node on the IK chain;

calculating a weighted pose matrix according to the skin total weight ratio, a preset global return coefficient and the IK pose matrix of the father node;

and accumulating the weighted pose matrix into the return cache matrix before updating.

5. The method of claim 3, wherein the mesh model includes at least one first vertex and the base skinning bone model includes at least one second vertex, and wherein computing the optimal local pose matrix for the current node from the mesh model and the base skinning bone model comprises:

acquiring a first group of point clouds and a second group of point clouds, wherein the second group of point clouds comprises a group of second vertexes, the group of second vertexes are second vertexes included in a covering region acted by the current node in the base covering skeleton model, the first group of point clouds comprises a group of first vertexes, and the group of first vertexes are first vertexes included in a region corresponding to the covering region in the mesh model;

solving an optimal rigid transformation matrix to ensure that the error between the point cloud after the second group of point clouds is transformed by the optimal rigid transformation matrix and the first group of point clouds is minimum;

and taking the optimal rigid transformation matrix as the optimal local pose matrix.

6. The method according to claim 2, wherein the normalized pass-back buffer matrix is a local pose matrix, and the updating the bone driving coefficients of the respective bone nodes according to the normalized pass-back buffer matrix of the respective bone nodes comprises:

acquiring a source value corresponding to a bone driving coefficient before updating of each bone node, wherein an initial value of the bone driving coefficient before updating is a bone driving coefficient corresponding to the base skin bone model;

determining a target value of each skeleton node according to the local pose matrix of each skeleton node;

calculating the difference value between the target value and the source value corresponding to each bone node, multiplying the difference value by a preset step length to obtain a product value, and adding the source value to the product value to obtain a result value of each bone node;

and taking the bone driving coefficient corresponding to the result value of each bone node as the updated bone driving coefficient of each bone node.

7. The method of claim 2, further comprising:

determining that the ending condition is reached when the updating times of the bone driving coefficient reach preset times; and/or the presence of a gas in the gas,

updating the second position coordinate according to the updated bone driving coefficient to obtain a third position coordinate; calculating an error value according to the first position coordinate and the third position coordinate, and determining that the ending condition is reached when the error value is converged; the first position coordinate is a position coordinate of the first vertex, and the second position coordinate is a position coordinate of the second vertex.

8. The method of any of claims 1-7, further comprising:

selecting the current bone driving coefficient of the bone node meeting a preset selection condition from the current bone driving coefficients of all the bone nodes to obtain the selected current bone driving coefficient;

and sending the selected current skeleton driving coefficient to a client so that the client generates a current skin skeleton driving model according to the selected current skeleton driving coefficient, and taking the current skin skeleton driving model as the virtual image.

9. An avatar generation apparatus comprising:

the system comprises an acquisition unit, a calculation unit and a display unit, wherein the acquisition unit is used for acquiring a grid model of an image and a preset base skin skeleton model, the base skin skeleton model comprises at least one skeleton node, and the grid model and the base skin skeleton model have the same topological structure;

a determining unit, configured to determine, according to the mesh model and the base skin bone model, a current bone driving coefficient of each bone node of the at least one bone node;

and the sending unit is used for sending the current bone driving coefficient to a client so that the client generates an avatar corresponding to the image according to the current bone driving coefficient.

10. The apparatus of claim 9, wherein the determining unit comprises:

the computing module is used for respectively taking each skeleton node as a current node, and computing a return cache matrix before updating of the current node according to the grid model and the base skin skeleton model;

the cache updating module is used for calculating an IK pose matrix of each level of father nodes on an IK chain where the current node is located by adopting a reverse motion IK resolver, and updating the return cache matrix before updating according to the IK pose matrix of the father nodes so as to obtain the updated return cache matrix of each bone node;

the normalization module is used for normalizing the updated postback cache matrix of each bone node to obtain the normalized postback cache matrix of each bone node;

the coefficient updating module is used for updating the bone driving coefficient of each bone node according to the normalized return cache matrix of each bone node until a preset end condition is reached;

and the determining module is used for determining the bone driving coefficient when the ending condition is reached as the bone driving coefficient to be adopted.

11. The apparatus of claim 10, wherein the computing module is specifically configured to:

12. The apparatus of claim 10, wherein the cache update module is specifically configured to:

13. The apparatus of claim 11, wherein the mesh model comprises at least one first vertex and the base skin bone model comprises at least one second vertex, the computing module being further specific to:

14. The apparatus according to claim 10, wherein the normalized backhaul buffer matrix is a local pose matrix, and the coefficient update module is specifically configured to:

15. The apparatus of claim 10, wherein the means for determining is further configured to:

16. The apparatus according to any one of claims 9 to 15, wherein the transmitting unit is specifically configured to:

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.