CN114527873A - Virtual character model control method and device and electronic equipment - Google Patents

Abstract

The virtual character model control method, device, and electronic device provided by the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models for processing to obtain different limb parts corresponding to different limb parts. The three-dimensional pose data of the first key point is then combined with each group of three-dimensional pose data of the first key point to control the same virtual character model to perform corresponding actions. In this way, the decoupling between the 3D pose data of the first key point corresponding to different limbs can be realized even when there are few training samples of the prediction model, so that the same set of 3D pose data of the first key point can be controlled together by using each group. Avoid incorrect limb linkages on the avatar model.

Description

Virtual character model control method, device and electronic device

technical field

The present application relates to the technical field of image processing, and in particular, to a virtual character model control method, device, and electronic device.

Background technique

In some image processing scenarios, key points can be identified and predicted on a two-dimensional video image of a person, so as to obtain the three-dimensional pose data (such as spatial position coordinates and attitude angle) of the key points of the person's limbs for modeling or model control. For example, in some live broadcast scenarios, the position of key points of human limbs can be identified on the two-dimensional live video image obtained from the host terminal, and then the three-dimensional pose data can be predicted according to the coordinate position data of the key points of the limbs in the two-dimensional image. Obtain the 3D pose data of the key points of the limbs, and finally drive the corresponding virtual character model to imitate the anchor's actions according to the 3D pose data. Among them, the action of predicting and obtaining three-dimensional pose data according to the two-dimensional coordinate data of key points is usually performed by a machine learning model. However, due to the limitation of the number of training samples of the machine learning model or the diversity of training sample data, it may cause in the prediction results. The coupling of the 3D pose data corresponding to the relatively independent limbs is too high, which makes the subsequent modeling or model control process produce wrong limb linkage, which affects the modeling or model control effect.

SUMMARY OF THE INVENTION

In order to overcome the above deficiencies in the prior art, the purpose of this application is to provide a virtual character model control method, the method comprising:

Obtain the key point two-dimensional coordinate data of the target person from the two-dimensional image;

For at least two limb parts of the human body, extract corresponding at least two sets of first key point data sets from the two-dimensional coordinate data of the key points respectively;

Inputting the at least two sets of first key point data sets into at least two different first prediction models for processing, respectively, to obtain three-dimensional pose data of first key points corresponding to the at least two limb parts respectively;

Control the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of the first key points in each group.

In a possible implementation manner, the step of extracting corresponding at least two sets of first key point data sets from the two-dimensional coordinate data of the key points for at least two limb parts of the human body, respectively, includes:

For each limb part of the at least two limb parts, extracting from the key point two-dimensional coordinate data includes the key point two-dimensional coordinate data corresponding to the limb part and the key point two-dimensional coordinate data corresponding to the torso part as the first keypoint dataset for this limb part.

In a possible implementation manner, the three-dimensional pose data of the first key point includes spatial position data and attitude angle data of preset joint points in the corresponding limb part.

In a possible implementation, the method further includes:

Taking the two-dimensional coordinate data of the key points of the target person as a whole as the second key point data set;

Inputting the second key point data set into a second prediction model for processing to obtain overall three-dimensional pose data, wherein the overall three-dimensional pose data includes the three-dimensional position of the third key point corresponding to the at least two limb parts pose data and the 3D pose data of the second key point corresponding to the torso part;

The step of controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of each set of the first key points includes:

Use the first key point three-dimensional pose data to replace the third key point three-dimensional pose data in the overall three-dimensional pose data, and use the replaced overall three-dimensional pose data to control the virtual character model to perform corresponding operations. action.

In a possible implementation manner, the at least two limb parts include a left arm and a right arm, and the at least two different first prediction models include a left arm prediction model and a right arm prediction model;

The left arm prediction model includes the first fully connected network of the left arm, the second fully connected network of the left arm, and the third fully connected network of the left arm connected in sequence; the input of the first fully connected network of the left arm is a 44-dimensional left arm. The first key point data set of the arm, the output of the first fully connected network of the left arm is 512-dimensional data; the input of the second fully connected network of the left arm is the output of the first fully connected network of the left arm The output of the left arm second fully connected network is 512-dimensional data; the input of the left arm third fully connected network is the 512-dimensional data output by the left arm second fully connected network , the output of the third fully connected network of the left arm is the 12-dimensional three-dimensional pose data of the first key point of the left arm;

The right arm prediction model includes the first fully connected network of the right arm, the second fully connected network of the right arm, and the third fully connected network of the right arm connected in sequence; the input of the first fully connected network of the right arm is a 44-dimensional right arm. The first key point data set of the arm, the output of the first fully connected network of the right arm is 512-dimensional data; the input of the second fully connected network of the right arm is the output of the first fully connected network of the right arm 512-dimensional data, the output of the second fully connected network of the right arm is 512-dimensional data; the input of the third fully connected network of the right arm is the 512-dimensional data output by the second fully connected network of the right arm , the output of the third fully connected network of the right arm is the 12-dimensional three-dimensional pose data of the first key point of the right arm;

The second prediction model includes the first fully connected network of the torso, the second fully connected network of the torso, and the third fully connected network of the torso; the input of the first fully connected network of the torso is the 48-dimensional second key point three-dimensional pose data, the output of the first fully connected network of the torso is 512-dimensional data; the input of the second fully connected network of the torso is the 512-dimensional data output by the first fully connected network of the torso, the said The output of the second fully connected network of the torso is 512-dimensional data; the input of the third fully connected network of the torso is the 512-dimensional data output by the second fully connected network of the torso, and the output of the third fully connected network of the torso is the 144-dimensional overall three-dimensional pose data.

In a possible implementation, the method further includes:

extracting a second keypoint data set including the keypoints of the torso part from the two-dimensional coordinate data of the keypoints;

Inputting the second key point data set into the second prediction model for processing to obtain the three-dimensional pose data of the second key point;

The step of controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of each set of the first key points includes:

The same virtual character model is controlled to perform corresponding actions by using the three-dimensional pose data of the first key point and the three-dimensional pose data of the second key point.

In a possible implementation manner, the step of obtaining the key point two-dimensional coordinate data of the target person from the two-dimensional image includes:

Acquire two-dimensional coordinate data of key points of the host user from the first live video image of the host user; the first live video image is the two-dimensional image;

The step of controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of each set of the first key points includes:

According to the obtained three-dimensional pose data of each set of the first key point, the virtual character model corresponding to the anchor user is controlled to perform corresponding actions, so that the virtual character model performs similar actions to the anchor user.

Another object of the present application is to provide a virtual character model control device, the device comprising:

The acquisition module is used to acquire the two-dimensional coordinate data of the key points of the target person from the two-dimensional image;

an extraction module, configured to extract corresponding at least two sets of first key point data sets from the two-dimensional coordinate data of the key points for at least two limb parts of the human body;

A prediction module, configured to input the at least two sets of first key point data sets into at least two different first prediction models for processing, and obtain the three-dimensional poses of the first key points corresponding to the at least two limb parts respectively data;

The model control module is configured to control the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of the first key points in each group.

Another object of the present application is to provide an electronic device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions are executed by the processor. When the control method of the virtual character model provided by the present application is realized.

Another object of the present application is to provide a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions, when executed by one or more processors, implement the present application Provided avatar model control method.

Compared with the prior art, the present application has the following beneficial effects:

The virtual character model control method, device, and electronic device provided by the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models for processing to obtain different limb parts corresponding to different limb parts. The three-dimensional pose data of the first key point is then combined with each group of three-dimensional pose data of the first key point to control the same virtual character model to perform corresponding actions. In this way, the decoupling between the 3D pose data of the first key point corresponding to different limbs can be realized even when the training samples of the prediction model are few, so that the same set of 3D pose data of the first key point can be controlled together by using each group. Avoid wrong limb linkages on the avatar model.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following drawings will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a schematic flowchart of steps of a method for controlling a virtual character model provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of a live broadcast system provided by an embodiment of the present application.

FIG. 3 is a schematic diagram of an electronic device provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of functional modules of an apparatus for controlling a virtual character model provided by an embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

Thus, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application as claimed, but is merely representative of selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "arrangement", "installation", "connection" and "connection" should be interpreted in a broad sense, for example, it may be a fixed connection, It can also be a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal communication between two components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood in specific situations.

According to the research of the inventor, it is found that the process of obtaining the three-dimensional pose data of the key points of a person's limbs according to the two-dimensional video image of the person is usually performed by a trained machine learning model. In the training process of the machine learning model, people wearing 3D pose sensors usually perform various actions first. camera) to acquire two-dimensional images to obtain two-dimensional coordinates of limb key points on the two-dimensional images. Then, the two-dimensional coordinates and three-dimensional pose data of each limb key point are used as training samples, and the machine learning model is trained to predict the three-dimensional pose data of the limb key points according to the two-dimensional coordinates of the limb key points.

In the above method, due to the display of the number of training samples or the diversity of training sample data, there may be a large number of scenes in which different limbs act simultaneously in the training samples, but there are few scenes in which a limb moves alone. For example, there may be a large number of scenes in which the left hand and the right hand are active together in the training sample, and there are few scenes in which only one hand is active. This will lead to the prediction result of the 3D pose data output by the trained machine learning model is always biased towards different limbs moving together, and there is a problem of transition coupling between the 3D pose data of different limbs, which will lead to subsequent model reconstruction or model control. An incorrect limb linkage occurred. For example, in the actual 2D image, there is only left hand motion, but the prediction result of the machine learning model is that the left hand motion is followed by a slight motion of the right hand.

In view of the discovery and research on the above-mentioned problems, this embodiment provides a solution that can reduce false limb linkages generated by a virtual character model. The solution provided by this embodiment is described in detail below.

Please refer to FIG. 1. FIG. 1 is a flowchart of a method for controlling a virtual character model provided in this embodiment. The method includes various steps in detail below.

Step S110: Acquire two-dimensional coordinate data of key points of the target person from the two-dimensional image.

In this embodiment, the limb key points may correspond to various limb joints of the target person, such as shoulders, elbows, wrists, and the like. In a possible implementation manner, a pre-trained key point recognition model may be used to perform image recognition on a two-dimensional image (such as a two-dimensional video image collected by a camera) containing the target person, so as to determine each of the target person The position of the limb key in the two-bit image, and then the two-dimensional coordinate data of the key point of the target person is obtained.

Step S120, for at least two limb parts of the human body, extract corresponding at least two sets of first key point data sets from the two-dimensional coordinate data of the key points, respectively.

In this embodiment, according to the recognition result of the human body key points of the two-dimensional image, for at least two human body parts that can move independently, at least two groups of corresponding first two-dimensional coordinate data can be extracted from the key point two-dimensional coordinate data respectively. Keypoint dataset. For example, the left arm (including the left shoulder, the upper left arm, the lower left arm, and the left hand) and the right arm (including the right shoulder, the upper right arm, the lower right arm, and the right hand) are two limbs that can move relatively independently. At least one set of first keypoint data sets corresponding to the left arm and a set of first keypoint data sets corresponding to the right arm are extracted from the keypoint two-dimensional coordinate data. The first key point data set may include two-dimensional coordinate data of key points of key joints corresponding to limb parts. For example, the first key point data set of the left arm at least includes the two-dimensional coordinate data of key points of the left elbow and the left wrist, and the right The first key point data set of the arm at least includes the two-dimensional coordinate data of the key points of the right elbow and the right wrist.

Optionally, in this embodiment, each of the first key point data sets may correspond to different limb parts, for example, the first key point data set corresponding to the left arm may not include the coordinates of each key point corresponding to the right arm. Data, the first key point data set corresponding to the right arm may not include the coordinate data of each key point corresponding to the left arm.

Step S130: Input the at least two sets of first key point data sets into at least two different first prediction models for processing, and obtain first key point three-dimensional pose data corresponding to the at least two limb parts respectively.

In this embodiment, the at least two different first prediction models may be machine learning models that do not share network parameters. It can be understood that, in some cases, the at least two different first prediction models may have the same model network structure, but may have different model parameters according to different training samples.

In this embodiment, the first key point data sets corresponding to different limb parts can be input into different first prediction models for relatively independent prediction, so the three-dimensional pose data of each group of first key points obtained by prediction are decoupled of. For example, the first keypoint data sets of the left arm and the right arm are respectively input into different first prediction models for prediction, so the two-dimensional coordinate data of the keypoint corresponding to the right arm will not affect the three-dimensional position of the first keypoint corresponding to the left arm. Attitude data, the 2D coordinate data of the key point corresponding to the left arm will not affect the 3D pose data of the first key point corresponding to the right arm, thus realizing the decoupling of the 3D pose data of the first key point between the left arm and the right arm .

Wherein, the three-dimensional pose data of the first key point may include spatial position data and attitude angle data of preset joint points in the corresponding limb part. The space position data may be three-dimensional space position coordinates corresponding to the joint points, and the attitude angle data may be represented by the rotation change angles of the joint points in three directions in the three-dimensional space compared with the initial attitude.

Step S140, controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional pose data of the first key points in each group.

In this embodiment, the limb parts corresponding to the virtual character model can be controlled respectively according to the three-dimensional pose data of the first key points in each group. Since each group of the 3D pose data of the first key point is decoupled, when using each group of the 3D pose data of the first key point to control the same avatar model together, the erroneous limb linkage of the avatar model is avoided.

In a possible implementation manner, in step S120, for each limb part of the at least two limb parts, the two-dimensional coordinate data of the key point including the two-dimensional coordinate of the key point corresponding to the limb part may be extracted from the key point two-dimensional coordinate data The data and the two-dimensional coordinate data of the key points corresponding to the body part are used as the first key point data set of the limb part.

For example, since the left arm is connected to the torso and has a strong linkage relationship, in this embodiment, when acquiring the first key point data set corresponding to the left arm, the left arm can be extracted from the two-dimensional coordinate data of the key point. The two-dimensional coordinate data of the key points corresponding to each key joint of the arm (such as the left elbow, the left wrist, etc.) and the two-dimensional coordinate data of the key points corresponding to the torso are used as the first key point data set of the left arm. In this way, when predicting the three-dimensional pose data of the left arm subsequently, the two-dimensional coordinate data of the key points of the left arm and the torso can be predicted together, so that the prediction result can be more accurate.

When controlling the virtual character model to perform activities, in addition to the three-dimensional pose data of the limbs, the three-dimensional pose data of the torso may also be required. Therefore, in a possible implementation manner, the method may further include the following steps.

Step S210, taking the whole two-dimensional coordinate data of the key points of the target person as the second key point data set.

Step S220, inputting the second key point data set into a second prediction model for processing to obtain overall three-dimensional pose data, wherein the overall three-dimensional pose data includes a third key corresponding to the at least two limb parts point three-dimensional pose data and the second key point three-dimensional pose data corresponding to the body part.

In this embodiment, since all limbs are connected to the torso, in this implementation, in order to accurately predict the three-dimensional pose data of the torso, the entire two-dimensional coordinate data of the key points of the target person can be used as the second key The point dataset is input to the second prediction model for processing. The data output by the second prediction model may include the three-dimensional pose data of the third key point corresponding to the at least two limb parts and the three-dimensional pose data of the second key point corresponding to the torso part. Wherein, the torso part may include parts of the upper body of the human body other than the left and right arms, such as the body, neck, and head.

It can be understood that, in this embodiment, due to the display of the number of training samples or the diversity of training samples, the three-dimensional pose data corresponding to the third key point of each of the limb parts may be over-coupled. Therefore, in step S140, the 3D pose data of the third key point in the overall 3D pose data can be replaced with the 3D pose data of the first key point, and the overall 3D pose data after the replacement process can be used to control The virtual character model performs corresponding actions. In this way, the decoupled prediction results (the first key 3D pose data) are used to replace the possibly over-coupled prediction results (the third key 3D pose data) in the overall 3D pose data, and then the overall 3D pose after replacement is used. The data controls the virtual character model to perform corresponding actions, which can prevent the virtual character model from causing incorrect linkage of body movements.

Specifically, in this embodiment, taking the prediction processing of three-dimensional pose data on the upper body of the target person as an example, the at least two limb parts include a left arm and a right arm, and the at least two different first prediction models Including left arm prediction model and right arm prediction model.

In this case, the left arm prediction model includes a first fully connected network of the left arm, a second fully connected network of the left arm, and a third fully connected network of the left arm, which are connected in sequence. The input of the first fully connected network for the left arm is the 44-dimensional first keypoint data set of the left arm, which may include two-dimensional coordinate data of 22 joints of the left arm and the torso. The output of the first fully connected network of the left arm is 512-dimensional data, the input of the second fully connected network of the left arm is the 512-dimensional data output by the first fully connected network of the left arm, and the first fully connected network of the left arm is 512-dimensional data. The output of the second fully connected network is 512-dimensional data. The input of the third fully connected network of the left arm is the 512-dimensional data output by the second fully connected network of the left arm, and the output of the third fully connected network of the left arm is the first fully connected network of the 12-dimensional left arm. Three-dimensional pose data of key points, which can include 6D pose data of two joints of the left elbow and the left wrist.

The right arm prediction model includes a first fully connected network for the right arm, a second fully connected network for the right arm, and a third fully connected network for the right arm that are connected in sequence. The input of the first fully connected network for the right arm is the 44-dimensional first keypoint data set of the right arm, which may include two-bit coordinate data of 22 joints of the right arm and the trunk. The output of the first fully connected network of the right arm is 512-dimensional data, the input of the second fully connected network of the right arm is the 512-dimensional data output by the first fully connected network of the right arm, the The output of the second fully connected network is 512-dimensional data. The input of the third fully connected network of the right arm is the 512-dimensional data output by the second fully connected network of the right arm, and the output of the third fully connected network of the right arm is the first fully connected network of the 12-dimensional right arm. Three-dimensional pose data of key points, which can include 6D pose data of two joints of the right elbow and the right wrist.

The second prediction model includes a first fully connected network of the torso, a second fully connected network of the torso, and a third fully connected network of the torso, which are connected in sequence. The input of the first fully connected network of the torso is the 48-dimensional three-dimensional pose data of the second key point, which may include two-dimensional coordinate data of 24 joints of the left arm, the right arm and the torso. The output of the torso first fully connected network is 512-dimensional data, the input of the torso second fully connected network is the 512-dimensional data output by the torso second fully connected network, and the torso second fully connected network The output is 512-dimensional data, the input of the third fully connected network of the torso is the 512-dimensional data output by the second fully connected network of the torso, and the output of the third fully connected network of the torso is the 144-dimensional Overall 3D pose data, which can include 6D pose data of 24 joints of the left arm, right arm and torso.

As another possible implementation manner, the method further includes the following steps.

Step S310, extracting a second key point data set including the key points of the torso part from the two-dimensional coordinate data of the key points.

Step S320, the second key point data set is input into the second prediction model for processing to obtain the three-dimensional pose data of the second key point.

And in step S140, the three-dimensional pose data of the first key point and the three-dimensional pose data of the second key point may be used to control the same virtual character model to perform corresponding actions.

Wherein, the torso part may include parts of the upper body of the human body other than the left and right arms, such as the body, neck, and head. In this implementation, by separately extracting the two-dimensional coordinate data of the key points corresponding to the torso part, and using a separate prediction model for processing, the predicted three-dimensional pose data of the key points of the torso part and other limb parts are also solutions. Coupling, further avoiding wrong limb or body linkage when modeling or controlling the model later.

In this embodiment, the above solution can be applied to avatar control in a live broadcast system. Wherein, the two-dimensional image including the target task may be a live video image obtained from a live broadcast terminal, and the virtual character model may be a virtual character image corresponding to the host.

Specifically, in a possible implementation manner, please refer to FIG. 2 , which is a schematic diagram of a live broadcast system. The live broadcast system may include a host terminal 201 , a server 202 and a viewer terminal 203 .

The host user may use the host terminal 201 to shoot live video images, and the live video images may include a half-body or full-body image of the host user.

The server 202 may be an independent device or a cluster composed of multiple devices that work together. The server 202 may obtain the two-dimensional coordinate data of the key point of the host user from the live video image of the host user, and predict and obtain the three-dimensional pose data of the first key point corresponding to each limb part according to the two-dimensional coordinate data of the key point. Then, according to the obtained three-dimensional pose data of each set of the first key point, the virtual character model corresponding to the anchor user is controlled to perform corresponding actions, so that the virtual character model performs similar actions to the anchor user. Then, the server 202 may send the second live video image containing the avatar to the viewer terminal 203 or the host terminal 201 for display.

Based on the same inventive concept, this embodiment also provides an electronic device, which may have a certain image processing capability. For example, the electronic device may be a personal computer or the server 202 shown in FIG. 2 .

Please refer to FIG. 3 , which is a schematic block diagram of the electronic device 100 . The electronic device 100 includes a virtual character model control apparatus 110 , a machine-readable storage medium 120 , and a processor 130 .

The elements of the machine-readable storage medium 120 , the processor 130 and the communication unit 140 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements may be electrically connected to each other through one or more communication buses or signal lines. The virtual character model control device 110 includes at least one device that can be stored in the machine-readable storage medium 120 in the form of software or firmware (firmware) or solidified in an operating system (operating system, OS) of the electronic device 100 . Software function modules. The processor 130 is configured to execute executable modules stored in the machine-readable storage medium 120 , such as software function modules and computer programs included in the virtual character model control device 110 .

The machine-readable storage medium 120 may be, but not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-Only Memory) , PROM), erasable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM) and so on. The machine-readable storage medium 120 is used for storing a program, and the processor 130 executes the program after receiving the execution instruction.

The processor 130 may be an integrated circuit chip with signal processing capability. The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), and the like. It may also be a digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Referring to FIG. 4 , this embodiment further provides a virtual character model control device 110 . The virtual character model control device 110 includes at least one functional module that can be stored in the machine-readable storage medium 120 in the form of software. In terms of functions, the virtual character model control device 110 may include an acquisition module 111 , an extraction module 112 , a prediction module 113 and a model control module 114 .

The acquiring module 111 is used for acquiring the two-dimensional coordinate data of the key points of the target person from the two-dimensional image.

In this embodiment, the acquiring module 111 may be configured to execute the step S110 shown in FIG. 1 , and for the specific description of the acquiring module 111 , please refer to the description of the step S110 .

The extraction module 112 is configured to extract corresponding at least two sets of first key point data sets from the two-dimensional coordinate data of the key points for at least two limb parts of the human body, respectively.

In this embodiment, the extraction module 112 may be configured to execute the step S120 shown in FIG. 1 , and for the specific description of the extraction module 112 , please refer to the description of the step S120 .

The prediction module 113 is configured to input the at least two sets of first key point data sets into at least two different first prediction models for processing, respectively, to obtain three-dimensional first key points corresponding to the at least two limb parts respectively. pose data.

In this embodiment, the prediction module 113 may be configured to execute the step S130 shown in FIG. 1 , and for the specific description of the prediction module 113 , please refer to the description of the step S130 .

The model control module 114 is configured to control the same virtual character model to perform corresponding actions according to the obtained sets of three-dimensional pose data of the first key points.

In this embodiment, the model control module 114 may be configured to execute the step S140 shown in FIG. 1 , and for the specific description of the model control module 114 , please refer to the description of the step S140 .

To sum up, the virtual character model control method, device, and electronic device provided by the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models for processing to obtain the same The three-dimensional pose data of the first key points corresponding to different limb parts are then combined with each group of three-dimensional pose data of the first key points to control the same virtual character model to perform corresponding actions. In this way, the decoupling between the 3D pose data of the first key point corresponding to different limbs can be realized even when the training samples of the prediction model are few, so that the same set of 3D pose data of the first key point can be controlled together by using each group. Avoid incorrect limb linkages on the avatar model.

In the embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architectures, functions and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present application. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above are only various embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, All should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (10)

1. A virtual character model control method is characterized by comprising the following steps:

acquiring key point two-dimensional coordinate data of a target person from the two-dimensional image;

aiming at least two limb parts in four limbs of a human body, respectively extracting at least two groups of corresponding first key point data sets from the key point two-dimensional coordinate data;

inputting the at least two groups of first key point data sets into at least two different first prediction models respectively for processing to obtain first key point three-dimensional pose data respectively corresponding to the at least two limb parts;

and controlling the same virtual character model to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points.

2. The method according to claim 1, wherein the step of extracting at least two corresponding sets of first keypoint data sets from the keypoint two-dimensional coordinate data for at least two limb portions of the human body, respectively, comprises:

and extracting key point two-dimensional coordinate data corresponding to the limb part and key point two-dimensional coordinate data corresponding to the body part from the key point two-dimensional coordinate data as a first key point data set of the limb part for each limb part of the at least two limb parts.

3. The method of claim 2, wherein the first keypoint three-dimensional pose data comprises spatial position data and pose angle data of a pre-determined joint point in the corresponding limb portion.

4. The method of claim 2, further comprising:

taking the two-dimensional coordinate data of the key points of the target person as a second key point data set as a whole;

inputting the second key point data set into a second prediction model for processing to obtain integral three-dimensional pose data; wherein the overall three-dimensional pose data comprises third keypoint three-dimensional pose data corresponding to the at least two limb portions and second keypoint three-dimensional pose data corresponding to a torso portion;

the step of controlling the same virtual character model to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points comprises the following steps:

and replacing the third key point three-dimensional pose data in the whole three-dimensional pose data by using the first key point three-dimensional pose data, and controlling the virtual character model to execute corresponding actions by using the replaced whole three-dimensional pose data.

5. The method of claim 4, wherein the at least two limb portions comprise a left arm and a right arm, and the at least two different first predictive models comprise a left arm predictive model and a right arm predictive model;

the left arm prediction model comprises a left arm first full-connection network, a left arm second full-connection network and a left arm third full-connection network which are connected in sequence; the input of the left arm first fully connected network is the first keypoint data set of the 44-dimensional left arm, and the output of the left arm first fully connected network is 512-dimensional data; the input of the left arm second fully-connected network is 512-dimensional data output by the left arm first fully-connected network, and the output of the left arm second fully-connected network is 512-dimensional data; the input of the left arm third fully-connected network is 512-dimensional data output by the left arm second fully-connected network, and the output of the left arm third fully-connected network is 12-dimensional first key point three-dimensional pose data of the left arm;

the right arm prediction model comprises a right arm first full-connection network, a right arm second full-connection network and a right arm third full-connection network which are connected in sequence; the input of the right arm first fully connected network is the first keypoint data set of the right arm with 44 dimensions, and the output of the right arm first fully connected network is data with 512 dimensions; the input of the right arm second fully-connected network is 512-dimensional data output by the right arm first fully-connected network, and the output of the right arm second fully-connected network is 512-dimensional data; the input of the right arm third fully-connected network is 512-dimensional data output by the right arm second fully-connected network, and the output of the right arm third fully-connected network is 12-dimensional first key point three-dimensional pose data of the right arm;

the second prediction model comprises a first trunk full-connection network, a second trunk full-connection network and a third trunk full-connection network which are connected in sequence; the input of the first trunk fully-connected network is 48-dimensional second key point three-dimensional pose data, and the output of the first trunk fully-connected network is 512-dimensional data; the input of the second trunk fully-connected network is 512-dimensional data output by the first trunk fully-connected network, and the output of the second trunk fully-connected network is 512-dimensional data; the input of the third fully-connected network of the trunk is 512-dimensional data output by the second fully-connected network of the trunk, and the output of the third fully-connected network of the trunk is 144-dimensional integral three-dimensional pose data.

6. The method of claim 2, further comprising:

extracting a second key point data set comprising body part key points from the key point two-dimensional coordinate data;

inputting the second key point data set into a second prediction model for processing to obtain second key point three-dimensional pose data;

the step of controlling the same virtual character model to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points comprises the following steps:

and controlling the same virtual character model to execute corresponding actions by using the three-dimensional pose data of the first key point and the three-dimensional pose data of the second key point.

7. The method of claim 1, wherein the step of obtaining the two-dimensional coordinate data of the key points of the target person from the two-dimensional image comprises:

acquiring key point two-dimensional coordinate data of a main broadcast user from a first direct broadcast video image of the main broadcast user; the first live video image is the two-dimensional image;

the step of controlling the same virtual character model to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points comprises the following steps:

and controlling a virtual character model corresponding to the anchor user to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points, so that the virtual character model executes actions similar to those of the anchor user.

8. A virtual character model control device is characterized in that the device comprises:

the acquisition module is used for acquiring key point two-dimensional coordinate data of a target person from the two-dimensional image;

the extraction module is used for extracting at least two groups of corresponding first key point data sets from the key point two-dimensional coordinate data aiming at least two limb parts in human limbs;

the prediction module is used for inputting the at least two groups of first key point data sets into at least two different first prediction models respectively for processing to obtain first key point three-dimensional pose data respectively corresponding to the at least two limb parts;

and the model control module is used for controlling the same virtual character model to execute corresponding actions according to the obtained three-dimensional pose data of each group of the first key points.

9. An electronic device comprising a processor and a machine-readable storage medium having stored thereon machine-executable instructions that, when executed by the processor, implement the method of any of claims 1-7.

10. A machine-readable storage medium having stored thereon machine-executable instructions which, when executed by one or more processors, perform the method of any one of claims 1-7.

Images