WO2024000480A1

WO2024000480A1 - 3d virtual object animation generation method and apparatus, terminal device, and medium

Info

Publication number: WO2024000480A1
Application number: PCT/CN2022/103020
Authority: WO
Inventors: 陈恭斌; 王磊; 刘薰裕; 程俊
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2024-01-04

Abstract

The present application is applicable to the technical field of image processing. Provided are a 3D virtual object animation generation method and apparatus, a terminal device, and a medium. The animation generation method comprises: successively performing three-dimensional reconstruction on a reference object in each video frame of a target video to obtain three-dimensional reconstruction information of the reference object in each video frame; according to the three-dimensional reconstruction information of the reference objects in all the video frames and a preset object model, generating a first animation sequence corresponding to the reference object, the preset object model comprising a preset skeleton; on the basis of the preset skeleton, performing skeleton retargeting on the target skeleton of the target virtual object to obtain a retargeted skeleton, the structure of the target skeleton being the same as or similar to the structure of the preset skeleton; and, on the basis of the first animation sequence, performing animation retargeting on the retargeted skeleton to obtain a second animation sequence corresponding to the target virtual object, thereby improving animation generation efficiency of the target virtual object.

Description

Animation generation method, device, terminal equipment and media for 3D virtual objects

Technical field

The present application belongs to the field of image processing technology, and in particular relates to a method, device, terminal equipment and medium for animation generation of 3D virtual objects.

Background technique

Currently, three-dimensional (Three Dimensions, 3D) technology is widely used in games, movies, virtual reality (Virtual Reality, VR) and other fields. In the application process of 3D technology, it is usually necessary to design 3D virtual objects and a series of actions of the 3D virtual objects according to the application scenarios of the 3D technology to generate animations of the 3D virtual objects. For example, when 3D technology is applied in the game field, it is usually necessary to design a 3D game character and a series of actions of the 3D game character based on the 3D game scene, and generate animation of the 3D game character based on a series of actions of the 3D game character.

When generating animations of 3D virtual objects, the usual method is to first create key animation frames corresponding to key actions of the 3D virtual object, and then generate animation frames corresponding to other actions of the 3D virtual object based on the key animation frames. However, the production of key animation frames requires professionals, and the production process of key animation frames is complex and time-consuming, which reduces the efficiency of animation generation of 3D virtual objects.

technical problem

The purpose of the embodiments of the present application is to provide an animation generation method, device, terminal device and medium for 3D virtual objects, aiming to solve the existing problem of low efficiency in animation generation of 3D virtual objects.

Technical solutions

In order to solve the above technical problems, the technical solutions adopted in the embodiments of this application are:

In a first aspect, embodiments of the present application provide a method for generating animations of 3D virtual objects, including:

Sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video to obtain three-dimensional reconstruction information of the reference object in each video frame;

Generate the first animation sequence corresponding to the reference object according to the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton;

Skeleton redirection is performed on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

Perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.

In an optional implementation of the first aspect, generating a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model includes:

Get the default object model;

Based on the three-dimensional reconstruction information of the reference object in each video frame, set the structural parameters of the preset object model in each first animation frame of the first animation sequence respectively;

According to the preset first animation frame rate, all the first animation frames are sequentially composed into the first animation sequence.

In an optional implementation of the first aspect, the three-dimensional reconstruction information includes grid point information, bone key point information, root node position information and shooting parameter information; the video frame is based on each frame The three-dimensional reconstruction information of the reference object in the first animation sequence is used to set the structural parameters of the preset object model in the first animation frame of each frame of the first animation sequence, including:

For each video frame, determine the spatial position of the preset object model in the corresponding first animation frame according to the shooting parameters of the reference object in the video frame and the position information of the root node;

Based on the spatial position of the preset object model in the corresponding first animation frame, adjust the position of the root node of the preset object model in the corresponding first animation frame;

Based on the grid point information and skeletal key point information of the reference object in the video frame, the posture of the preset object model in the corresponding first animation frame is set.

In an optional implementation of the first aspect, before performing skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton, the method further includes:

Determine the bones to be merged from the preset skeleton, merge the bones to be merged, and obtain the merged bones;

Obtain a target object model of the target virtual object; the target object model includes an initial skeleton;

Based on the merged bones, corresponding bones in the initial skeleton are set to obtain the target bones of the target virtual object.

In an optional implementation of the first aspect, the preset object model further includes a preset mesh; the target virtual object model further includes a target mesh; and the pair of objects based on the preset skeleton is The target skeleton of the target virtual object performs skeleton redirection to obtain the redirected skeleton, including:

Match the bone key points of the target skeleton with the bone key points of the preset skeleton.

In an optional implementation of the first aspect, the three-dimensional reconstruction of the reference object in each video frame of the target video is performed sequentially to obtain the three-dimensional reconstruction information of the reference object in each video frame. ,include:

All video frames of the target video are imported into the preset three-dimensional reconstruction model frame by frame for processing, and the three-dimensional reconstruction information of the reference object in each video frame is obtained respectively.

In a second aspect, embodiments of the present application provide an animation generation device for 3D virtual objects, including:

A three-dimensional reconstruction unit, configured to sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video, and obtain three-dimensional reconstruction information of the reference object in each video frame;

An animation sequence generation unit, configured to generate a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model; the preset object model includes a preset skeleton;

a skeleton redirection unit, configured to perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

An animation redirection unit is configured to perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.

In an optional implementation of the second aspect, the animation sequence generating unit includes:

The first acquisition unit is used to acquire the preset object model;

A parameter setting unit configured to separately set the structural parameters of the preset object model in each first animation frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each frame of the video frame. Make settings;

A sorting unit, configured to sequence all the first animation frames into the first animation sequence according to a preset first animation frame rate.

In a third aspect, embodiments of the present application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the Steps in any optional manner as described above in the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, any optional method as described in the first aspect is implemented. Methods.

In a fifth aspect, embodiments of the present application provide a computer program product, which when the computer program product is run on a terminal device, causes the terminal device to execute the method described in the above first aspect or any alternative method of the first aspect.

The 3D virtual object animation generation method, device, terminal equipment, computer-readable storage medium and computer program product provided by the embodiments of the present application have the following beneficial effects:

The animation generation method of 3D virtual characters provided in this embodiment obtains the three-dimensional reconstruction information of the reference object in each video frame by sequentially performing three-dimensional reconstruction of the reference object in each video frame of the target video, and then based on all The three-dimensional reconstruction information of the reference object in the video frame and the preset object model generate a first animation sequence corresponding to the reference object; since the structure of the preset skeleton of the preset object model is the same or similar to the structure of the target skeleton of the target virtual object, therefore , convert the target video into the corresponding first animation sequence, which can provide a reference basis for the subsequent animation simulation of the target virtual object; then, first perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain the redirected skeleton, and then Perform animation redirection on the redirected skeleton based on the first animation sequence to obtain the second animation sequence corresponding to the target virtual object, so that the actions of the reference object in the target video can be mapped to the virtual object. That is, this solution can be used to automatically realize the target video The animation sequence of the target virtual object is obtained by simulating the motion of the reference object. Compared with the traditional method, there is no need for professionals to produce key animation frames of the target virtual object, thus improving the efficiency of animation generation of the target virtual object.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments or exemplary technologies will be briefly introduced below. Obviously, the drawings in the following description are only for the purpose of the present application. For some embodiments, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is an implementation flow chart of a method for generating animations of 3D virtual objects provided by an embodiment of the present application;

Figure 2 is a schematic diagram of a target virtual object provided by an embodiment of the present application;

Figure 3 is a schematic diagram of a first animation sequence generation process provided by an embodiment of the present application;

Figure 4 is an implementation flow chart of a method for generating a first animation sequence provided by an embodiment of the present application;

Figure 5 is an implementation flow chart of a skeleton redirection process provided by an embodiment of the present application;

Figure 6 is a schematic diagram of a second animation sequence generation process provided by an embodiment of the present application;

Figure 7 is a schematic structural diagram of a 3D virtual object animation generation device provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

Embodiments of the invention

It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. In the description of the embodiments of this application, unless otherwise specified, "plurality" refers to two or more than two, and "at least one" and "one or more" refer to one, two or more than two. The terms “first” and “second” are used for descriptive purposes only and shall not be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, defining "first" and "second" features may explicitly or implicitly include one or more of these features.

Reference in this specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in other embodiments", etc. appearing in different places in this specification are not necessarily References are made to the same embodiment, but rather to "one or more but not all embodiments" unless specifically stated otherwise. The terms “including,” “includes,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized.

The animation generation method of 3D virtual objects provided by the embodiment of the present application can be applied in three-dimensional (Three Dimensions, 3D) digital creation or virtual reality (Virtual Reality, VR) scenes. Among them, 3D digital creation can include but is not limited to 3D film and television creation or 3D game creation.

For example, in a 3D game creation scenario, the animation generation method of 3D virtual objects provided by the embodiments of the present application can be used to generate an animation sequence of 3D virtual objects in the game. For example, when you need to construct a target 3D virtual character in a 3D game scene, you can first record a video containing the character. The postures of the characters in different video frames in the video can be different. Three-dimensional reconstruction: obtain the three-dimensional reconstruction information of the character in each video frame, and then generate the first animation sequence corresponding to the character based on the three-dimensional reconstruction information of the character in all video frames and the preset object model; due to the skeleton structure of the preset object model It is the same as or similar to the skeleton structure of the target 3D virtual character. Therefore, converting the video into the first animation sequence can provide a reference basis for subsequent construction of the animation model of the target 3D virtual character; after that, the prediction based on the preset object model is first Assume that the skeleton performs skeleton redirection on the target skeleton of the target 3D virtual character to obtain the redirected skeleton, and then performs animation redirection on the redirected skeleton based on the first animation sequence to obtain the second animation sequence corresponding to the target 3D virtual character, so that the redirected skeleton can be obtained The actions of the characters in the video are mapped to a certain target 3D virtual character in the 3D game scene.

The execution subject of the animation generation method for 3D virtual objects provided by the embodiments of the present application may be a terminal device. Terminal devices can include electronic devices such as mobile phones, tablets, laptops, and desktop computers. Please refer to Figure 1. Figure 1 is an implementation flow chart of a 3D virtual object animation generation method provided by an embodiment of the present application. The 3D virtual object animation generation method may include S101~S104, as detailed below:

In S101, three-dimensional reconstruction is performed on the reference object in each video frame of the target video in sequence, and three-dimensional reconstruction information of the reference object in each video frame is obtained.

In a target scene (for example, a 3D game creation scene), when it is necessary to create a 3D virtual object and generate an animation sequence of the 3D virtual object, the terminal device can obtain the target video. The target video may include several video frames, and each video frame may include a reference object. The postures of the reference objects in different video frames may be the same or different.

The reference object refers to the object that the target virtual object refers to when simulating actions. Therefore, the skeleton structure of the reference object needs to be the same or similar to the skeleton structure of the target virtual object. The target virtual object may be any 3D virtual object among the 3D virtual objects that need to be created. For example, when the target virtual object is a human-like object (for example, 21 or 22 in Figure 2), the reference object may be a human.

The target video can be a video containing a reference object recorded in real time, a video containing a reference object that is pre-recorded or produced, a video containing a reference object downloaded from the Internet, or it can be obtained through other methods. Videos containing reference objects. There are no special restrictions on how to obtain the target video.

After the terminal device obtains the target video, it can process the target video into frames to obtain each video frame in the target video. For example, as shown in Figure 3, if 31 in Figure 3 is a target video, the terminal device can obtain multi-frame video frames 32 after performing frame segmentation processing on the target video 31.

After the terminal device obtains each video frame in the target video, it can sequentially perform three-dimensional reconstruction of the reference object in each video frame. The purpose of the terminal device performing three-dimensional reconstruction of the reference object in each video frame is to obtain the three-dimensional reconstruction information of the reference object in each video frame.

Wherein, performing three-dimensional reconstruction of the reference objects in each video frame sequentially means performing three-dimensional reconstruction of the reference objects in each video frame according to the order in which each video frame is arranged in the target video.

In one embodiment of the present application, the terminal device can use a three-dimensional reconstruction model to process each video frame in sequence to obtain the three-dimensional reconstruction information of the reference object in each video frame. The 3D reconstruction model can be used to quickly and efficiently obtain the 3D reconstruction information of the reference object in each video frame.

Among them, the three-dimensional reconstruction model can be obtained by training a deep learning model using a preset sample data set. Each piece of data in the preset sample data set includes a sample image and three-dimensional reconstruction information of the sample image. Among them, the sample image is an image containing a reference object. When using this preset sample data set to train the deep learning model, the sample image in each sample data is used as the input of the deep learning model, and the three-dimensional reconstruction information of the sample image in each sample data is used as the input of the deep learning model. Output, train the deep learning model, so that after the deep learning model training is completed, the deep learning model can learn the correspondence between the image containing the reference object and the three-dimensional reconstruction information of the reference object. The terminal device can determine the trained deep learning model as a three-dimensional reconstruction model.

As an example and not a limitation, when the reference object is a human, the three-dimensional reconstruction information of the reference object may include: grid point information, bone key point information, root node information, and shooting parameter information.

Among them, the grid point information is used to describe the outline of the reference object.

Skeleton keypoint information is used to describe the pose of the reference object. For example, the bone key point information may be the axis angle information of the bone key point. Among them, the bone key points can include the starting point and end point of each bone of the human body. The axis angle information of the bone key point is used to describe the three-dimensional space rotation information of the bone key point relative to its parent node.

The root node information is used to describe the pelvic position of the reference object. The root node information can be described by the coordinates of the reference object's pelvic key points in the preset coordinate system. The preset coordinate system can be set according to actual needs.

The shooting parameter information is used to describe the camera parameters when the reference object is photographed. The shooting parameter information may include but is not limited to the zoom ratio of the camera.

In the embodiment of this application, after the terminal device obtains the three-dimensional reconstruction information of the reference object in each video frame, it can store the bone key point information and root node information of the reference object in each video frame in an array respectively. And the arrays corresponding to all video frames are stored in the first preset file according to the order of each video frame in the target video. As an example and not a limitation, the first preset file may be a file in PKL format. PKL format is a file format used by python software to save files, and python is a computer programming language.

In S102, a first animation sequence corresponding to the reference object is generated based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model; the preset object model includes a preset skeleton.

Default object models refer to standard 3D reference object templates. Preset object models can be described by preset skeletons and preset meshes. Among them, the preset skeleton is the same as or similar to the skeleton of the reference object. For example, if the reference object is a human, the default object model is a standard 3D human body template.

It should be noted that the preset object model has a preset initial posture. For example, please continue to refer to Figure 3. 33 in Figure 3 is a standard 3D human body template, and its current posture is the above-mentioned initial posture.

The terminal device can adjust the posture and/or position of the preset object model according to the three-dimensional reconstruction information of the reference object in each video frame to obtain the first animation frame corresponding to each video frame and the third animation frame corresponding to all video frames. One animation frame constitutes the first animation sequence. As shown in FIG. 3 , the terminal device adjusts the posture and/or position of the preset object model 33 according to the three-dimensional reconstruction information of the reference object in each video frame 32 to obtain the first animation frame 34 corresponding to each frame 32 . , the first animation frames 34 corresponding to all the video frames 32 constitute the first animation sequence.

In an embodiment of the present application, S102 may include S41~S43 as shown in Figure 4, as detailed below:

In S41, the preset object model is obtained.

In a possible implementation manner, the preset object model can be stored in the object model library in advance. Based on this, the terminal device can obtain the pre-stored preset object model from the object model library.

In S42, based on the three-dimensional reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each first animation frame of the first animation sequence are respectively set.

Specifically, setting the structural parameters of the preset object model in each first animation frame may include setting the position and posture of the preset object model in each animation frame.

In a possible implementation, the terminal device can first generate multiple first animation frames containing the preset object model, and the number of the first animation frames is consistent with the number of video frames included in the target video; and then set the first animation frame of each frame. The initial position of the preset object model in an animation frame; and then adjusting the position and posture of the preset object model in the corresponding first animation frame based on the three-dimensional reconstruction information of each video frame.

Among them, the position information of the preset object in the first animation frame can be determined by the shooting parameters of the reference object in the corresponding video frame and the position information of the root node; the posture of the preset object model can be determined by the grid points of the reference object in the corresponding video frame. information and bone keypoint information determined.

In a possible implementation, S42 may include the following steps:

Step a: For each video frame, determine the spatial position of the preset object model in the corresponding first animation frame according to the shooting parameters and the position information of the root node of the reference object.

In this implementation, after the terminal device obtains the shooting parameters and the location information of the root node of the reference object in each video frame, it can combine weak perspective projection based on the scaling ratio in the shooting parameters and the location information of the root node. Relationship, calculate the spatial position of the preset object model in the first animation frame corresponding to each video frame.

Step b: Adjust the root node position of the preset object model in the first animation frame based on the spatial position of the preset object model in the corresponding first animation frame.

After the terminal device obtains the spatial position of the preset object model in the first animation frame of each frame, it adjusts the position of the root node of the preset object model in the first animation frame of each frame so that the root node of the preset object model The position of the node is consistent with the spatial position of the preset object model in the corresponding first animation frame.

Step c: Set the posture of the preset object model in the corresponding first animation frame based on the grid point information and the skeletal key point information.

After the terminal device adjusts the spatial position of the preset object model in the first animation frame of each frame, it can based on the grid point information and bone key point information of the reference object in the video frame corresponding to the first animation frame of each frame, The posture of the preset object model in the first animation frame of each frame is adjusted so that the posture of the preset object model in the first animation frame of each frame is the same as the posture of the reference object model in the corresponding video frame.

Specifically, the terminal device can first calculate the axis angle of each bone key point of the preset object model in the first animation frame of each frame based on the bone key point information of the reference object in the video frame corresponding to the first animation frame of each frame. Adjust so that the axis angle of each bone key point of the preset object model in the first animation frame of each frame is the same as the axis angle of each bone key point of the reference object model in the corresponding video frame; after that, the terminal device can be based on The grid point information of the reference object in the video frame corresponding to the first animation frame of each frame is adjusted to the grid points of the preset object model in the first animation frame of each frame, so that the first animation frame of each frame The grid points of the preset object model in are the same as the grid points of the reference object model in the corresponding video frame.

In S43, all the first animation frames are sequentially composed into the first animation sequence according to the preset first animation frame rate.

In this embodiment of the present application, the preset first animation frame rate can be set according to actual needs. For example, the preset first animation frame rate may be the same as the frame rate of the target video, or the preset first animation frame rate may be n times the frame rate of the target video, and n may be an integer greater than 1.

The terminal device can sort all the first animation frames according to the order of the video frames corresponding to the first animation frames of each frame in the target video, and all the sorted first animation frames constitute the first animation sequence. Specifically, the first animation frame corresponding to the first video frame is regarded as the first frame of the first animation sequence, the first animation frame corresponding to the second video frame is regarded as the second frame of the first animation sequence, and so on. , that is, the corresponding first animation frames of each of the above video frames can be composed into a first animation sequence in the above order.

After the terminal device obtains the first animation sequence, it can store the first animation sequence in the second preset file. For example, the second preset file may be a file in FBX format. FBX format is a file format used by Filmbox software to save files. Filmbox is a software that allows mutual guidance of models, materials, actions and camera information between 3D design software.

In the above scheme, by obtaining the preset object model and setting the structural parameters of the above-mentioned preset object in the first animation frame corresponding to each frame based on the three-dimensional reconstruction information of the reference object of each video frame, the video frame can be The action pose of the reference object in the animation is quickly transferred to the preset object in the first animation frame. Then, the first animation frames of each frame after the above-mentioned structural parameter setting are arranged in order. All the first animation frames after sorting constitute the first animation sequence, so that the target video can be mapped to the corresponding animation sequence, and each first animation frame can be mapped to the corresponding animation sequence. The posture and position of the reference object in the video frame are quickly mapped to the preset object model in the corresponding first animation frame, so that the preset object model in the first animation frame of each frame has the same characteristics as the reference object in the corresponding video frame. attitude and position.

In S103, perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the redirected skeleton is the same as or similar to the structure of the preset skeleton.

The target virtual object can be any 3D virtual object that needs to be created in the target scene, for example, it can be a 3D virtual character. Target scenes can include 3D game creation scenes or VR scenes, etc.

Each target virtual object corresponds to a default target object model. The target object model can be described by an initial skeleton and an initial mesh.

The terminal device can first determine the target skeleton of the target virtual object based on the preset skeleton and the initial skeleton of each target virtual object, and then perform skeleton redirection on the target skeleton of the target virtual object based on the above preset skeleton to obtain the redirected skeleton.

The structure of the target skeleton is the same as or similar to the structure of the preset skeleton.

In the embodiment of the present application, the skeleton reorientation process on the target skeleton refers to the process of corresponding the same bone key points of the target skeleton with the preset skeleton.

In one embodiment of the present application, the terminal device can obtain the target skeleton of each target virtual object through S51~S53 as shown in Figure 5, as detailed below:

In S51, bones to be merged are determined from the preset skeleton, and the bones to be merged are merged to obtain merged bones.

In this embodiment of the present application, the above-mentioned skeleton to be merged can be determined based on the actual situation of the preset skeleton. The bones to be merged in different skeletons can be different. The embodiment of the present application does not specifically limit the merging rules and merging methods of the bones to be merged in the preset skeleton. The skeleton composed of the merged bones can describe the action posture of the reference object.

In S52, a target object model of the target virtual object is obtained; the target object model includes an initial skeleton.

In a possible implementation manner, the target object model of the target virtual object can be pre-stored in the virtual object model library corresponding to the target scene. Based on this, the terminal device can obtain the target object model of the target virtual object from the virtual object model library.

In S53, the corresponding bones in the initial skeleton are set based on the merged bones to obtain the target skeleton of the target virtual object.

In the embodiment of this application, after the terminal device merges the bones to be merged in the preset skeleton, it can merge the corresponding bones in the initial skeleton based on the merge rules and merge methods of the merged bones, and the initial skeleton after the skeleton is merged Identifies the target skeleton as the target virtual object.

In one embodiment of the present application, since the naming rules of the key points of the target skeleton and the naming rules of the key points of the preset skeleton may be different, it is necessary to distinguish the key points of the target skeleton and the key points of the preset skeleton. Bone key points are matched to match the target skeleton with the same bone key points of the preset skeleton, thereby reorienting the target skeleton.

In S104, perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.

Animation redirection refers to copying action data from one skeleton to another. Therefore, animation redirection of the redirected skeleton based on the first animation sequence means copying the action data of the preset object model in the first animation sequence from The preset skeleton is copied to the redirected skeleton of the target object model, thereby obtaining a second animation sequence of the target object model.

In this embodiment of the present application, the terminal device may first generate multiple frames of second animation frames containing the target object model after skeleton redirection, where the number of second animation frames is consistent with the number of first animation frames; and then set each The initial position of the target object model in the second animation frame of each frame. For example, the initial position of the target object model in the second animation frame of each frame can be set at the center of the second animation frame of each frame; and then based on each frame The position and posture of the preset object model in the first animation frame are adjusted to the movement and posture of the target object model in the second animation frame.

Wherein, the position of the target object model in the second animation frame can be determined by the position of the root node of the preset object model in the corresponding first animation frame; the posture of the target object model can be determined by the posture of the preset object model in the corresponding first animation frame. Sure.

In a possible implementation, S104 may include the following steps:

Step d: For each second animation frame, adjust the spatial position of the target object model in the second animation frame according to the spatial position of the preset object model in the first animation frame of each frame.

In this implementation, the terminal device can obtain the spatial position of the preset object model in the first animation frame of each frame, and adjust the spatial position of the target object model in the second animation frame corresponding to each frame, so that the preset object The spatial position of the model in the first animation frame is the same as the spatial position of the target object model in the second animation frame, that is, even if the position of the root node of the preset object in the first animation frame is the same as the position of the target object model in the second animation frame The location of the root node is the same.

Step e: For each second animation frame, the terminal device adjusts the posture of the target object model so that the posture of the target object model is the same as the posture of the reference object model.

After the terminal device adjusts the spatial position of the target object model in each second animation frame, it can use the grid point information and bone key of the preset object model in the first animation frame corresponding to each second animation frame. point information, adjust the posture of the target object model in the second animation frame of each frame, so that the posture of the target object model in the second animation frame of each frame corresponds to the posture of the preset object model in the first animation frame same.

Specifically, the terminal device may first calculate the axes of each bone key point of the target object model in each second animation frame based on the bone key point information of the reference object in the first animation frame corresponding to each second animation frame. The angle is adjusted so that the axis angle of each bone key point of the target object model in the second animation frame of each frame is the same as the axis angle of each bone key point of the corresponding preset object model in the first animation frame; after that, the terminal The device can adjust the grid points of the target object model in each second animation frame based on the grid point information of the preset object in the first animation frame corresponding to each second animation frame, so that each The grid points of the target object model in the second animation frame are the same as the corresponding grid points of the preset object model in the first animation frame.

Step f: According to the preset second first animation frame rate, all the adjusted second first animation frames are sequentially composed into the second animation sequence.

In this embodiment of the present application, the preset second animation frame rate can be set according to actual needs. For example, the preset second animation frame rate may be the same as the first animation frame rate, or the preset second animation frame rate may be n times the first animation frame rate, and n may be an integer greater than 1.

The terminal device can sort all the second animation frames according to the order in which the second animation frames correspond to the first animation frames, and all the sorted second animation frames constitute the second animation sequence. Specifically, the second animation frame corresponding to the first animation frame of the first frame is regarded as the first frame of the second animation sequence, and the second animation frame corresponding to the first animation frame of the second frame is regarded as the second frame of the second animation sequence. , and by analogy, the corresponding second animation frames of each of the above first animation frames can be composed in the above order to form a second animation sequence.

After the terminal device obtains the second animation sequence, it can store the second animation sequence in the third preset file. For example, the third preset file may be a file in FBX format. FBX format is a file format used by filmbox software to save files. Filmbox is a software that allows mutual guidance of models, materials, actions and camera information between 3D design software.

For example, as shown in Figure 6, if 611 in Figure 6 is multiple video frames of the target video 61, the first animation frame obtained according to each video frame can be 621, and all the first animation frames constitute The first animation sequence 61; the second animation frame obtained according to each first animation frame may be 631, and all the second animation frames 631 constitute the second animation sequence 63.

The above scheme obtains the three-dimensional reconstruction information of the reference object in each video frame by sequentially performing three-dimensional reconstruction of the reference object in each video frame of the target video, and then based on the three-dimensional reconstruction information of the reference object in all video frames Generate a first animation sequence corresponding to the reference object with the preset object model; since the structure of the preset skeleton of the preset object model is the same or similar to the structure of the target skeleton of the target virtual object, the target video is converted into the corresponding first animation sequence. The animation sequence can provide a reference basis for the subsequent animation simulation of the target virtual object; then, first perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain the redirected skeleton, and then perform the redirected skeleton based on the first animation sequence. Animation redirection obtains the second animation sequence corresponding to the target virtual object, so that the actions of the reference object in the target video can be mapped to the virtual object. That is, this solution can be used to automatically simulate the actions of the reference object in the target video and obtain the target Compared with traditional methods, the animation sequence of virtual objects does not require professionals to produce key animation frames of the target virtual object, thereby improving the efficiency of animation generation of the target virtual object.

Based on the animation generation method of 3D virtual objects provided by the above embodiments, embodiments of the present application further provide an animation generation device for 3D virtual objects that implements the above method embodiments. Please refer to Figure 7. Figure 7 shows the animation generation method of 3D virtual objects provided by the embodiments of the present application. A schematic structural diagram of an animation generation device for 3D virtual objects. As shown in FIG. 7 , the animation generation device of 3D virtual objects may include: a three-dimensional reconstruction unit 71 , an animation sequence generation unit 72 , a skeleton reorientation unit 73 and an animation reorientation unit 74 . in:

The three-dimensional reconstruction unit 71 is configured to sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video to obtain three-dimensional reconstruction information of the reference object in each video frame.

The animation sequence generating unit 72 is configured to generate a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all video frames and a preset object model; the preset object model includes a preset skeleton.

The skeleton redirection unit 73 is configured to perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the redirected skeleton is the same as or similar to the structure of the preset skeleton.

The animation redirection unit 74 is configured to perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.

Optionally, the animation sequence generating unit 72 includes a first obtaining unit, a parameter setting unit and a sorting unit. in:

The first acquisition unit is used to acquire the preset object model.

The parameter setting unit is configured to separately set the structural parameters of the preset object model in each first animation frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each frame of the video frame. Make settings.

The sorting unit is configured to sequence all the first animation frames into the first animation sequence according to a preset first animation frame rate.

Optionally, the parameter setting unit includes a position determination unit, a position adjustment unit and an attitude adjustment unit. in:

The position determination unit is configured to determine, for each video frame, the space of the preset object model in the corresponding first animation frame according to the shooting parameters of the reference object in the video frame and the position information of the root node. Location.

The position adjustment unit is configured to adjust the position of the root node of the preset object model in the corresponding first animation frame based on the spatial position of the preset object model in the corresponding first animation frame.

The posture adjustment unit is configured to set the posture of the preset object model in the corresponding first animation frame based on the grid point information and skeletal key point information of the reference object in the video frame.

Optionally, the 3D virtual object animation generation device also includes a bone merging unit, a second acquisition unit, and a bone setting unit. in:

The bone merging unit is used to determine the bones to be merged from the preset skeleton, and merge the bones to be merged to obtain the merged bones.

The second acquisition unit is used to acquire the target object model of the target virtual object; the target object model includes an initial skeleton.

The bone setting unit is used to set corresponding bones in the initial skeleton based on the merged bones to obtain the target skeleton of the target virtual object.

Optionally, the skeleton redirection unit 73 is specifically used for:

Optionally, the three-dimensional reconstruction unit 71 is specifically used for:

All video frames of the target video are imported into a preset three-dimensional reconstruction model frame by frame for processing, and the three-dimensional reconstruction information of the reference object in each video frame is obtained respectively.

It should be noted that the information interaction, execution process, etc. between the above units are based on the same concept as the method embodiments of the present application. For details of their specific functions and technical effects, please refer to the method embodiments section, which will not be discussed here. Again.

Please refer to FIG. 8 , which is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in Figure 8, the terminal device 8 provided by this embodiment may include: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and runable on the processor 80, such as a method for generating animations of 3D virtual objects. corresponding program. When the processor 80 executes the computer program 82, the steps in the above embodiment of the animation generation method applied to 3D virtual objects are implemented, such as S101 to S104 shown in FIG. 1 . Alternatively, when the processor 80 executes the computer program 82 , the functions of each module/unit in the above embodiment of the animation generation device for 3D virtual objects are implemented, such as the functions of the units 71 to 74 shown in FIG. 7 .

For example, the computer program 82 may be divided into one or more modules/units, and one or more modules/units are stored in the memory 81 and executed by the processor 80 to complete the present application. One or more modules/units may be a series of computer program instruction segments capable of completing specific functions. The instruction segments are used to describe the execution process of the computer program 82 in the terminal device 8 . For example, the computer program 82 can be divided into a three-dimensional reconstruction unit, an animation sequence generation unit, a skeleton redirection unit, and an animation redirection unit. For the specific functions of each unit, please refer to the relevant descriptions in the corresponding embodiment of FIG. 7 and will not be described again here. .

Those skilled in the art can understand that FIG. 8 is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8. It may include more or less components than shown in the figure, or some components may be combined, or different components may be used. .

Processor 80 may be a central processing unit (central processing unit). processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (field-programmable) gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The memory 81 may be an internal storage unit of the terminal device 8 , such as a hard disk or memory of the terminal device 8 . The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card or a flash memory card (flash card) equipped on the terminal device 8. card) etc. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store computer programs and other programs and data required by the terminal device. The memory 81 may also be used to temporarily store data that has been output or is to be output.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units is used as an example. In practical applications, the above functions can be allocated to different functional units according to needs, that is, The internal structure of the power supply control device is divided into different functional units to complete all or part of the functions described above. Each functional unit in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware. , can also be implemented in the form of software functional units. In addition, the specific names of each functional unit are only for the convenience of distinguishing each other and are not used to limit the protection scope of the present application. For the specific working processes of the units in the above system, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be described again here.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When executed by a processor, the computer program can implement the steps in each of the above method embodiments.

Embodiments of the present application provide a computer program product. When the computer program product is run on a terminal device, the terminal device implements the steps in each of the above method embodiments.

In the above embodiments, each embodiment is described with its own emphasis. For parts that are not detailed or recorded in a certain embodiment, you can refer to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

The above-described embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still implement the above-mentioned implementations. The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of this application, and should be included in within the protection scope of this application.

Claims

An animation generation method for 3D virtual objects, which is characterized by including:

Sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video to obtain three-dimensional reconstruction information of the reference object in each video frame;

Generate the first animation sequence corresponding to the reference object according to the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton;

Skeleton redirection is performed on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

Perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.
The animation generation method of 3D virtual objects according to claim 1, characterized in that the first animation corresponding to the reference object is generated according to the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model. Sequences, including:

Get the default object model;

Based on the three-dimensional reconstruction information of the reference object in each video frame, set the structural parameters of the preset object model in each first animation frame of the first animation sequence respectively;

According to the preset first animation frame rate, all the first animation frames are sequentially composed into the first animation sequence.
The animation generation method of 3D virtual objects according to claim 2, characterized in that the three-dimensional reconstruction information includes grid point information, bone key point information, root node position information and shooting parameter information; Frame the three-dimensional reconstruction information of the reference object in the video frame, and separately set the structural parameters of the preset object model in the first animation frame of each frame of the first animation sequence, including:

For each video frame, determine the spatial position of the preset object model in the corresponding first animation frame according to the shooting parameters of the reference object in the video frame and the position information of the root node;

Based on the spatial position of the preset object model in the corresponding first animation frame, adjust the position of the root node of the preset object model in the corresponding first animation frame;

Based on the grid point information and skeletal key point information of the reference object in the video frame, the posture of the preset object model in the corresponding first animation frame is set.
The animation generation method of 3D virtual objects according to claim 1, characterized in that, before performing skeleton reorientation on the target skeleton of the target virtual object based on the preset skeleton, it further includes:

Determine the bones to be merged from the preset skeleton, merge the bones to be merged, and obtain the merged bones;

Obtain a target object model of the target virtual object; the target object model includes an initial skeleton;

Based on the merged bones, corresponding bones in the initial skeleton are set to obtain a target skeleton of the target virtual object.
The animation generation method of 3D virtual objects according to claim 4, characterized in that the preset object model further includes a preset mesh; the target virtual object model further includes a target mesh; The above-mentioned preset skeleton performs skeleton redirection on the target skeleton of the target virtual object to obtain the redirected skeleton, including:

Match the bone key points of the target skeleton with the bone key points of the preset skeleton.
The animation generation method of 3D virtual objects according to any one of claims 1 to 5, characterized in that the reference object in each video frame of the target video is sequentially three-dimensionally reconstructed to obtain the said Three-dimensional reconstruction information of the reference object in the video frame, including:

All video frames of the target video are imported into the preset three-dimensional reconstruction model frame by frame for processing, and the three-dimensional reconstruction information of the reference object in each video frame is obtained respectively.
An animation generation device for 3D virtual objects, which is characterized by including:

A three-dimensional reconstruction unit, configured to sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video, and obtain three-dimensional reconstruction information of the reference object in each video frame;

An animation sequence generation unit, configured to generate a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model; the preset object model includes a preset skeleton;

a skeleton redirection unit, configured to perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

An animation redirection unit is configured to perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.
The animation generation device for 3D virtual objects according to claim 7, wherein the animation sequence generation unit includes:

The first acquisition unit is used to acquire the preset object model;

A parameter setting unit configured to separately set the structural parameters of the preset object model in each first animation frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each frame of the video frame. Make settings;

A sorting unit, configured to sequence all the first animation frames into the first animation sequence according to a preset first animation frame rate.
The animation generation device for 3D virtual objects according to claim 8, wherein the parameter setting unit includes:

A position determination unit configured to, for each video frame, determine the position of the preset object model in the corresponding first animation frame according to the shooting parameters of the reference object in the video frame and the position information of the root node. Spatial location;

A position adjustment unit configured to adjust the position of the root node of the preset object model in the corresponding first animation frame based on the spatial position of the preset object model in the corresponding first animation frame;

A posture adjustment unit configured to set the posture of the preset object model in the corresponding first animation frame based on the grid point information and skeletal key point information of the reference object in the video frame.
The device for generating animation of 3D virtual objects according to claim 7, characterized in that the device for generating animation of 3D virtual objects further includes:

A bone merging unit is used to determine the bones to be merged from the preset skeleton, and merge the bones to be merged to obtain the merged bones;

a second acquisition unit, configured to acquire the target object model of the target virtual object; the target object model includes an initial skeleton;

A skeleton setting unit is configured to set the corresponding bones in the initial skeleton based on the merged skeleton to obtain the target skeleton of the target virtual object.
The animation generation device for 3D virtual objects according to claim 10, wherein the skeleton reorientation unit is specifically used to:

Match the bone key points of the target skeleton with the bone key points of the preset skeleton.
The animation generation device for 3D virtual objects according to any one of claims 7 to 11, wherein the three-dimensional reconstruction unit is specifically used for:

All video frames of the target video are imported into the preset three-dimensional reconstruction model frame by frame for processing, and the three-dimensional reconstruction information of the reference object in each video frame is obtained respectively.
A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, the following steps are implemented:

Sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video to obtain three-dimensional reconstruction information of the reference object in each video frame;

Generate the first animation sequence corresponding to the reference object according to the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton;

Skeleton redirection is performed on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

Perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.
The terminal device according to claim 13, wherein generating the first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model includes:

Get the default object model;

Based on the three-dimensional reconstruction information of the reference object in each video frame, set the structural parameters of the preset object model in each first animation frame of the first animation sequence respectively;

According to the preset first animation frame rate, all the first animation frames are sequentially composed into the first animation sequence.
The terminal device according to claim 14, characterized in that the three-dimensional reconstruction information includes grid point information, bone key point information, root node position information and shooting parameter information; the video frame is based on each frame The three-dimensional reconstruction information of the reference object in the first animation sequence is used to set the structural parameters of the preset object model in the first animation frame of each frame of the first animation sequence, including:

For each video frame, determine the spatial position of the preset object model in the corresponding first animation frame according to the shooting parameters of the reference object in the video frame and the position information of the root node;

Based on the spatial position of the preset object model in the corresponding first animation frame, adjust the position of the root node of the preset object model in the corresponding first animation frame;

Based on the grid point information and skeletal key point information of the reference object in the video frame, the posture of the preset object model in the corresponding first animation frame is set.
The terminal device according to claim 13, wherein when the processor executes the computer program and before implementing the skeleton redirection of the target skeleton of the target virtual object based on the preset skeleton, the processor further includes:

Determine the bones to be merged from the preset skeleton, merge the bones to be merged, and obtain the merged bones;

Obtain a target object model of the target virtual object; the target object model includes an initial skeleton;

Based on the merged bones, corresponding bones in the initial skeleton are set to obtain a target skeleton of the target virtual object.
The terminal device according to claim 16, wherein the preset object model further includes a preset mesh; the target virtual object model further includes a target mesh; and the pair of objects based on the preset skeleton is The target skeleton of the target virtual object performs skeleton redirection to obtain the redirected skeleton, including:

Match the bone key points of the target skeleton with the bone key points of the preset skeleton.
The terminal device according to any one of claims 13 to 17, characterized in that the reference object in each video frame of the target video is sequentially three-dimensionally reconstructed to obtain the reference object in each video frame. 3D reconstruction information of the object, including:

All video frames of the target video are imported into a preset three-dimensional reconstruction model frame by frame for processing, and the three-dimensional reconstruction information of the reference object in each video frame is obtained respectively.
A computer-readable storage medium stores a computer program, wherein the computer program implements the following steps when executed by a processor:

Sequentially perform three-dimensional reconstruction of the reference object in each video frame of the target video to obtain three-dimensional reconstruction information of the reference object in each video frame;

Generate the first animation sequence corresponding to the reference object according to the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton;

Skeleton redirection is performed on the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton;

Perform animation redirection on the redirected skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.
The computer-readable storage medium according to claim 19, wherein the first animation sequence corresponding to the reference object is generated based on the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model, include:

Get the default object model;

Based on the three-dimensional reconstruction information of the reference object in each video frame, set the structural parameters of the preset object model in each first animation frame of the first animation sequence respectively;

According to the preset first animation frame rate, all the first animation frames are sequentially composed into the first animation sequence.