CN117788672A - Real-time dynamic human body new perspective rendering method and system based on multi-view video - Google Patents

Abstract

The invention discloses a real-time dynamic human body new perspective rendering method and system based on multi-view video. This method estimates the posture parameters of the human body based on the multi-view human body image information; constructs the three-dimensional geometric field and texture feature field of the human body based on the posture parameters; based on The implicit neural network performs volume rendering on the shooting perspective image to obtain the shooting perspective rendering image, and constructs a consistency constraint between the shooting perspective rendering image and the shooting perspective image, so as to obtain the three-dimensional geometric field and texture feature field as optimized variables. The optimized texture feature field; based on the optimized texture feature field and human body image information, the human body is rendered from a new perspective to obtain a new perspective rendering image. The invention can realize a new perspective rendering of a dynamic three-dimensional human body with a three-dimensional effect.

Description

Real-time dynamic human body new perspective rendering method and system based on multi-view video

Technical field

The invention relates to the technical fields of computer vision and computer graphics, and in particular to a real-time dynamic human body new perspective rendering method and system based on multi-view videos.

Background technique

In production and life, real-time remote video call technology between people has been widely used. As people's pursuit of immersion and experience in remote communication continues to increase, realizing three-dimensional, three-dimensional, real-time remote communication with freely changing viewing angles has become a new technical requirement. Dynamic three-dimensional human body reconstruction technology has broad application prospects and important application value in fields such as virtual reality, augmented reality, remote communications, film and television animation, etc. In practical applications, people often have real-time communication needs. Compared with conventional two-dimensional videos, three-dimensional videos with a more three-dimensional feel can bring a more immersive experience. Therefore, how to implement new perspective rendering is a gap in current technology.

Contents of the invention

The present invention aims to solve one of the technical problems in the related art, at least to a certain extent.

To this end, the present invention proposes a real-time dynamic human body new perspective rendering method based on multi-view videos, which can achieve real-time three-dimensional reconstruction of the target three-dimensional human body through multi-view human body movement videos taken by users, and render out any perspective of the human body. images to achieve a three-dimensional presentation effect.

Another object of the present invention is to provide a real-time dynamic human body new perspective rendering system based on multi-perspective video.

In order to achieve the above objectives, on the one hand, the present invention proposes a real-time dynamic human body new perspective rendering method based on multi-view videos, including:

Estimating the posture parameters of the human body based on multi-view human body image information;

Construct a three-dimensional geometric field and texture feature field of the human body based on the posture parameters;

Based on the implicit neural network, the shooting perspective image is volume rendered to obtain the shooting perspective rendering image, and a consistency constraint between the shooting perspective rendering image and the shooting perspective image is constructed to combine the three-dimensional geometric field and the texture feature field. The optimized texture feature field is obtained as an optimized variable;

Based on the optimized texture feature field and the human body image information, a new perspective rendering of the human body is performed to obtain a new perspective rendering image.

The real-time dynamic human body new perspective rendering method based on multi-view videos according to the embodiment of the present invention may also have the following additional technical features:

In one embodiment of the present invention, estimating posture parameters of a human body based on multi-view human body image information includes:

Obtain multi-view human body image information;

Use two-dimensional human posture estimation tools to calculate the two-dimensional coordinates of human joint points in human body image information;

Solve the three-dimensional human body posture based on the two-dimensional coordinates of the human body joint points and the multi-view geometric information of the human body

In one embodiment of the present invention, the three-dimensional geometric field and texture feature field of the human body are constructed based on the posture parameters, including:

Constructing three-dimensional voxels for representing three-dimensional geometric fields and texture feature fields;

The three-dimensional voxels are used to represent the three-dimensional human body posture to obtain human body geometric information and human body surface texture characteristics in standard space.

In one embodiment of the present invention, the three-dimensional geometric field records the signed distance function value of the current three-dimensional voxel, that is, the distance value of the current three-dimensional voxel position to the nearest point on the human body surface; for the three-dimensional voxels inside the human body, the distance value is negative, and for the three-dimensional voxels outside the human body, the distance value is positive.

In one embodiment of the present invention, volume rendering is performed on the shooting perspective image based on an implicit neural network to obtain a shooting perspective rendering image, including:

Project a ray in the direction of a pixel of the shooting perspective image, sample a number of light points, and calculate the color value of the light point through an implicit neural network;

Obtain the density information of the light point by querying the three-dimensional geometric field;

Weighted integration is performed based on the color value and density information of the light points to obtain the color data of the pixels to obtain the shooting perspective rendering image.

To achieve the above object, the present invention further provides a real-time dynamic human body new perspective rendering system based on multi-perspective video, comprising:

The posture parameter estimation module is used to estimate the posture parameters of the human body based on multi-view human body image information;

A human body feature characterization module, used to construct a three-dimensional geometric field and texture feature field of the human body based on the posture parameters;

A feature variable optimization module is used to perform volume rendering on the shooting perspective image based on an implicit neural network to obtain a shooting perspective rendering image, and to construct a consistency constraint between the shooting perspective rendering image and the shooting perspective image to reduce the The three-dimensional geometric field and texture feature field are used as optimized variables to obtain the optimized texture feature field;

A perspective image rendering module is used to perform a new perspective rendering of the human body based on the optimized texture feature field and the human body image information to obtain a new perspective rendering image.

The real-time dynamic human body new perspective rendering method and system based on multi-view videos according to the embodiment of the present invention can realize real-time three-dimensional reconstruction of the target three-dimensional human body through multi-view human body movement videos taken by users, and render images from any perspective. Achieve a three-dimensional presentation effect.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

Figure 1 is a flow chart of a real-time dynamic human body new perspective rendering method based on multi-view videos according to an embodiment of the present invention;

Figure 2 is a schematic network architecture diagram of real-time dynamic human body new perspective rendering based on multi-view videos according to an embodiment of the present invention;

Figure 3 is a schematic structural diagram of a real-time dynamic human body new perspective rendering system based on multi-view videos according to an embodiment of the present invention.

Detailed ways

It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of the present invention.

The real-time dynamic human body new perspective rendering method and system based on multi-view videos proposed according to the embodiments of the present invention will be described below with reference to the accompanying drawings.

Figure 1 is a flow chart of a real-time dynamic human body new perspective rendering method based on multi-view videos according to an embodiment of the present invention.

As shown in Figure 1, the method includes but is not limited to the following steps:

S1, estimate the posture parameters of the human body based on multi-view human body image information.

It can be understood that the present invention estimates the posture parameters of the human body through multi-view human movement videos.

Specifically, multi-view human body image information is obtained, a two-dimensional human body posture estimation tool is used to calculate the two-dimensional coordinates of human joint points in the human body image information, and the three-dimensional human body posture is solved based on the two-dimensional coordinates of human joint points and the multi-view geometric information of the human body.

In one embodiment of the present invention, the present invention uses the two-dimensional human body posture estimation tool OpenPose to calculate the two-dimensional coordinates of human body joints, and then combines multi-view geometric information to solve the three-dimensional human body posture.

In one embodiment of the present invention, solving the posture of the dynamic human body is beneficial to aligning the human body in different postures into the standard "big" glyph space as shown in Figure 2, so as to integrate different human postures in a unified standard space. Information.

S2, construct the three-dimensional geometric field and texture feature field of the human body based on the posture parameters.

It can be understood that the present invention further constructs a three-dimensional geometric field and texture feature field of the human body.

Specifically, three-dimensional voxels are constructed to represent the three-dimensional geometric field and texture feature field, and the three-dimensional voxels are used to represent the three-dimensional human body posture to obtain human body geometric information and human body surface texture features in standard space.

In one embodiment of the present invention, in order to achieve modeling of three-dimensional human body geometry and surface texture, the present invention introduces a three-dimensional geometric field and a texture feature field, both of which are stored in the form of three-dimensional voxels and are represented in the standard space of the human body's "big" shape.

Among them, the three-dimensional geometric field records the directional distance function value of each voxel, that is, the distance value of the voxel position from the closest point to the human body surface. For voxels inside the human body, the distance value is negative, and for voxels outside the human body, the distance value is negative. , the distance value is positive. Through this three-dimensional geometric field, the human body geometry in standard space can be determined.

Furthermore, the texture feature field stores the texture features of the human body surface, which are used to assist in new perspective image rendering.

S3. Perform volume rendering on the shooting perspective image based on the implicit neural network to obtain the shooting perspective rendering image, and construct a consistency constraint between the shooting perspective rendering image and the shooting perspective image to combine the three-dimensional geometric field and the texture feature field. The optimized texture feature field is obtained as an optimized variable.

Specifically, after the present invention obtains the three-dimensional geometric field and texture feature field, the present invention uses an implicit neural network to perform volume rendering under the shooting perspective to obtain a series of rendered images.

In one embodiment of the present invention, for a pixel in the image, the present invention projects a light ray in the direction of the pixel, samples several light points, and queries the color value of the light point through an implicit neural network. At the same time, the density of light points can be obtained by querying the three-dimensional geometric field. Combining density and color, a weighted integration can be performed to obtain the color of the pixel. The implicit neural network here inputs the texture features at the ray point and outputs the color of the ray point.

After rendering the image, you can construct a consistency constraint between the rendered image and the captured image, requiring the pixel-by-pixel L2 error of the two images to be as small as possible, and using the three-dimensional geometric field and texture feature field as optimizable variables to achieve Iterative optimization of both.

S4, performing new perspective rendering of the human body based on the optimized texture feature field and the human body image information to obtain a new perspective rendering image.

Specifically, in order to achieve a higher-quality new perspective rendering effect, the present invention further combines the texture feature field optimized in step S3 with the features in the input image to perform image rendering.

It is understandable that the texture feature field obtained through multi-frame fusion optimization provides relatively complete information, but lacks texture details; on the other hand, the input multi-view image features contain more texture details, but some of them have not been observed. The information in these areas is missing.

It can be understood that the present invention combines the advantages of both, inputs the texture features at the light point and the image features obtained by projection through an implicit neural network, outputs the color of the light point, and renders the human body image.

So far, the present invention has achieved real-time reconstruction of the target three-dimensional human body and supports rendering from a new perspective with a three-dimensional sense.

Figure 2 is an architecture diagram of the present invention. As shown in Figure 2, the posture parameters of the human body are estimated through multi-view human movement videos. Construct the three-dimensional geometric field and texture feature field of the human body. Optimize the three-dimensional geometric field and texture feature field of the human body. Real-time rendering is performed based on input image information. The present invention uses multi-view human body motion video as input, realizes real-time dynamic three-dimensional human body reconstruction, and supports real-time new-view video rendering with a specific three-dimensional sense.

The real-time dynamic human body new perspective rendering method based on multi-perspective video of the embodiment of the present invention can realize real-time three-dimensional reconstruction of the target three-dimensional human body through the multi-perspective human body movement video shot by the user, and render the image at any perspective, thereby achieving a presentation effect with a three-dimensional stereoscopic sense.

In order to implement the above embodiment, as shown in Figure 3, this embodiment also provides a real-time dynamic human body new perspective rendering system 10 based on multi-view video. The system 10 includes a posture parameter estimation module 100, a human body feature characterization module 200, Feature variable optimization module 300 and perspective image rendering module 400;

The posture parameter estimation module 100 is used to estimate the posture parameters of the human body based on multi-view human body image information;

The human body feature representation module 200 is used to construct the three-dimensional geometric field and texture feature field of the human body based on posture parameters;

The feature variable optimization module 300 is used to perform volume rendering on the shooting perspective image based on the implicit neural network to obtain the shooting perspective rendering image, and construct a consistency constraint between the shooting perspective rendering image and the shooting perspective image to combine the three-dimensional geometric field and texture The feature field is used as an optimized variable to obtain the optimized texture feature field;

The perspective image rendering module 400 is used to render the human body from a new perspective based on the optimized texture feature field and human body image information to obtain a new perspective rendering image.

Further, the attitude parameter estimation module 100 is also used to:

Obtain multi-view human body image information;

Use two-dimensional human posture estimation tools to calculate the two-dimensional coordinates of human joint points in human body image information;

Solve the 3D human posture based on the 2D coordinates of human joints and multi-view geometric information of the human body.

Furthermore, the human body feature characterization module 200 is also used for:

Construct three-dimensional voxels used to represent three-dimensional geometric fields and texture feature fields;

Three-dimensional voxels are used to represent the three-dimensional human body posture to obtain human body geometric information and human body surface texture characteristics in standard space.

Furthermore, the three-dimensional geometric field records the directed distance function value of the current three-dimensional voxel, that is, the distance value of the current three-dimensional voxel position from the closest point to the human body surface; for the three-dimensional voxels inside the human body, the distance value is negative, and for the three-dimensional voxels outside the human body, the distance value is negative. 3D voxels, the distance value is positive.

Further, the feature variable optimization module 300 is also used to:

Project a ray in the direction of a pixel of the shooting perspective image, sample a number of light points, and calculate the color value of the light point through an implicit neural network;

Get the density information of light points by querying the three-dimensional geometric field;

Based on the color value and density information of the light points, weighted integration is performed to obtain the color data of the pixels to obtain the shooting perspective rendering image.

The real-time dynamic human body new perspective rendering system based on multi-view videos according to the embodiment of the present invention can realize real-time three-dimensional reconstruction of the target three-dimensional human body through multi-view human body movement videos taken by users, and render images from any viewing angle, achieving the goal of Three-dimensional rendering effect.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.

In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.

Claims (10)

1. A real-time dynamic human body new perspective rendering method based on multi-perspective video, characterized in that the method comprises the following steps: Estimate human body posture parameters based on multi-view human image information; Construct a three-dimensional geometric field and texture feature field of the human body based on the posture parameters; Based on the implicit neural network, the shooting perspective image is volume rendered to obtain the shooting perspective rendering image, and a consistency constraint between the shooting perspective rendering image and the shooting perspective image is constructed to combine the three-dimensional geometric field and the texture feature field. The optimized texture feature field is obtained as an optimized variable; A new perspective rendering of the human body is performed based on the optimized texture feature field and the human body image information to obtain a new perspective rendering image. 2. The method according to claim 1, characterized in that estimating posture parameters of a human body based on multi-view human body image information comprises: Obtain multi-view human body image information; Use two-dimensional human posture estimation tools to calculate the two-dimensional coordinates of human joint points in human body image information; The three-dimensional human body posture is solved based on the two-dimensional coordinates of the human body joint points and the multi-view geometric information of the human body. 3. The method according to claim 2, characterized in that constructing the three-dimensional geometric field and texture feature field of the human body based on the posture parameters includes: Constructing three-dimensional voxels for representing three-dimensional geometric fields and texture feature fields; The three-dimensional voxels are used to represent the three-dimensional human body posture to obtain human body geometric information and human body surface texture characteristics in standard space. 4. The method according to claim 3, characterized in that the three-dimensional geometric field records the directional distance function value of the current three-dimensional voxel, that is, the distance value of the current three-dimensional voxel position from the closest point to the human body surface; for the human body For internal three-dimensional voxels, the distance value is negative, and for three-dimensional voxels outside the human body, the distance value is positive. 5. The method according to claim 4, characterized in that, performing volume rendering on the shooting perspective image based on an implicit neural network to obtain the shooting perspective rendering image, including: Project a ray in the direction of a pixel of the shooting perspective image, sample a number of light points, and calculate the color value of the light point through an implicit neural network; Obtain the density information of the light point by querying the three-dimensional geometric field; Weighted integration is performed based on the color value and density information of the light points to obtain the color data of the pixels to obtain the shooting perspective rendering image. 6. A real-time dynamic human body new perspective rendering system based on multi-view videos, which is characterized by: The posture parameter estimation module is used to estimate the posture parameters of the human body based on multi-view human body image information; A human body feature characterization module, used to construct a three-dimensional geometric field and texture feature field of the human body based on the posture parameters; A feature variable optimization module is used to perform volume rendering on the shooting perspective image based on an implicit neural network to obtain a shooting perspective rendering image, and to construct a consistency constraint between the shooting perspective rendering image and the shooting perspective image to reduce the The three-dimensional geometric field and texture feature field are used as optimized variables to obtain the optimized texture feature field; A perspective image rendering module is used to perform a new perspective rendering of the human body based on the optimized texture feature field and the human body image information to obtain a new perspective rendering image. 7. The system according to claim 6, characterized in that the posture parameter estimation module is further used for: Obtain multi-view human body image information; Use two-dimensional human posture estimation tools to calculate the two-dimensional coordinates of human joint points in human body image information; The three-dimensional human body posture is solved based on the two-dimensional coordinates of the human body joint points and the multi-view geometric information of the human body. 8. The system according to claim 7, characterized in that the human body feature characterization module is also used for: Construct three-dimensional voxels used to represent three-dimensional geometric fields and texture feature fields; The three-dimensional voxels are used to represent the three-dimensional human posture to obtain human body geometry information and human body surface texture features in a standard space. 9. The system according to claim 8, characterized in that the three-dimensional geometric field records the directional distance function value of the current three-dimensional voxel, that is, the distance value of the current three-dimensional voxel position from the closest point to the human body surface; for the human body For internal three-dimensional voxels, the distance value is negative, and for three-dimensional voxels outside the human body, the distance value is positive. 10. The system according to claim 9, characterized in that the feature variable optimization module is also used to: Project a ray in the direction of a pixel of the shooting perspective image, sample a number of light points, and calculate the color value of the light point through an implicit neural network; Obtaining density information of the light point by querying the three-dimensional geometric field; Weighted integration is performed based on the color value and density information of the light points to obtain the color data of the pixels to obtain the shooting perspective rendering image.