CN103269423A - Scalable 3D display remote video communication method - Google Patents

Abstract

The invention discloses an expandable three-dimensional display remote video communication method. According to the expandable three-dimensional display remote video communication method, the image of a first user is obtained by means of an RGB-D camera at the data transmission end, the image of the first user comprises a texture image and a depth image, facial feature information and expression feature information are extracted from the texture image, physical feature information is extracted from the depth image, and point clouds of the first user are reconstructed; all the feature information is optimized by means of the point clouds, the optimized feature information is obtained to generate a three-dimensional model A, the three-dimensional model A is projected on a corresponding texture image, the corresponding texture information is extracted, and acquired voice information, the optimized feature information and texture data are sent to a second user; at the data receiving end, the second user receives the data from the first user and extracts the optimized feature information from the received data to generate a three-dimensional model B, the three-dimensional model B is rendered by means of the texture data, and a rendering result is output through a three-dimensional display device, and the voice information is played.

Description

Scalable 3D display remote video communication method

technical field

The invention relates to the field of remote video communication, in particular to an expandable three-dimensional display remote video communication method.

Background technique

Remote video communication systems are playing an increasingly important role in today's society. The industry has adopted many technologies to improve the effect of virtual reality in the system, so that participants in remote conferences can obtain a more immersive experience. Three-dimensional display technology has experienced unprecedented development in both academia and industry, and it can be used in remote video communication systems. Since the latest 3D display technology can provide natural visualization effects and important visual cues, such as eye contact, facial expressions, body language, etc., the advantages of applying it to remote video communication systems are self-evident.

There are already some 3D display technologies applied to remote video communication in the academic circle, among which the parallax-type self-viewing 3D display is a relatively mature method. This multi-view self-viewing 3D display provides high-resolution 3D images at some fixed viewing angles. However, as the number of observation positions increases, the display effect of this method will be problematic. It cannot provide smooth motion parallax in a large range of observation angles, which means that the freedom of movement of participants in remote video communication is seriously restricted. restricted. In addition, some volumetric three-dimensional display systems designed based on the principle of screen rotation limit their display volume due to the structural design of their rotation mechanism, and also limit their application in commercial remote video communication systems in terms of cost, simplicity and scalability. .

A market-oriented 3D remote video communication system needs to meet some functional requirements. First of all, the most basic requirement is to enable participants to talk, walk, and do some actions. The less restrictions in this regard, the better. Second, there is a need to provide a natural, three-dimensional display effect with smooth binocular parallax and motion parallax. At the same time, it is also necessary to quickly capture information such as the appearance and language of the participants so that they can be displayed in real time. From a commercial point of view, any end-to-end 3D display remote video communication system is required to be low in cost, simple and easy to deploy, so as to facilitate large-scale market promotion.

In terms of real-time 3D information capture of people, the emergence of RGB-D cameras provides a convenient and low-cost potential solution to this problem. However, there are very few reports on its application to video communication, especially 3D video communication.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of existing remote video communication solutions in terms of virtual reality effect, user freedom and implementation cost, and provide a market-oriented scalable three-dimensional display remote video communication system and method.

An expandable three-dimensional display remote video communication method, the implementation steps are as follows:

Data sending end: obtain the character image and voice information of the first user, and extract the feature information and texture in the character image

1) Use the RGB-D camera to acquire the character image of the first user, the character image includes texture images and depth images;

Extract facial feature information and expression feature information from the texture image;

Extract body feature information from depth images and reconstruct character point clouds;

2) Using the character point cloud to optimize each feature information in step 1) to obtain optimized feature information, and generate a 3D model A through the optimized feature information;

3) Project the 3D model A to the corresponding texture image, extract the texture data corresponding to the 3D model A; acquire voice information while collecting the character image, and combine the voice information with the optimized feature information, and sending the extracted texture data to the second user;

4) Cycle operation steps 1) to 3), update the data at the previous time with the data at the next time and send it to the second user;

Data receiver:

After receiving the data from the first user through the network, the second user extracts the optimized feature information therein to generate a 3D model B, uses the extracted texture data to render the 3D model B, and outputs the rendering result through a 3D display device, And play the voice message.

Obtain the character image and voice information of the first user at the data sending end, extract the feature information and texture data in the character image, and then send it to the data receiving end, and the second user at the data receiving end performs modeling according to the received data and rendering processing, output through the three-dimensional display device, and play the corresponding voice information to realize remote video communication between the data sending end and the data receiving end.

At the data sending end, when using the RGB-D camera to acquire the person image of the first user, change the angle of view of the RGB-D camera relative to the first user at different times to obtain the person image of the first user at different angles, and then according to the image of the person Each feature information of the system is used to build a 3D model to achieve the effect of 3D display.

The voice information, the optimized feature information and the extracted texture data are compressed and packaged and sent to the second user through the network. Among them, voice information and optimized feature information have a small amount of data, and compression processing may not be performed, which will have a great impact on the speed of data transmission. However, the texture data has a large amount of data and takes up more resources. If it is not streamlined Compared with compression processing, the transmission speed is relatively slow, and there is a large time difference between the first user and the second user, which affects the real-time performance of video communication.

When using the data at the next moment to update the data at the previous moment, use the optimized feature information at the next moment to update the optimized feature information at the previous moment, and correspondingly generate the updated 3D model A, Project the updated 3D model A to the texture image at the next moment, and extract the updated texture data.

Preferably, the three-dimensional display device includes a projection module and a directional scattering screen module, the directional scattering screen module is a screen with bidirectional scattering characteristics, and the projection module is a projection array. The projection array is an array composed of projectors, or composed of a two-dimensional display and a lens array. The 3D display device is a real-time spatial 3D presentation system using integrated light field 3D display technology, which outputs a spatial 3D scene image with a large field of view and continuous motion parallax, enabling the observer to form a 3D vision through binocular parallax to achieve a real 3D video Effect.

The present invention combines integrated light field 3D display technology with RGB-D camera-based real-time character capture technology, and each terminal device uses integrated light field 3D display technology to provide real-time realism, with fine binocular parallax and motion parallax 3D At the same time, the use of RGB-D cameras to capture character information and interconnection through the Internet has brought into play the visual advantages of the 3D display system and the low cost of RGB-D cameras and the advantages of high-efficiency character capture.

The main advantages of the present invention are:

Integrate relevant resources, use the cheap and easy-to-use RGB-D depth camera to collect and process in real time, get better quality character interaction information, and simplify and compress the information to reduce its network transmission bandwidth requirements, so that it can Real-time transmission on the Internet, and the application of a high-performance real-time light field three-dimensional display device can achieve a good virtual reality effect, so that participants in remote video communication can obtain an immersive interactive experience.

Description of drawings

FIG. 1 is a schematic diagram of the basic logic structure of the three-dimensional display remote video communication method in the present invention.

FIG. 2 is a flow chart of the data sending end acquiring interactive data.

Fig. 3 is a flowchart of generating a virtual reality scene by using interactive data at the data receiving end.

Detailed ways

As shown in Figure 1, the system that realizes the three-dimensional display remote video communication method of the present invention includes a data sending end 1 and a data receiving end 2 connected to the data sending end 1 through a network, and the data sending end 1 and the data receiving end 2 perform data processing in real time. Interaction to realize remote video communication. The data sending end 1 and the data receiving end 2 both receive and send data at the same time, and the logical structure of the two terminals is the same. For the convenience of description of the present invention, they are artificially divided into data sending end 1 and data receiving end 2.

The data sending end 1 includes a main control computer 8 , a three-dimensional display device 5 , and a person capture device 6 , wherein the main control computer 8 includes a network transmission module 9 , a three-dimensional display software module 4 and a person capture software module 3 . The three-dimensional display device 5 and the character capture device 6 enter the three-dimensional display software module 4 and the character capture software module 3 of the main control computer 8 respectively, and the three-dimensional display software module 4 and the character capture software module 3 are connected with the network transmission module 9 respectively, and the network transmission The module 9 is connected to the data receiving terminal 2 through the Internet, the three-dimensional display device 5 outputs image data in the communication interaction area 7, the character capture software module 3 is used to obtain the character image in the communication interaction area 7, the visible area and The capture areas of the character capture devices 6 must be able to cover the communication interaction area 7 .

The network transmission module 9 simplifies and compresses the data that needs to be transmitted on the network, and transmits it to the data receiving end 2 through the Internet; at the same time, it accepts data sent by other terminals through the Internet. The data includes character expression parameters, body movement parameters, character textures that need to be updated, and voice information. The amount of data is relatively small, and it can be compressed to meet real-time network transmission.

The three-dimensional display device 5 is a real-time spatial three-dimensional rendering system using integrated light field three-dimensional display technology, which is one of the advantages of the present invention. The principle of light field 3D display is to reconstruct the spatial distribution of light emitted by a 3D scene. The 3D display device using this principle mainly includes a projection module and a directional scattering screen module. The projection module and the directional scattering screen module have a variety of arrangement modes. Here, a multi-projection light field display system that realizes 360-degree panoramic viewing is taken as an example. The projectors distributed in a ring around the screen project the combined images of the three-dimensional model or three-dimensional scene to be displayed at various angles into the central area of the ring screen. The ring screen is a directional scattering screen with two-way scattering characteristics. That is, a scattering screen structure with a specific scattering angle in the horizontal direction and a larger scattering angle in the vertical direction, and a rotationally symmetrical distribution. Thus, the images projected by the annularly distributed projectors are converted into 360-degree panoramic and visible spatial three-dimensional scene images for viewers in the viewing area surrounding the screen. According to the principle of light field reconstruction and the characteristics of the annular directional scattering screen, each position in the observation area can only watch a narrow image projected by a projector corresponding to this position, and each position can see more The combined image of multiple narrow strip images projected by a projector forms a complete picture of the position, and the complete image information is presented through this mode similar to the integrated stitching light field. Then different images corresponding to the corresponding positions can be viewed at different positions in the observation area, which can ensure that the image information viewed by the eyes of the audience in the observation area is different, and three-dimensional vision is formed through binocular parallax, which is also Motion parallax can be obtained by moving between different positions in the lateral direction. The dependence of the 3D display device 5 on the 3D display software module 4 is only that the 3D display device 5 requires the 3D display software module 4 to provide models, textures and other necessary data and parameters for rendering 3D effects. The specific implementation method of the three-dimensional display software module 4 has nothing to do with the three-dimensional display device 5 .

The main function of the three-dimensional display remote video communication system is to realize real-time information exchange in two directions. The first direction is the acquisition and transmission of interactive data such as appearance, expression, voice, and action of video communication participants. The character capture device 6 captures the appearance, action and voice of the communication participating users in the communication interaction area 7 to obtain basic image and audio data. These data are sent to the character capture software module 3, and the character capture software module 3 calculates the shape, voice and facial expressions of the character through some algorithms, and optimizes and accumulates it to obtain information that is more in line with the real scene. And send these data to network transmission module 9. The network transmission module 9 compresses and packs the data, and sends it to another remote communication terminal (that is, the data receiving end) through the Internet, so as to complete the data transmission in this direction. The second direction is to receive interactive data such as appearance, expression, voice, and action of video communication participants, and generate virtual reality scenes. The network transmission module 9 obtains the communication user data sent by another terminal through the Internet, and transmits the decompressed data to the three-dimensional display software module 4 . The three-dimensional display software module 4 calculates the three-dimensional model and texture information that the three-dimensional display device 5 needs to present through the obtained user appearance, expression, action, and voice data, and transmits it to the three-dimensional image display device 5, so that it presents a corresponding three-dimensional effect and plays Voice, for the participants in the communication interaction area 7 to watch. The completion of real-time information exchange in these two directions enables users of different terminals of the three-dimensional display remote video communication system to observe the other party's appearance, expression, movement and other three-dimensional effects in real time, hear the other party's voice, and also can real-time Feedback this information to the other party.

The specific implementation process of real-time information exchange in the above two directions is introduced in detail below, including the following steps:

As shown in Figure 2, the data sender:

1) Use the RGB-D camera to acquire the character image of the first user, the character image includes texture images and depth images;

Extract facial feature information and expression feature information from the texture image;

Extract body feature information from depth image and reconstruct character point cloud.

The character capture device uses an inexpensive, easy-to-use, and easy-to-deploy RGB-D camera. When using the RGB-D camera to capture the first user's character image, change the angle of view of the RGB-D camera relative to the first user at different times. In this embodiment The RGB-D camera specifically used in is Kinect. Kinect can collect the color and depth data stream of the scene in real time, and send them to the character capture software module as color frames and depth frames. These data streams themselves are the texture information and depth information of the scene part, which contains the texture and depth data of the characters. The character capture software module 3 extracts the character's facial features and expression feature information from the character's texture data in real time, and extracts the character's body feature information from the character's depth data, which defines the character's geometric shape, facial expression and body movements Information, the point cloud of the character can be reconstructed from the depth data, which can reflect the original geometric data of the character.

In this embodiment, Kinect is used as the RGB-D camera. For Kinect software development, Microsoft provides Kinect's Software Development Kit (SDK). Using the interface in the KinectSDK, feature information can be extracted and character point clouds can be reconstructed.

2) Using the person point cloud to optimize each feature information in step 1) to obtain optimized feature information, and generate a 3D model A through the optimized feature information.

The feature information here includes the above-mentioned facial feature information, expression feature information, and body feature information. The reconstructed character point cloud is used to optimize each feature signal to obtain the optimized special name information, and then use the optimized feature information to generate a 3D model. Model A.

Feature information optimization: define the energy equation, and use the optimized feature information as the unknown. The equation describes the degree of difference in geometry between the model generated by the optimized feature information and the model generated by the feature information before optimization, and the degree of difference in geometry between the model generated by the optimized feature information and the point cloud data. Sum. Find the solution that makes this energy equation obtain the minimum value, that is, obtain the optimized characteristic information.

Generate a 3D model based on feature information: Model deformation methods are used, including Morph technology and Skinned Mesh technology. All are conventional techniques.

3) Project the 3D model A to the corresponding texture image, extract the texture data corresponding to the 3D model A; acquire voice information while collecting the character image, and combine the voice information with the optimized feature information, And the extracted texture data is sent to the second user.

The three-dimensional model A is projected into the texture image collected by Kinect, and the texture data corresponding to the three-dimensional model A is extracted, wherein the voice information can be directly obtained by the sound collection device in the Kinect, and then the network transmission module 9 converts the voice information, optimized After the feature information and the extracted texture data are compressed and packaged, they are sent to the second user. For each color frame captured by Kinect, the original texture of the person can be extracted. We accumulate and optimize the original texture data of the person collected many times to improve its robustness and obtain a character texture with better quality.

Texture optimization: Each character has a basic texture, and every time new texture data is extracted, it will be used to update the basic texture data. This update is a cumulative method that can optimize the texture. The specific update method is: for each pixel, a probability distribution is estimated according to the accumulated texture data collected, and the color value of the pixel is updated with the expected value of the probability distribution.

4) Cycle operation steps 1) to 3), update the data at the previous time with the data at the next time and send it to the second user.

By changing the angle of view multiple times to obtain character images, and accumulating and optimizing the original texture data of the characters collected multiple times, combined with the corresponding 3D model, you can get the complete shape, expression, and action data of the character.

As shown in Figure 3, the data receiving end:

After receiving the data from the first user through the network, the second user extracts the optimized feature information therein to generate a 3D model B, uses the extracted texture data to render the 3D model B, and outputs the rendering result through a 3D display device, And play the voice message.

The network transmission module in the data receiving end receives the data packets sent over the Internet, decompresses the data therein, and completes the work of data synchronization. The network transmission module provides the data that needs to be presented in virtual reality to the three-dimensional display software module. These data include the characteristic information of the geometric shape of the character, the characteristic information of the facial expression, the characteristic information of the body movement, the character texture information that needs to be updated, and the character information. voice message. The implementation process of the three-dimensional display software module is as follows: the characteristic information of the character's geometric shape, facial expression and body movement are brought into the algorithm as parameters, and the standard character model pre-stored in the three-dimensional display software module is deformed. Through the deformation operation, a character model conforming to the above static and dynamic feature descriptions can be obtained. The 3D display software module initially stores the basic texture of the character model, and every time the character texture information that needs to be updated is obtained, the 3D display software module will update the basic texture of the character model and continuously accumulate it to obtain the currently usable texture information. character texture. Character voice data can be used directly. The 3D display software module submits the calculated character model data and character texture data to the 3D display device for virtual reality presentation and simultaneously plays character voice.

Claims (5)

1. can expand formula three-dimensional display remote video communication method for one kind, it is characterized in that implementation step is as follows:

Data sending terminal:

1) utilizes the RGB-D camera to obtain first user's character image, comprised texture image and depth image in this character image;

From texture image, extract facial characteristics information and expressive features information;

From depth image, extract the limbs characteristic information and reconstruct people's object point cloud;

2) utilize described people's object point cloud that each characteristic information in the step 1) is optimized characteristic information after being optimized, the characteristic information after optimizing by this generates threedimensional model A;

3) described threedimensional model A is projected to corresponding texture image, the corresponding data texturing of this threedimensional model A is extracted; Obtain voice messaging when gathering character image, together with the characteristic information after optimizing, and the data texturing that extracts sends to second user with this voice messaging;

4) cycling step 1)～3), utilize back one data constantly that the data of previous moment are upgraded and send to second user;

Data receiver:

Second user receives from after first user's data by network, characteristic information after the extraction optimization wherein generates threedimensional model B, utilize the described data texturing that extracts that threedimensional model B is played up, by three-dimensional display apparatus output rendering result, and play described voice messaging.

2. the formula three-dimensional display remote video communication method of expanding as claimed in claim 1, it is characterized in that, when utilizing back one data constantly that the data of previous moment are upgraded, characteristic information after utilizing characteristic information after the optimization constantly of back one to the optimization of previous moment upgrades processing, and the corresponding threedimensional model A that generates after upgrading, threedimensional model A after upgrading is projected to the texture image in this back moment, extract the data texturing after the renewal.

3. the formula three-dimensional display remote video communication method of expanding as claimed in claim 2 is characterized in that, when utilizing the RGB-D camera to obtain first user's character image, at relative first user's of different time changing RGB-D cameras visual angle.

4. the formula three-dimensional display remote video communication method of expanding as claimed in claim 3, it is characterized in that the characteristic information after described voice messaging, the optimization and the data texturing that extracts are compressed, the processing back of packing sends to described second user by network.

5. the formula three-dimensional display remote video communication method of expanding as claimed in claim 4, it is characterized in that, described three-dimensional display apparatus comprises projection module and directional scattering panel module, described directional scattering panel module is to have two screens to scattering properties, and described projection module is projected array.

Images