CN104392045B - A kind of real time enhancing virtual reality system and method based on intelligent mobile terminal - Google Patents

Abstract

The invention relates to a real-time augmented virtual reality system and method based on an intelligent mobile terminal, comprising: a processor, a depth information acquisition card, an RGB information acquisition card, a storage unit and a display screen, and the depth information acquisition card is used to obtain the three-dimensional depth of the objective environment information; use the RGB information acquisition card to obtain the basic information of the objective environment; input the above-mentioned three-dimensional depth information and the basic information of the objective environment into the processor for processing, and create a three-dimensional model of the objective world; store the result data of the processing, that is, the three-dimensional model data of the objective world in In the storage unit; the interface circuit is used to display the augmented virtual reality to realize human-computer interaction. The system of the present invention has reasonable design, simple structure, high real-time performance, less equipment required, less loss of scene information, strong operability, wide application range, low cost, etc., enabling users to truly appreciate the benefits that augmented virtual reality brings to people. Infinite charm, the error between the constructed 3D model and the actual objective environment model is small.

Description

A real-time augmented virtual reality system and method based on an intelligent mobile terminal

technical field

The invention relates to a generation technology of a virtual reality system, in particular to a real-time augmented virtual reality system and method based on an intelligent mobile terminal.

Background technique

Augmented reality technology (Augmented Reality, AR) is to apply virtual information to the real world through computer technology. The real environment and virtual objects are superimposed on the same picture or space in real time and exist simultaneously. Augmented reality provides a virtual environment that generates a realistic sense of sight, hearing, force, touch and movement, which is different from the information that humans can perceive. It not only shows the information of the real world, but also displays the virtual information at the same time, and the two kinds of information complement and superimpose each other. Augmented reality uses computer graphics technology and visualization technology to generate virtual objects that do not exist in the real environment, and organically integrates virtual objects into the real environment through sensing technology. Direct natural interaction. It is a brand-new human-computer interaction technology. Using such a technology, it can simulate the real on-site landscape. It is an advanced computer human-computer interface with interactivity and conception as its basic features. Users can not only feel the "immersive" realism experienced in the objective physical world through the virtual reality system, but also break through space, time and other objective constraints, and experience experiences that cannot be experienced in the real world. . Therefore, AR has three basic elements, namely the combination of reality and virtuality, real-time interaction and three-dimensional positioning.

To achieve the combination of virtual reality and reality in AR, users need to use some kind of device to watch. Currently popular technologies mainly include transparent devices and opaque devices. The former uses transparent devices, such as 3D glasses and 3D projection devices, to project virtual images onto the device, allowing users to experience the scene of augmented virtual reality, but requires multiple Special equipment is used in conjunction with limited places and environments, which have gradually failed to meet people's needs for augmented virtual reality anytime and anywhere; Intelligent mobile terminals such as smartphones and tablets. Since smart mobile terminals have complex computing capabilities, video recording, image display, and multiple functions such as GPS, network connection, touch control, and inclination detection, the price is gradually reduced. There are more and more studies.

At present, the mainstream method of augmented virtual reality technology based on opaque devices is to collect objective environmental information through image and video acquisition cards. The feature extraction of the same scene is carried out, and then the feature matching algorithm is used for scene matching to obtain the 3D information of the objective environment, and the 3D reconstruction is carried out to obtain the 3D model of the objective environment, and finally the augmented virtual reality is realized. The disadvantages of this method are: the amount of calculation is huge, and a large amount of cloud data processing and calculation is required. The acquisition of 3D information requires a very high processor capability, and this method can hardly perform real-time 3D modeling, and it is difficult to achieve real-time 3D modeling in scene features. In the process of 3D reconstruction, a lot of useful information in the objective environment will be lost.

In order to establish a real-time three-dimensional model, the present invention adopts the RGB+D model to construct the three-dimensional model, directly uses the depth information acquisition card to obtain the three-dimensional depth information (Depth Information) of the objective environment, and uses the CCD video image acquisition card to obtain the basic information of the objective environment ( Such as color (RGB, Red, Green and Blue), texture information, grayscale information and intensity information, etc.), which liberates the calculation of three-dimensional depth information in the objective environment information from the processor, greatly improving the performance of augmented virtual reality. real-time.

In an augmented reality environment, users can see computer-generated augmented information while seeing the real environment around them. Since augmented reality bridges the gap between virtual reality and the real world, the application potential of augmented reality is quite huge, and it can be widely used in intelligent robot navigation and obstacle avoidance, military, model driving, three-dimensional navigation , medicine, manufacturing and maintenance, games and entertainment and many other fields.

Contents of the invention

The acquisition and calculation of 3D information in the prior art requires very high processor capabilities and it is almost impossible to perform real-time 3D modeling, and a lot of useful information in the objective environment will be lost in the process of scene feature extraction and 3D reconstruction. Insufficient, the technical problem to be solved in the present invention is to provide a kind of real-time augmented virtual reality based on intelligent mobile terminal that can liberate the calculation of the three-dimensional depth information in the objective environment information from the processor and greatly improve the real-time performance of augmented virtual reality. systems and methods.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

A kind of real-time augmented virtual reality system based on intelligent mobile terminal of the present invention comprises:

Processor, depth information acquisition card, RGB information acquisition card, storage unit, and display screen, wherein the processor receives the 3D depth information of the objective environment collected by the depth information acquisition card and the RGB information acquisition card and processes the basic information of the objective environment. The model is built, and the processed result data is stored in the storage unit; the processor is connected to the display screen through the interface circuit.

The depth information acquisition card is a camera capable of directly computing depth information, and can directly read 3D depth information of an objective environment.

A kind of real-time augmented virtual reality method based on intelligent mobile terminal of the present invention comprises the following steps:

Use the depth information acquisition card to obtain the three-dimensional depth information of the objective environment;

Use the RGB information acquisition card to obtain the basic information of the objective environment;

Input the above-mentioned 3D depth information and basic information of the objective environment into the processor for processing, and create a 3D model of the objective world;

storing the processed result data, that is, the three-dimensional model data of the objective world in the storage unit;

The interface circuit is used to display the augmented virtual reality to realize human-computer interaction.

Creating a 3D model of the objective world includes the following steps:

Information filtering: filter the three-dimensional depth information and the basic information of the objective environment, that is, the RGB+D image at the same time, and remove the noise image;

Information tracking algorithm: estimate the next image based on all the information of the denoised RGB+D information;

3D reconstruction based on RGB+D information: using estimated parameters of 3D world information and typical triangle mosaic method to pair Perform interpolation and splicing to obtain the final 3D model of the objective world.

Simultaneous filtering of RGB+D images is achieved by the following formula:

(1)

in, is the RGB+D image after filtering and denoising; for image pixels neighborhood, is the weight coefficient of the filter; is a noise-containing RGB+D image, m , n are neighbors respectively The coordinate value of each point in .

Using a typical Gaussian filter, that is

(2)

in, is the coordinates of each pixel in the image, for the filter in pixels adopt neighborhood collection of is the standard deviation of the Gaussian function.

For all the information of the denoised 3D world information, the estimation of the next image is:

Using maximum likelihood estimation, that is

(3)

In the formula, (4)

(5)

in, is the current RGB+D image; To estimate the new RGB+D image; is the Lie group algebraic operator of the current image and the estimated image; for image elements, ; is the pose estimation model; Depth information acquired for the depth information acquisition card; for standard deviation of is a norm, and defines

(6)

in, is a parameter, s is a variable, where, .

The 3D reconstruction based on RGB+D information is performed by the following formula:

(7)

in, ( p ) is the smallest unit of three-dimensional reconstruction; is an energy function, and Satisfy minimum; For the ground plane estimation of the three-dimensional reconstruction of the intelligent mobile terminal, a simple low-pass filter is used here, that is, the ground plane estimation is performed using RGB+D information; is the energy function The estimated parameters of for the gradient of It is a collection of RGB+D information.

The inventive method also comprises the following steps:

If the data processing capability of the processor of the mobile terminal is insufficient, upload the entire processing process to the network server or personal PC for processing through the mobile network or wifi, and then download the processed data to the mobile smart phone through the mobile network or wifi. terminal.

The inventive method also comprises the following steps:

If multiple users use mobile smart terminals, users can use the networking function to share AR data, and multiple users can enjoy augmented virtual reality applications at any place and any time.

The present invention has the following beneficial effects and advantages:

1. The system of the present invention has reasonable design, simple structure, high real-time performance, less equipment required, strong operability, wide application range, low cost, etc., enabling users to truly experience the infinite charm that augmented virtual reality brings to people .

2. The present invention uses the RGB+D model to construct a three-dimensional model, directly uses the depth information acquisition card to obtain the three-dimensional depth information (Depth Information) of the objective environment, uses the CCD video image acquisition card to obtain the basic information of the objective environment, and converts the objective environment information into The calculation of the 3D depth information is freed from the processor, with high real-time performance, less loss of scene information, and small errors between the constructed 3D model and the actual objective environment model.

3. The present invention can be widely used in many fields such as intelligent robot navigation and obstacle avoidance, military affairs, model driving, three-dimensional navigation, medicine, manufacturing and maintenance, games and entertainment, etc.

Description of drawings

Fig. 1 is a schematic structural diagram of the system of the present invention;

Fig. 2 is a functional block diagram of a real-time augmented virtual reality system based on an intelligent mobile terminal;

Fig. 3 is the main flowchart of the method of the present invention;

Fig. 4 is a flowchart of the three-dimensional reconstruction algorithm in the method of the present invention.

Among them, 101 is a mobile terminal; 102 is a processor; 103 is a storage unit; 104 is an interface circuit; 105 is an RGB video image acquisition card; 106 is a depth information acquisition card; 107 is a light source; 108 is a display screen.

detailed description

The present invention will be further elaborated below in conjunction with the accompanying drawings of the description.

As shown in Figures 1 and 2, the real-time augmented virtual reality system based on the intelligent mobile terminal of the present invention includes: a processor, a depth information acquisition card, an RGB information acquisition card, a storage unit, and a display screen, wherein the processor receives information collected by the depth The three-dimensional depth information and basic information of the objective environment collected by the RGB information acquisition card and the RGB information acquisition card are processed, the three-dimensional model is constructed, and the processed result data is stored in the storage unit; the processor is connected to the display screen through the interface circuit.

In this embodiment, the mobile terminal 101 can be a smart phone, a tablet computer, a palmtop computer, etc., and the processor 102, the storage unit 103, and the interface circuit 104 adopt the internal components of the smart terminal, and the present invention does not require high data processing and computing capabilities. , therefore, the computing capability of most intelligent terminal processors can meet the requirement at present. 105 is an RGB video image acquisition card, and an ordinary CCD camera can meet the requirements. The depth information acquisition card 106 adopts a Kinect camera with direct calculation capability of depth information, which can directly read the three-dimensional depth information of the objective environment. The light source 107 is used to compensate the impact of insufficient natural light on objective environmental information.

The present invention utilizes the processor 102 to perform calculation and processing of data, builds a three-dimensional model of the environment in real time, and realizes the application of augmented virtual reality, stores the processed result data in the storage unit 103, and utilizes the interface circuit 104 to perform augmented virtual reality display 108 , to achieve human-computer interaction.

As shown in Figure 3, a kind of real-time augmented virtual reality method based on intelligent mobile terminal of the present invention comprises the following steps:

Use the depth information acquisition card to obtain the three-dimensional depth information of the objective environment;

Use the RGB information acquisition card to obtain the basic information of the objective environment;

Input the above-mentioned 3D depth information and basic information of the objective environment into the processor for processing, and create a 3D model of the objective world;

storing the processed result data, that is, the three-dimensional model data of the objective world in the storage unit;

The interface circuit is used to display the augmented virtual reality to realize human-computer interaction.

In order to establish a real-time three-dimensional model, the present invention adopts the RGB+D model to construct the three-dimensional model, directly uses the depth information acquisition card to obtain the three-dimensional depth information (Depth Information) of the objective environment, and uses the CCD video image acquisition card to obtain the basic information of the objective environment ( Such as color (RGB, Red, Green and Blue), texture information, grayscale information and intensity information, etc.), which liberates the calculation of three-dimensional depth information in the objective environment information from the processor and improves the real-time performance of augmented virtual reality. sex.

The main steps of the inventive method have:

Step 1: Use the depth information acquisition card and RGB information acquisition card in Figure 2 to obtain 3D world information, which will be represented by RGB+D in the future;

Step 2: Input the RGB+D information into the intelligent mobile terminal processor for processing, process the objective world RGB+D information collected by the depth information acquisition card and RGB information acquisition card, and finally obtain the 3D model of the objective world, as shown in the figure 4 shows:

(1) RGB+D information filtering: filter and denoise the intensity image and depth image at the same time:

(1)

in, is the RGB+D image after filtering and denoising; for image pixels neighborhood, is the weight coefficient of the filter; is a noisy RGB+D image, for the image The coordinate set of the neighborhood, m and n are respectively the neighborhood The coordinate value of each point in . A typical Gaussian filter is used here, namely

(2)

in, is the coordinates of each pixel in the image, for the filter in pixels adopt neighborhood collection of is the standard deviation of the Gaussian function.

(2) Tracking algorithm based on RGB+D: use all the RGB+D information of the current image collected by the depth information acquisition card and RGB information acquisition card to estimate the next image, and use the maximum likelihood estimation, that is,

(3)

In the formula, (4)

(5)

in, is the current RGB+D image; To estimate the new RGB+D image; is the Lie group algebraic operator of the current image and the estimated image; for image elements, is the information set of RGB+D image, namely ; is the pose estimation model; Depth information acquired for the depth information acquisition card; for standard deviation of is a norm, and defines

(6)

in, is a parameter, s is a variable, where,

Through the above algorithm, it can be estimated whether the video frame of the objective world obtained in real time by the depth information acquisition card and RGB information acquisition card of the acquisition card is a key frame, and if it is a key frame, update the RGB+D image of this frame to the 3D reconstruction model , if it is not a key frame, then re-track until the key frame is captured. Finally, the present invention uses the key frames of the objective world captured by the depth information acquisition card and the RGB information acquisition card to reconstruct the 3D model, and the key frame contains It is all objective world information, that is, RGB+D information, thereby ensuring the integrity of the reconstructed 3D model information in the objective world. In this way, the three-dimensional information of the objective world can be estimated by using this algorithm, which is characterized in that all the information of the objective world is used, and no feature extraction and feature matching are performed, and all information in the objective world is included. It has high computing efficiency and high real-time performance. The current popular algorithms are based on image features, which require feature extraction, feature matching and other processes on the image. The disadvantages are large amount of calculation, low efficiency, and poor real-time performance. In addition, a lot of objective world information will be lost in the process of feature extraction.

(3) 3D reconstruction algorithm based on RGB+D information:

(7)

in, The smallest unit for three-dimensional reconstruction; is an energy function, and Satisfy minimum; For the ground plane estimation of the three-dimensional reconstruction of the intelligent mobile terminal, a simple low-pass filter is used here, that is, the ground plane estimation is estimated by using RGB+D information; is the energy function The estimated parameters of for the gradient of It is a collection of RGB+D information.

Finally, using the typical triangle splicing method to Interpolation and splicing are performed to obtain the final 3D objective world model.

Step 3: The data obtained after processing based on steps 1 and 2 can be stored in the storage unit.

Step 4: Display the real-time augmented virtual reality application based on the smart mobile terminal on the display screen of the smart mobile terminal.

If the data processing capacity of the processor of the mobile terminal is insufficient, the processing process of steps 1, 2, and 3 can be uploaded to the network server or personal PC for processing through the mobile network or wifi, and then the processed data can be processed through the mobile terminal. The network or wifi is downloaded to the mobile intelligent terminal, which can improve the data processing capability of the system, as shown in the extended storage unit and extended processor (such as a network server or PC) in Figure 2.

If the capacity of the storage unit of the mobile intelligent terminal is limited, the data processing result is uploaded to the extended storage unit through the mobile network or wifi to increase the information storage capacity.

If multiple users have mobile smart terminals, users can use the networking function to share AR data, so that multiple users can enjoy augmented virtual reality applications at any place and any time.

Claims (5)

1. A real-time augmented virtual reality method based on an intelligent mobile terminal is characterized by comprising the following steps:

acquiring three-dimensional depth information of an objective environment by using a depth information acquisition card;

obtaining basic information of an objective environment by using an RGB information acquisition card;

inputting the three-dimensional depth information and the basic information of the objective environment into a processor for processing, and creating a three-dimensional model of an objective world;

storing the processed result data, namely the three-dimensional model data of the objective world, in a storage unit;

the interface circuit is used for displaying the augmented virtual reality to realize human-computer interaction;

creating a three-dimensional model of an objective world includes the steps of:

information filtering: filtering the three-dimensional depth information and the basic information of the objective environment, namely RGB + D images, simultaneously to remove noise images;

information tracking algorithm: estimating the next image based on all the denoised RGB + D information;

three-dimensional reconstruction based on RGB + D information: carrying out interpolation splicing on the minimum unit of the three-dimensional reconstruction by utilizing the estimation parameters of the three-dimensional world information and a typical triangular splicing method to obtain a final objective world three-dimensional model;

the simultaneous filtering of the RGB + D image is achieved by the following formula:

wherein, I' (x, y) is an RGB + D image after filtering and denoising; epsilon is the neighborhood of the image pixel (x, y), and w (m, n) is the weight coefficient of the filter; i (m, n) is an RGB + D image containing noise, and m, n are coordinate values of each point in a neighborhood epsilon respectively;

three-dimensional reconstruction based on RGB + D information is performed by the following formula:

wherein u (p) is the minimum unit of three-dimensional reconstruction,in the form of a picture element or a picture element,e (u) is an energy function, and u satisfies Emin (u); pi (p) is the ground plane estimation of the three-dimensional reconstruction of the intelligent mobile terminal, and a simple low-pass filter is adopted, namely the ground plane estimation is carried out by utilizing RGB + D informationα and V_πFor the estimated parameters of the energy function E (u),is a gradient of u, Ω_DIs the RGB + D information set.

2. The real-time augmented virtual reality method based on intelligent mobile terminal according to claim 1,

the method is characterized in that: using a classical Gaussian filter, i.e.

Wherein, (x, y) is the coordinate of each pixel point of the image, (m, n) is the set of neighborhood epsilon adopted by the filter at the pixel point (x, y), and sigma is²Is the standard deviation of the gaussian function.

3. The real-time augmented virtual reality method based on the intelligent mobile terminal according to claim 1, characterized in that: and estimating the next image of all the denoised three-dimensional world information as follows:

using maximum likelihood estimation, i.e.

In the formula, r_p(p,ξ_j,i)＝I_i(p)-I_j(ω(p,D_i(p),ξ_j,i)) (4)

Wherein, I_iIs the current RGB + D image; i is_jfor new estimation of RGB + D image ξ_j,iLie group algebra operators for the current image and the estimated image;in the form of a picture element or a picture element,omega is a pose estimation model; d_iDepth information acquired by a depth information acquisition card; v_iIs D_iStandard deviation of (d); i | · | purple wind_δIs a norm, and defines

Wherein, delta is a parameter, s is a variable, in the formula,

4. the real-time augmented virtual reality method based on the intelligent mobile terminal according to claim 1, characterized in that: further comprising the steps of:

if the data processing capacity of the processor of the mobile terminal is insufficient, the whole processing process is uploaded to a network server or a personal computer for processing through a mobile network or wifi, and then the processed data is downloaded to the mobile intelligent terminal through the mobile network or wifi.

5. The real-time augmented virtual reality method based on the intelligent mobile terminal according to claim 1, characterized in that: further comprising the steps of:

if a plurality of users use the mobile intelligent terminal, the users share the AR data by using the networking function, and the plurality of users enjoy the application of the augmented virtual reality at any place and any time.