WO2018045759A1

WO2018045759A1 - Method and device for lighting rendering in augmented reality, and mobile terminal

Info

Publication number: WO2018045759A1
Application number: PCT/CN2017/081402
Authority: WO
Inventors: 邵红胜
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-09-07
Filing date: 2017-04-21
Publication date: 2018-03-15
Also published as: CN107808409B; CN107808409A

Abstract

A method and device for lighting rendering in augmented reality, and a mobile terminal. The method comprises: determining position and brightness of a light source in a real scene; and performing lighting rendering on a virtual object in a virtual scene according to the position and brightness of the light source in the real scene. The present invention can obtain information of a light source in a real scene in augmented reality in real time, and dynamically perform lighting rendering on a virtual object in a virtual scene in augmented reality according to the information of the light source in the real scene, so that the virtual object in the virtual scene can be better combined with the real scene, thereby providing visual experience closer to reality for a user.

Description

Method, device and mobile terminal for performing illumination rendering in augmented reality

Technical field

The present disclosure relates to the field of augmented reality technologies, and in particular, to a method, an apparatus, and a mobile terminal for performing illumination rendering in an augmented reality.

Background technique

Augmented Reality (AR) is a technique for calculating the position and angle of a camera image in real time and adding a corresponding virtual model. The goal of this technology is to load the virtual model into the real world and perform it on the screen. interactive. However, the problem of rendering virtual models in augmented reality has always been the key to affecting the visual experience. In particular, the lighting rendering of virtual models has the greatest impact on the visual experience. In the current augmented reality technology, the virtual model of the augmented reality is simply rendered by the fixed source, so that the realism of the virtual model cannot meet the requirements.

Summary of the invention

The technical problem to be solved by the present disclosure is to provide a method, a device and a mobile terminal for performing illumination rendering in an augmented reality, which enhances the fidelity of a virtual object and improves the visual effect of the augmented reality.

The technical solution adopted by the present disclosure is that the method for performing illumination rendering in the augmented reality includes:

Determine the position and brightness of the light source in the real scene;

Illuminating the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.

Optionally, the determining the position and brightness of the light source in the real scene includes:

Determining a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object and its shadow in the real scene;

The brightness value of all the position points in the real scene is calculated, and the maximum brightness value in the real scene is the brightness of the light source in the real scene.

Optionally, the determining, according to the positional relationship between any real object in the real scene and the shadow thereof, the three-dimensional coordinate position of the light source in the real scene, including:

Determining the physical boundary line and the shadow boundary line of any real object in the real scene;

Selecting feature points on the physical boundary line and the shadow boundary line of any of the real objects;

Determining a feature vector according to a correspondence between a feature point on the physical boundary line and a feature point on the shadow boundary line;

The position of the intersection of the extension line or the inverse extension line of all the feature vectors is determined as the three-dimensional coordinate position of the light source in the real scene.

Optionally, the feature points include: an inflection point and an extreme point.

Optionally, the feature vector is that any feature point on the boundary line of the entity points to the shadow boundary line A vector of feature points corresponding to any of the feature points.

Optionally, the illuminating the virtual object in the virtual scene according to the position and brightness of the light source in the real scene, including:

Converting the position of the light source in the real scene to the position of the light source in the virtual scene;

Illuminating the virtual object in the virtual scene according to the position of the light source in the virtual scene and the brightness of the light source in the real scene.

Optionally, the converting the position of the light source in the real scene to the position of the light source in the virtual scene comprises:

Obtaining a realistic model transformation matrix M according to the parallel tracking and mapping algorithm, and converting the position (x, y, z) of the light source in the real scene to the position of the light source in the virtual scene by using the real model transformation matrix M (u, v, w); where (u, v, w) = M * (x, y, z).

The present disclosure also provides an apparatus for performing illumination rendering in augmented reality, including:

a light source determining module for determining a position and brightness of the light source in a real scene;

The illumination rendering module is configured to perform illumination rendering on the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.

Optionally, the light source determining module includes:

a position determining unit, configured to determine a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object and a shadow thereof in the real scene;

The brightness determining unit is configured to calculate a brightness value of all the position points in the real scene, and the maximum brightness value in the real scene is the brightness of the light source in the real scene.

Optionally, the location determining unit is configured to:

Optionally, the feature vector is a vector of any feature point on the solid boundary line pointing to a feature point corresponding to the any feature point on the shadow boundary line.

Optionally, the illumination rendering module includes:

a conversion unit, configured to convert a position of the light source in the real scene to a position of the light source in the virtual scene;

And a rendering unit, configured to perform illumination rendering on the virtual object in the virtual scene according to the position of the light source in the virtual scene and the brightness of the light source in the real scene.

Optionally, the conversion unit is configured to:

Obtaining a realistic model transformation matrix M according to a parallel tracking and mapping algorithm, and using the real model transformation matrix M Converting the position (x, y, z) of the light source in the real scene to the position (u, v, w) of the light source in the virtual scene; where (u, v, w) = M * (x, y) ,z).

The present disclosure also provides a mobile terminal, including the apparatus for performing illumination rendering in the augmented reality described above.

The present disclosure also provides a non-transitory computer readable storage medium having stored therein computer program instructions that, when executed by one or more processors of a terminal device, perform the computer program instructions as described above A method of lighting rendering in augmented reality.

With the above technical solutions, the present disclosure has at least the following advantages:

The method, device and mobile terminal for performing illumination rendering in the augmented reality of the present disclosure can obtain the light source information in the real scene of the augmented reality in real time, and dynamically virtualize the augmented reality according to the light source information in the real scene. The virtual objects in the scene are rendered by illumination, so that the virtual objects in the virtual scene can be better integrated with the real scene, bringing the user a visual experience closer to reality.

DRAWINGS

1 is a flowchart of a method for performing illumination rendering in an augmented reality according to a first embodiment of the present disclosure;

2 is a flowchart of a method for performing illumination rendering in augmented reality according to a second embodiment of the present disclosure;

3 is a schematic structural diagram of a device for performing illumination rendering in an augmented reality according to a third embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of an apparatus for performing illumination rendering in an augmented reality according to a fourth embodiment of the present disclosure.

detailed description

The present disclosure will be described in detail below with reference to the accompanying drawings and preferred embodiments.

A first embodiment of the present disclosure, a method for performing illumination rendering in augmented reality, as shown in FIG. 1, includes the following steps:

Step S101: Determine the position and brightness of the light source in the real scene.

Optionally, step S101 includes:

Step A1: Determine the three-dimensional coordinate position of the light source in the real scene according to the positional relationship between any real object and its shadow in the real scene.

Optionally, step A1 includes:

Step A11: Determine the physical boundary line and the shadow boundary line of any real object in the real scene.

Step A12: Select feature points respectively on the physical boundary line and the shadow boundary line of any of the real objects. The feature points include: an inflection point and an extreme point, for example, a highest point, a lowest point, and a vertex on the boundary line.

Step A13: Determine a feature vector according to a correspondence relationship between a feature point on the physical boundary line and a feature point on the shadow boundary line. The feature vector is a vector of any feature point on the solid boundary line pointing to a feature point corresponding to the any feature point on the shadow boundary line.

Step A14: determining the position of the intersection of the extension line or the reverse extension line of all the feature vectors as the light source in the true The position of the three-dimensional coordinates in the real scene.

Step A2: Calculate the brightness value of all the position points in the real scene, and the maximum brightness value in the real scene is the brightness of the light source in the real scene.

Step S102: Perform light rendering on the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.

Optionally, step S102 includes:

Step B1: Converting the position of the light source in the real scene to the position of the light source in the virtual scene.

Step B2: Perform light rendering on the virtual object in the virtual scene according to the position of the light source in the virtual scene and the brightness of the light source in the real scene.

Optionally, step B1 includes:

A second embodiment of the present disclosure, a method for performing illumination rendering in an augmented reality, as shown in FIG. 2, includes the following steps:

Step S201: Acquire a real-world video image by using a camera.

Step S202: Decomposing the video image of the real world to obtain a series of image frames.

Step S203: Calculate the position of the light source in each of the image frames.

Optionally, step S203 includes:

Step C1: Determine a physical boundary line and a shadow boundary line of any real object in the image frame.

Step C2: Select feature points on the physical boundary line and the shadow boundary line, respectively.

The feature points include: an inflection point and an extreme point, for example, a highest point, a lowest point, and a vertex on the boundary line.

Step C3: Determine a feature vector according to a correspondence relationship between a feature point on the solid boundary line and a feature point on the shadow boundary line. The feature vector is a vector of any feature point on the solid boundary line pointing to a feature point corresponding to the any feature point on the shadow boundary line.

Step C4: Determine the position of the light source in the image frame according to the feature vector. The position of the light source in the image frame is the intersection of the extension line or the inverse extension line of all of the feature vectors.

Step S204: Calculate the brightness of the light source in each of the image frames.

Optionally, step S204 includes:

The luminance value of each pixel in any of the image frames is calculated, and the maximum luminance value in any of the image frames is the luminance of the light source in any of the image frames.

Step S205: Converting the position of the light source in the image frame to the position of the light source in the three-dimensional model of the augmented reality.

Optionally, step S205 includes:

According to PTAM (Parallel Tracking and Mapping) algorithm, the real model is transferred. Converting the matrix M, using the real model transformation matrix M to convert the position (x, y, z) of the light source in the image frame to the position (u, v, w) of the light source in the augmented reality three-dimensional model , where (u, v, w) = M * (x, y, z).

Step S206: Perform illumination rendering on the virtual object in the three-dimensional model of the augmented reality according to the position of the light source in the three-dimensional model of the augmented reality and the brightness of the light source in the image frame.

Step S207: Place the virtual object rendered by the illumination into the real world video image acquired by the camera.

As the camera continuously acquires the real-world video image, the light source information in the real-world video image is dynamically extracted, and the virtual object is dynamically rendered according to the light source information.

A third embodiment of the present disclosure, an apparatus for performing illumination rendering in augmented reality, as shown in FIG. 3, includes the following components:

1) A light source determining module 301 for determining the position and brightness of the light source in a real scene.

Optionally, the light source determining module 301 includes:

The position determining unit is configured to determine a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object and its shadow in the real scene.

Optionally, the location determining unit is configured to:

Determines the physical boundary line and shadow boundary line of any real object in the real scene.

Feature points are respectively selected on the physical boundary line and the shadow boundary line of any of the real objects. The feature points include: an inflection point and an extreme point, for example, a highest point, a lowest point, and a vertex on the boundary line.

The feature vector is determined according to a correspondence relationship between a feature point on the solid boundary line and a feature point on the shadow boundary line. The feature vector is a vector of any feature point on the solid boundary line pointing to a feature point corresponding to the any feature point on the shadow boundary line.

2) The illumination rendering module 302 is configured to perform illumination rendering on the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.

Optionally, the illumination rendering module 302 is configured to:

a conversion unit for converting a position of the light source in the real scene to a position of the light source in the virtual scene.

Optionally, the conversion unit is configured to:

Obtaining a realistic model transformation matrix M according to the parallel tracking and mapping algorithm, and converting the position (x, y, z) of the light source in the real scene to the position of the light source in the virtual scene by using the real model transformation matrix M (u, v,w); Where (u, v, w) = M * (x, y, z).

A fourth embodiment of the present disclosure, an apparatus for performing illumination rendering in augmented reality, as shown in FIG. 4, includes the following components:

1) A video acquisition module 401, configured to acquire a real-world video image by using a camera.

2) The video decomposition module 402 is configured to decompose the real-world video image to obtain a series of image frames.

3) A position calculation module 403 for calculating the position of the light source in each of the image frames.

Optionally, the calculating module 403 includes:

A boundary line determining unit is configured to determine a physical boundary line and a shadow boundary line of any real object in the image frame.

A feature point determining unit is configured to respectively select feature points on the physical boundary line and the shadow boundary line. The feature points include: an inflection point and an extreme point, for example, a highest point, a lowest point, and a vertex on the boundary line.

And a feature vector determining unit, configured to determine a feature vector according to a correspondence between a feature point on the solid boundary line and a feature point on the shadow boundary line. The feature vector is a vector of any feature point on the solid boundary line pointing to a feature point corresponding to the any feature point on the shadow boundary line.

And a light source determining unit configured to determine a position of the light source in the image frame according to the feature vector. The position of the light source in the image frame is the intersection of the extension line or the inverse extension line of all of the feature vectors.

4) A brightness calculation module 404 for calculating the brightness of the light source in each of the image frames.

Optionally, the brightness calculation module 404 is configured to:

A luminance value is calculated for each point in any of the image frames, and the maximum luminance value in any of the image frames is the luminance of the light source in any of the image frames.

5) A position conversion module 405 for converting the position of the light source in the image frame to the position of the light source in the three-dimensional model of the augmented reality.

Optionally, the location conversion module 405 is configured to:

Obtaining a real model transformation matrix M according to the PTAM algorithm, and converting the position (x, y, z) of the light source in the image frame into a position of the light source in the augmented reality three-dimensional model by using the real model transformation matrix M ( u, v, w), where (u, v, w) = M * (x, y, z).

6) The illumination rendering module 406 is configured to perform illumination rendering on the virtual object in the augmented reality three-dimensional model according to the position of the light source in the three-dimensional model of the augmented reality and the brightness of the light source in the image frame.

7) The video restoration module 407 is configured to place the virtual object rendered by the illumination into the real world video image acquired by the camera.

A fifth embodiment of the present disclosure is a mobile terminal provided with an apparatus for performing illumination rendering in an augmented reality according to a third embodiment of the present disclosure.

The present disclosure also provides a non-transitory computer readable storage medium having stored therein computer program instructions When the one or more processors of the terminal device execute the computer program instructions, the terminal device performs the method of performing illumination rendering in the augmented reality as described in the first or second embodiment.

The method, device and mobile terminal for performing illumination rendering in the augmented reality introduced in the embodiment of the present disclosure can obtain the light source information in the real scene of the augmented reality in real time, and dynamically augment the reality according to the light source information in the real scene. The virtual objects in the virtual scene are rendered by the light, so that the virtual objects in the virtual scene can be better integrated with the real scene, bringing the user a visual experience closer to reality.

The description of the specific embodiments is intended to provide a more thorough understanding of the embodiments of the invention, and the .

Industrial applicability

The method for performing illumination rendering in the augmented reality provided by the embodiment of the present disclosure may be applied to the light source information in the real scene of the augmented reality acquired in real time by the terminal device, and dynamically enhanced according to the dynamic information of the light source in the real scene. The virtual objects in the realistic virtual scene are rendered by the light, so that the virtual objects in the virtual scene can be better integrated with the real scene, bringing the user a visual experience closer to reality.

Claims

A method of performing illumination rendering in augmented reality, including:

Determine the position and brightness of the light source in the real scene;

Illuminating the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.
The method for performing illumination rendering in augmented reality according to claim 1, wherein the determining the position and brightness of the light source in the real scene comprises:

Determining a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object and its shadow in the real scene;

The brightness value of all the position points in the real scene is calculated, and the maximum brightness value in the real scene is the brightness of the light source in the real scene.
The method for performing illumination rendering in augmented reality according to claim 2, wherein the determining a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object in the real scene and the shadow thereof, including :

Determining the physical boundary line and the shadow boundary line of any real object in the real scene;

Selecting feature points on the physical boundary line and the shadow boundary line of any of the real objects;

Determining a feature vector according to a correspondence between a feature point on the physical boundary line and a feature point on the shadow boundary line;

The position of the intersection of the extension line or the inverse extension line of all the feature vectors is determined as the three-dimensional coordinate position of the light source in the real scene.
The method of performing illumination rendering in augmented reality according to claim 3, wherein the feature points comprise: an inflection point and an extreme point.
The method for performing illumination rendering in augmented reality according to claim 3, wherein the feature vector is any feature point on the solid boundary line pointing to the shadow boundary line and any one of the feature points The vector of the corresponding feature point.
The method for performing illumination rendering in augmented reality according to claim 1, wherein the rendering of the virtual object in the virtual scene according to the position and brightness of the light source in the real scene comprises:

Converting the position of the light source in the real scene to the position of the light source in the virtual scene;

Illuminating the virtual object in the virtual scene according to the position of the light source in the virtual scene and the brightness of the light source in the real scene.
The method for performing illumination rendering in augmented reality according to claim 6, wherein the converting the position of the light source in the real scene to the position of the light source in the virtual scene comprises:

Obtaining a realistic model transformation matrix M according to the parallel tracking and mapping algorithm, and converting the position (x, y, z) of the light source in the real scene to the position of the light source in the virtual scene by using the real model transformation matrix M (u, v, w); where (u, v, w) = M * (x, y, z).
An apparatus for performing illumination rendering in augmented reality, comprising:

a light source determining module configured to determine a position and brightness of the light source in a real scene;

The illumination rendering module is configured to perform illumination rendering on the virtual object in the virtual scene according to the position and brightness of the light source in the real scene.
The apparatus for performing illumination rendering in augmented reality according to claim 8, wherein the light source determining module comprises:

a position determining unit configured to determine a three-dimensional coordinate position of the light source in the real scene according to a positional relationship between any real object and its shadow in the real scene;

The brightness determining unit is configured to calculate a brightness value of all the position points in the real scene, and the maximum brightness value in the real scene is the brightness of the light source in the real scene.
The apparatus for performing illumination rendering in augmented reality according to claim 9, wherein the position determining unit is configured to:

Determining the physical boundary line and the shadow boundary line of any real object in the real scene;

Selecting feature points on the physical boundary line and the shadow boundary line of any of the real objects;

Determining a feature vector according to a correspondence between a feature point on the physical boundary line and a feature point on the shadow boundary line;

The position of the intersection of the extension line or the inverse extension line of all the feature vectors is determined as the three-dimensional coordinate position of the light source in the real scene.
The apparatus for performing illumination rendering in augmented reality according to claim 10, wherein the feature points comprise: an inflection point and an extreme point.
The apparatus for performing illumination rendering in augmented reality according to claim 10, wherein the feature vector is any feature point on the solid boundary line pointing to the shadow boundary line and the any feature point The vector of the corresponding feature point.
The apparatus for performing illumination rendering in augmented reality according to claim 8, wherein the illumination rendering module comprises:

a conversion unit configured to convert a position of the light source in the real scene to a position of the light source in the virtual scene;

And a rendering unit configured to perform illumination rendering on the virtual object in the virtual scene according to the position of the light source in the virtual scene and the brightness of the light source in the real scene.
The apparatus for performing illumination rendering in augmented reality according to claim 13, wherein the conversion unit is configured to:

Obtaining a realistic model transformation matrix M according to the parallel tracking and mapping algorithm, and converting the position (x, y, z) of the light source in the real scene to the position of the light source in the virtual scene by using the real model transformation matrix M (u, v, w); where (u, v, w) = M * (x, y, z).
A mobile terminal comprising the apparatus for performing illumination rendering in an augmented reality according to any one of claims 8 to 14.