CN101739710A - Outdoor scene illumination parameter restoration device and working method thereof - Google Patents

Abstract

The invention discloses an outdoor scene illumination parameter restoration device and a working method thereof. The device comprises a computer and a video camera. The basic working method comprises the following steps: the image or video stream which is shot by the video camera is input to the computer and processed by the computer, and the restoration result of the illumination parameters of the scene is output. The process is divided into a preprocessing phase and an on-line phase. The device has the working principle that any outdoor scene image can be presented as a linear combination of a sunlight and skylight base image, so that the problem of the restoration of the incident light brightness values of the sunlight and the skylight in the image can be included as a problem of optimizing the real-time solving. The method does not need three-dimensional geometric information and material information of the scene, and can acquire the incident light brightness values of the sunlight and the skylight in the outdoor scene in real time. The method can be directly used in an augmented reality system, and can also be used for outdoor scene video sequence preprocessing, such as image segmentation, pattern recognition and the like.

Description

An outdoor scene illumination parameter restoration device and its working method

technical field

The present invention relates to outdoor scene lighting parameter recovery, in particular to an outdoor scene lighting parameter recovery device and a working method thereof.

Background technique

Synthesis of a computer-generated virtual object into a photo of a real scene was first proposed by E. Nakamae in 1986 [NHIN86]. The author establishes equations through the shadow points and non-shadow points of some diffuse materials in the user's simple interaction scene to solve the values of the red, green, and blue channels of the sun light and sky light, and then use the restored two light sources to draw Virtual objects, so that the lighting of virtual objects and real photos looks consistent. This method is still operable for the lighting restoration of one or several frames of images, but it cannot be interactively operated for online video streams, so it cannot be processed at all. Since then, since the appearance of the scene is largely affected by the incident lighting, there have been some works to obtain lighting parameters in the fields of reverse rendering, shape recovery from light and shade, and intrinsic image analysis. These methods are briefly introduced below.

1. Reverse drawing

In reverse rendering, in order to avoid the complexity of direct solution, [LDR2000] uses manual measurement method to obtain the position, color and intensity of the light source in the indoor scene. With the aid of instruments, this method can obtain more accurate lighting parameters. But obviously, this method is not feasible for obtaining outdoor lighting. To obtain lighting parameters in complex scenes and complex lighting environments, [DYB98] proposes an image-based method. This method puts a mirror ball into the scene to obtain the omnidirectional (360°) incident light where the mirror ball is located. By shooting several images with different exposures on the mirror ball, the omnidirectional high dynamic range of the scene can be obtained. Lighting, the virtual object can be drawn by mapping this image onto a sphere or cube. This image-based acquisition method of lighting environment has been widely used in rendering algorithms, which greatly facilitates the acquisition of lighting environment in complex environments. However, this method is not suitable for restoring the lighting environment of outdoor scenes. There are two reasons: 1) In outdoor lighting, the intensity of sunlight in sunny weather is very large. For existing cameras, even if the minimum exposure speed is used, the reflected light of the sun on the mirror ball will still be oversaturated and difficult to recover. The correct brightness of the sun light, causing the light recovery to fail. 2) In outdoor scenes, there are often clouds drifting by, or there is a breeze blowing, so it is difficult to ensure that the pixels of multiple photos taken with different exposures are completely still or completely aligned, so the obtained high dynamic range images are often inconsistent. precise.

2. Physics-based methods

The appearance of an object in an image is determined by the material properties (BRDF) of the object's surface, the geometry of the object, and the lighting. The lighting of a scene can be solved if the geometry and material properties of the scene are known. This physics-based solution method often uses the brightness information of image pixels, shadows, key points, etc. to obtain lighting parameters. Using a controllable light source, [NKGR06] decomposes a scene image into two images, the scene image under direct light illumination and the scene image under global illumination. In addition, in shape from shading, there are also some lighting parameters for work recovery scenes, such as [SJ08]. The above lighting solution based on physical model can achieve satisfactory results in indoor environments such as controlled laboratories. However, since most of these methods require 3D modeling of the scene, they are not suitable for lighting analysis of outdoor scenes

3. Intrinsic image analysis

In recent years, intrinsic image analysis has received increasing attention from computer graphics and computer vision researchers [TFA05]. Intrinsic image decomposition technology decomposes a natural image into a material image (intrinsic image) and a lighting image, where the lighting image is the product of lighting parameters and scene geometry. The main purpose of intrinsic image analysis is to extract the invariant amount of illumination in the scene, that is, the material properties, which can allow users to perform further editing operations on the material image. Since the lighting image is the product of the lighting parameters and the scene geometry, this method cannot explicitly solve the lighting parameters. In addition, the existing intrinsic image technology for recovering scenes from a single image is not robust and accurate enough, and the computational complexity is high, which cannot be performed in real time.

4. Illumination analysis of outdoor scenes

In recent years, the lighting analysis of outdoor scenes has received more and more attention. [SMPR07] decomposes a video volume of a static scene taken over time into 3 material images, two base curves, and a compressible shadow representation , any image in the video body can be reconstructed from the above elements. Due to the influence of assumptions, that is, in sunny weather, the appearance vectors of all pixels are the same in the sense of difference of an offset and a scale factor, so this method is only suitable for sunny weather. Video breakdown. Furthermore, [SRM08] proposes to represent the color distribution of outdoor light sources as a two-dimensional linear subspace in three-dimensional linear space, and obtain the chromaticity of scene light sources through methods such as shadow extraction. The above two methods analyze the outdoor scene images, but do not obtain clear lighting parameters and do not use lighting parameters to draw virtual objects. In addition, since these two methods are solved on the video volume, they are only suitable for post-processing the video, and cannot process the video online and in real time.

The work most relevant to the present invention is [AJM06]. The goal of this article is also to add virtual objects to a real scene, which can change the lighting of virtual objects in real time when the lighting of the scene changes. This work simulates sunlight as parallel light and sky light as floodlight. By building a 3D model of the scene, the scene point corresponding to each pixel on the image can be determined, and the outdoor light is simulated as a Phong model, and a set of equations is established to solve it. The value of the sun light and sky light. Through a series of preprocessing in the off-line stage, and the use of image-based rendering methods, the embedding of virtual objects can achieve real-time effects. The disadvantage of this method is that the 3D model of the known scene is required, and the modeling of the outdoor scene as mentioned above is very difficult, so it has great limitations in practical applications.

There are two main challenges in the illumination analysis of outdoor scenes: (1) geometric complexity. Compared to indoor scenes, the scale of outdoor scenes is much larger. In addition, outdoor scenes usually include some objects with complex shapes such as trees and terrain that are difficult to model. However, the existing indoor lighting algorithms generally require the 3D representation of the known scene, so the indoor lighting analysis method is not suitable for the lighting analysis of the outdoor scene. (2) Lighting complexity. Natural light sources (sun light and sky light) in outdoor scenes have some special properties. For example, the lighting of outdoor scenes is greatly affected by clouds and weather conditions, and cannot be controlled artificially. Due to the complex structure of the scene, it is directly based on the accurate lighting model It is difficult to reversely calculate the lighting parameters, and it is even more difficult to meet the real-time requirements.

Contents of the invention

The object of the present invention is to provide an outdoor scene illumination parameter recovery device and its working method to address the deficiencies of the prior art. The device is simple and easy to operate, and consists of a computer and a video camera. This method can obtain the lighting parameters of the outdoor scene in real time without knowing the geometric information and material information of the 3D scene, so that the virtual objects generated by the computer can be seamlessly integrated into the image sequence of the real scene at any time . Lighting parameters refer to the brightness values of the sun light and sky light.

An outdoor scene lighting parameter recovery device includes a computer and a camera, and the computer and the camera are connected through a data line.

A working method of an outdoor scene lighting parameter recovery device, the method is as follows:

1) A cloudy image of the scene captured by the camera is used as a sky light-based image and input to the computer;

2) According to the sky light-based image, the unit of the sky light brightness value is determined in an interactive manner;

3) The camera shoots the image sequence of the scene under any weather conditions, and transmits it to the computer in real time, and the computer processes the current frame;

4) Calculate the weight value corresponding to each pixel of the current frame through the weight function;

5) Calculate the sky brightness value according to the weight values corresponding to all the pixels in the current frame;

6) The pixel value of the current frame image is subtracted from the product of the sky luminance value and the corresponding pixel value of the sky light base image to obtain the sunlight component value of the corresponding pixel of the current frame image;

7) Determine whether the current frame is the first frame of the image sequence? If yes, set the value of the sunlight brightness to 1, and the value of the sunlight component is the sunlight-based image; otherwise, divide the sunlight component of the current frame image by the sunlight-based image to obtain the value of the sunlight brightness, and pass the current frame image’s sun Light component value, to get the updated sun light base image;

8) Determine whether the current frame is the last frame? If yes, the work is finished; if not, the next frame is used as the current frame, and steps (3) to (7) are repeated.

Introduce five aspects of the present invention in detail:

1) Image linear representation model under outdoor lighting scene

Under the assumption that the sunlight is a parallel light source and the sky light is a uniformly distributed surface light source, the luminance L(x, λ) at any three-dimensional point x in an outdoor scene can be expressed as the brightness part C _sun irradiated by sunlight Linear combination of (x, λ) and the brightness component C _sky (x, λ) illuminated by the sky light. I(x, λ) is the pixel value of the image pixel x in the λ channel, where λ=1, 2, and 3 are the three color channels of red, green, and blue respectively, and there are:

I(x,λ)=L _sun (λ)C _sun (x,λ)+L _sky (λ)C _sky (x,λ) (1)

In the above formula, C _sun (x, λ) and C _sky (x, λ) respectively represent the comprehensive effect of the shadow at x, the geometry, and the occlusion relationship between the sky light and the sun light at x. For a given sun orientation, no matter how the sun light and sky light change, these two items are constant, so they are called the base images of the sun light and sky light respectively. In general, due to the blooming nature of the sky light, its base image remains unchanged for any sun orientation. Any image of an outdoor scene can be represented as a linear combination of the sun-based image and the sky-based image.

2) Acquisition of sky light-based image

An image of a scene on a cloudy day is captured by the camera, and used as the sky light-based image C _sky (x, λ) of the scene.

3) Solve the sky luminance value of each frame according to the characteristics of the sun-based image

其权重函数为

再对下面的能量函数对不同的颜色通道λ分别求最小解以获取参数a(λ)：Record the current frame image I(x, λ) taken at any time, and calculate the weight value of each pixel of the current frame image

Its weight function is

Then find the minimum solution for the following energy function for different color channels λ to obtain the parameter a(λ):

Where Ω is the set of all pixels in I(x), then L _sky (λ)=a(λ). In this method, for an image with a gray level of 255, r ₁ =r ₂ =18 is generally taken.

4) According to the restored sky brightness value, restore the sun brightness value

After the sky luminance L _sky (λ) is recovered, the sunlight component value L _sun (λ)C _sun (x, λ) of the current frame can be obtained from formula (1). Because the sun moves slowly around the equatorial plane, the difference between the sun positions of two adjacent frames in the online image sequence is very small, so the solar light-based images of two adjacent frames are considered to be the same. If the current frame is the first frame, its sunlight component value L _sun (λ)C _sun (x, λ) is its sunlight base image (so the relative sunlight brightness is 1), recorded as C _sun ⁰ (x, λ); otherwise, divide the sunlight component value of the current frame by the sunlight base image c _sun ⁰ (x, λ) of the first frame to obtain the sunlight brightness value of the current frame, and the sunlight component L _{sun of} the current frame (λ)C _sun (x, λ) is divided by the brightness of the sun to obtain the sun-based image of the current frame, and use it as a new sun-based image.

5) the objective function defined in the present invention as shown in formula (2) has following three characteristics:

a) Using all the pixels in the image I(x) to solve the sky luminance value L _sky (λ), instead of using a single pixel to solve, improves the robustness of the algorithm to noise. On the other hand, although according to the definition C _sun (x, λ) should be a non-negative value, considering that the image noise tends to appear in the shadow area, the objective function requires the objective function to be the smallest in a square sense, which further improves the algorithm’s ability to resist noise. robustness.

b) There is no need to perform explicit binary shadow detection on I(x), and a weight function is used to control the weight of pixels with different brightness in the objective function.

c) The objective function has an analytical solution:

Compared with background technology, the present invention has the advantages of:

The present invention adopts an image-based linear model method, does not need three-dimensional geometric information of the scene, thus avoids various technical challenges faced when reconstructing a large-scale outdoor scene, and makes the illumination recovery method more practical. After the initial setting is completed, the method does not require human-computer interaction processing in the online stage, thereby ensuring that the method can process video streams. Due to the guaranteed lighting consistency, virtual objects can be seamlessly blended into the background video sequence of the real scene.

Description of drawings

Figure 1 is a flowchart of the method.

Figure 2. Schematic diagram of the device.

Detailed ways

The present invention will be further described below in conjunction with accompanying drawing and example.

Example:

An outdoor scene lighting parameter recovery device, as shown in Figure 2, includes a computer and a camera, and the computer and the camera are connected through a data line.

A working method of an outdoor scene lighting parameter recovery device, as shown in Figure 1, the method is as follows:

1) A cloudy image of the scene captured by the camera is used as a sky light-based image and input to the computer;

2) According to the sky light-based image, the unit of the sky light brightness value is determined in an interactive manner;

3) The camera shoots the image sequence of the scene under any weather conditions, and transmits it to the computer in real time, and the computer processes the current frame;

4) Calculate the weight value corresponding to each pixel of the current frame through the weight function;

5) Calculate the sky brightness value according to the weight values corresponding to all the pixels in the current frame;

6) The pixel value of the current frame image is subtracted from the product of the sky luminance value and the corresponding pixel value of the sky light base image to obtain the sunlight component value of the corresponding pixel of the current frame image;

7) Determine whether the current frame is the first frame of the image sequence? If yes, set the value of the sunlight brightness to 1, and the value of the sunlight component is the sunlight-based image; otherwise, divide the sunlight component of the current frame image by the sunlight-based image to obtain the value of the sunlight brightness, and pass the current frame image’s sun Light component value, to get the updated sun light base image;

8) Determine whether the current frame is the last frame? If yes, the work is finished; if not, the next frame is used as the current frame, and steps (3) to (7) are repeated.

Claims (2)

1. An outdoor scene lighting parameter recovery device, comprising a computer and a video camera, characterized in that the computer and the video camera are connected by a data line. 2. the working method of a kind of outdoor scene illumination parameter recovery device claimed in claim 1, is characterized in that, working method is as follows: 1) A cloudy image of the scene captured by the camera is used as a sky light-based image and input to the computer; 2) According to the sky light-based image, the unit of the sky light brightness value is determined in an interactive manner; 3) The camera shoots the image sequence of the scene under any weather conditions, and transmits it to the computer in real time, and the computer processes the current frame; 4) Calculate the weight value corresponding to each pixel of the current frame through the weight function; 5) Calculate the sky brightness value according to the weight values corresponding to all the pixels in the current frame; 6) The pixel value of the current frame image is subtracted from the product of the sky luminance value and the corresponding pixel value of the sky light base image to obtain the sunlight component value of the corresponding pixel of the current frame image; 7) Determine whether the current frame is the first frame of the image sequence? If yes, set the value of the sunlight brightness to 1, and the value of the sunlight component is the sunlight-based image; otherwise, divide the sunlight component of the current frame image by the sunlight-based image to obtain the value of the sunlight brightness, and pass the current frame image’s sun Light component value, to get the updated sun light base image; 8) Determine whether the current frame is the last frame? If yes, the work is finished; if not, the next frame is used as the current frame, and steps (3) to (7) are repeated.

Images