KR100834157B1 - Method for Light Environment Reconstruction for Image Synthesis and Storage medium storing program therefor. - Google Patents

Abstract

Disclosed is a recording medium on which an illumination environment reconstruction method and program for image synthesis are recorded. According to an embodiment of the present invention, a method for reconstructing an illumination space for image synthesis includes: calculating a plurality of photographing positions corresponding to an input image of a target space; Classifying characteristics of illumination of the target space by using an input image; And reconstructing the three-dimensional illumination space using characteristics of the plurality of photographing positions and illumination.

According to the present invention, the position of the camera and the lighting is automatically calculated without a preliminary correction process, and there is an advantage that the lighting information can be accurately applied to a moving object.

Video synthesis, rendering, lighting

Description

Method for Light Environment Reconstruction for Image Synthesis and Storage medium storing program therefor.}

1 is a flowchart illustrating a lighting environment configuration method according to an embodiment of the present invention.

2 is a diagram illustrating a multiple exposure image photographed by varying a shutter speed of a camera according to an exemplary embodiment of the present invention.

3 is a diagram illustrating a response function of a camera estimated using multiple exposure images according to an embodiment of the present invention.

4 is a plan view of an image acquisition system according to an embodiment of the present invention;

5 is a diagram illustrating a front hemisphere image obtained at the shooting position C ₁ and C ₂ according to an embodiment of the present invention.

6 is a diagram illustrating a direction vector for setting a search region of a corresponding point for estimating global illumination according to an embodiment of the present invention.

FIG. 7 is a diagram illustrating illumination and sampled corresponding points sampled between L-L unit images according to an embodiment of the present invention. FIG.

FIG. 8 illustrates an image in which layers corresponding to illuminations sampled in LLI2 and corresponding points in LLI2 corresponding to LLI1 according to an embodiment of the present invention are superimposed.

9 is a view showing a result of synthesizing a virtual object in a target space by a method for reconfiguring an illumination environment according to an embodiment of the present invention.

The present invention relates to a recording medium on which a method and a program for configuring a lighting environment are recorded. More specifically, a 3D (Dimension) spatial analysis and a method and program for configuring a lighting environment for realistic synthesis of virtual objects in real world space are recorded. To a recorded recording medium.

The technique of photo-realistic synthesis of virtual objects in photorealistic images has long been studied in computer vision and graphics. In order to obtain a realistic composite image, the geometric characteristics between the real camera and the virtual object must be matched, and the same brightness of light applied to other objects in the background image must be applied to the virtual object.

In general, determining the location and brightness of the lights in the object space requires a great deal of time and effort, and thus it is very difficult to obtain realistic scenes. This is because it is difficult to fully consider the effects of complex lighting in the real world such as direct and indirect lighting, shadows, reflections and refraction by light sources. Many rendering tools to automate this compositing process have been proposed so far, but most of the research has a problem that it is limited to a static situation in which the light source and the object do not move.

Therefore, the conventional lighting environment reconstruction method has a problem that the dynamic situation through the reconstruction of the lighting information is not considered.

In addition, the conventional lighting environment reconstruction method has a problem that it is necessary to pre-calibration to know the position and internal parameters of the camera photographing the target space.

In order to solve the above-mentioned problems, the present invention provides a method for reconfiguring an illumination environment for image synthesis, and a recording medium on which a lighting environment reconstruction method and a program for applying illumination information can be accurately applied to a moving object. To propose.

In addition, the present invention proposes a recording medium in which a method and a program for automatically reconstructing an illumination environment by automatically estimating the need for pre-calibration to know a camera position and an internal parameter photographing a target space are recorded.

Still other objects of the present invention will be readily understood through the following description of the embodiments.

In order to achieve the above object, according to an aspect of the present invention, a method for reconstructing the illumination space for image synthesis, comprising: calculating a plurality of shooting positions corresponding to the input image of the target space; (b) classifying characteristics of illumination of the object space by using the input image; And (c) reconstructing a three-dimensional illumination space using the plurality of photographing positions and the characteristics of the illumination.

The input image is a multiple exposure image photographed at different shutter speeds of a camera equipped with a fisheye lens, and (d) the response of the camera using the multiple exposure image prior to step (a). The method may further include estimating a response function and generating a high dynamic range (HDR) image.

In the step (a), (a1) the input image is a front and rear hemi-sphere images corresponding to the plurality of photographing positions, respectively, and constructing a hemisphere coordinate system having the plurality of photographing positions as the origin. ; (a2) calculating a projected center point corresponding to the first and second center points of the hemisphere images acquired at the first and second photographing positions using a preset algorithm; And (a3) calculating a second photographing position relative to the first photographing position by using the first and second center points and the projected center point, and corresponding to the number of the plurality of photographing positions. The plurality of photographing positions may be calculated by repeating steps (a2) to (a3).

The preset algorithm may be at least one of a feature point extraction algorithm and a matching algorithm.

를 이용하여

각(∠AC1B) 및

각(∠ C1 BC2)를 각각 구하는 단계 - C1 및 C2는 제1 및 제2 촬영위치이고, A 및 B는 C1 및 C2에서 취득한 반구영상의 중앙점이며,

는 렌즈의 초점거리이며,

은 각각의 반구영상의 중앙점과 상대 중앙점이 사영된 중앙점간의 거리임 -; 상기

각(∠AC1B), 상기

각(∠ C1 BC2) 및 수학식

을 이용하여 제1 촬영위치 C1에 상대적인 제2 촬영위치 C2를 계산하는 단계-상기

는 상기 제1 촬영위치 C1로부터 상기 중앙점 B를 통과하는 선분, 상기

는 상기 제2 촬영위치 C2로부터 상기 중앙점 A를 통과하는 선분, 상기

및 상기

는 임의의 수

및

에 상응하여 각각 변화하는 상기

및

의 방향벡터임-를 포함할 수 있다. Step (a3), the equation

Using

Angle (∠AC1B) and

Obtaining angles (∠ C1 BC2), respectively: C1 and C2 are first and second photographing positions, A and B are center points of the hemisphere image acquired at C1 and C2,

Is the focal length of the lens,

Is the distance between the center point of each hemisphere image and the center point where the relative center point is projected-; remind

Angle (∠AC1B), above

Angle (∠ C1 BC2) and equation

Calculating a second photographing position C2 relative to the first photographing position C1 using

Is a line passing through the center point B from the first photographing position C1,

Is a line segment passing through the center point A from the second photographing position C2,

And said

Is any number

And

Corresponding to each change corresponding to

And

It may be a direction vector of-.

Prior to the step (b), (e1) the input image is a front and rear hemi-sphere images corresponding to the plurality of shooting positions, and the LL (Latitude- Creating a unit image; (e2) calculating positions of a plurality of light sources of the L-L unit image by using structured importance sampling; (e3) constructing a hemisphere coordinate system each having the plurality of shooting positions as an origin; And (e4) calculating unit direction vectors corresponding to positions of the plurality of illumination sources, respectively.

Step (b) may include: (f1) converting a first direction vector of the plurality of unit direction vectors calculated from the first L-L unit image corresponding to the first photographing position into a rectangular coordinate system; (f2) calculating a corresponding point in the second L-L unit image by using a preset algorithm in the second L-L unit image corresponding to the second photographing position and the converted direction vector; And (f3) calculating the corresponding points in the second LL unit image by repeating steps (f1) to (f2) corresponding to the number of unit direction vectors calculated in the first LL unit image. However, in accordance with the number of LL unit images corresponding to the plurality of photographing positions, repeating the step (f1) and (f3), each of the one or more corresponding points in the LL unit images corresponding to the plurality of photographing positions, respectively. Can be calculated.

를 이용하여

을 산출하는 단계-

는 상기 제1 방향 벡터가 직각 좌표계로 변환된 것이고,

는 임의의 수임 -; (g2) 상기

와 수학식

을 이용하여

을 산출하는 단계; 및 (g3) 상기

을 이용하여 상기 제2 L-L 단위영상에서 대응 점을 산출할 수 있다. Step (f2) is (g1) equation

Using

To calculate the

Is the first direction vector is converted to a rectangular coordinate system,

Is any number-; (g2) above

And equations

Using

Calculating; And (g3) above

The corresponding point may be calculated from the second LL unit image.

을 구면 좌표계로 변환하여

을 산출하는 단계; (h2) 상기

가 상기 제2 L-L 단위영상에 대응되는 위치를 산출하는 단계; (h3) 상기 산출된 위치를 중심으로 미리 지정된 크기의 검색영역 내에서 밝기에 대한 SSD(Sum of Squared Differences)가 가장 적은 값에 대응되는 위치를 대응점으로 산출하는 단계를 포함할 수 있다. Step (g3) is (h1)

To a spherical coordinate system

Calculating; (h2) above

Calculating a position corresponding to the second LL unit image; (h3) calculating a position corresponding to a value with the smallest sum of squared differences (SSD) for brightness in a search area having a predetermined size based on the calculated position as a corresponding point.

Step (b) comprises: (i1) sampling a plurality of lights in a second L-L unit image corresponding to the second photographing position; (i2) dividing a plurality of layers of the second L-L unit image according to the number of the plurality of lights; And (i3) after the step (f3), calculating the illumination corresponding to the floor not including the corresponding point as the local lighting.

을 이용하여 복수개의 3차원 단위 방향벡터로 각각 변환하는 단계; 및 (j2) 상기 촬영위치 및 상기 복수개의 3차원 방향벡터를 미리 설정된 알고리즘에 적용하여 전역 조명의 좌표를 산출하는 단계를 포함하되, 상기 복수개의 촬영위치에 대응하는 상기 L-L 단위 영상의 개수에 상응하여 상기 (j1)단계 내지 (j2)단계를 반복하여 상기 전역조명의 좌표를 각각 산출할 수 있다.Step (c) is performed by (j1) calculating a plurality of unit direction vectors corresponding to the first and second photographing positions.

Converting each into a plurality of three-dimensional unit direction vectors by using; And (j2) calculating coordinates of global illumination by applying the photographing position and the plurality of three-dimensional direction vectors to a preset algorithm, and corresponding to the number of the LL unit images corresponding to the plurality of photographing positions. By repeating steps (j1) to (j2) to calculate the coordinates of the global illumination, respectively.

을 이용하여

및

을 각각 산출하는 단계 -

및

는 각각 제1 및 제2 촬영위치의 3차원 단위 방향벡터; 및 (k2) 상기

및 상기

의 교차점을 전역 조명의 좌표로 산출하는 단계를 포함하되, 상기 3차원 단위 방향벡터

및

의 개수에 상응하여 상기 (k1)단계 내지 (k2)단계를 반복하여 상기 전역 조명의 좌표를 각각 산출할 수 있다. Step (j2) is (k1) equation

Using

And

Calculating each of-

And

Are the three-dimensional unit direction vectors of the first and second photographing positions, respectively; And (k2) above

And said

Computing the intersection of the coordinates of the global illumination, the three-dimensional unit direction vector

And

The coordinates of the global illumination may be calculated by repeating steps (k1) to (k2) corresponding to the number of.

및 상기

의 교차점이 존재하지 않는 경우 상기

및 상기

간에 거리가 각각 가장 가까운 위치를 전역조명의 좌표로 산출하는 단계를 더 포함할 수 있다. The step (k2) is

And said

Above if no intersection of

And said

The method may further include calculating, as coordinates of the global lights, positions where distances are closest to each other.

을 이용하여 복수개의 3차원 단위 방향벡터로 각각 변환하는 단계; (k2) 상기 복수개의 3차원 단위 방향벡터의 교차점을 지역조명의 좌표로 산출하는 단계; (k3) 상기 복수개의 촬영위치에 대응하는 상기 L-L 단위 영상의 개수에 상응하여 상기 (k1)단계 내지 (k2)단계를 반복하여 상기 L-L 단위영상에서 상기 지역조명의 좌표를 각각 산출하는 단계를 더 포함할 수 있다. Step (c) may be performed by (1) calculating a plurality of unit direction vectors corresponding to the first and second photographing positions.

Converting each into a plurality of three-dimensional unit direction vectors by using; (k2) calculating intersections of the plurality of three-dimensional unit direction vectors as coordinates of area lights; (k3) repeating steps (k1) to (k2) corresponding to the number of the LL unit images corresponding to the plurality of photographing positions, and calculating the coordinates of the local lighting in the LL unit image, respectively. It may include.

According to another aspect of the present invention, a program of instructions that can be executed by a digital image processor is tangibly implemented for reconstructing an illumination environment for image synthesis, and a recording medium recording a program that can be read by the digital image processor. (A) calculating a plurality of shooting position corresponding to the input image of the target space; (b) classifying characteristics of illumination of the object space by using the input image; And (c) recording a program for executing a step of reconstructing a three-dimensional illumination space using the plurality of photographing positions and the characteristics of the illumination.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In the present specification, an embodiment of the present invention will be described using the LL environment map acquired at the first photographing position and the LL environment map acquired at the second photographing position as an example. It will be apparent to those skilled in the art that this may be repeated as many times as possible.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, the same reference numerals will be used for the same means regardless of the reference numerals in order to facilitate the overall understanding.

1 is a flowchart illustrating a lighting space reconstruction method according to an embodiment of the present invention.

Referring to FIG. 1, the illumination space reconstruction method according to an embodiment of the present invention acquires an omni-directional image of a target space for analyzing lighting information in step S101.

For example, a digital camera equipped with a fisheye lens can acquire an omnidirectional image of the target space, and the front and rear hemisphere images to obtain illumination information for all spaces in 360 degrees of three-dimensional space. (hemi-sphere) can be obtained.

At this time, the apparatus for photographing the image in the target space is not limited to a digital camera equipped with a fisheye lens, and it is apparent to those skilled in the art that the apparatus includes all devices for acquiring illumination information in the target space regardless of the name of the apparatus. However, in the present specification, a digital camera equipped with a fisheye lens will be described as an example to facilitate the understanding and explanation of the present invention.

Here, since the pixel value of the image taken by the digital camera is fixed according to the input condition at the time of the omnidirectional image acquisition in the target space, limited illumination information is obtained rather than actual radiance. As the position of the camera changes, it is difficult to express a new scene. Therefore, in order to solve this problem in step S102, the response function of the camera is estimated from the multiple exposure images photographed at different shutter speeds for the target scene, and a high dynamic range (HDR) image is generated.

Next, the position of the camera which acquired the omnidirectional image of the target space in step S103 is calculated, and the characteristic of illumination is classified in step S104.

Subsequently, in step S104, a corresponding point for estimating global lighting affecting most of the target space is calculated (S 105), and a step of estimating local lighting having a direction that has a limited influence on some regions. (S106).

Subsequently, three-dimensional coordinates of global and local lights are calculated in step S107 to construct a three-dimensional lighting environment, and in step S108, an image is synthesized and rendered into a virtual scene.

So far, the flow of the illumination space reconstruction method according to an embodiment of the present invention has been described. Hereinafter, each step of the illumination space reconstruction method according to an embodiment of the present invention will be described in detail with reference to the drawings. First, an operation (S102) of estimating a response function of a camera to generate an HDR image will be described.

2 is a diagram illustrating a multiple exposure image photographed by varying a shutter speed of a camera according to an embodiment of the present invention, and FIG. 3 is a view illustrating a camera estimated using a multiple exposure image according to an embodiment of the present invention. A diagram illustrating a response function.

Referring to FIG. 2, an image 206 of photographing a target space by gradually decreasing a shutter speed from an image 201 of photographing a target space by increasing a shutter speed of a digital camera is illustrated.

In general, when taking an image with a digital camera, when the dynamic range is narrow and the shutter speed is slowed to secure a large amount of light (for example, the 201 image of FIG. 2), the dark portion is expressed but the bright portion is not captured in white. On the contrary, when the shutter speed of the digital camera is increased, the bright part is well expressed, but the dark part is darkened, and the distinction becomes unclear. This is because digital cameras cannot express all of the wide lighting range of the real world.

Therefore, in an embodiment of the present invention, as a method for overcoming this, a plurality of pieces of the target space at the same position are taken with different exposures of the digital camera. In addition, the pixel value for each pixel position of the image generated by photographing is divided into values of a wider area rather than between 0 and 255. When the image is expressed by the data of the finely divided wide area, the HDR image is obtained. When the pixel value of the finely divided illumination is graphed, the camera response function 300 of FIG. 3 is obtained.

More specifically, as illustrated in FIG. 2, when the target space at the same location is photographed with different exposures of the digital camera to generate a multiple exposure image, a pixel value (between 0 and 255) corresponding to a specific location is generated. You can see how it changes as the shutter speed changes.

For example, it can be seen that the pixel value at a specific position changes from 255 to values of 200, 180, 90, 50 and 0.

In this case, each of the above data may be collected for every pixel, and the digital camera response function 300 may be estimated using mathematical calculations. Actual lighting value).

Here, since the method of estimating the response function of the digital camera using the pixel value of the specific position is already known, a detailed description thereof will be omitted.

Using the response function 300 of the digital camera shown in FIG. 3, the pixel value of the image taken by the camera may be converted into an actual lighting value, and thus an HDR image may be generated.

So far, the step (S102 of FIG. 1) of estimating the response function of the camera is generated with reference to FIGS. 2 and 3. Hereinafter, the step of calculating the position of the digital camera photographing the target space ( S103 of FIG. 1 is demonstrated.

4 is a plan view of an image acquisition system according to an embodiment of the present invention, and FIG. 5 is a front view of the front surface acquired at the photographing positions C ₁ 401 and C ₂ 402 according to an embodiment of the present invention. A diagram illustrating a hemisphere image.

In general, environment maps, which have lighting information for all 360-degree parts of the target space, are used to create scenes that are normally illuminated by light sources in distant places. Obtain the distribution of existing illumination sources.

Since the real world is composed of complex structures, the radiance distribution obtained at a fixed point cannot express all the effects of various light sources in the scene. This is because there are many light sources with local characteristics that cannot be sampled in a single environment map in a complex structured space.

Therefore, in one embodiment of the present invention, a plurality of HDR images are acquired by placing cameras at different positions in order to analyze illumination sources having regional characteristics.

Here, the process of generating the HDR image using the response function of the camera estimated from the multiple exposure images photographed at different shutter speeds has been described with reference to FIGS. 2 and 3, and thus redundant descriptions thereof will be omitted.

Referring to FIG. 4, C _i (i = 1, 2, 3,..., N) (401 to 404) is the projection center point of each digital camera, and at each point, all the cameras rotate about the vertical axis. Take a picture of the object space. In one embodiment of the present invention, in order to facilitate the understanding and explanation of the present invention, the object space is assumed to be in the form of a cube, but it is apparent to those skilled in the art that the object space to which the present invention is applicable is not limited thereto.

In addition, although FIG. 4 illustrates four locations (401 to 404), for example, the position of the camera (hereinafter, referred to as a 'shooting position'), it may be variously changed according to an environment and an object to which the present invention is applied. Self-explanatory

Latitude-Longitude environment map (hereinafter referred to as LL unit image) by performing a panoramic conversion of the hemi-sphere images of the front and rear images taken at each position according to an embodiment of the present invention. And then position the lights by structured importance sampling.

Here, panoramic transformation is a technique for converting three-dimensional image information into a two-dimensional panorama image. Since the 2D panoramic image is a 2D image but has all 3D information, when a specific position is selected in the 2D panoramic image, it is possible to know where the position exists in the 3D image. Since panorama conversion is a known technique, a description of the conversion process is omitted herein.

The 2D panoramic image generated at this time is a 2D image, but is an HDR image having all 3D information.

In this case, when all the lighting information included in the 2D panoramic image is rendered, the same image as the actual image can be obtained, but the calculation time becomes very large. Therefore, in an embodiment of the present invention, structured importance sampling may be applied to sample a limited number of lights that best represent the lighting of a space. At this time, the position value of the sampled lights can be calculated in three dimensions by performing the inverse transformation of the panoramic transformation, and the degree of brightness of the lighting there.

At this time, it is apparent to those skilled in the art that the sampling that can be applied in the course of the present invention is not limited thereto.

Subsequently, a hemisphere coordinate system whose origin is the camera position (for example, 401 to 404) is configured, and a unit direction vector for an illumination source existing in the hemisphere image is calculated. The calculated direction vector is then used in the step of estimating the global illumination and the local illumination and estimating the respective positions (steps S104 to S107 of FIG. 1), which will be described later with reference to FIGS. 6 to 8.

According to an embodiment of the present invention, in order to calculate a plurality of photographing positions, a photographing position C ₁ 401 located at a center point of a space is defined as an origin and the relative position between the other points (for example, C ₂ to C ₄ ). Calculate your location. Hereinafter, for convenience of understanding and description of the present invention will be described with reference to FIGS. 4 and 5 taking the photographing position C ₁ (401) and C ₂ as an example.

According to an embodiment of the present invention, the positions of the center points A and B of the front hemisphere image acquired by C ₁ 401 and C ₂ 402 are (0, L) 431 and (Δx, L) 432. Can be defined as The projected center points A '511 and B' 512 are calculated using a feature point extraction algorithm and a matching algorithm.

The feature point extraction algorithm is a well-known technique for extracting a point where a large difference in brightness from a neighboring pixel or a point where a plurality of linear components intersect using a differential operator or the like is omitted.

In addition, since the matching algorithm is a known technique for obtaining a corresponding position in the right image 502 with respect to a specific position of the left image 501 using a mask, description thereof will be omitted.

각(∠AC₁B)(421) 및

각(∠ C₁ BC₂)(422)를 하기의 수학식 1을 이용하여 산출한다.According to an embodiment of the present invention using the center point A 431 and B 432 and the projected center point A '511 and B' 512

Angle (∠AC ₁ B) 421 and

An angle ∠ C ₁ BC ₂ 422 is calculated using Equation 1 below.

는 렌즈의 초점 거리(focal length)이며,

(521) 및

(522)는 각각 영상의 중심점과 상대 중심점이 사영된 지점 간의 거리이다. 본 발명의 일 실시예에 따라

각(421)과

각(422)를 계산한 후, 방향 벡터

및

와 라인

및

들을 하기의 수학식을 이용하여 정의할 수 있다. here,

Is the focal length of the lens,

521 and

Reference numeral 522 denotes a distance between the center point and the relative center point of the image, respectively. In accordance with one embodiment of the present invention

Angle (421)

After calculating the angle 422, the direction vector

And

And lines

And

These can be defined using the following equation.

는 제1 촬영위치 C₁(401)로부터 중앙점 B(432)를 통과하는 선 분, 상기

는 제2 촬영위치 C₂(402)로부터 중앙점 A(431)를 통과하는 선분, 방향벡터

및

는 임의의 수

및

에 상응하여 각각 변화하는 선분

및

의 방향벡터이다. here,

Is a line segment passing through the center point B 432 from the first photographing position C ₁ 401, wherein

Is a line segment and direction vector passing from the second photographing position C ₂ 402 to the center point A 431.

And

Is any number

And

Corresponding line segments

And

Direction vector of.

가 y=L인 지점을 교차할 때 Δx(411)를,

가 x=Δx인 지점을 교차할 때 Δy(412)를 각각 산출할 수 있고, 따라서 제2 촬영위치인 C₂의 위치(402)를 산출 할 수 있다.Then, the line segment according to the embodiment of the present invention

Δx (411) when we intersect a point where y = L,

When y crosses a point where x = Δx, each of Δy 412 can be calculated, and thus, the position 402 of C ₂ , which is the second photographing position, can be calculated.

So far, the method of calculating the position of the second photographing position C ₂ 402 with the first photographing position C ₁ 401 as the origin has been described. It will be apparent to those skilled in the art that C _i can also be calculated in the same manner as above.

Up to now, the step of calculating the position of the camera (step S103 of FIG. 1) according to an embodiment of the present invention has been described with reference to FIGS. 4 and 5, the following step of classifying the characteristics of the illumination (of FIG. A step of calculating a corresponding point (S105 in FIG. 1) to estimate global illumination in step S104 will be described.

According to an embodiment of the present invention, the characteristics of the lights acquired in the target space may be classified into global lighting and local lighting. Global illumination directly affects all regions within the target space and is acquired from all L-L unit images, while local illumination only affects specific regions within the target space. When a virtual object moves to a specific area within the target space, the lighting space is reconstructed by classifying the light characteristics from L-L unit images acquired at a plurality of photographing positions in order to be affected by not only global lighting but also local lighting.

Hereinafter, a process of calculating a corresponding point for estimating global illumination (step S105 of FIG. 1) will be described first.

6 is a diagram illustrating a direction vector for setting a search region of a corresponding point for estimating global illumination according to an embodiment of the present invention, and FIG. 7 is sampled between LL unit images according to an embodiment of the present invention. The figure illustrates the lighting and the detected corresponding point.

In general, the three-dimensional coordinates of global lights can be estimated using the corresponding points between the L-L unit images. However, in the L-L unit image, it is difficult to obtain the corresponding points effectively by the existing feature extraction algorithm because of the distortion caused by the spherical projection.

Therefore, in an embodiment of the present invention, a feature point extraction algorithm using an adaptive search region based on a direction vector of an illumination position is defined and applied to search for a corresponding point in an L-L unit image.

First, according to an embodiment of the present invention, the position of the light sampled from the LL unit basic image (hereinafter referred to as 'LLI1') acquired at the first photographing position C ₁ 401 which is the center position of the object space is estimated as global illumination. can do. This is because the lighting information acquired from the center point in the target space can be assumed to affect all positions in the target space.

Subsequently, in order to calculate the exact position of the illumination obtained from the LLI1 by the global illumination, information about the global illumination in the LL unit image acquired in another region (for example, the second photographing position C ₂ 402) is required. Because the global lighting position estimated by LLI1 is a vector that indicates the direction of global illumination, the distance can be positioned in 3D space by using the information about global illumination acquired from other regions.

However, it will be apparent to those skilled in the art that the first photographing position is not limited to the center point of the object space and may be any position, and thus a method for calculating global illumination may be added.

를 산출하여, 하기의 수학식 3에 의해 3차원 선분

(601)을 산출 할 수 있다. According to an embodiment of the present invention, by converting a rectangular vector coordinate system to a direction vector (which is a direction vector with respect to the illumination source in the hemispherical coordinate system whose camera position is the origin, among the illumination sources sampled in LLI1).

To calculate the three-dimensional line segment by the following equation

601 can be calculated.

(601)로 향하는 방향벡터

(602)을 하기의 수학식 4에 의하여 산출할 수 있다. Subsequently, in order to set a search region for searching for a corresponding point on the LL unit image (hereinafter referred to as 'LLI2') acquired at the second photographing position C ₂ 402 according to one embodiment of the present invention, From shooting location C ₂ (402)

Direction vector to (601)

(602) can be calculated by Equation 4 below.

(602)는 임의의 수

에 따라 변화할 수 있다. Here, as illustrated in FIG. 6

602 is any number

Can change accordingly.

Referring to FIG. 7, FIG. 7A is a diagram illustrating an illumination sampled at LLI1 according to an embodiment of the present invention, and FIG. 7B is an arbitrary representation of LLI1 according to an embodiment of the present invention. FIG. 7C is a diagram illustrating corresponding points sampled in LLI2 for any illumination of LLI1 according to an embodiment of the present invention.

(602)의 궤적을 LLI2에서의 위치를 산출할 수 있다(702). According to an embodiment of the present invention calculated using the direction vector and the equation (4) of any illumination 701 on the LLI1

The locus of 602 can be calculated at location 702 in LLI2.

(602)의 위치를 중심으로 미리 지정된 마스크(mask) 속의 영상의 밝기에 대한 SSD(sum of Squared Difference)를 구해 가장 적은 값이 나오는 지점을 LLI2상의 대응점으로 정할 수 있다.Here, any illumination 701 on LLI1 and LLI2 on

The point where the smallest value is obtained may be determined as a corresponding point on the LLI2 by obtaining a sum of squared difference (SSD) of brightness of an image in a predetermined mask based on the position of 602.

(602)의 위치를 중심으로 매크로 블록(macro block) 5Ⅹ5 크기의 마스크 내에서 영상의 밝기에 대한 SSD(sum of Squared Difference)를 구해 가장 적은 값이 나오는 지점을 LLI2상의 대응점으로 정할 수 있다.For example, if the size of a predetermined mask is a macro block 5Ⅹ5 size, the arbitrary illumination 701 on the LLI1 and the LLI2 on

The point where the smallest value is obtained may be determined as the corresponding point on the LLI2 by obtaining a sum of squared difference (SSD) for the brightness of an image in a macroblock 5 × 5 mask based on the position of 602.

Subsequently, according to the embodiment of the present invention, the lighting of the LLI1 is changed by varying arbitrary illuminations on the LLI1 and the corresponding points of the LLI2 are calculated and illustrated on the LLI2, respectively, as shown in FIG. 7C.

In the exemplary embodiment of the present invention, since most of the radiance energy incident to the camera in the experimental image is reflected from the ceiling light to the blue screen, many lights on the blue screen of FIG. Sampled to the corresponding point.

Thus, it will be apparent to those skilled in the art that the corresponding point between the L-L unit images photographed at the plurality of photographing positions can be sampled by the above-described method.

So far, the step of calculating the corresponding point for estimating the global illumination (step S105 of FIG. 1) has been described with reference to FIGS. 6 and 7. Hereinafter, a method of estimating local lighting according to an exemplary embodiment of the present invention will be described.

In order to more realistically express the scene in which the object moves from the first shooting position C ₁ 401 to the second shooting position C ₂ 402 in the synthesized image, only the second shooting position C ₂ 402 is affected. State lighting should be considered. The lights sampled on the LLI2 include global lights as well as local lights.

According to an embodiment of the present invention, after detecting a plurality of illumination sources in the LL unit image corresponding to the second photographing position C ₂ 402 in order to detect the area lighting, a layer generated as many as the number of sampled illumination sources ( strata) Si information can be used.

FIG. 8 is a diagram illustrating an image in which layers corresponding to LLI2 corresponding to LLI1 and strata for the illumination sampled at LLI2 are superimposed according to an embodiment of the present invention.

First, referring to the strata shown in FIG. 8 (for example, 801 and 802 of FIG. 8), the layer is generated in the process of sampling a plurality of lights in LLI2, and the number of lights sampled is generated. do.

According to one embodiment of the invention, when sampling the structural importance in LLI2, one illumination may be sampled for each floor. At this time, the illumination may represent illumination information of the area included in the layer. That is, the sampled illumination represents the illumination brightness calculated by averaging the sum of all pixel values of each pixel position included in the layer.

Referring to FIG. 8, corresponding images calculated in LLI2 overlap with images of LLI2, layers generated in the process of sampling structural importance in LLI2, and illumination of LLI1.

Here, according to an embodiment of the present invention, local lighting may be estimated as the following pseudo code.

For i = 1, 2, 3,... , k

For j = 1, 2, 3,... , n

If Cp _j ∈ Si then false

else true

Loop j

If true then Si is the stratum of local light

Loop i

Where Cp is the corresponding point in LLI2 corresponding to the illumination of LLI1, n is the number of Cp, and k is the number of illumination calculated by the structural importance sampling in LLI2.

Referring to the above pseudo code according to an embodiment of the present invention, a layer (eg, 801 of FIG. 8) that does not include a corresponding point among a plurality of layers of LLI2 may be estimated by local lighting, and the layer may be The lighting of the representative LLI2 can be calculated as the location of the local lighting.

Up to now, the method of calculating the LLI2 local light in the relationship between the image of the LLI1 and the LLI2 has been described. The above-described method is repeated in the corresponding LL unit image by changing the plurality of shooting positions. It will be apparent to those skilled in the art that the calculations can be made separately, so that redundant description will be omitted to clarify the gist of the present invention.

So far, the method of calculating the local lighting has been described, and the following describes the step of calculating the three-dimensional coordinates of the global lighting and the local lighting (S107 of FIG. 1).

According to one embodiment of the present invention, the photographing position is calculated (S103 in FIG. 1), and the characteristics for the illumination are calculated after the global illumination (S105 in FIG. 1) and the local illumination (S106 in FIG. 1), respectively. Three-dimensional coordinates can be calculated to construct a three-dimensional lighting environment (S107 in FIG. 1).

First, a method of calculating three-dimensional coordinates of global lights will be described using LLI1 and LLI2 as examples.

According to an embodiment of the present invention, the three-dimensional coordinate values of the global lights may be calculated by a stereo algorithm.

Since a stereo algorithm is a known technique for calculating a corresponding point of each other using two images and then determining the intersection point of each vector as a three-dimensional position, a detailed description thereof will be omitted.

를 산출할 수 있다. According to an embodiment of the present invention, the two-dimensional coordinate value of the corresponding point calculated in the LL unit image is a three-dimensional direction vector according to Equation 5 below.

Can be calculated.

와 LLI2에서 산출된 대응점에 대한 방향벡터

을 수학식 5를 이용하여 산출하고,

,

및 하기의 수학식 6을 이용하여 선분

및

을 산출 할 수 있다. A plurality of direction vectors for the global illumination calculated in LLI1 according to an embodiment of the present invention

Direction vectors for the corresponding points computed from and LLI2

Is calculated using Equation 5,

,

And a line segment using Equation 6 below.

And

Can be calculated.

은 전역조명의 개수이다.here

Is the number of global lights.

및

의 교차점을 전역조명의 3차원 좌표로 산출 할 수 있다. Subsequently, the line segment is performed by a stereo algorithm according to an embodiment of the present invention.

And

The intersection of can be calculated by 3D coordinate of global lighting.

However, due to an error occurring in the L-L unit image acquisition process, two line segments may not intersect correctly. In this case, the point where the distance between the two line segments is closest may be determined as 3D coordinates of the global illumination. At this time, the distance in three dimensions of the two line segments can be calculated by the following equation (7).

The three-dimensional coordinates of the local light according to an embodiment of the present invention may be obtained by calculating a position at which the direction vectors of each L-L unit image cross in a virtual space.

Global lighting is a point light, where the light extends into the radiation and affects all object spaces, while local lighting is a directional light.

For example, the virtual space may be placed in a cube shape, and the first area light may be positioned at a point where the direction vector of the first L-L unit image contacts the cube.

In this case, since the first area light is a directional light, only the first photographing position corresponding to the first L-L unit image is affected.

It will be apparent to those skilled in the art that the above-described method can calculate the position where each direction vector in each L-L unit image intersects the virtual space by local lighting.

Up to now, the step (S107 of FIG. 1) for constructing a 3D lighting environment for image synthesis by calculating 3D coordinates of global lighting and local lighting has been described with reference to FIG. 9. After the three-dimensional lighting environment is configured, the step of synthesizing and rendering the virtual object in the target space (S108 of FIG. 1) is well known to those skilled in the art, and thus, detailed description thereof will be omitted.

FIG. 9 is a diagram illustrating a result of synthesizing a virtual object in a target space by a method of reconfiguring an illumination environment according to an exemplary embodiment of the present invention. More specifically, in order to determine whether or not the realistic expression of the dynamic situation is a synthetic image, the virtual object is rendered while moving from the first photographing point C ₁ 401 to the second photographing point C ₂ 402.

Referring to FIG. 9, FIG. 9A illustrates an image rendered at the first photographing point C ₁ 401, and FIG. 9B illustrates using only global illumination at the second photographing point C ₂ 402. The image is a rendered image, and FIG. 9C is an image rendered by using both global and local lighting at the second photographing point C ₂ 402.

Since the image rendered at the first photographing point C ₁ 401 according to an embodiment of the present invention was assumed to have only global illumination, there was little difference from the case of considering some local illumination (FIG. 9A). Reference). However, as the object moves to the second photographing point C ₂ 402, the effect of local lighting may be taken into consideration to express a brighter and more realistic scene (see FIG. 9C). This is because there are many indirect lights on the blue screen of the object space as illustrated in FIG. Therefore, as virtual objects such as spheres, toruss, and tables move to the second photographing point C ₂ 402 close to the blue screen, natural illumination is achieved by not only global illumination but also local illumination as shown in FIG. 9 (c).

One embodiment of the present invention described above is for the purpose of illustration, those skilled in the art will be able to various modifications, changes, additions within the spirit and scope of the present invention, such modifications, changes and Additions should be considered to be within the scope of the following claims.

As described above, the recording medium on which the lighting environment reconstruction method and program according to the present invention is recorded has an advantage that the lighting information can be applied accurately to a moving object.

In addition, the present invention has the advantage that it is possible to automatically reconstruct the lighting environment by automatically estimating it without the need for pre-calibration to know the camera position and the internal parameters photographing the object space.

Claims (15)

In the method for reconstructing the lighting space for image synthesis, (a) calculating a plurality of photographing positions corresponding to the input image of the target space; (b) classifying and calculating illumination of the target space into global illumination or local illumination using the input image; And (c) calculating a three-dimensional illumination space including the calculated three-dimensional coordinates of the global light or the local light and the three-dimensional coordinates of the plurality of photographing positions. The method of claim 1, The input image is a multiple exposure image photographed by different shutter speeds of a camera equipped with a fisheye lens. (d) estimating a response function of the camera and generating a high dynamic range (HDR) image by using the multiple exposure image prior to step (a). The method of claim 1 Step (a) is (a1) the input image is a front and rear hemi-sphere images corresponding to the plurality of photographing positions, respectively, and constructing a hemisphere coordinate system having the plurality of photographing positions as origins; (a2) calculating a projected center point corresponding to the first and second center points of the hemisphere images acquired at the first and second photographing positions using a preset algorithm; And (a3) calculating a second photographing position relative to the first photographing position using the first and second center points and the projected center point; And calculating the plurality of photographing positions by repeating steps (a2) to (a3) corresponding to the number of the photographing positions. The method of claim 3 And the predetermined algorithm is at least one of a feature point extraction algorithm and a matching algorithm. The method of claim 3, wherein Step (a3), Equation

Using

Angle (∠AC ₁ B) and

Obtaining angles (∠ C ₁ BC ₂ ), respectively: C1 and C2 are first and second photographing positions, A and B are center points of the hemispherical images acquired at C1 and C2,

Is the focal length of the lens,

Is the distance between the center point of each hemisphere image and the center point where the relative center point is projected-; remind

Angle (∠AC ₁ B), above

Angle (∠ C ₁ BC ₂ ) and equation

Calculating a second photographing position C ₂ relative to the first photographing position C ₁ using

Is a line passing from the first photographing position C ₁ to the center point B,

Is a line segment passing through the center point A from the second photographing position C ₂ ,

And said

Is any number

And

Corresponding to each change corresponding to

And

And a direction vector of the illumination space reconstruction method. The method of claim 1, Prior to step (b) above, (e1) the input image is a front and rear hemi-sphere images corresponding to the plurality of photographing positions, and generating a Latitude-Longitude (L-L) unit image by panoramic converting the input image; (e2) calculating positions of a plurality of light sources of the L-L unit image by using structured importance sampling; (e3) constructing a hemisphere coordinate system each having the plurality of shooting positions as an origin; And and (e4) calculating unit direction vectors corresponding to positions of the plurality of illumination sources, respectively. The method of claim 6, In step (b), (f1) converting the first direction vector among the plurality of unit direction vectors calculated from the first L-L unit image corresponding to the first photographing position into a rectangular coordinate system; (f2) calculating a corresponding point in the second L-L unit image by using a preset algorithm in the second L-L unit image corresponding to the second photographing position and the converted direction vector; And (f3) repeating steps (f1) to (f2) corresponding to the number of unit direction vectors calculated in the first LL unit image to calculate corresponding points in the second LL unit image, respectively. , Repeating step (f1) and step (f3) corresponding to the number of LL unit images corresponding to the plurality of shooting positions, respectively, calculating one or more corresponding points in the LL unit images corresponding to the plurality of shooting positions, respectively. Illumination space reconstruction method, characterized in that. The method of claim 7, wherein Step (f2) is (g1) Equation

Using

To calculate the

Is the first shooting position vector,

Is the three-dimensional direction vector for any light starting at the first camera position C1,

Is the first direction vector is converted to a rectangular coordinate system,

Is any number-; (g2) above

And equations

Using

To calculate-

Is the second shooting position vector,

At the second shooting position

Vector towards end of; And (g3) above

Computing a corresponding point in the second LL unit image by using a. The method of claim 8, Step (g3) is (h1) above

To a spherical coordinate system

Calculating; (h2) above

Calculating a position corresponding to the second LL unit image; and (h3) calculating a position corresponding to a value having the smallest sum of squared differences (SSD) for brightness in a search area having a predetermined size based on the calculated position as a corresponding point. Spatial reconstruction method. The method of claim 7, wherein Step (b) is (i1) sampling a plurality of lights from a second L-L unit image corresponding to a second photographing position; (i2) dividing a plurality of layers of the second L-L unit image according to the number of the plurality of lights; And  (i3) after the step (f3), further comprising the step of calculating the illumination corresponding to the floor not including the corresponding point as the local lighting. The method of claim 6 Step (c) is (j1) Equation of a plurality of unit direction vectors corresponding to the first and second photographing position

Converting each into a plurality of three-dimensional unit direction vectors by using; And (j2) calculating coordinates of global illumination by applying the photographing position and the plurality of three-dimensional direction vectors to a preset algorithm; And repeating steps (j1) to (j2) to calculate coordinates of the global illumination, respectively, corresponding to the number of L-L unit images corresponding to the plurality of photographing positions. The method of claim 11 Step (j2) is (k1) Equation

Using

And

Calculating each of-

And

Are the three-dimensional unit direction vectors of the first and second photographing positions, respectively,

And

Are the first and second shooting position vectors,

And

Are the three-dimensional vectors for the i-th illumination starting from the first and second shooting position vectors, respectively, and t and s are respectively

And

Is a scaling factor of; And (k2) above

And said

Calculating an intersection of with coordinates of the global illumination, The 3D unit direction vector

And

And repeating steps (k1) to (k2) corresponding to the number of to calculate the coordinates of the global illumination, respectively. The method of claim 12, Step (k2) is remind

And said

Above if no intersection of

And said

Comprising the step of calculating the coordinates of the global illumination with the position of the distance between the most close to each other. The method of claim 6 Step (c) is (l1) Equation of a plurality of unit direction vectors corresponding to the first and second photographing position

Converting each into a plurality of three-dimensional unit direction vectors by using;  (k2) calculating intersections of the plurality of three-dimensional unit direction vectors as coordinates of area lights; (k3) repeating steps (k1) to (k2) corresponding to the number of the LL unit images corresponding to the plurality of photographing positions, and calculating the coordinates of the local lighting in the LL unit image, respectively. Illumination space reconstruction method comprising a. In the recording medium recording a program that can be read by the digital image processor, a program of instructions that can be executed by the digital image processor is tangibly implemented for reconstructing the lighting environment for image synthesis. (a) calculating a plurality of photographing positions corresponding to the input image of the target space; (b) classifying and calculating illumination of the target space into global illumination and / or local illumination using the input image; And and (c) calculating a three-dimensional illumination space including the calculated three-dimensional coordinates of the global or local illumination and three-dimensional coordinates of the plurality of photographing positions.

Images