KR100259837B1 - Method for displaying three dimensional image from two dimensional image - Google Patents

Abstract

PURPOSE: A three dimensional image display method out of two dimensional image is provided to get a three dimensional image for an object having shading out of one image by displaying left eye image and right eye image in turns. CONSTITUTION: A convex curved surface area that exists in two dimensional input image is detected to presume a direction of illumination light(100). A convex shape is approximated to agree very well with an image intensity of these areas for the convex areas that is detected above(200). A direction of illumination light is decided through the process of shape approximation(300). A depth is calculated out of two dimensional input image having the direction of light source and shading(400). An observer's left eye image and right eye image is reorganized from two dimensional input image by using depth calculated above(500). The left eye and right eye image reorganized above is displayed in turns at the observer's left eye and right eye(600).

Description

3D image display method from 2D image

The present invention relates to an image processing method for generating a three-dimensional image from a two-dimensional image, in particular, to estimate the direction of the illumination light from the shading (shading) information, and to calculate the two images corresponding to the left and right eyes from the calculated depth It's about how to create it.

Conventional techniques for obtaining information about the direction of a light source from an image include a method of approximating the shape of an object at all points in the image with a sphere.

In the above methods, the direction of the light source was calculated from the distribution of the image derivatives. Estimate.

However, in this conventional method, since the approximation to the spherical shape is assumed in all regions in the image, it is somewhat applicable to an image composed only of a smooth curved surface such as a spherical surface, but there is a discontinuous shape such as an angled part. In this case, the accuracy is significantly lower, and even if all the regions are smoothly curved, an error occurs when the distribution of surface vertical vectors is not uniform in all directions, such as in a cylindrical or rugby ball shape.

In addition, since the conventional method uses the derivative value of the image data, its accuracy is significantly reduced due to the noise component included in the image data.

In addition, in the conventional method for obtaining a 3D image from a 2D image, two images obtained from different viewing angles are used. In this case, an image input device having two different viewing directions must be operated simultaneously to obtain an input image. To this end, the distance between the two input devices must be kept constant at all times, and there is an inconvenience in that the direction and focal length of the input device must be properly adjusted according to the distance where the object is located.

The present invention was devised to solve the above problems, and in order to estimate the direction of the illumination light from the image, the regions of convex surfaces distributed in the image are found, and a bump shape approximation is applied to these regions. By using the method to obtain a more accurate direction of the illumination light, and using the light source direction obtained by the method, to obtain the depth of the object area from the image having the shading information, and to reconstruct the left and right eyes image alternately left and right eyes The object of the present invention is to provide a method of obtaining a three-dimensional image of an object having shading from one image.

1 is a view showing the relationship between the surface vertical direction, the illumination light direction and the observation direction,

2 is a 7-by-7 filter for selecting convex planes, showing an x-differential filter, a y-differential filter, and a Gaussian filter,

3 is a diagram showing an example of a convex region selected from an input image;

4 is a view showing a relationship between a light source direction vector and an inclination angle and an inclination angle,

5 is a view showing an image calculated from an input image of a bump shape and a calculated shape coefficient and a light source direction image;

6 is a view showing an image calculated from a spherical input image and a calculated shape coefficient and a light source direction image;

7 is a diagram showing an example of an integrated region and a corresponding point in an x-y space and a p-q space;

8 is a diagram showing a characteristic curve and an initial curve of integration;

9 is a view showing a concept of coordinate transformation for reconstructing the right eye image,

10 illustrates an input image and a reconstructed left and right eye image,

11 is a process diagram for generating a three-dimensional image from a two-dimensional image according to the present invention.

According to the present invention devised to achieve the above object, the three-dimensional image display method from the two-dimensional image, the first process of detecting the convex curved areas existing in the two-dimensional input image to estimate the direction of the illumination light 100 and; A second process (200) of approximating a convex shape for the convex regions detected in the first process (100) to best match the image intensity of these regions; A third process (300) of determining a direction of illumination light through a shape approximation process of the second process (200); A fourth step 400 of calculating a depth from a two-dimensional input image having a light source direction and shading estimated in the third step 300; A fifth process (500) of reconstructing the left eye image and the right eye image of the observer from the two-dimensional input image using the depth calculated in the fourth process (400); And a sixth step 600 of alternately displaying the left and right eye images reconstructed in the fifth step 500 of the observer's left and right eyes.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

In general, an image generated by a reflecting object is formed by reflecting light from an illumination light on an object surface and being observed by an observer. Even if the reflectance of the object surface is the same, the amount of light reflected by an angle formed by the illumination light and the object surface is the same. This will change the shading.

Since the shading information is closely related to the shape of the object, it is very useful as information for restoring the shape of the object.

However, in the methods of obtaining the shape information from the shading information of the object, information on the direction of the illumination light is essential, but since the information on the direction of the light source is not separately given to the image, estimating the direction of the light source from the image is a shape. It is an essential process for restoration.

In the present invention, since information on the direction of the illumination light is to be obtained by using the convex portion in the image, a process of selecting a region having the convex shape in the image is necessary.

To this end, the present invention uses a filter suitable for the detection of convex shapes, and uses two differential filters and one gaotian filter.

This filter is created when the light source illuminates a certain convex shape having a Lemberian characteristic in the (1,0,1), (0,1,1) and (0,0,1) directions of the image coordinates. Use the same type of image that is being used.

More specifically,

If there is a convex shape described by the following equation, the filter for the (1,0,1) direction, the (0,1,1) direction and the (0,0,1) direction is F101 (x, y), respectively. , F011 (x, y), F001 (x, y), they are obtained as follows.

The filter values are the same as the result of scalar multiplying the surface vertical direction and the light source direction at each point, where the convex faces are illuminated in three different directions, as described above.

On the other hand, when considering a filter having a specific size S, as shown in the flat plane, the average value of the filter element values is subtracted from each filter element value so that the intensity of the image is zeroed to a filtered value in a constant region. .

That is, the above filters have the same shape as the image of the convex shape, thereby exhibiting the optimum filter performance in selecting the convex region.

Variables a and b describing the filter are related to the size of the convex surface in the image. If the convex surface is large, a small value is used. If the convex surface is small, a large value is used, but in general, a = 1.0 And b = 1.0.

As the size of the filter, a filter having several sizes, such as 5x5, 7x7, or 9x9, can be selected and used, and FIG. 2 shows a filter having a 7x7 size.

In the figure of FIG. 2, the values of the filter elements have a medium brightness of zero, a part brighter than the medium brightness positive, a part darker than the medium brightness negative and the overall average zero.

A method of selecting a convex region using the three filters is as follows.

Filters F ' ₁₀₁ , F'011 and F' ₁ are independently applied to the input image h _{(x, y)} , respectively, for three different filtered values (F '(h _{(x, y)} )). The average value of the filtered values (F '(h _{(x, y) avg} )), the minimum value (F' (h _{(x, y) min} )), and the maximum value (F '(h _{(x, y) max} )) Obtain each.

Use these values to delete the h _{(x, y)} region that simultaneously meets the following three conditions for the three filtered values.

At this time, A is a variable that controls the size of the deletion area. As A increases, a larger area is deleted.

In the present invention, 0.3 to 0.7 is usually used.

The region having a convex shape may be selected through the above deletion process.

An example of this is shown in FIG. 3.

On the other hand, the calculation method of the light source direction through the shape approximation to the selected convex shape region is as follows.

In order to approximate the convex shape using the quadratic equations for the image coordinates x and y, the height of the shape plane at position (x, y) is Z (x, y), which usually includes the first and second terms. It can be expressed by the following equation.

Thus, the slope p with respect to the x-axis and the slope q with respect to the y-axis of the shape plane at points x and y are

It is expressed as

In other words,

On the other hand, the surface vertical vector at that point can be calculated from the inclination p, q at each point.

라 하면 다음 식과 같이 표현된다.Face down vector

Is expressed as the following equation.

라 하고, 고려하는 형상면이 렘버시안이라 하면, 이미지 인텐시티 h_(x,y)는

으로 표현된다.At this time, the vector representing the direction of the light source

If the shape plane under consideration is Lemversian, the image intensity h _{(x, y)} is

It is expressed as

Thus, we can write

Using the above relation, we can rewrite it in p-q space as follows.

Observing the above equation, it can be seen that the image intensity at a specific position (x, y) is determined by the inclination (p, q) and the light source direction with respect to the x and y axes of the shape at that point.

를 계산할 수 있다.Since the image intensity is already given a value from the input image, once we know the slope (p, q) determined by the shape at that point, the light source direction vector

Can be calculated.

를 계산하기 위해, 도 4에 나타낸 것과 같은 p-q 공간에서의 반경 R인 원 내부에서의 상기 식의 양변에 대한 적분을 고려해보자.Light source direction vector

To calculate, consider the integration of both sides of the equation inside a circle with radius R in the pq space as shown in FIG. 4.

The light source direction component t ₃ can be calculated from the integral below.

Of the three terms on the right side of the equation, the two terms including p and q have integral functions, so the integral value inside the circle with a certain radius becomes zero.

Thus, only one term for t3 remains.

Therefore, the third component t ₃ in the light source direction is calculated as follows.

Further, the light source direction components t ₁ and t ₂ can be calculated by multiplying the integral function by p and q respectively and integrating them.

That is, when t _{1 is} to be calculated, the left and right sides are multiplied by p and then integrated.

Therefore, t ₁ is calculated as follows.

t ₂ can also be calculated by multiplying the left and right sides by q and then integrating.

Therefore, the light source direction vectors t ₁ , t ₂ , t ₃ are expressed as follows.

과

은 반경 R을 갖는 원에 대한 적분으로, 다음과 같이 반경 R에 대한 함수로 계산된다.At this time,

and

Is the integral for the circle with radius R, which is calculated as a function of radius R as

와

및

의 계산 과정에서는 점 (p,q)에서의 h_(x,y)값을 알아야 한다.Meanwhile,

Wow

And

In the calculation of, we need to know the value of h _{(x, y)} at point (p, q).

Therefore, the point (x, y) corresponding to the point (p, q) should be obtained by using Equation (8).

If p ₀ = 0 and q ₀ = 0, the corresponding point (x, y) for the point (p, q) is calculated as follows.

Therefore, the integral value is calculated using h _{(x, y)} at the point (x, y) obtained in the above equation.

However, in the real image, since the point (x, y) is quantized to an integer value, if the point (x, y) calculated in the above equation is not an integer value, the interpolation method in two dimensions is used to h _{(x, y). y} ) is calculated.

The light source direction (t ₁ , t ₂ , t ₃ ) obtained by integration in a region of radius R in pq space, selected by the method, is determined by the constants m ₁₁ , m ₁₂ and m ₂₂ describing the shape. Will be.

Therefore, in order to obtain an optimal shape constant, the image intensity calculated by using the calculated light source direction and the surface vertical vector obtained from the shape is made through an optimization process that best matches the input image intensity.

If the input image intensity of the selected area S in the image is I _{(x, y)} and the image intensity calculated by the shape coefficient and the light source direction is h _{(x, y)} , the error at each point in the area is Sum of values

We can obtain the direction of the light source by finding the shape variable that minimizes.

Next, the calculated light source directions are calculated from images obtained at different light source directions for some bumps and spheres, and the results are shown in Tables 1 and 2.

The direction of the light source was indicated by tilt and slant, respectively, and their definitions are shown in FIG. 5.

Examples of images obtained using the shape coefficient and the light source direction calculation are shown in FIGS. 6 and 7 together with the input image.

The obtained light source direction is represented by a tilt angle and a slant, respectively, and their definitions are shown in FIG. 7.

The light source direction obtained by the method is used to obtain the shape information at each position of the object area in the image together with the shading information of the input image.

The shape information can be obtained by calculating the plane vertical vector or depth at each position of the object region.

라 하고, 물체 영역의 특정 위치에서의 면수직 벡터를

이라 하고, 이때의 물체 표면이 한 방향으로 입사한 빛이 모든 방향으로 균등하게 산란되는 렘버시안이라면, 물체영역의 각 위치에서의 이미지 인텐시티 R은 다음 식으로 표현된다.The obtained light source direction vector

Let's say that the surface vertical vector at a specific position

In this case, if the object surface at this time is a Lemversian in which light incident in one direction is uniformly scattered in all directions, the image intensity R at each position of the object region is expressed by the following equation.

와 이미지 좌표(x,y) 및 입력 이미지 인텐시티(h) 사이에는 다음과 같은 방정식에 의해 기술된다.At this time, the plane vertical vector is calculated starting from a specific point in the image.

And between image coordinates (x, y) and input image intensity (h) are described by the following equation.

In the above formula, R _n1 , R _n2 and h _x , h _y and the like represent differentiation for n ₁ , n ₂ and x, y, respectively.

Further, the variable t is a variable representing the length on the characteristic curve at the image coordinates irrelevant to the light source direction vector (see Fig. 8).

은 형상의 표면 (x,y,z) 상의 적분초기 특성곡선에 의한 미분 벡터와는 항상 수직인 관계가 있다.Vector vertical

There is always a vertical relationship with the derivative vector by the integrating initial characteristic curve on the silver-shaped surface (x, y, z).

This is because the integral initial curve, s, is always defined on the object surface.

의 조건이 적분 초기 곡선상의 모든 점에서 만족되어야 한다.therefore,

The condition of must be satisfied at all points on the initial curve of integration.

By using this condition, the accuracy of the plane vertical vector can be improved, so that the value of depth Z is also improved.

The calculation of the plane vertical vector by this added condition is possible using the following formula.

The subscript s, on the other hand, means the derivative of the length variable s on the initial curve of integration.

Through the above calculation process, when the surface vertical vector information is known at a specific position in the image, it is possible to calculate the surface vertical vector and the depth for the other peripheral region.

On the other hand, if there is no initial information on the plane vertical vector at a specific position in the image, the above method is difficult to apply, and thus an additional method for calculating the depth is needed.

If the obtained tilt angle tilt in the light source direction is τ and the slant angle is σ, the image intensity h in the tilt space p-q of the shape is expressed by the following equation.

In this case, p _s and q _s are values for the direction of the light source, and p _s = (cosτsinσ) / cosσ and q _s = (sinτsinσ) / cosσ.

On the other hand, p and q defined as derivatives in the image coordinate space for depth Z can be written as

The equation for the image intensity can be expressed as follows.

Using a linear approximation and iterative calculation with the Taylor series expansion of the function (f) for a given depth (Z ^n-1 ) at a given value, at a given image intensity (h) and fixed point (x, y), Will be available.

In other words,

Depth Z in the n-th calculation is expressed as follows.

At this time,

Therefore, when the integral initial value for the surface vertical vector in the image region is not known, information on the object shape can be obtained by obtaining the depth Z through the above iterative calculation method.

A method for restoring depth, which is shape information, from an image having shading information described above, and restoring an image in a different viewing direction from the restored depth.

First, the calculated depth is called Z (x, y), and it is always assumed that the input of the image is made in the plane perpendicular to the viewing direction.

First, if the surface of an object with depth Z (x, y) is the intensity of the input image in the direction perpendicular to xy, h (x, y), if the surface is Lemversian, even if the angle of observation changes, Intensity remains the same.

Therefore, there is no change in intensity at each point due to the change in the observation direction.

In FIG. 9, only one variation of the depth Z of the surface appearing on the x-y plane is shown along the x-axis.

As shown in FIG. 9, the image input direction and the new viewing direction are different by θ, which is similar to the case of observation through the right eye.

Considering the position of the point x ₀ , when the observation direction is inclined by θ, the vertical plane to the new viewing direction is inclined by θ, and the position x ₀ 'in the new plane x'-y' becomes x ₀ cosθ. .

On the other hand, the degree of movement at this time is influenced not only by the magnitude of the observation angle but also by the depth Z at that point.

That is, when the depth is Z, the effect Δ1 _z ′ is Zsinθ.

When the observation angle θ defined in the above figure has a positive sign, the effect of depth gives a shift in the -x axis direction, and as a result, the following relationship is established.

At this time, there is no movement of the points in the y direction.

This is because the change in the observation direction is made with the y-axis as the rotation axis.

On the other hand, considering the case where the observation direction is observed through the left eye (theta has a negative sign), the sign is similar to the above case, but only the influence by the depth is changed.

Therefore, the following relationship holds.

On the other hand, when the x-coordinate is converted by the relational expression above, a case in which the same x ₀ 'value occurs by a combination of variables x ₀ , θ, and Z occurs.

That is, a case where a plurality of x-coordinate values have the same x'-coordinate value occurs.

The above figure shows an example where the points of x ₀ and x ₁ coincide with one point of x ₀ '.

In this case, it should be determined whether the intensity of x ₀ or x ₁ should be finally assigned to the x ₀ 'position.

In the case of the right eye observation above, the case where the value of the x-coordinate is often large is final, so the arrangement order can be advanced from the lower side of the x-value to the higher. You must do it.

Another problem is that when all the coordinate values are quantized when dealing with the above problem in a real image, no placement may occur at a particular point on the x'-coordinate.

In this case, there arises a problem of how to specify the intensity value at the point where no arrangement occurs.

After performing the placement of the whole, first, select the unplaced points and take the left and right average values using the left and right values, or if the arrangement is not performed continuously at several points, use the interpolation method. You need to allocate the calculated intensity.

FIG. 10 shows a picture obtained by reconstructing a left eye observation image and a right eye observation image from a depth reconstructed using a Mozart image.

At this time, the observation distance is 30 cm and the distance between both eyes is 7 cm, and the observation angles of both eyes at this time are 6.7 °, respectively.

Thus, the small viewing angle results in a very small difference between the original image and the image reconstructed at the new viewing angle.

On the other hand, a method of displaying a three-dimensional stereoscopic image using the left eye image and the right eye image of FIG. 8 is possible by installing a shutter in front of the left eye and the right eye so that the left eye image and the right eye image alternately enter the left eye and the right eye. .

As described in detail above, in order to estimate the direction of the illumination light from the image, the present invention finds regions of convex surfaces distributed in the image, and calculates the illumination light direction through the bump-shaped approximation for these regions. Therefore, a more accurate direction of the illumination light can be obtained compared to the method based on the assumption that it can be approximated to the shape of a sphere at all points of the entire image area.

In particular, since only three variables used for optimization to calculate the light source direction are used as variables describing the bump shape, a simpler optimization process can be used.

Depth of the object area from the image with shading information using the light source direction obtained in the method, and by reconstructing the left eye and right eye images and displaying the left and right eyes alternately, the three-dimensional image of the object with shading from one image It allows you to get a video.

This method enables the display of three-dimensional images with a much simpler image input device than when using two or more images.

Claims (6)

In the method of displaying a three-dimensional image by estimating the direction of the illumination light from the shading region in the two-dimensional input image and calculating the depth of the object having shading by using the estimated light source direction and the information of the shading region, the direction of the illumination light is estimated. A first process (100) of detecting convex curved areas existing in the two-dimensional input image to produce; A second process (200) of approximating a convex shape for the convex regions detected in the first process (100) to best match the image intensity of these regions; A third process (300) of determining a direction of illumination light through a shape approximation process of the second process (200); A fourth step 400 of calculating a depth from a two-dimensional input image having a light source direction and shading estimated in the third step 300; A fifth process (500) of reconstructing the left eye image and the right eye image of the observer from the two-dimensional input image using the depth calculated in the fourth process (400); And a sixth step (600) of alternately displaying the left and right eye images reconstructed in the fifth step (500) in the observer's left and right eyes. The method of claim 1, wherein the first process 100 includes a (1,0,1) direction, a (0,1,1) direction, and ( A first step of obtaining the conversion equations F ₁₀₁ (x, y), F ₁₁ (x, y) and F ₁ (x, y) for the 0,0,1) direction as shown in the following equation;

A second step of normalizing the sum of the conversion equations to zero; The value (F converted by said conversion equation _{'([h (x, y} )] avg [h (x, y)]) mean value F) for a ", the minimum value (F' [h _{(x, y)] min} ) And a third step of obtaining the maximum value F '[h _{(x, y)} ] _max ); The average value (F '[h _{(x, y)} ] _avg ) obtained above, the minimum value (F' [h _{(x, y)} ] _min ), and the maximum value (F '[h _{(x, y)} ] _max ) are used. 3 conditions under which A value is 0.3 ~ 0.7

Detecting a region of image intensity h _{(x, y)} that satisfies simultaneously; And a fifth step of selecting the remaining area from which the image intensity h _{(x, y)} area is deleted from the two-dimensional input image as a convex shape area. The method of claim 1, wherein the calculation process of the light source direction vector (t ₁ , t ₂ , t ₃ ) in the detected convex region comprises: the slope space p, q of the depth through approximation to the shape.

In the pq space described by, the method is obtained by integrating a circle with a radius R, which is expressed as

At this time,

And φ (R) are

Comprising a step of calculating the light source direction vector calculated by the three-dimensional image display method from a two-dimensional image. The method of claim 1, wherein the viewer's left eye image and right eye image are reconstructed to display a three-dimensional stereoscopic image using a depth calculated from the input image having the light source direction and shading. x, y) and the depth is Z (x, y), the corresponding reconstruction of the left eye image assumes that the reconstructed image intensity is h '(x', y '). using the input image intensity h (x, y) for y ') = (x cosθ + Z sin θ, y); Reconstruction of the right eye image comprises using the input image intensity h (x, y) at the reconstructed image coordinates (x ', y') = (x cosθ + Z sinθ, y) Image display method. The method of claim 4, wherein the method of reconstructing the left eye image and the right eye image of the observer comprises: when a plurality of input image intensities correspond to the same coordinates of the image to be reconstructed, the value of the x-coordinate in the case of the left eye image is increased. Using the input image intensity of the smallest case as the intensity of the final reconstructed image; A method of displaying a three-dimensional image from a two-dimensional image, comprising using the input image intensity when the value of the x-coordinate is the largest in the right eye image as the intensity of the final reconstructed image. The method of claim 4, wherein the method of reconstructing the left eye image and the right eye image of the observer comprises: reconstructing the image to calculate the intensity at the specific coordinates when none of the input image coordinates correspond to the specific coordinates of the image to be reconstructed. A method of displaying a three-dimensional image from a two-dimensional image, comprising the step of using a two-dimensional interpolation method using the intensity of peripheral coordinates.

Images