KR20070073647A - Method and apparatus of generating three dimensional scene in virtual view point - Google Patents

Abstract

A method and an apparatus for generating a 3D image in a virtual viewpoint are provided to receive real images from a plurality of real image input devices, generate foreground projection images from the real images, project the foreground projection images to a virtual 3D space sought in a certain virtual viewpoint, produce a cross point between the projected images, embody a virtual hull, and simply and accurately embody a virtual 3D image in the certain virtual viewpoint by the virtual hull. A plurality of real image input devices(10) acquires real images in inter-different viewpoints. A foreground projection image generating device(20) receives a plurality of the real images from the real image input devices(10), and generates foreground projection images from the respective real images. A contour information extractor(30) extracts foreground contour information from the foreground projection images generated in the foreground projection image generating device(20). A virtual hull and virtual 3D image generator(50) generates a virtual hull and a 3D virtual image corresponding to a certain viewpoint by the foreground projection images and the contour information.

Description

Method and apparatus for generating three dimensional scene in virtual view

1 is an overall configuration diagram of a 3D image generating apparatus in a virtual view according to an embodiment of the present invention;

2 is an exemplary view illustrating an example of a configuration of a live action image input device;

3 is a flowchart schematically illustrating a 3D image generating method at a virtual viewpoint according to an exemplary embodiment of the present invention.

4 is a view showing an image of a state in which the foreground projected image is generated by separating the foreground and the background by the foreground projected image generating apparatus 20, and then the contour information is extracted.

5 to 10 are diagrams for describing a method of generating a virtual 3D image at an arbitrary time point by the virtual skin and the virtual 3D image generator 50 based on the virtual skin technique.

<Explanation of symbols for the main parts of the drawings>

10: live image input device

20: foreground projection image generating device

30: contour information extractor

40: 3D modeling information database

50: virtual envelope and virtual 3D image generator

60: display device

The present invention relates to a method and apparatus for generating a 3D image from a virtual viewpoint. More particularly, the present invention relates to a method for generating a 3D image from a virtual viewpoint, and more specifically, receives a real image from a plurality of real image input devices, and selects a random image from the real image. A method and apparatus for generating a 3D image at a virtual viewpoint of the present invention.

Currently, due diligence-based three-dimensional object modeling techniques can be divided into active and passive methods. A representative example of the active method is using a 3D scanner, and a representative case of the passive method is the geometry of a 2D image taken by a camera. Analyze the structure to construct a 3D model.

In the method using a 3D scanner, various patterns (lasers, etc.) designed artificially are projected onto an object to analyze a pattern changing according to a shape to extract a 3D geometric structure, and to obtain color information obtained through a CCD camera. The 3D model is constructed by using a 3D model. However, this method has an accurate measurement result and a relatively short measurement time, but there is a big disadvantage that it is expensive and difficult to apply to dynamic objects by using expensive separate measuring equipment. .

On the other hand, a method of constructing a 3D model by analyzing the geometry of a 2D image taken with a camera is relatively economical without constructing a 3D model without expensive equipment, compared to a method using a 3D scanner. As a result, the accuracy is low and the time required for constructing the 3D model is large.

On the other hand, among the techniques for representing the current three-dimensional object, there are techniques that show the various aspects of the object by taking a still cut from various angles with respect to the object. For example, a single camera is used to continuously photograph an object on a rotating table, or a plurality of cameras are placed around the object to simultaneously photograph and then create a scene through post-processing. However, these techniques have a limitation in that they cannot provide a dynamic scene of an object or a scene from a virtual viewpoint that is adaptive to a user's request.

The present invention has been made in view of the above-described problems, and receives a real image from a plurality of real image input devices, generates a foreground projection image from these real images, and then views them from an arbitrary virtual view. The present invention relates to a method and apparatus for realizing a virtual envelope by projecting in a dimensional space and calculating an intersection point between the projected images, and enabling a simple and accurate implementation of a virtual three-dimensional image at any virtual viewpoint by the virtual envelope.

In order to achieve the above object, according to an aspect of the present invention, in the method for generating a three-dimensional image at a virtual view by receiving a live image from a plurality of live image input device, from the plurality of live image input device Generating a foreground projection image of the received photorealistic image, extracting contour information of the foreground object from a constantly generated foreground projection image, and using the generated foreground projection image and contour information at an arbitrary virtual viewpoint A method of generating a 3D image at a virtual view may include generating a virtual envelope of the image and interpolating image information at the virtual view of the generated virtual envelope.

In a preferred embodiment, the step of generating a foreground projection image of the live image received from the synchronized plurality of live image input devices, characterized in that further comprising the step of distinguishing the foreground and background of the received live image do. The generating of the foreground projection image of the live image received from the plurality of synchronized photo image input devices may further include performing morphological filtering and convex hull filtering. It is characterized by. In addition, the step of distinguishing the foreground and background of the received live image, characterized in that for setting a predetermined threshold value using the brightness value and the color component value of the live image and distinguishing the foreground and the background by the set threshold value. It is done. In addition, the live images input from the live image input apparatus are synchronized in time. The generating of the virtual envelope at any virtual viewpoint using the generated foreground projected image and contour information may include projecting the plurality of foreground projected images on a three-dimensional space at an arbitrary virtual viewpoint. Calculating intersections of the projected images. The interpolation of the image information at the virtual viewpoints of the generated virtual envelope may include: when the distance between any one of a plurality of live images and the arbitrary virtual viewpoint is less than or equal to a predetermined threshold, It is characterized by using the information as it is. The interpolation of the image information at the virtual point of time with respect to the generated virtual envelope may include: when the pixel value of the real picture is not obtained, the foreground projected image of the live image input device at a point closest to the distance Weighting is proportional to the pixel values of the in proportion to the distance.

According to another aspect of the present invention, in the apparatus for generating a three-dimensional image from a virtual viewpoint by receiving the photo-realistic image, a plurality of photo-realistic image input device for acquiring the photorealistic image from different viewpoints, a plurality of from the photoreal image input A foreground projection image generating device for generating foreground projection images on each of the actual images, and a contour information extractor extracting contour information of the foreground object from the foreground projection image generated by the foreground projection image generating device And a virtual envelope and a virtual three-dimensional image generator for generating a virtual envelope and a three-dimensional virtual image corresponding to an arbitrary virtual viewpoint based on the foreground projected image and the contour information. can do.

In a preferred embodiment, a brightness value and a color component value are obtained for each of the plurality of live image images input from the live image input device, a difference between the brightness value and the color component value of the background image without an object is obtained, and a predetermined threshold value is obtained. Set to include a background / foreground separator for separating the background and foreground. The foreground projection image generating apparatus may further include an image filtering processor configured to perform at least one of morphology filtering to remove small image noise or convex filter filtering to fill small holes in the foreground projection image. It is done. The live image input apparatus may further include a synchronization unit configured to perform temporal synchronization with respect to each of the plurality of real images acquired by the plurality of live image input apparatuses. The method may further include a database storing at least one of a live image obtained by the live image input apparatus, a foreground projected image generated by the foreground projected image generating apparatus, and contour information generated by a contour information extractor. It features. In addition, the virtual envelope and the virtual three-dimensional image generator is characterized in that for calculating the intersections of the projected images by projecting the plurality of foreground projection images in a three-dimensional space at any virtual viewpoint. In addition, the virtual envelope and the virtual three-dimensional image generator is characterized in that for interpolating the image information from the virtual point of view for the generated virtual envelope. The virtual envelope and the virtual 3D image generator may use an image of the actual image as it is when the distance between any one of a plurality of photorealistic images and the arbitrary virtual view is less than or equal to a predetermined threshold. And interpolating the information. In addition, the virtual envelope and the virtual three-dimensional image generator is weighted to be proportional to the distance to the pixel values of the foreground projected image of the live image input apparatus at the closest distance when the pixel value of the real image is not obtained. And interpolating the image information at the virtual point of view with respect to the generated virtual envelope.

Next, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating a 3D image from a virtual viewpoint according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the 3D image generating apparatus at the virtual view of the present embodiment may include a real image input apparatus 10, a foreground projected image generating apparatus 20, an outline information extractor 30, and a 3D modeling information database ( 40, a virtual envelope and a virtual 3D image generator 50, and a display device 60.

The live image input apparatus 10 refers to a device for collecting a live image to be used as a reference image, for example, a plurality of cameras or camcorders. A plurality of image collection apparatuses, for example, a plurality of cameras, which constitute the live image input apparatus 10, photograph the real images at different viewpoints, and transmit them to the foreground projected image generating apparatus 20, which will be described later. Each of the plurality of cameras has a preset position and distance relationship and is synchronized in time and space. Such multiple cameras can be configured as shown in FIG.

Referring to FIG. 2, a plurality of cameras are composed of a total of six, each of which is connected to two hubs 11 and 12, three of which are connected to a synchronization unit 13. Photorealistic images collected from a plurality of cameras through the hubs 11 and 12 may be synchronized in time and space by the synchronization unit 13 and transmitted to the computer 14 for later image processing. Here, the computer 14 may perform a separate image processing and transmit an image to the foreground projected image generating apparatus 20 separately provided, or the computer itself may be the foreground projected image generating apparatus 20. It may be.

As described above, the foreground projecting image generating apparatus 20 generates a foreground projecting image necessary to obtain a reference image for generating a virtual envelope from each of the actual images transmitted from the actual image input apparatus 10, The generated foreground projection image is transmitted to the contour information extractor 30. The foreground projected image generating apparatus 20 generates brightness and color component values Cb and Cr for each of the actual images inputted from the real image input apparatus 10 to generate the foreground projected image. The background / foreground separator 21 extracting only the foreground object from the image by setting a predetermined threshold value after calculating the difference between the brightness value and the color component value of the pure background image without the object, and the background / foreground separator 21. The image filtering processor 22 performs a filtering process on the image obtained through. In the image filtering processor 22, image filtering processing includes morphological filtering to remove small image noise, and convex hull filtering to fill small holes in the foreground projected image. Can be performed.

The contour information extractor 30 extracts the contour information of the foreground object from the foreground projected image generated by the background / foreground separator 21 and the image filtering processor 22 using the obtained foreground and background boundary information. Then, the foreground projected image and the contour information transmitted from the foreground projected image generating apparatus 20 are stored in the 3D modeling information database 40 and transmitted to the virtual envelope and the virtual 3D image generator 50.

The 3D modeling information database 40 stores the result data generated by the real image input device 10, the foreground projected image generating device 20, and the contour information extractor 30, that is, the foreground projected image and the contour information. do.

The virtual envelope and the virtual 3D image generator 50 reconstruct a 3D virtual image requested at an arbitrary virtual view based on the virtual envelope technique using data stored in the 3D modeling information database 40.

The display device 60 displays the reconstructed three-dimensional scene to the user.

FIG. 3 is a flowchart schematically illustrating a 3D image generating method at a virtual view according to an exemplary embodiment of the present invention, which has the configuration as shown in FIG. 1.

Referring to FIG. 3, the foreground projection image generating apparatus 20 receives a live image from a plurality of synchronized image input apparatuses as shown in FIG. 2 (S301).

The background / foreground separator 21 of the foreground projected image generating apparatus 20 extracts only the foreground by separating the background and the foreground from the image by using the input brightness value and the color component value (S303). As a method of separating a background and a foreground from an image, the following method may be used. First, in a live image input from a plurality of live image input apparatus 10, by using a brightness value and a color component value of the image together, a predetermined threshold value to distinguish the background and the foreground is set, and the background and foreground are set by the set threshold value. To separate.

The equation for calculating the brightness value and color component value is as follows.

Brightness value = 0.29900R + 0.58700G + 0.11400B

Blue color value = -0.16874R-0.33126G + 0.50000B

Red color value = 0.50000R-0.41869G-0.08131B

In the existing RGB color space, each color component contains a brightness component, which is affected by the brightness change caused by lighting or shadows, and causes a lot of noise. Therefore, the brightness component and the color component are separated and used for the brightness change. The impact can be reduced.

After setting a predetermined threshold value using the brightness value and the color component value, the foreground object is classified using the following equation.

는 배경 참조 영상에서의 해당 픽셀의 성분값을 의미하고,

는 전경체가 존재하는 현재 입력된 영상에서의 해당 픽셀 의 성분값을 의미한다.

는 문턱값을 의미한다. 이와 같이 밝기 성분과 컬러성분값을 동시에 이용함으로써, 조명이나 그림자에 의한 미세한 변화에 의한 오차를 극복할 수 있으며, 그림자 등이 전경체로 구분되는 것을 효과적으로 제거할 수 있다. In the above formula

Means the component value of the corresponding pixel in the background reference image,

Denotes the component value of the corresponding pixel in the currently input image in which the foreground exists.

Means threshold. In this way, by using the brightness component and the color component value at the same time, it is possible to overcome the error due to the minute change caused by the lighting or shadow, it is possible to effectively remove the shadow and the like divided into the foreground.

Next, the image filtering processor 22 performs morphological filtering and convex hull filtering (S305). Here, morphology filtering is for removing small image noise, and convex pod filtering is for filling small holes inside the foreground projection image.

When the foreground projection image is generated by performing the filtering process in the image filtering processor 22, the foreground projection image generating apparatus 20 transmits the generated foreground projection image to the contour information extractor 30, and the contour information extractor. 30 extracts contour information of the foreground object from the foreground projected image using boundary information of the foreground and the background (S307). 4 shows an image of a state in which the foreground projecting image is generated by separating the foreground and the background by the foreground projecting image generating apparatus 20, and then the contour information is extracted.

After extracting the contour information, the contour information extractor 30 stores the foreground projection image and the contour information in the 3D modeling information database 40 (S309, S311), or the virtual envelope and the virtual three-dimensional image generator 50. send. Of course, the 3D modeling information database 40 and the virtual envelope and the virtual 3D image generator 50 may be simultaneously transmitted.

Next, the virtual envelope and the virtual three-dimensional image generator 50 reconstructs the virtual three-dimensional image at any point of time based on the virtual envelope technique using data stored in the 3D modeling information database 40 ( S313). A detailed method of reconstructing a virtual 3D image at an arbitrary time point based on the virtual envelope technique will be described below with reference to FIG. 5.

When a virtual 3D image is generated at an arbitrary time point by the virtual envelope and the virtual 3D image generator 50, the display apparatus 60 displays the reconstructed 3D scene to the user (S315).

5 to 10 are diagrams for describing a method of generating a virtual 3D image at an arbitrary time point by the virtual skin and the virtual 3D image generator 50 based on the virtual skin technique.

First, an algorithm for newly generating a virtual 3D image at an arbitrary virtual point of view may be summarized as pseudo code.

   for each pixel p in desired image do

distanceAlongRay = 0

while (isOutsideVisualHull)

distanceAlongRay ++

if (intersectionFound)

for each realCamera

if (notOccluded && close)

sampleColor ()

pixelColor = weightMean (sample)

else

pixel = background

Reconstructing a virtual three-dimensional image at an arbitrary point of time may be performed by separating foreground and background from a live image obtained from a plurality of cameras, performing filtering and the like, and then extracting contour information from a plurality of foreground projection images (see Image) is projected on a three-dimensional space at an arbitrary point of time, that is, the point at which a virtual three-dimensional image is to be generated, to obtain intersection points in three-dimensional space of reference images, and a virtual envelope is formed by these intersections. Image mapping such as color seen at an arbitrary time is performed by mapping textures by interpolation or the like.

5 and 6 are diagrams for explaining the relationship between the foreground projection image and the virtual envelope.

Referring to FIG. 5, when a live image is obtained from four cameras, a concept of implementing a virtual jacket is shown. The definition of the virtual envelope is as follows.

은 각각의 카메라에서 입력된 실사영상 각각을 의미하며,

은 다수의 카메라에서 입력된 실사영상의 집합을 의미한다. 그리고

은 각각의 카메라 중심점에서 전경체 사영영상의 중심점을 향해 가상의 빛을 쏘았을 때 생성되는 원뿔모형의 3차원 형상 각각을 의미한다. 즉 가상외피(Visual Hull)는 원뿔모양의 3차원 형상들의 교집합이라 정의할 수 있다.In the above formula

Means each of the live images input from each camera,

Refers to a set of live-action images input from a plurality of cameras. And

Denotes each of the three-dimensional shapes of the cone model generated when a virtual light is shot from the center point of the camera toward the center point of the foreground projected image. In other words, the virtual hull can be defined as the intersection of three-dimensional cone-shaped shapes.

Referring to FIG. 6, when real image images are acquired by two cameras, the points of intersection of projection images are illustrated when they are projected in three-dimensional space, and these intersections generate a virtual envelope in a virtual three-dimensional space. You can see that.

FIG. 7 and FIG. 8 are diagrams for describing generating a virtual envelope at an arbitrary virtual viewpoint and generating a virtual 3D image based on the virtual envelope.

Referring to FIG. 7, when a real image is obtained from two cameras and the images are projected onto a three-dimensional space, a virtual skin is generated by an intersection point between an arbitrary virtual viewpoint and these projection images, and a virtual three-dimensional image at the virtual viewpoint. Create an image.

Referring to FIG. 8, as shown in FIG. 7, a virtual envelope generated by an intersection point between an image projected from two cameras and an arbitrary virtual viewpoint and a virtual envelope image viewed from the virtual viewpoint, and having a three-dimensional object with a sense of volume The concept of constructing Solid Geometry (SCG), which can easily express modeling, is introduced, and Boolean subtraction is performed to form a virtual image.

9 and 10 illustrate actual implementation screens of a virtual 3D image generated at an arbitrary virtual viewpoint based on the virtual envelope technique as described above, and FIG. 9 is a photo-realistic image obtained from three cameras of different views. Are shown at different time points, i.e., at t = 1, t = 2, t = 3, and t = 4, and FIG. 10 generates virtual skins by the live image shown in FIG. It shows a virtual three-dimensional image screen viewed from another arbitrary virtual viewpoint of.

As described above, when a virtual envelope is generated from an arbitrary viewpoint, information necessary for realizing other images such as color information and brightness information at an arbitrary viewpoint may be implemented by interpolating from the obtained live image through a process such as texture mapping. Will be. In this case, when an arbitrary viewpoint is close to the viewpoint of the actual image acquired, the information of the actual image may be used as it is. On the other hand, if the pixel value of the live image cannot be obtained due to the obstruction of another object, the pixel value of the foreground projected image of the live image input apparatus of the closest distance can be replaced. The weight may be proportional to the distance.

The present invention is not limited to the above embodiments, and many variations are possible by those skilled in the art within the spirit of the present invention.

According to the present invention, after receiving a live image from a plurality of live image input apparatus to generate a foreground projection image from these live image, and project them in a virtual three-dimensional space viewed from any virtual viewpoint and between the projected image It is possible to provide a method and apparatus for calculating a crossing point to implement a virtual envelope, and by using the virtual envelope, a simple and accurate implementation of a virtual 3D image at any virtual viewpoint.

According to the present invention, a plurality of viewpoints are generated by generating a plurality of foreground projection images of a dynamic non-rigid object, a virtual envelope from the generated foreground projection images, and an arbitrary virtual viewpoint scene using an image interpolation technique. The 3D image in Esau can be quickly implemented by a video input device such as a limited number of cameras.

In addition, according to the present invention, since an image-based photorealistic image is used as a reference image, expensive equipment such as a 3D laser scanner is not required, and since a sensor such as a marker is not used, more natural and dynamic movement of the object is induced. In addition, by using a synchronized camera, the image quality can be improved and the system can be economically inexpensive.

Claims (17)

In the method for receiving a live image from a plurality of live image input apparatus for generating a three-dimensional image from a virtual view, Generating a foreground projection image of the live image received from the plurality of live image input devices; Extracting contour information of the foreground from a constantly generated foreground projection image; Generating a virtual envelope at any virtual viewpoint using the generated foreground projected image and contour information; And Interpolating image information at the virtual view point of the generated virtual envelope 3D image generation method in a virtual view including a. The method of claim 1, The generating of the foreground projection image of the live image received from the plurality of synchronized image input apparatuses may include distinguishing the foreground and the background of the received live image from the virtual viewpoint. 3D image generation method. The method of claim 1, Generating the foreground projection image of the live image received from the plurality of synchronized image input apparatuses further includes performing morphological filtering and convex hull filtering. 3D image generating method in a virtual view characterized in that. The method of claim 2, The step of distinguishing the foreground and the background of the received live image, characterized in that for setting a predetermined threshold value using the brightness value and the color component value of the live image and distinguishing the foreground and the background by the set threshold value 3D image generation method in a virtual view. The method of claim 1, 3. The method of claim 3, wherein the live images inputted from the live image input device are synchronized in time. The method of claim 1, Generating a virtual envelope at any virtual viewpoint using the generated foreground projection image and the contour information, And projecting the plurality of foreground projected images on a three-dimensional space at an arbitrary virtual viewpoint to calculate intersections of the projected images. The method of claim 1, Interpolating the image information at the virtual virtual view with respect to the generated virtual envelope, 3. The method of claim 3, wherein when the distance between any one of the plurality of live images and the arbitrary virtual view is less than or equal to a predetermined threshold, the information of the live image is used as it is. The method of claim 1, The interpolation of the image information at the virtual view of the generated virtual envelope may include: when the pixel value of the live image cannot be obtained, the foreground projected image of the live image input device at a point closest to the distance 3. The method of claim 3, wherein the pixel values are weighted to be proportional to the distances. In the apparatus for receiving a live image and generating a three-dimensional image from a virtual view, A plurality of live image input apparatuses for obtaining a live image at different viewpoints; A foreground projecting image generating device which receives a plurality of live image inputs from the live image input apparatus and generates a foreground projecting image on each of the actual image; A contour information extractor for extracting contour information of the foreground from the foreground projected image generated by the foreground projected image generating device; And A virtual envelope and a virtual three-dimensional image generator for generating a virtual envelope and a three-dimensional virtual image corresponding to an arbitrary virtual viewpoint based on the foreground projected image and contour information 3D image generating apparatus in a virtual view including a. The method of claim 9, The foreground projection image generating device, A brightness value and a color component value are obtained for each of the plurality of live image images input from the live image input device, a difference between the brightness value and the color component value of the background image without an object is obtained, and a predetermined threshold is set to set the background and Background / foreground separator to separate foreground 3D image generating apparatus in a virtual view, characterized in that it comprises a. The method of claim 9, The foreground projection image generating device, An image filtering processor that performs at least one of morphological filtering to remove small image noise or convex pod filtering to fill small holes in the foreground projected image. 3D image generating apparatus in a virtual view, characterized in that it comprises a. The method of claim 9, The live image input device, A synchronization unit that performs temporal synchronization with respect to each of the plurality of live image images acquired by the plurality of live image image input apparatuses 3D image generating apparatus in a virtual view further comprising. The method of claim 9, A database storing at least one of a live image obtained by the live image input apparatus, a foreground projected image generated by the foreground projected image generating apparatus, and contour information generated by a contour information extractor 3D image generating apparatus in a virtual view further comprising. The method of claim 9, The virtual envelope and the virtual three-dimensional image generator 3 calculates intersections of the projected images by projecting the plurality of foreground projected images on a three-dimensional space at an arbitrary virtual viewpoint. Dimensional image generating device. The method of claim 9, And the virtual envelope and the virtual three-dimensional image generator interpolate the image information of the virtual viewpoint with respect to the generated virtual envelope. The method of claim 15, The virtual envelope and the virtual three-dimensional image generator, when the distance between any one of the plurality of live images and the arbitrary virtual view is less than a predetermined threshold value by using the information of the live image as it is image information at the arbitrary virtual view 3D image generating apparatus in a virtual view, characterized in that to interpolate. The method of claim 15, When the virtual envelope and the virtual 3D image generator cannot obtain the pixel value of the live image, the virtual weight and the virtual 3D image generator are weighted proportionally to the distance of the pixel values of the foreground projected image of the live image input apparatus at the closest distance. And 3D image generating apparatus at the virtual viewpoint, wherein the image information of the generated virtual envelope is interpolated.

Images