KR20060021566A - Reconstruction 3d scene of virtual viewpoint using foreground projection-image and virtual-shap - Google Patents

Abstract

The present invention proceeds without capturing a marker on a non-rigid object such as a human body input through a plurality of cameras, and through a video interpolation algorithm, a scene corresponding to a virtual camera viewpoint that does not actually exist is 3 A method for reconstructing into a dimensional model and a computer-readable recording medium having recorded thereon a program for realizing the method.

Complex reality, foreground image, virtual envelope

Description

Reconstruction 3D Scene of Virtual Viewpoint using Foreground Projection-Image and Virtual-Shap}             

1 is a system-wide configuration diagram of a method and system for reconstructing a virtual viewpoint 3D scene based on a foreground image and a virtual envelope according to the present invention.

FIG. 2 is a diagram illustrating an embodiment of obtaining a foreground projection image by dividing a background and a foreground from an actual image input from a camera in a virtual viewpoint 3D scene reconstruction system based on a foreground projection image and a virtual envelope according to the present invention.

3 and 4 is an embodiment conceptual diagram illustrating the relationship between the foreground projection image and the virtual envelope in a virtual viewpoint three-dimensional scene reconstruction system based on the foreground projection image and the virtual envelope according to the present invention.

5 and 6 is an embodiment conceptual diagram for obtaining a virtual envelope in a virtual viewpoint three-dimensional scene reconstruction system based on the foreground projection image and the virtual envelope according to the present invention.

7 is a flowchart illustrating an embodiment of a virtual viewpoint 3D scene reconstruction system and method thereof based on a foreground projection image and a virtual envelope according to the present invention.

8 and 9 are diagrams illustrating exemplary embodiments of an image received from a camera and a resultant image generated through a whole process in a virtual viewpoint 3D scene reconstruction system based on a foreground projected image and a virtual envelope according to the present invention.

The present invention obtains the motion information of the foreground object and the projected image of the foreground object from the actual image inputted from a plurality of cameras whose position and distance relationship are adjusted, and applies an interpolation algorithm to the projected image to fit the required virtual observation point. The present invention relates to a method for reconstructing and providing various scenes and a system for realizing the method.

Currently, due diligence-based object modeling techniques can be divided into active and passive methods. A representative case of the active method is a 3D scanner, and a representative case of the passive method is to analyze the geometry of a 2D image taken by a camera. To construct a three-dimensional model.

Projecting various patterns (lasers, etc.) artificially designed on the object using a 3D scanner to analyze patterns changing according to the shape, extracting the 3D geometric structure and using the color information acquired through the CCD camera The method of constructing the model is accurate, and the measurement time is relatively short, but there is a big disadvantage that it is difficult to apply to dynamic objects because the cost is increased by using expensive separate measuring equipment.

Although image-based methods can construct a three-dimensional model economically without expensive equipment compared to the method using a scanner, it is relatively inaccurate and requires a long time to construct a three-dimensional model, and the number of cameras (or references) If the number of images is small, the quality of the reconstructed image is greatly degraded.

In addition, among the techniques for representing the current three-dimensional object, there are techniques that show the various aspects of the object by taking a still cut at various angles to the object. It is a technique of continuously photographing an object on a table rotated by one camera or by simultaneously placing a plurality of cameras around an object and shooting at the same time, followed by post-processing. However, these techniques cannot provide a dynamic scene of an object or a scene from a virtual viewpoint that is adaptive to a user's request.

The present invention is a method and system proposed to solve and supplement the above problems, and generates a scene of a virtual view that cannot be obtained by a real camera based on a live image of a dynamic non-rigid object. It is also an object of the present invention to provide a method of providing a three-dimensional image according to a user's request by using scenes of a virtual view and a computer-readable recording medium on which a program for realizing the method is recorded.

That is, the present invention effectively obtains a foreground projection image from each real image of a dynamic non-rigid object input from a plurality of cameras whose position and distance relation are controlled and synchronized in time and space. In this case, the foreground projection image uses a background subtraction method that uses the difference between the background reference image without the object and the image in which the object exists. The foreground and projection images are more stably distinguished from the background and foreground using both brightness and color component values. . In addition, the system is configured to generate a scene image at a virtual view point that cannot be obtained by an actual camera by using a fast image interpolation technique based on the acquired foreground projected image. Here, the individual pixel values of the virtual view scene are obtained by using an interpolation technique that weights the pixel values according to the distance between the virtual view point and the camera center point by referring to images of two cameras close to the virtual view point.

In order to achieve the above object, the present invention provides an apparatus for adaptively generating and providing a scene of a virtual view according to a user's request. Dividing the background and the foreground by inputting the images and setting a threshold value by merging the intensity value and the color component value of the image for each image; Obtaining a projected image of a dynamic non-rigid foreground using the background and foreground separation method and removing a small noise present in the foreground projected image using an image filtering technique; Generating a reference projection image for a virtual viewpoint by applying an image interpolation method to the projection images; Generating a virtual envelope for a target object by calculating intersection points between cone-shaped shapes generated by shooting virtual light at one point of the camera position on each of the foreground projected images obtained by the above method. do.

The above objects, features and advantages will become more apparent from the following description taken in conjunction with the accompanying drawings.

1 is a diagram illustrating an embodiment of a method and system for reconstructing a virtual viewpoint 3D scene based on a foreground projection image and a virtual envelope according to the present invention.

As shown in the drawing, the apparatus for implementing a virtual viewpoint 3D scene reconstruction system based on the foreground projected image and the virtual envelope of the present invention includes an image input device 10 for collecting a live image to be used as a reference image, each of which is inputted. Foreground projective image generating device 11 for obtaining a reference image required for generating a virtual envelope from the real image, and using the obtained foreground projected images and the image interpolation technique to generate a scene seen from the virtual viewpoint 3D modeling database 13 and database for storing the result data generated by the virtual view scene generator 12, the live image input device 10 and the foreground object generating device 11, and the virtual view scene generator 12 Virtual envelope generator 14 for reconstructing three-dimensional scenes based on virtual envelope techniques using modeling data in the And a device 15 for displaying the scene.

The foreground projection image generating apparatus 11 receives a live image from an image input device and obtains brightness and color component values Cr for each image to obtain brightness values of pure background images without an object. Morphological filtering to remove small image noises from the image obtained through the background and foreground separator 100 and the background and foreground separator that extract only the foreground from the image by obtaining the difference from the color component values and setting the threshold. And an image filtering processor 101 for performing convex hull filtering to fill small holes in the foreground projected image.

In the virtual viewpoint scene generator 12, the method 102 proceeds to the virtual envelope generator 14 without storing the data in the 3D modeling information database and reconstructs the 3D scene in real time.

FIG. 2 sets a threshold for distinguishing a background from a foreground by using a brightness value and a color component value of an image in a live image input from a camera, and shows a projected image of a foreground separated by the threshold.

The difference between the brightness value and the color component value of the (a) image in which the object of FIG. 2 exists and the background reference image (b) in which the object does not exist is determined, and the difference between the threshold value and the threshold value is determined. Distinguish The equation for calculating the brightness value and color component value is as follows.

Intensity = 0.29900R + 0.58700G + 0.11400B

Color value (cr) = 0.50000R-0.41869G-0.08131B

The equation for identifying the foreground is:

In the above equation, the threshold value is set by the following equation.

In the equation for calculating the threshold value, C is a constant and adjusts the size according to the environment in which the live image is input. The minimum value of the constant is one.

Using the brightness component and the color component at the same time as described above can effectively eliminate the separation of the shadow and the like if the error can be overcome by the slight change in the illumination.

3 and 4 are a plan view and a three-dimensional view showing the relationship between the foreground projection image and the virtual envelope. The definition of the virtual envelope is as follows.

In the above formula, R means a set of live images received from the camera, r means each of the live images, and VSr is a conical model 3 generated when a virtual light is shot from the camera center to the center of the foreground projection image. Each of the dimensional shapes means. In other words, virtual-shape may be defined as an intersection of three-dimensional shapes having a cone shape.

5 is a view showing a scene in which a virtual ray starting from one point corresponding to a camera center point position is shot toward a center point of a plurality of foreground projected images, and a cone shape formed by the ray intersects each other. FIG. 6 is a view illustrating the formation of a virtual envelope by introducing a SCG (Constructive Solid Geometry) concept capable of easily expressing a modeling of a three-dimensional object having a sense of feeling in FIG. 5, and performing a Boolean combination operation. Example.

7 is to obtain a projected image of the foreground from the live image input from a plurality of cameras according to the present invention, and to reconstruct and provide a variety of scenes suitable for the required virtual observation point by applying a virtual envelope concept and interpolation algorithm to the projected image This is a flowchart to explain the whole process of how to give.

In FIG. 7, an algorithm for generating a virtual new scene by adapting to a virtual view scene request is summarized as follows.

for each pixel p in desired image do

  distanceAlongRay = 0;

  while (isOutsideVisualHull)

      distanceAlongRay ++;

      if (intersectionFound)

          for each realCamera

              if (notOccluded && close)

                 sampleColor ();

          pixelColor = weightdMean (sample);

    else

pixel = background;

8 is a view showing images inputted from a time-synchronized camera having a controlled distance and positional relationship, and FIG. 9 is a scene example of a virtual view generated through the entire process of the images input from FIG. It is shown.

As described above, the method of the present invention may be implemented as a program and stored in a computer-readable recording medium.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the spirit of the present invention for those of ordinary skill in the art to which the present invention pertains, the foregoing embodiments and the accompanying drawings. It is not limited to.

As described above, the present invention provides a virtual view scene using a plurality of foreground projection images and fast image interpolation techniques for dynamic non-rigid objects in order to generate a scene of a new view which is essential for showing a scene adaptive to a user's demand in real time. Create By reconstructing the three-dimensional scene by combining the concept of the virtual envelope based on the virtual viewpoint scene obtained as described above, it generates a random virtual viewpoint scene as well as a limited scene to the real image input from the camera as in the conventional method. It is effective to provide various scenes quickly.

In addition, since an image-based photorealistic image is used as a reference image, expensive equipment such as a laser scanner is not required, and since a sensor such as a marker is not used, there is an effect of inducing more natural and dynamic movement of an object, and By using time-synchronized cameras, the number of cameras can be maintained to a certain level while the image quality can be further improved, making the system economically inexpensive.

Claims (5)

What is claimed is: 1. A method of creating and displaying various scenes in real time in accordance with an arbitrary virtual viewpoint of a user, and a system for realizing the method, A first step of effectively obtaining a foreground projection image of the live image received from a plurality of cameras; A second step of constructing a virtual envelope of the 3D target object using the foreground projected image; A third step of generating in real time a scene of a virtual view adaptive to a user's request by using a fast image interpolation technique based on the virtual envelope; A method and system for generating a virtual viewpoint scene based on real image. The method of claim 1, wherein the foreground projection image is obtained by dividing the foreground and the background of the image received from the camera. A method and system for obtaining a foreground projection image for setting a threshold using color information as well as brightness information of an image. The method of claim 1, wherein the image interpolation technique is based on a virtual envelope. When the pixel value of the silhouette image of the camera to be referenced cannot be obtained due to the obstruction of another object, the weight of the pixel values is proportional to the distance when the pixel value of the foreground projection image of the closest camera is replaced with the pixel value. How and how to give. The method of claim 1, wherein the actual images inputted from the camera enable more images to be used as reference images than the reference images obtained by the limited number of cameras by the camera's temporal synchronization. Method and system. The method and system of claim 4, wherein the reference image obtained by temporal synchronization is used to obtain a more precise and detailed shape of the object by matching feature points with the object photographed at a previous time using motion information of the object.

Images