KR101665049B1 - Image processing apparatus and method - Google Patents

Abstract

An image processing apparatus is provided. The image processing apparatus determines a video processing mode by referring to a plurality of frame images included in the input video data and a target point to be rendered. When the image processing mode is the first mode for generating a 3D model using at least a part of the plurality of frame images, the image processing apparatus generates a 3D model using at least a part of the plurality of frame images. If the image processing mode is the second mode for performing 2D conversion using at least a part of the plurality of frame images, the image processing apparatus performs 2D conversion using at least a part of the plurality of frame images .

Description

[0001] IMAGE PROCESSING APPARATUS AND METHOD [0002]

And more particularly to an image processing apparatus and method for rendering a stereoscopic and / or multi-view image at an arbitrary target time point from input video frames, and more particularly, to a method and apparatus for generating a 3D model using image synthesis, And an image processing apparatus and method for generating a multi-view image using a 2D image frame adjacent to a target time point.

Recently, interest in 3D (3 Dimensional) images is increasing. However, there are 2D video data that are conventionally produced by a 3D camera set or not produced by 3D model rendering.

In this case, if the quality of the image processing capable of providing the 3D effect is improved by using the existing images which are not possessed of the modeled 3D object information or are not made of the 3D image, the market of the 3D image may be faster have.

Accordingly, there is an increasing interest in the field of generating a stereoscopic or multi-view image by generating an image at a certain time from a plurality of input 2D images captured at different points in time.

There is provided an image processing apparatus and method for generating stereoscopic image and / or multi-view image at an arbitrary time by receiving video data including images of a plurality of frames, generating a 3D model, and using the generated 3D data.

There is also provided an image processing apparatus and method capable of recycling at least a part of images of a plurality of frames included in input video data to generate a stereoscopic image and / or a multi-view image.

According to an aspect of the present invention, there is provided a video processing apparatus including a mode determination unit for determining an image processing mode by referring to a plurality of frame images included in input video data and a target point to be rendered, And a 3D model generation unit that generates a 3D model using at least a part of the plurality of frame images in a first mode for generating a 3D model using the plurality of frame images.

According to an embodiment of the present invention, when the image processing mode is the second mode in which the image processing mode performs 2D conversion using at least a part of the plurality of frame images, at least a part of the plurality of frame images And generating a 2D information corresponding to the target point by performing a 2D conversion using the 2D image data.

The mode determining unit may include a feature point extracting unit that extracts feature points from at least a part of the plurality of frame images, a feature point matching unit that analyzes the correspondence between the feature points by matching the extracted feature points, A time estimator for analyzing a motion pattern at a time point corresponding to each of the plurality of frame images of the input video data, and an optimizer for determining the image processing mode in consideration of the motion pattern and the target viewpoint .

In this case, the viewpoint estimating unit may analyze the motion pattern of the viewpoint using a structure from motion (SFM) technique.

Meanwhile, when the image processing mode is the first mode, the 3D model generating unit generates a 3D image using the sparse 3D structure, which is restored in the process of determining the image processing mode, An image alignment unit for performing polar rectification of the dense three-dimensional structure, an image matching unit for restoring a dense three-dimensional structure using the sparse three-dimensional structure, A 3D structure estimating unit for restoring the texture, and a 3D modeling unit for generating the 3D model using the 3D surface and the texture.

According to an embodiment of the present invention, the image processing apparatus further includes a rendering unit that renders an image at the target point by using the 3D model generated by the 3D modeling unit.

According to another embodiment of the present invention, the image processing apparatus may further comprise a rendering unit that renders the image at the target point using the 3D model generated by the 3D modeling unit and the 2D information generated by the static image adjustment unit, And a rendering unit.

According to another aspect of the present invention, there is provided a video processing method including: a mode determining step of determining an image processing mode by referring to a plurality of frame images included in input video data and a target point to be rendered; A 3D model generation step of generating a 3D model using at least a part of the plurality of frame images in a first mode for generating a 3D model using a part of the plurality of frame images, The image processing method comprising the steps of:

Even if information of the 3D model is not given, it is possible to quickly generate the 3D model through the video data including the images of the plurality of frames.

Also, a stereoscopic image and / or a multi-view image at an arbitrary time point are efficiently generated using the generated 3D model.

Further, at least a part of the images of the plurality of frames included in the input video data are utilized for generation of the stereoscopic image and / or the multi-view image, thereby improving the image processing speed and providing the real-time realistic 3D image .

In terms of processors, computational complexity is reduced and parallel computation is possible, which enables the final three-dimensional model to be generated more quickly.

FIG. 1 illustrates an image processing apparatus according to an embodiment of the present invention.
2 is a conceptual diagram illustrating exemplary video frames input to an image processing apparatus according to an embodiment of the present invention.
3 is a conceptual diagram showing a viewpoint of a multi-view image to be generated according to an embodiment of the present invention.
4 illustrates a detailed structure of a mode determination unit of an image processing apparatus according to an embodiment of the present invention.
FIG. 5 illustrates a detailed structure of a 3D model generation unit of an image processing apparatus according to an embodiment of the present invention.
6 illustrates an image of an exemplary plurality of frames utilized by the 3D model generator in accordance with an embodiment of the present invention.
FIG. 7 illustrates minutiae points extracted from images of a plurality of frames of FIG. 6 according to an embodiment of the present invention.
FIG. 8 illustrates an image processing method according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

FIG. 1 shows an image processing apparatus 100 according to an embodiment of the present invention.

The image processing apparatus 100 includes a mode determination unit 110, a 3D model generation unit 120, a static image adjustment unit 130, and a rendering unit 140.

The mode determination unit 110 may determine an image processing mode for rendering an image at a target viewpoint.

In this case, the image processing mode may be a 3D modeling mode for generating a 3D model, or a static image adjusting mode for processing 2D homography using an existing input image frame at a near point.

Meanwhile, it is also possible to enhance the quality of the image by complementing the respective disadvantages of the two modes in parallel, which corresponds to parallel progression (not shown) of the two modes. The parallel mode progression will be described later with reference to FIG.

The detailed structure of the mode determination unit 110 will be described later with reference to FIG.

When the mode for the image rendering at the target time point is determined as the 3D modeling mode, the 3D model generating unit 120 generates a 3D model using a plurality of frame images of the input video data, Will be described later in detail.

On the other hand, if the mode for rendering the target viewpoint image is determined as the static image adjustment mode, the static image adjustment unit 130 performs 2D homography by static image adjustment.

In this process, the static image adjustment unit 130 separates the foreground and the background using the feature points of the images extracted by the mode determination unit 110, and performs a salient region extraction.

Then, the static image adjustment unit 130 performs a static image adjustment using defocused region extraction, fore / background separation, and single-view 3D reconstruction. Find candidate regions that need 3D effects on the image.

Then, the static image adjustment unit 130 determines the degree of the 3D effect and the virtual 3D depth to be applied based on the extracted information based on the single static image, and generates the virtual view image corresponding to the target point.

Then, the rendering unit 140 renders the stereoscopic and / or multi-view images at the target point using the 3D model and / or the static image adjustment result. The operation of the rendering unit 140 in this process will be described later in detail with reference to FIG.

2 is a conceptual diagram illustrating exemplary video frames input to an image processing apparatus according to an embodiment of the present invention.

The viewpoints 101 to 119 correspond to the camera viewpoints of the images of the plurality of frames included in the input video data, respectively.

The data photographed at the respective viewpoints are 2D images, and no separate 3D model information about the object, such as geometry information of the object, is given.

According to an embodiment of the present invention, a 3D model for an object is generated by using images of frames of at least a part of each frame of video data obtained and input as input at the time (101 to 119 and the like) And generates a stereoscopic image and / or a multi-view image corresponding to arbitrary target time points using the generated 3D model.

According to another embodiment of the present invention, prior to the 3D model generation, the image processing apparatus 100 determines whether images corresponding to the target time points exist in the input video data, And may selectively perform image processing using images of at least one frame corresponding to target points in a frame of an existing input video instead of generating a 3D model.

Hereinafter, the operation of the image processing apparatus according to one embodiment of the present invention will be described in more detail with reference to FIG.

3 is a conceptual diagram showing a viewpoint of a multi-view image to be generated according to an embodiment of the present invention.

The time points 101 to 105 correspond to the shooting time of the 2D image of the plurality of frames existing in the input video data. As the shooting progresses from the viewpoint 101 to the viewpoint 105, it is observed that the viewpoint of the camera moves from the upper left to the lower right.

However, if a nine view (nine view) image is generated centering on the viewpoint 103, the image corresponding to the viewpoint 103 can be used as it is, but the image corresponding to the remaining viewpoints 301 to 308 In order to generate each image, the existing image can not be used as it is.

If an image of a plurality of frames in the previously input video data exists at a time point similar to a specific time point to be rendered, it can be utilized as it is and used for generating parallax with other images.

However, these methods have a limitation that a real multi-view image can be generated only when parallax corresponding to target points to be rendered exists among each frame image of the existing input video data.

Thus, due to this constraint, the range of input video data available in the method is significantly narrowed.

As a simple example, when the camera moves mainly forward and backward when photographing input video data, it is impossible to apply the present invention because there is no left and right parallax information. Even if it is not necessarily straight, It is difficult to apply it even when it does not exist.

For example, in the case of rendering a stereoscopic image centered on the viewpoint 115 of FIG. 2, other frame images having appropriate parallax information on the left or right of the viewpoint 115, that is, And the frame image of the viewpoint 116, it is possible to provide a stereoscopic image directly without generating a separate 3D model.

However, as shown in FIG. 3, when frame images having time difference information corresponding to the target time points 103 and 301 to 308 are not present in the existing input video data, Acquisition of time difference information and image rendering are required.

Further, in the case of the frame at the time 101 photographed at the leftmost end of the cut, parallax information corresponding to the time point to the left of the cut can not be obtained. The time difference information corresponding to the time point on the right side can not be obtained even in the case of the time point 119 at which the image is photographed from the rightmost side.

Therefore, according to an embodiment of the present invention, when the input video is received and the multi view viewpoint, i.e., the target viewpoints to be rendered, is determined, the mode determination unit of the image processing apparatus 100 refers to each frame image of the input video data And determines an image processing mode for rendering an image corresponding to the target time points.

The mode corresponds to each of the above-mentioned two cases. One is that when the frame image at the time point similar to the target time points is already present in the existing input video data, since the time difference information is secured, the 2D image is rendered using 2D homography without generating a separate 3D model Is a static image adjustment mode.

3, when it is determined that a frame image at a time point similar to the target time points does not already exist in the existing input video data, In the 3D modeling mode.

The detailed operation of the mode determination unit 110 will be described later with reference to FIG. 4. When the mode determination unit 110 determines the image processing mode as the 3D modeling mode, the detailed operation of the 3D model generation unit 120 will be described with reference to FIG. 5 Will be described later.

FIG. 4 illustrates a detailed structure of a mode determination unit 110 of an image processing apparatus according to an embodiment of the present invention.

When the input video data is received, the mode determination unit 110 determines an image processing mode to be utilized for image rendering at the target viewpoints. The image processing mode may be a 3D modeling mode for directly generating a 3D model and rendering an image at a target time point, or a static image processing unit for 2D-homography of an image of an existing input video frame, This is the image adjustment mode.

According to an embodiment of the present invention, once the image processing mode is determined, a mode determined continuously may be used. However, according to another embodiment of the present invention, 0.0 > mode decision. ≪ / RTI >

According to an embodiment of the present invention, when video data is input, the feature point extracting unit 410 extracts feature points from each frame image of input video data.

6 illustrates an image of an exemplary plurality of frames utilized by the 3D model generator in accordance with an embodiment of the present invention.

Referring again to FIG. 4, the feature point matching unit 420 determines pairs of corresponding feature points through feature point matching between images of respective frames. In this way, motion estimation of the viewpoint between each frame image is possible by extracting each feature point and searching for the correspondence relationship.

The process of extracting or matching feature points can be understood as restoring a sparse 3D structure between objects.

FIG. 7 illustrates minutiae points extracted from images of a plurality of frames of FIG. 6 according to an embodiment of the present invention.

A feature point extraction result 710 from the view point image 610 and a feature point extraction result 720 from the view point image 620 are shown.

The extracted feature points 711 to 718 correspond to the extracted feature points 721 to 728, but the feature point 719 or the feature point 729 and the like do not have corresponding feature points.

Referring again to FIG. 4, the viewpoint estimating unit 430 analyzes a motion pattern of the viewpoint of the input video data through a structure from motion (SFM) technique.

In the analysis of the motion pattern, whether the movement of the camera viewpoint is in the horizontal direction, the vertical direction, or the forward / backward direction such as zoom in / zoom out is analyzed.

Then, the optimizing unit 440 determines the image processing mode considering the given target time points. As described above, the optimizing unit analyzes the viewpoint motion characteristic of the input video data and the characteristic of the target point to be rendered, and may change the mode from time to time when it is determined that the mode change is necessary.

FIG. 5 shows a detailed structure of a 3D model generation unit 120 of an image processing apparatus according to an embodiment of the present invention.

When the mode determining unit 110 determines the image processing mode to be a 3D modeling mode, the image aligning unit 510 of the 3D model generating unit 120 generates a sparse 3D structure that is repeatedly restored in the mode determination process. 3D structure) can be used to perform the polar rectification of the input image.

According to one embodiment of the present invention, the image arranging unit 510 performs the image sorting process in accordance with M. Pollefeys, R. Koch, and L. Van Gool, "A Simple and efficient rectification method for general motion" 1999), and algorithm of image alignment method can be used.

The image matching unit 520 reconstructs a dense 3D structure using the image alignment result. According to an embodiment of the present invention, in this process, KJ Yoon and IS Kweon's paper entitled "Locally Adaptive The method of "Support-Weight Approach for Visual Correspondence Search" (CVPR 2005) and / or PF Felzenzwalb and DP Huttenlocher's "Efficient Belief Propagation for Early Vision" (IJCV 2006) .

According to an embodiment of the present invention, the image matching unit 520 performs cost computation and optimization (Belief propagation) in the process of estimating the correspondence relationship for dense three-dimensional structure restoration.

In this case, the image matching unit 520 applies the correspondence in the sparse three-dimensional structure as prior knowledge so that the disparity candidates of the sparse correspondence exist, The penalty is given to candidates that do not satisfy the sparse correspondence among the candidates, thereby increasing the accuracy of the process of estimating the correspondence relationship and increasing the convergence speed of the optimization process.

The scope of the present invention is not to be interpreted as being limited to these specific exemplary methods, but should be construed according to the scope of the claims.

When the dense three-dimensional structure is restored through the image matching, the 3D structure estimator 530 performs a 3D surface and texture restoration process, and the 3D modeling unit 550 generates a 3D model using the restoration result . According to one embodiment of the present invention, the method of M. Kazhdan, M. Bolitho and H. Hoppe, "Poisson Surface Reconstruction" (Eurographics Symposium on Geometry Processing, 2006) .

In this process, the filter unit 540 removes the outlier or performs appropriate data filtering in the generation of the dense three-dimensional structure or the 3D model, thereby reducing the error.

When the 3D model is created, the rendering unit 140 renders the image at the target point using the 3D model. These target times correspond to the respective times of the stereoscopic and / or multi-view images to be displayed at present.

According to an embodiment of the present invention, the rendering unit 140 temporarily renders the target viewpoint image based on the depth information of the 3D model, and performs a 2D transformation (2D As a result based on homography, another target viewpoint image is temporarily rendered, and the optimized image can be rendered by fusing them and eliminating the discontinuity.

When there is no existing image having a time point sufficiently close to the target point in the input video data, the error of the parallax information is large, so that the 3D modeling mode is selected. In addition, the static video mode is also performed in parallel, and depth discontinuity Is to overcome the parallax error where it exists.

In this case, since the depth information verified by the 3D model is used, not only accurate parallax can be calculated where the depth information is close to the target point, but also an initial dense correspondence The parallax information in the case where there is no image close to the target point can be accurately calculated.

In addition, according to another embodiment of the present invention, the rendering unit 140 performs a segment-based method in which the reference image is over-segmented in order to reduce the lost region information, image shifting can also be used.

The two embodiments of the rendering unit 140 have characteristics opposite to each other, and the images with high parallax accuracy obtained through the preceding embodiment are used first, and in the image obtained through the latter embodiment, By complementing the parts of the regions, it is possible to obtain virtual viewpoint images with high parallax accuracy and completeness.

Meanwhile, a variety of methods are available for the rendering unit 140 to render an accurate image by merging the results of the two modes.

According to one embodiment of the present invention, the rendering unit 140 may be implemented in a manner similar to that described in AA Efros and WT Freeman, "Image Quilting for Texture Synthesis and Transfer" (SIGGRAPH 2001) or V. Kwatra, A Schodl, I. Essa, G. It is possible to use a method of selecting an optimal boundary line using overlapping regions according to a method disclosed in Turk and A. Bobick's paper entitled " Graphcut Textures: Image and Video Synthesis Using Graph Cuts "(SIGGRAPH 2003).

In addition, according to another embodiment of the present invention, the rendering unit 140 generates a temporary image of the target viewpoint based on the depth information of the 3D model in a static image adjustment mode It is possible to smoothly convert the entire color using the information of the corresponding position of the obtained target temporal temporal image and utilize the inclination information of each image. Such a method is presented in P. Perez, M. Gangnet and A. Blake, "Poisson Image Editing" (SIGGRAPH 2003).

In this way, the rendering unit 140 effectively removes the discontinuity of the resultant image, and can generate a complete optimal virtual viewpoint image with a more accurate parallax at the boundary line portion.

FIG. 8 illustrates an image processing method according to an embodiment of the present invention.

In step 810, the mode determination unit 110 of the image processing apparatus 100 may determine an image processing mode for rendering an image at a target point in time.

In this case, the image processing mode may be a 3D modeling mode for generating a 3D model, or a static image adjusting mode for processing 2D homography using an existing input image frame at a near point.

Meanwhile, as described above with reference to FIG. 7, two modes may be performed in parallel to improve the quality of the image by complementing the respective disadvantages. In this case, the mode corresponds to parallel progression (not shown) of the two modes .

The operation of the mode determination unit 110 is as described above with reference to FIG. 2 to FIG.

If the mode is determined, the 3D modeling mode of step 840 is performed in step 820 to generate a 3D model.

The process of generating the 3D modeling is as described above with reference to FIGS.

In the case of the static image adjustment mode, 2D homography is performed by static image adjustment in step S830.

In this process, the static image adjustment unit 130 separates the foreground and the background using the feature points of the images extracted by the mode determination unit 110, and performs a salient region extraction.

Then, the static image adjustment unit 130 performs a static image adjustment using defocused region extraction, fore / background separation, and single-view 3D reconstruction. Find candidate regions that need 3D effects on the image.

Then, the static image adjustment unit 130 determines the degree of the 3D effect and the virtual 3D depth to be applied based on the extracted information based on the single static image, and generates the virtual view image corresponding to the target point.

In step 850, the rendering unit 140 renders the image at the target point using the result. The operation of the rendering unit 140 in this process is as described above with reference to FIG.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: image processing device
110:
120: 3D model generation unit
130: Static image adjustment unit
140:

Claims (14)

The image processing mode is determined according to whether a plurality of frame images included in the input video data and a target time point to be rendered exist and a frame at a time point similar to the target time point exists in the plurality of frame images based on the time difference information ;
A 3D model generation unit for generating a 3D model using at least a part of the plurality of frame images when the image processing mode is a first mode for generating a 3D model using at least a part of the plurality of frame images; And
And a rendering unit for rendering a stereoscopic image at the target time point using the 3D model generated by the 3D model generation unit,
And the image processing apparatus. The method according to claim 1,
When the image processing mode is a second mode for performing 2D conversion using at least a part of the plurality of frame images, performing 2D conversion using at least a part of the plurality of frame images to generate 2D A static image adjustment unit
Further comprising: The method according to claim 1,
Wherein the mode determination unit determines,
A feature point extracting unit that extracts feature points from at least a part of the plurality of frame images;
A feature point matching unit for matching the extracted feature points and analyzing a correspondence relationship between the feature points;
A time estimator for analyzing a motion pattern at a time point corresponding to each of the plurality of frame images of the input video data through a correspondence relationship between the minutiae points; And
An optimization unit for determining the image processing mode in consideration of the motion pattern and the target time point,
And the image processing apparatus. The method of claim 3,
Wherein the viewpoint estimating unit analyzes a motion pattern of the viewpoint using a structure from motion (SFM) technique. The method according to claim 1,
Wherein the 3D model generating unit comprises:
When the image processing mode is the first mode, a polar rectification of an input image is performed using a sparse 3D structure that is restored in the process of determining the image processing mode by the mode determining unit An image alignment unit;
An image matching unit for reconstructing a dense three-dimensional structure using the sparse three-dimensional structure;
A 3D structure estimating unit for restoring a 3D surface and a texture using the dense three-dimensional structure; And
A 3D modeling unit for generating a 3D model using the 3D surface and the texture,
And the image processing apparatus. delete 3. The method of claim 2,
A rendering unit for rendering the image at the target point using the 3D model generated by the 3D model generating unit and 2D information generated by the static image adjusting unit according to the 2D deformation,
Further comprising: The image processing mode is determined according to whether a plurality of frame images included in the input video data and a target time point to be rendered exist and a frame at a time point similar to the target time point exists in the plurality of frame images based on the time difference information Determining a mode;
A 3D model generation step of generating a 3D model using at least a part of the plurality of frame images when the image processing mode is a first mode for generating a 3D model using at least a part of the plurality of frame images; And
Rendering the stereoscopic image at the target point using the 3D model
And an image processing method. 9. The method of claim 8,
When the image processing mode is a second mode for performing 2D conversion using at least a part of the plurality of frame images, performing 2D conversion using at least a part of the plurality of frame images to generate 2D Static image adjustment step to generate information
Further comprising the steps of: 9. The method of claim 8,
Wherein the mode determination step comprises:
Extracting feature points from at least a portion of the plurality of frame images;
Analyzing a correspondence relationship between the minutiae points by matching the extracted minutiae;
Analyzing a motion pattern at a time point corresponding to each of the plurality of frame images of the input video data through correspondence between the minutiae points; And
Determining the image processing mode in consideration of the motion pattern and the target time point
And an image processing method. 11. The method of claim 10,
Wherein analyzing the motion pattern comprises analyzing the motion pattern of the viewpoint using a structure from motion (SFM) technique. 9. The method of claim 8,
In the 3D model generation step,
When the image processing mode is the first mode, a polar rectification of the input image is performed using a sparse 3D structure restored in the process of determining the image processing mode of the mode determination step ;
Reconstructing a dense 3D structure using the sparse three-dimensional structure;
Reconstructing a three-dimensional surface and a texture using the dense three-dimensional structure; And
Generating a 3D model using the 3D surface and texture
And an image processing method. 10. The method of claim 9,
Rendering the image at the target point by using the 2D information generated in the static image adjustment step by the 3D model and the 2D transformation generated in the 3D model creation step
Further comprising the steps of: A computer-readable recording medium containing a program for performing the image processing method according to any one of claims 8 to 13.

Images