KR100795481B1 - Method and apparatus for processing multi-view images - Google Patents

Abstract

The present invention adjusts the horizontal and vertical parallaxes of the images displayed on the left and right eyes and generates intermediate images between the two viewpoints to reduce the eye fatigue of the viewer and stereoscopic flicker phenomenon when changing the viewpoints to be viewed. The present invention relates to a method and an apparatus for processing a multi-viewpoint image to reduce the number of images. A multi-view image processing apparatus according to the present invention comprises: a viewpoint image extracting unit for extracting a first image set including two or more viewpoint images consecutive from multi-view image data; An intermediate image generation unit for generating an intermediate image constituting an intermediate image set inserted in the middle of the first image set and a second image set newly extracted by the viewpoint image extracting unit according to a viewpoint change of the first image set ; And an image output unit for outputting an image according to the first or second image set and the intermediate image set. By using the multi-view image processing apparatus or method according to the present invention, it is possible to provide a more stable image by adjusting the vertical parallax and horizontal parallax of the image displayed on the left eye and the right eye, and by generating an intermediate image when the user's viewpoint changes. This can prevent eye fatigue caused by sudden parallax changes and increase the sense of reality.

Stereoscopic broadcast, intermediate video, multi-view

Description

Method and apparatus for multi-view image {Method and apparatus for processing multi-view images}

1 is a block diagram of an embodiment of a multi-view image processing apparatus according to the present invention.

2 is a detailed block diagram of a viewpoint image disparity adjusting unit of an embodiment of a multi-view image processing apparatus according to the present invention.

3 is a detailed block diagram of an intermediate image generating unit of an embodiment of a multi-point image processing apparatus according to the present invention.

4 is a view showing a sequence of multi-view images in which L intermediate images are inserted in an embodiment of a multi-view image processing apparatus according to the present invention.

<Description of the symbols for the main parts of the drawings>

10: multi-view image processing apparatus 12: multi-view image input unit

14: viewpoint image extraction unit 16: viewpoint image disparity control unit

18: intermediate image generating unit 20: image output unit

22: dynamic vector extraction unit 24: vertical parallax calculation unit

26: horizontal parallax calculation unit 28: crystal image generation unit

32: number of intermediate image determination unit 34: variation map generation unit

36: depth map calculator 38: intermediate image processing unit

The present invention relates to a method and apparatus for processing a multi-view image, and specifically, to adjust the horizontal parallax and vertical parallax of an image displayed on the left eye and the right eye, and to change the viewpoint to view an intermediate image between two viewpoints. The present invention relates to a method and apparatus for processing multi-view images that reduce eye strain of a viewer and reduce stereoscopic flicker.

Multi-view images are view images obtained by photographing the same object with a large number of synchronized cameras (for example eight) arranged in parallel or arc shape. Such multi-view images can be applied not only to stereoscopic display devices but also to stereoscopic, realistic, 3D DMB, FTV (free-view TV), etc. It is a broad technology.

However, in the multi-view images developed so far, accurate stereoscopic images are not provided due to the horizontal parallax and the vertical parallax existing in the right eye and the left eye, so that stable stereoscopic images are not realized and eyes are easy to view even for short time images. There was a downside to fatigue.

In addition, since a finite number of cameras are used to implement a stereoscopic image, when the viewpoint of the image changes, the screen suddenly changes so that the stereoscopic feeling and the real feeling are deteriorated.

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for processing a multi-view image, which provides a stable stereoscopic image by adjusting the horizontal parallax and the vertical parallax present in two images to be presented to the left eye and the right eye.

In addition, an object of the present invention is to provide a method and apparatus for processing a multi-view image to reduce the eye fatigue and increase the sense of reality that occurs in the process of changing the viewpoint by showing the intermediate image when changing the viewpoint in the multi-view image.

In order to achieve this object, a multi-view image processing apparatus according to the present invention comprises: a viewpoint image extracting unit 14 for extracting a first image set including two or more viewpoint images consecutive from multi-view image data; An intermediate image generation unit for generating an intermediate image constituting an intermediate image set inserted in the middle of the first image set and a second image set newly extracted by the viewpoint image extracting unit according to a viewpoint change of the first image set (18); And an image output unit 20 for outputting an image corresponding to the first or second image set and the intermediate image set.

Preferably, the multi-view image processing apparatus further includes a multi-view image input unit 12 for receiving multi-image data again. The multi-view image input unit 12 includes a buffer to store a plurality of multi-view images and transfer image data to the viewpoint image extracting unit 14 or the like as necessary.

The multi-view image processing apparatus further includes a viewpoint image disparity control unit 16 that calculates horizontal parallax and vertical parallax with respect to the two or more images, and corrects some of these images according to the calculated horizontal parallax and vertical parallax. It can be provided further. Preferably, the multi-view image processing apparatus calculates the horizontal parallax and the vertical parallax for two images for the left eye and the right eye, and more preferably corrects the image of the right eye based on the image of the left eye.

Preferably, the viewpoint image disparity adjusting unit 16 includes: a moving vector generation unit 22 for extracting moving vectors from two or more input images; A vertical parallax calculation unit (24) for obtaining the vertical parallax from an average vertical dynamic vector component of the dynamic vectors; A horizontal parallax calculator 26 for obtaining the horizontal parallax from an average horizontal dynamic vector component of the dynamic vectors; And a corrected image generation unit 28 for moving the image by the vertical parallax and the horizontal parallax to obtain the corrected image.

Preferably, the intermediate image generation unit 18 includes: an intermediate image number determination unit 32 for determining the number of intermediate images between the viewpoints of two images; A disparity map generator 34 extracting disparity for each pixel or block between the two images; A depth map calculator 36 for calculating a depth value from the variation map unit; And an intermediate image processor 38 applying the depth value to a perspective projection technique to generate an intermediate image suitable for the viewpoint of the intermediate image.

Extraction of the dynamic vector may be performed by a block-based motion prediction method, or may be performed by an optical flow method.

In another aspect of the present invention, a multi-view image processing method according to the present invention comprises the steps of: extracting a first image set including two or more viewpoint images consecutive from the multi-view image data; Generating an intermediate image constituting the first image set and an intermediate image set inserted between the first image set and a second image set newly extracted according to a change of viewpoint of the first image set; And outputting an image according to the first or second image set and the intermediate image set. Preferably, the multi-view image processing method further includes receiving the multi-view image.

Also, preferably, the multi-view image processing method includes: calculating horizontal parallax and vertical parallax for the two or more images; And correcting some of these images in accordance with the calculated horizontal parallax and vertical parallax.

The calculating of the horizontal parallax and the vertical parallax may include extracting a motion vector from two or more input images; Obtaining the horizontal parallax from an average horizontal dynamic vector component of the dynamic vectors; And calculating the vertical parallax from an average vertical dynamic vector component of the dynamic vectors.

Preferably, the step of modifying a part of the image is a step of modifying the image by moving by the average horizontal dynamic vector component and the average vertical dynamic vector component.

On the other hand, the step of generating the intermediate image, the step of determining the number of the intermediate image between the viewpoint of the two images; Generating a disparity map extracting disparity for each pixel or block between the two images; Calculating a depth value from the disparity map unit to obtain a depth map; And applying the depth value to a perspective projection technique to generate an intermediate image suitable for the viewpoint of the intermediate image.

Preferably, the extraction of the motion vector is performed by a block-based motion prediction method or an optical flow method.

Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. This embodiment is only an aspect of the present invention and should not be understood as limiting the scope of the present invention.

1 is a block diagram of a multi-image processing apparatus according to an embodiment of the present invention. The multi-view image input unit 12 receives data on the multi-view images F ₁ , F ₂ , ..., F _N from N multi-view cameras. The multi-view image input unit 12 has a buffer and can store the required number of images.

Then, the viewpoint image extracting unit 14 determines the viewpoint image F _K to be displayed among the N images of the multi-view image input unit 12 (where, K ∈ {1, 2, ..., N-1}). ) The set of images {F _K , F _{K +1} } is formed by receiving the viewpoint image F _K and the subsequent F _{K + 1} . Here, F _K is an image for the left eye and F _{K + 1} is an image for the right eye.

The extracted image is input to the viewpoint image disparity adjusting unit 16 again. The viewpoint image parallax adjustment unit calculates the horizontal parallax and the vertical parallax of the image to be displayed on the left eye and the right eye, and removes the vertical parallax and the horizontal parallax by moving the image of the right eye according to the obtained parallax value. The configuration and function of the viewpoint image parallax adjustment unit 16 will be described later.

Subsequently, the viewpoint image extraction unit 14 extracts the image sets {F ₁ , F _{1 + 1} } of the new viewpoint. The horizontal image parallax and the vertical parallax can also be removed or reduced through the viewpoint image parallax adjustment unit 16 for the image set {F ₁ , F _{1 + 1} } of the new viewpoint. When two image sets {F _K , F _{K +1} } and {F _l , F _{l + 1} } are obtained, the intermediate image generation unit 18 generates an intermediate image to be inserted between the two image sets. By inserting an intermediate image between two sets of images in this way, the user can experience a smooth three-dimensional image change when the viewpoint changes. The detailed configuration and function of the intermediate image generating unit 18 will be described later.

2 is a block diagram of the viewpoint image parallax adjustment unit 16 according to the present embodiment. When the multi-view images F _K and F _{K +1} are input from the multi-view image input section 12, the motion vector extracting section 22 first extracts a motion vector between the two images. For this, a block-based motion estimation method or an optical flow method may be used.

The block-based motion prediction method finds the most similar block in the next frame for a block of one frame and extracts the vector for these two blocks. To find the most similar block, you can compare all the blocks within a certain range, find the most suitable one among them, use the wide search grid to find the closest grid, and then reduce the grid size to find the best block. How to find it.

Meanwhile, in the mineral oil method, a Gaussian filter is applied to an input image, and then a laplacian filter is applied to perform edge detection. The mineral oil method then finds similar portions in both images by patch patching the two edge sensed images.

Assume that the motion vector obtained for the blocks of the two images by the above-described method is V = (u, v). Here u denotes a horizontal dynamic vector component and v denotes a vertical dynamic vector component. The horizontal parallax calculation unit 26 calculates an average horizontal dynamic vector component from the copper vector thus obtained using Equation 1 below.

Here, M is the number of dynamic vectors in the dynamic vector field.

On the other hand, in the vertical parallax calculation unit 24, the average vertical dynamic vector component of the dynamic vector obtained by the dynamic vector extraction unit 22 is obtained by using Equation 2 below.

_

_

The corrected image generating unit 28 moves the right image F _{K + 1} by the average horizontal dynamic vector component and the average vertical dynamic vector component thus obtained to obtain a corrected right image F ' _{K + 1} .

Hereinafter, a detailed configuration and function of the intermediate image generation unit 18 according to the present embodiment will be described with reference to FIG. 3. The intermediate image generation unit 18 of the present embodiment is constituted by the number of intermediate image generation units 32, the variation map generation unit 34, the depth map calculation unit 36, and the intermediate image processing unit 38. As described above, the intermediate image generation unit 18 generates one or more intermediate images to be inserted between two image sets {F _K , F _{K + 1} } and {F _l , F _{l +1} }. First, the number of intermediate images determining unit 32 is F _{K + 1} and F _l The number L of intermediate images to be created is determined.

Subsequently, the variation map generation unit 23 performs an image F _{K + 1} and Create a map of the pixel-by-block variation of F _l . To this end, a block-based motion prediction method or a mineral oil method, as used in the viewpoint image disparity control unit 16, may be used. To generate a map for such a variation, {F _K , F _l }, {F _{K +1} , F _l }, {F _K , F _{l +1} }, {F _{K +1} , F _{l + 1} } It is also possible to use a disparity map for one of the set of images of or a disparity map incorporating the disparity maps for these four sets of images.

The depth map calculator 36 of the intermediate image generator 18 calculates a depth map from the disparity value of each pixel or block obtained by the disparity map generator 34. Depth map means a map of depth value (z) of how far the component of the image is from the camera. An example of a method for generating such a depth map is a method disclosed in a paper entitled Stereoscopic analysis of multiple images published in 1993 by Sebastien Roy in the International Journal of Computer Vision.

The intermediate image processing unit 38 generates the intermediate images B ₁ , B ₂ ,..., B _L by the number L of viewpoints determined by the intermediate image number determination unit 32 using the depth value. The relationship between the generated intermediate image and the original image is as shown in FIG. In order to generate the intermediate image as described above, the intermediate image processor 38 generates the intermediate image by using a perspective projection technique. Since this perspective projection technique is widely known in the art, a detailed description thereof will be omitted. The generated image is transferred to the image output unit 20 of the multi-point image processing apparatus, and through the monitor {F _K , F _{K +1} }, {F _{K +1} , B ₁ }, {B ₁ , B ₂ } A set of pictures of ..., {B _{L -1} , B _L }, {B _L , F ₁ }, {F _l , F _{l +1} } are displayed in order.

The multi-view image processing apparatus may be configured in the form of a computer and software loaded and operated therein, or may be configured in the form of an ASIC chip or an electronic circuit composed of several circuit elements.

The present invention is not limited to the above embodiments and various modifications may be made within the technical scope of the present invention. For example, in the above embodiment, the number of intermediate images is determined by the intermediate image number determining unit. However, the intermediate image number determining unit always generates a predetermined number of intermediate images according to the computing power of the multi-view image processing apparatus without a separate intermediate image number determining unit. It may be configured. In addition, other displacement calculation methods other than the block-based motion prediction method or the mineral oil method may also be used to generate the displacement map.

By using the multi-image processing apparatus or method according to the present invention, it is possible to provide a more stable image by adjusting the vertical parallax and horizontal parallax of the image displayed on the left and right eyes, thereby reducing the user's fatigue.

In addition, when the viewpoint is changed by using the apparatus or method for multi-view image of the present invention, by generating an intermediate image, it is possible to prevent eye fatigue caused by a sudden change in parallax and increase the sense of reality.

Claims (14)

A multi-view image processing apparatus for displaying images for different viewpoints from multi-view image data, A viewpoint image extracting unit which extracts a first image set including two or more viewpoint images consecutive from the multi-view image data; An intermediate image generation unit for generating an intermediate image constituting an intermediate image set inserted in the middle of the first image set and a second image set newly extracted by the viewpoint image extracting unit according to a viewpoint change of the first image set ; And And an image output unit for outputting an image corresponding to the first or second image set and the intermediate image set. The multi-view image processing apparatus for displaying images for different viewpoints from multi-view image data. The multi-view image processing apparatus according to claim 1, further comprising a multi-view image input unit for receiving the multi-view image data. The apparatus of claim 1, wherein the multi-view image processing apparatus includes a viewpoint image disparity control unit configured to calculate horizontal parallax and vertical parallax with respect to the two or more images, and correct some of these images according to the calculated horizontal parallax and vertical parallax. A multi-point image processing apparatus further comprising. The method of claim 3, wherein the viewpoint image parallax adjustment unit A dynamic vector generator for extracting a dynamic vector from two or more input images; A vertical parallax calculation unit for obtaining the vertical parallax from an average vertical dynamic vector component of the dynamic vectors; A horizontal parallax calculation unit for obtaining the horizontal parallax from an average horizontal dynamic vector component of the dynamic vectors; And And a corrected image generation unit which moves one image by the vertical parallax and the horizontal parallax to obtain the corrected image. The method of claim 3, wherein the intermediate image generating unit An intermediate image number determining unit which determines the number of intermediate images between the first image set and the second image set; A disparity map generator for extracting pixel or block variation between an image belonging to the first image set and an image belonging to a second image set; A disparity map calculator for calculating a depth value from the disparity map generator; And And an intermediate image processor configured to apply the depth value to a perspective projection technique to generate an intermediate image suitable for the viewpoint of the intermediate image. 5. The multi-view image processing apparatus according to claim 4, wherein the extraction of the motion vector is performed by a block-based motion prediction method. 5. The multi-imaging image processing apparatus according to claim 4, wherein the extraction of the dynamic vector is performed by an optical flow method. A multi-view image processing method for displaying an image for a different viewpoint from multi-view image data, Extracting a first image set including two or more viewpoint images consecutive from the multi-view image data; Generating an intermediate image constituting the first image set and an intermediate image set inserted between the first image set and a second image set newly extracted according to a change of viewpoint of the first image set; And And outputting an image corresponding to the first or second image set and the intermediate image set. The multi-view image processing method for displaying an image for a different viewpoint from multi-view image data. 9. The method of claim 8, further comprising the step of receiving the multi-view image data. 10. The method of claim 9, wherein the multi-image processing method is Calculating horizontal parallax and vertical parallax for the two or more images; And And correcting some of these images in accordance with the calculated horizontal parallax and vertical parallax. The method of claim 10, wherein calculating the horizontal parallax and the vertical parallax Extracting a motion vector from two or more input images; Obtaining the horizontal parallax from an average horizontal dynamic vector component of the dynamic vectors; And Obtaining the vertical parallax from an average vertical dynamic vector component of the dynamic vectors, And modifying a part of the image comprises moving the image by the horizontal motion vector component and the vertical motion vector component. The method of claim 11, wherein the generating of the intermediate image comprises: Determining a number of intermediate pictures between the first and second picture sets; Generating a disparity map extracting disparity for each pixel or block between an image belonging to the first image set and an image belonging to a second image set; Calculating a depth value from the disparity map unit to obtain a depth map; And And applying the depth value to a perspective projection technique to generate an intermediate image suitable for the viewpoint of the intermediate image. 12. The method of claim 11, wherein the extraction of the motion vector is performed by a block-based motion prediction method. 12. The method of claim 11, wherein the extraction of the dynamic vector is performed by an optical flow method.

Images