KR101240497B1 - Method and apparatus for manufacturing multiview contents - Google Patents

Abstract

PURPOSE: A multi-viewpoint stereoscopic image manufacturing method and an apparatus thereof are provided to remarkably reduce time for hole reconstruction in conversion. CONSTITUTION: A multi-viewpoint stereoscopic image manufacturing apparatus performs an object rotoscoping process and a depth map process on an original image. The apparatus generates the last viewpoint image of the largest size having the greatest depth. After a hole reconstruction process on the last viewpoint image, the apparatus generates more than one middle viewpoint image. The apparatus performs an automatic hole reconstruction of the generated middle viewpoint image by using the last viewpoint image in which a hole reconstruction process is completed. [Reference numerals] (AA) Start; (BB) Performing an object rotoscoping process on an original image; (CC) Performing a depth map process; (DD) Generating the last viewpoint image with the greatest perceptional depth; (EE) Reconstructing holes in the last viewpoint image; (FF) Generating a middle viewpoint image by using the original image and the last viewpoint image; (GG) Reconstructing holes in the middle viewpoint image by using the last viewpoint image with the holes reconstructed; (HH) Finishing a multi-viewpoint image

Description

Method and apparatus for producing multiview stereoscopic images {Method and apparatus for manufacturing multiview contents}

The present invention relates to a method and apparatus for producing a multiview stereoscopic image, and more particularly, to produce an image of different parallaxes using a 2D image (image), the image having the largest depth (the movement of an object is the most Hole filling by generating a generated image, generating an image of an intermediate view using an original image and a depth map image, and generating an image of the generated intermediate view from the image of the last time of the hole reconstruction. The present invention relates to a method and apparatus for producing a multiview stereoscopic image, which can greatly reduce the time required for reconstructing a hole when generating a multiview image by automatically reconstructing a hole.

A multiview image refers to an image that provides a user with various viewpoints in various directions, and the multiview image has a feature of increasing freedom of view to a user.

Currently, multi-view images have limited use due to difficulty in production and are mainly used for lenticular images using software. A lenticular image converts a still image into several viewpoints and reconstructs it into a print / output. The lenticular image has a three-dimensional stereoscopic effect that can sense a sense of space and realism in a single image.

The conventional method of producing a multiview image, such as a lenticular image, is a method of performing hole filling on a multiview image generated after all images of various viewpoints are made through object rotoscoping and depth fab processes. Is used, and the process is as illustrated in FIG. For example, if you are making a 6-view image, go through the object rotoscoping process and the depth map process to create all 5-view images except the original, and then manually repeat the hole restoration for each multi-view image. do.

In the process of producing a multiview stereoscopic image as described above, when generating an image of different parallaxes using a 2D original image, an empty space generated when an object is moved appears. This is an empty space created when the invisible part becomes visible. To create another parallax image, the object of the original image is separated (rotoscoping), and each of the separated objects is moved (shifted) to the left or right according to the three-dimensional effect. do. As illustrated in FIGS. 2A and 2B, the images shifted left and right according to the three-dimensional effect are displayed in the form of a stretched or blank form because there is no image information as a background. Such a phenomenon that occurs in the form of sagging or blank is called a hole (also referred to as a hole or an art effect or a missing area), and is shifted (shifted) left and right as illustrated in FIGS. 3A and 3B. The work of restoring (correcting) the area of the hole portion generated in the hole is called hole filling (reole, clean up, gap filling, etc. according to the user).

However, most of these hole restoration processes are performed manually, and even in the case of multiview stereoscopic images, the number of viewpoints generated (n-1 for n time points; for example, for 6 time stereoscopic images, the original 1 time point image is used). Hall restoration process is performed manually by hand. In the hole restoration process, the hole portion must be restored by copying and pasting using image information of the place where the background image is used. Also, in case of video, if there is no information of background image to be used in the current frame, it is necessary to copy and use the background image to be moved to another frame, that is, before and after frame, or if there is no image information necessary for before and after frame, Workers must draw and restore themselves. Due to the difficulty of the manual work that needs to be elaborately performed, a lot of time and modification work must be carried out, resulting in a lot of work costs.

This method of reconstructing multiple image holes takes a lot of time and money to produce multi-view images, so some companies ignore the hole reconstruction process and deteriorate the quality of stereoscopic images.

The technical problem to be solved by the present invention is to create a depth map image by dividing the original image into a plurality of layers, applying a depth map to the separated layer, and applying the depth map image to the original image, which has the greatest depth. Restores the hole by generating an image of the viewpoint, generates an image of the intermediate viewpoint using the original image and the depth map image, and automatically restores the hole of the generated intermediate viewpoint using the image of the last viewpoint which was restored. It is to provide a multi-view stereoscopic image production method and apparatus that can significantly reduce the manpower and time required to restore a hole when generating a multi-view image to reduce the production cost.

According to one embodiment of the present invention for achieving the above object, in a multi-view stereoscopic image production method, (a) the object rotoscoping process and the depth map process is performed on the original image depth map operation is completed for the original image Generating a depth map image; (b) generating a last-view image having the greatest depth using the original image and the depth map image; (c) performing a hole restoration process on the last view image; (d) generating one or more intermediate view images using the original image and the depth map image, wherein the depth of the intermediate view image is set to have a smaller depth based on the depth applied to the last viewpoint image. being; (e) a method of producing a multiview stereoscopic image, wherein the generated intermediate view image is automatically reconstructed using a last view image at which the hole restoration process is completed.

Another embodiment of the present invention for achieving the above object is a depth map for generating a depth map image by dividing an original image into a plurality of layers and applying a depth map to the separated layer in a multi-view three-dimensional image production apparatus Generation unit 100; A last view image generator 200 generating the last view image having the greatest depth by applying the depth map image to the original image; A last view image hole restoration and correction unit 300 for applying the original image and the depth map image to restore the hole for the last view image; An intermediate view image generator 400 generating one or more intermediate view images using the original image and the depth map image, wherein the depth of the intermediate view image is smaller than that of the depth image applied to the last viewpoint image. Set to have a sense; And an intermediate viewpoint image hole restoration and correction unit 500 for restoring a hole for the intermediate viewpoint image by using the hole restored last viewpoint image.

The last view image may be any one selected from a last view image shifted to the left based on the original image, a last view image shifted to the right, and a last view image in each direction shifted in both directions. do.

The depth of the one or more intermediate view images may be set to be greater than the depth of the original view image and smaller than the depth of the last view image, and the depth of the original image, the depth of the one or more intermediate view images, and the depth of the last view image. It is characterized in that the depth difference between the senses is set to be equal.

According to the method of manufacturing a multiview stereoscopic image according to the present invention, when converting a 3D stereoscopic image to create a multiview stereoscopic image, the image is generated at the last point in which the most movement occurs and the hole restoration is performed, Since it is possible to reconstruct an intermediate view of a hole by applying the image, it is possible to drastically reduce the time required for reconstructing a hole when converting a multiview stereoscopic image. There is an advantage that can be lowered.

1 is an operation flowchart illustrating a multi-view stereoscopic image production process according to the prior art.
2 and 3 are reference pictures for explaining the hole and the hole restoration operation generated during the movement.
Figure 4 is a block diagram illustrating the configuration of a multi-view stereoscopic image production apparatus according to the present invention.
5 is an operation flowchart illustrating a multi-view stereoscopic image production process according to the present invention.
6A to 6C are reference pictures exemplarily illustrated to explain a hole restoration process according to a moving direction in the process of producing a multiview stereoscopic image according to the present invention.
7A to 7C are reference pictures exemplarily illustrated to describe a hole state according to a moving direction in the process of producing a multiview stereoscopic image according to the present invention.
8A to 8E are reference pictures exemplified for explaining a hole restoration process according to a moving direction in the process of producing a multiview stereoscopic image according to the present invention.
9A to 9G are reference pictures illustrating an image generated at each step in the process of producing a multiview stereoscopic image according to the present invention.
10 is a diagram for explaining a hole map.

Hereinafter, the structure and operation of a multi-view stereoscopic image production apparatus and a method thereof according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

The terms or words used in this specification and claims are not to be construed as limiting in their usual or dictionary meanings, and the inventors may properly define the concept of terms in order to best explain their invention in the best way possible. It should be interpreted as meaning and concept corresponding to the technical idea of the present invention. Therefore, since the embodiments shown in the specification and the configuration shown in the drawings is only one of the most preferred embodiment of the present invention, it is understood that there may be various equivalents and modifications that can replace them at the time of the present application shall.

4 is a block diagram illustrating a configuration of a multiview stereoscopic image producing apparatus according to the present invention. The multiview stereoscopic image producing apparatus of the present invention includes a depth map generator 100, a last view image generator 200, And a last view image hole restoration and correction unit 300, a mid view image generation unit 400, and a mid view image hole restoration and correction unit 500.

The depth map generator 100 separates (rotoscopes or masks) one original image into multiple layers and masks a depth map on each of the separated layers as illustrated in FIG. 8A through a mask operation. Create a depth map image as illustrated. This depth mapping is filled with gray color and the object in front is brighter and the object behind is darker.

The last view image generator 200 applies the depth map image generated by the depth map generator 100 to the original image, and as shown in the lower left photograph of FIG. (shift) produces the image of the last time point with the most. Here, the last view image is an image of the last time point left shifted (see FIGS. 6B and 7A) based on the original image, the image of the last time point shifted to the right (see FIGS. 6A and 7B), and a bidirectional shift (see FIG. 6C). It may be any one image selected from the images of the last viewpoint in each direction.

For example, when the image is shifted in either the left or right direction, the last image generated is one as illustrated in FIGS. 6A and 6B. When the image is shifted in both the left and right directions, the final image is illustrated in FIG. 6C. Likewise, since there exists one for each direction, there are two final images generated. When the shift (shift) operation is completed, one or more viewpoints are generated so that the actual stereoscopic image can be confirmed.

 When the final image is generated, the stereoscopic image can be confirmed.

For example, when the viewpoint n for making the 3D stereoscopic image of n viewpoints is set for the case where the last viewpoint image generator 200 is shifted in one of the left and right directions on the viewpoint generation option, the last viewpoint image generator 200 according to the set value Generates an image of the last (n-1) viewpoint for making a stereoscopic image of n viewpoints, and generates the image of the last viewpoint for each of both directions when the image is set for the case of shifting in both left and right directions. For example, when the view n is set to 6 on the view generation option, the last view image generator 200 generates a new image of the fifth view when shifted in either the left or right direction. In the case of shifting in both directions, an image of a second viewpoint to the left and a third to the right or a third to the left and a right is generated. The reason for generating a new image of the last viewpoint is that the image of the last viewpoint is stretched the most during the shift.

Meanwhile, in the present invention, the term "depth" is a term for expressing the numerical value of the distorted portion (hole) that occurs when the original image is shifted (shifted). Means that the size of is large.

Such a sense of depth can be expressed in units of pixels in an image. As a method of expressing a sense of depth, an integrated depth can be expressed as one unit or divided into two values, a depth value and a protrusion value.

The last view image hole restoration and correction unit 300 applies the original image and the depth map image to the image of the last view generated by the last (n-1) view image generator 200, and shows the image on the bottom right of FIG. 8C. As shown, the hole for the image of the last time point is restored.

The midpoint image generator 400 generates an image of one or more intermediate viewpoints as illustrated in FIG. 8D by using the original image and the depth map image. In this case, the intermediate view image generator 400 may be provided with an input interface means for inputting a view point n for making a stereoscopic image of n views on the original image, the depth map image, and the frame generation option. For example, when a 3D stereoscopic image of 6 views is to be made, the value of n may be set to 6, in which case, the intermediate view image generator 400 uses four original images and a depth map image to interpolate four intermediate images. The image of the viewpoint can be generated. If the left and right are shifted in both directions, three intermediate views are generated.

When the mid-view image is generated, the depth should be set for each image. The depth of the mid-view image is set larger than the depth of the original image, and smaller than the depth of the last view image, and the depth of the original image. The depth difference between the one or more intermediate view image depth and the last view image depth is preferably set to be equal.

For example, as shown in FIG. 9A, four intermediate view images are generated, and since the depth of the last view is set to 50 pixels, the depth of the intermediate view images is the original image depth. It is set to be larger than (0 pixels), smaller than the last depth of view (50 pixels), and set to have depths of 10 pixels, 20 pixels, 30 pixels, and 40 pixels so that the difference in depth of each image is even.

The mid-view image hole restoration and correction unit 500 holes the generated mid-view image as illustrated in FIG. 9D by using the image of the last viewpoint restored by the last view image hole restoration and correction unit 300. Restore

Referring to a method of restoring a hole of an intermediate view image, first, a hole area existing in the intermediate view image is automatically deleted (removed) using a hole map. In addition, the image of the last view of the hole reconstruction is positioned at the bottom of the intermediate view image from which the hole area is deleted. In this case, the portion where the hole area is deleted from the mid-view image is filled with the image of the last reconstructed view image. At this time, since the position of the last point of the restoration and the movement value of the intermediate image can be automatically derived using the depth map and the depth value of the entire image, the task of finding the position information of the image can be performed automatically without manual work. Do.

The hole map is generated along with the intermediate view image after the rendering process in which the image is shifted as shown in the right figure of FIG. 10, and refers to an image in which only a hole part is displayed in black on a white background. If only the black portion of the hole map is designated as the selection region and the selection region is removed from the generated mid-view image, the hole portion is deleted from the mid-view image, and all of these processes can be automatically implemented.

Another method of restoring the hole of the mid-view image is to first delete the hole area existing in the mid-view image. The region corresponding to the same position as the deleted hole region is extracted from the image of the last time of hole restoration. The hole restoration may be completed by covering the extracted hole area with a portion deleted from the intermediate view image.

5 is an operation flowchart illustrating a multi-view stereoscopic image production process according to the present invention, the multi-view stereoscopic image production method of the present invention is a depth map for the original image by performing an object rotoscoping process and a depth map process to the original image Generating the depth map image after the work is completed, generating the image of the last point of time having the greatest depth by applying the original image and the depth map image, and restoring the hole of the last point image by applying the original image and the depth map image. Generating an intermediate view image using the original image and the depth map image, wherein the depth of the intermediate view image is set to have a smaller depth based on the depth applied to the last view image. Automatically restoring the generated mid-point image using the last-restored image of the hole Achieved.

In the step of creating a depth map image in which the depth map operation is completed for the original image, a mask operation (Roto operation) separates the original image into several layers, and the depth map is applied to the separated layer by applying a depth map to the separated layer. Create an image.

In the step of generating the image of the last point of time with the greatest depth, the image of the last point of time with the highest depth of view, ie, the object shifted, is shifted to the left or right by applying the depth map image to the original image. Generates the image of the last time point that occurs. When shifted in either the left or right direction, one image of the last view is generated, and when shifted in both the left and right directions, an image of the last point in each direction is generated.

In the step of performing the hole restoration process for the image of the last viewpoint, the hole restoration is performed for the image of the last viewpoint to which the original image and the depth map image are applied.

In the image generation step of the intermediate view, the source image and the depth map image and the view point n are inputted through the input interface means, so that the image of the intermediate view, that is, the original image and the last view, is input using the original image and the depth map image. It is possible to generate an image of the intermediate view except for the image (n-2 if shifted in either the left or right direction and n-3 images when shifted in both the left and right directions).

In the automatic hole restoration step for the image of the mid-point view, the intermediate point (n-2 or n-3) images generated in the previous step are hall-restored using the image of the last time point that was restored. A method of setting a depth sense in generating a mid-view image and a method of performing hole reconstruction of the mid-view image are as described above.

6a to 6c are reference pictures exemplarily illustrated to explain a hole restoration process according to a moving direction in the process of producing a multiview stereoscopic image according to the present invention, and FIG. FIG. 6B is a diagram illustrating a last view image, FIG. 6B is a diagram illustrating a last view image, a mid-view image, and an original image when left shifted, and FIG. 6C is a view of two last views when a bidirectional (left and right) shift is performed. FIG. 9 is a diagram illustrating an image, an original image, and an intermediate view image therebetween.

7A to 7C are reference pictures exemplarily illustrated to explain a hole state according to a moving direction in the process of manufacturing a multiview stereoscopic image according to the present invention, and FIG. 7A illustrates an original image and a last view image when left shifted. FIG. 7B is a diagram illustrating an original image and a last viewpoint image when right shifted, and FIG. 7C is a diagram illustrating an original image and final viewpoint images in each direction when bidirectional (left and right) shifts. Drawing.

8A to 8E are reference pictures for explaining a hole restoration process according to a moving direction in the process of producing a multiview stereoscopic image according to the present invention, and FIG. 8A shows an image after the rotoscoping process is completed to the original image. 8B is a diagram illustrating an image after the rotoscoping process and the depth map process are completed in the original image. FIG. 8C is a final view image in which the original image and the hole are generated, and the last view image in which the hole is restored. FIG. 8D is a screen illustrating five different viewpoint images created by using an original image and a depth map image, and FIG. 8E is a view of the last extended point most increased among the five different viewpoint images illustrated in FIG. 8D. It is a figure which illustrates an image.

9A to 9G are reference pictures illustrating an image generated at each step in the process of producing a multiview stereoscopic image according to the present invention, and FIG. 9A is a diagram illustrating a distorted portion generated during image shift, and FIG. 9B. 9 is a diagram illustrating a state before and after a hole restoration for the last viewpoint image, and FIG. 9C is a diagram illustrating a state after correcting an image distortion portion of the last viewpoint image. 9E and 9F illustrate an operation of restoring an image distortion part of an intermediate view using a hole-restored last view image, FIG. 9G is a diagram illustrating a state after restoring an image of another view by using a restored portion of the hall-restored last view image.

The overall operation of the apparatus and method for producing a multiview stereoscopic image according to the present invention having such a configuration will be described.

For example, to create a 3D stereoscopic image of n = 6 viewpoints, first, the depth map generator 100 separates the image into multiple layers (rotoscoping or masking) as illustrated in FIG. 8A through a mask operation. )do.

In the next step, the depth map image is generated by applying a depth map to each layer separated through the roto work as illustrated in FIG. 8B.

Next, the last view image generator 200 applies the depth map image to the original image to generate an image of the last point of view that stretches the most when the image shifts, as illustrated in the lower left picture of FIG. 8C.

Thereafter, the last view image hole restoration and correction unit 300 performs a correction operation for restoring the hole for the image of the last view through a cleanup as illustrated in the lower right picture of FIG. 8C.

Next, the mid-view image generator 400 generates an image of the mid-view as illustrated in FIG. 8D by using the original image and the depth map image generated by the depth map generator 100.

Finally, the mid-view image hole restoration and correction unit 500 images the mid-view image generated by the mid-view image generation unit 400 and the last-view image of the last time the hole is restored by the mid-view image hole restoration and correction unit 300. Use to automatically restore the hole.

As described above, the present invention first generates an image of the last point in time when the movement of the most objects is generated when converting the 3D stereoscopic image to create a multi-view stereoscopic image, and then restores and corrects the hole, and the last point in which the movement is the most. Since the image of the intermediate view is automatically restored by applying the restoration and correction data for the image of the image, the time required for the post-correction operation during the conversion of the multi-view stereoscopic image can be greatly reduced.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, it is intended that the scope of the invention be defined by the claims appended hereto, and that all equivalent or equivalent variations thereof fall within the scope of the present invention.

100: depth map generator
200: last view image generator
300: last view image hole restoration and correction unit
400: mid-view image generator
500: mid-point image hole restoration and correction unit

Claims (6)

In the method of producing a multi-view stereoscopic image,
(a) generating a depth map image in which a depth map operation is completed on the original image by performing an object rotoscoping process and a depth map process on the original image;
(b) generating a last-view image having the greatest depth using the original image and the depth map image;
(c) performing a hole restoration process on the last view image;
(d) generating one or more intermediate view images using the original image and the depth map image, wherein the depth of the intermediate view image is set to have a smaller depth based on the depth applied to the last viewpoint image. being;
(e) automatically reconstructing the generated intermediate view image using the last view image at which the hole restoration process is completed;
Multi-view three-dimensional image production method characterized in that consisting of.
The method of claim 1,
In the step (b), the last view image,
And a last viewpoint image shifted to the left, a last viewpoint image shifted to the right, and a last viewpoint image in each direction shifted in both directions based on the original image.
The method of claim 1,
In the step (d) the depth of the one or more intermediate view image,
The depth of the original image is greater than the depth of the original image and smaller than the depth of the last view image, and the depth difference between the original image depth, at least one intermediate view image depth, and the last view image depth is set to be equal. Multi-view stereoscopic image production method.
In the multi-view stereoscopic image production apparatus,
A depth map generator 100 for separating the original image into a plurality of layers and applying a depth map to the separated layer to generate a depth map image;
A last view image generator 200 generating the last view image having the greatest depth by applying the depth map image to the original image;
A last view image hole restoration and correction unit 300 for applying the original image and the depth map image to restore the hole for the last view image;
An intermediate view image generator 400 generating one or more intermediate view images using the original image and the depth map image, wherein the depth of the intermediate view image is smaller than that of the depth image applied to the last viewpoint image. Set to have a sense; And
And a mid-view image hole restoration and correction unit (500) for restoring a hole for an intermediate view image by using the hole-restored last view image.
5. The method of claim 4,
The last view image generated by the last view image generation unit 200,
And a last viewpoint image shifted to the left, a last viewpoint image shifted to the right, and a last viewpoint image in each direction shifted in both directions, based on the original image.
5. The method of claim 4,
Depth of the one or more intermediate view image generated by the intermediate view image generator 400,
The depth of the original image is greater than the depth of the original image and smaller than the depth of the last view image, and the depth difference between the original image depth, at least one intermediate view image depth, and the last view image depth is set to be equal. Multi-view stereoscopic image production apparatus.

Images