KR20060036230A - 2d/3d converting method of compressed motion pictures using motion vectors therein, and device thereby - Google Patents

Abstract

The present invention relates to a method of converting a two-dimensional image to a three-dimensional image using a motion vector of a compressed video, the first step of extracting a motion vector of each block from the compressed video, and similar among the extracted motion vector A second step of extracting objects by grouping motion vectors, a third step of determining the perspective of each object from the movement of each extracted object, and a fourth step of separating the layers based on the determined perspective of each object And a fifth step of constructing a 3D image by sequentially overlapping the layers.

2D, 3D, Video, Motion Vector, Layer, Object, Depth, Parallax, MPEG, MOTION VECTOR, DISPARITY

Description

{2D / 3D CONVERTING METHOD OF COMPRESSED MOTION PICTURES USING MOTION VECTORS THEREIN, AND DEVICE THEREBY}

1 is an illustration of a 2D / 3D conversion apparatus according to the prior art 1.

2A is a block diagram of an MPEG decoder device for performing 2D / 3D conversion in accordance with a preferred embodiment of the present invention.

2B is a flow chart for each method of extracting an object from a motion vector according to a preferred embodiment of the present invention.

FIG. 2C illustrates a motion vector in connection with FIG. 2B. FIG.

FIG. 3A is a flow chart of a method of determining a layer for each object following the method of FIG. 2B according to a preferred embodiment of the present invention. FIG.

3B is an exemplary diagram for explaining a case where objects overlap in the method of FIG. 3A.

4A is a flow chart for each method of synthesizing 3D image data for each layer following the method of FIG. 3A according to a preferred embodiment of the present invention.

4B is an exemplary diagram for explaining a layer configuration procedure in the method of FIG. 4A.

5 is an exemplary diagram for describing a method of obtaining a motion vector using similarity of surrounding motion vectors, according to a preferred embodiment of the present invention.

The present invention relates to a method of converting a 2D image (2D) to a 3D image (3D), in particular, using a motion vector (Motion Vector) information included in a compressed video such as MPEG, H.264, etc. The present invention relates to a method of converting a two-dimensional image into a three-dimensional image by configuring a layer for each object.

With the advent of the high-definition interactive multimedia era, researches on the production technology of 3D stereoscopic images and the display device for displaying the same have been actively conducted in order to meet the needs of the visual information of the auditorium. In addition, studies on a method of converting a 2D image into a 3D image are also being conducted in consideration of the fact that most image contents are produced in a 2D form.

Sanyo's Korean Patent No.345630, "Method for Determining Before and After Positional Relationship of Subject and Converting 2D Image to 3D Image," as a representative technology of the 3D conversion method of 2D image, which is widely used (2002.11) .30 registration) (hereinafter referred to as "prior art 1") and Korean Patent No. 230447 "A device and method for converting two-dimensional continuous images to 3D image" (1999.8.23) (hereinafter referred to as "prior art 2") There is).

The prior art 1 is commonly referred to as a modified time difference (MTD) method, and is based on the theory that a three-dimensional image is felt by making an image of left and right views using a current image and a delayed image, respectively. That is, the MTD method obtains delay information using motion information from a compressed video, determines a delay range within the image, and then synthesizes a binocular image using a delayed image.

1 shows a configuration of a 2D / 3D converter according to the prior art 1.

Referring to FIG. 1, a normal two-dimensional image signal for displaying a two-dimensional image is input to an input terminal 1, and the two-dimensional image signal input to the input terminal 1 is an image switching circuit 2 and a field. It is supplied to the memory 5, respectively. The two-dimensional image signal input to the field memory 5 is output by delaying a predetermined number of fields and supplied to the image switching circuit 2. The delay of the field memory 5 is variably controlled by the memory control circuit 6 in units of one field. The image switching circuit 2 is connected to an output terminal 3 for outputting the left image signal L and an output terminal 4 for outputting the right image signal R, and controlled to switch the output state according to the same direction of the subject. .

In addition, the two-dimensional image signal input to the input terminal 1 is supplied to the moving vector detection circuit 7. The motion vector detection circuit 7 detects motion vectors (motion vectors) according to the movement between the image fields, that is, the movement amount (moving speed) of the subject. The detected copper vector is supplied to the CPU 8. The CPU 8 extracts a horizontal component of the detected motion vectors, and controls the memory control circuit 6 accordingly. That is, when the movement of the subject is large and the motion vector is large, the amount of delay of the field memory 5 is controlled to be small. When the movement of the subject is small or when the motion vector is small, such as during slow motion playback, the control is controlled to increase the amount of delay.

When the direction of the horizontal component of the vector is from left to right, the CPU 8 controls the image switching circuit 2 according to the detection target of the vector, so that the detection target of the vector is a subject in the front position of the background. When it is determined, the video switching circuit 2 is controlled so that the delayed video signal becomes the right eye video signal. On the other hand, when it is determined that the detection target of the vector is the background at the rear position of the subject, the video switching circuit 2 is controlled so that the delayed video signal is the left eye video signal. Therefore, for a scene in which the subject or the background moves in the left and right directions in the 2D image signal, parallax occurs in which the subject is always in the forward position with respect to the background due to the parallax according to the movement speed.

However, the above-described MTD method of the prior art 1 determines one frame among the previous K frames as the delayed image based on the motion information, so that, for example, the moving object may have a three-dimensional effect, but the motion may be reduced as in the background. There is a problem that can not feel the presence of a small object.

The prior art 2 improves the above-described disadvantages of the prior art 1, and does not generate a delay image from one frame before the current image, but generates a delay image by synthesizing block data from a plurality of previous frames for each block.

That is, the prior art 1 and the prior art 2 basically converts a 2D image into a 3D image using a delay parallax. Since the conventional conversion method using the delay parallax determines the delay range and classifies the left and right images accordingly, the delay range or the similarity between the delayed image and the current image changes according to the change of motion, so that the actual observer feels very tired. There is a problem that the three-dimensional feeling is significantly reduced. In addition, the conventional MTD method has a disadvantage in that a memory size required for implementation is increased because a plurality of previous frames must be stored in a frame memory to determine a delay image.

In addition, according to the related art, in order to reproduce a 3D image, an apparatus for acquiring a 3D image, a compression algorithm, and a new storage method are required, and thus there is a problem that additional hardware must be provided.

In order to solve the above problems, an object of the present invention is to enable the conversion of a two-dimensional image to a three-dimensional image in a conventional two-dimensional image player without the addition of hardware.

In order to achieve the above object, according to an aspect of the present invention, there is provided a method for converting a two-dimensional image to a three-dimensional image using a motion vector of the compressed video, extracting the motion vector of each block from the compressed video A second step of extracting an object by grouping similar motion vectors among the extracted motion vectors, a third step of determining the perspective of each object from the movements of the extracted objects, and the determined A fourth step of separating the layers based on the perspective of each object, and a fifth step of forming a three-dimensional image by sequentially overlapping the layers.

In this case, the first step may include setting the motion vectors of the neighboring blocks already extracted as the motion vectors of the current block.

The second step may include removing a noise error by using a displacement of a motion vector of a neighboring block in the group of similar motion vectors, and setting a group of motion vectors from which the noise error is removed as an inner region of an object. The method may include setting an outline of an object by using boundary area data of the 2D image.

The fourth step may include setting a depth value of each object corresponding to the determined perspective of each object, and separating the extracted objects into respective layers corresponding to the set depth value. In the setting of the depth value, it is preferable to assign a representative depth value to objects having a slight difference in depth.

 Finally, the fifth step may include configuring a background screen using background data, and sequentially overlapping the layer on the background screen according to a depth value. Compensating for the blank area of the background screen generated by the movement of the object by using the background data of the previous frame image.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 2A illustrates a configuration of an MPEG (MPEG) decoder device capable of 2D / 3D conversion according to a preferred embodiment of the present invention, and is quantized from an encoded bitstream of MPEG compressed data. An entropy decoder 110 that extracts a quantized transform coefficient, a motion vector (MV), and the like, and an quantized transform coefficient to dequantize and inverse discrete cosine transform (IDCT) to reconstruct an error image. A motion compensator for reconstructing the motion vector using the inverse quantization and inverse discrete cosine transform unit 120, the motion vector MV output from the entropy decoder 110, and a previous image frame stored in a frame buffer to be described later ( 130, an adder 140 for restoring a left image frame by adding the error image and the motion vector reconstructed from the motion compensator 130, and For the restoration of the video frame consists of a frame buffer 150 for storing the left image frame outputted from the addition unit 140.

The content of compressed two-dimensional video data generally has reference image data, and based on this data, several sequential frames are processed by reconstructing the image reconstructed by using the motion vector information and the motion vector from the original image. It consists of data. That is, the reference image data is usually called an I frame, and the image data constructed using the reference image is called a P or B frame. In the case of the P frame and the B frame, the video may be reconstructed from the motion vector information and the difference image data according to the structure of the MPEG decoder described above.

Referring back to FIG. 2A, the MPEG decoder apparatus includes a right image restoration unit 200 for restoring a right image, and the right image restoration unit 200 is output from the entropy decoder 110 described above. An object extracting unit 210 for extracting an object from a previous image frame stored in the motion vector MV and the frame buffer 150, a motion determining unit 220 for determining a motion for each extracted object, and the motion plate Per-object perspective determination unit 230 that determines the perspective of each object according to the object-specific movement determined by the government, and an object that separates each object by layer according to the perspective information of each object determined by the per-object perspective determination unit. The image layer synthesizing unit 250 configured to sequentially combine each layer determination unit 240 and each layer separated from the object layer determination unit to form 3D image data and output as a right image. It is configured.

In relation to the function of the object extractor 210, FIG. 2B illustrates a method of extracting an object from a motion vector by the object extractor 210 of FIG. 2A.

Referring to FIG. 2B, first, a motion vector is extracted from information reconstructed from the entropy decoder 110 of FIG. 2A (S260). In this case, by using the correlation vector and the motion vector of the neighboring block, the current motion vector may be extracted with the minimum operation by setting the motion vectors of the neighboring blocks extracted as the virtual motion vectors of the current block.

Meanwhile, a compressed 2D video has motion vector information of various sizes, ranging from 16x16 pixel macroblocks to 4x4 size blocks according to each standard, and a motion vector reconstructed from an MPEG compressed image is different from an optical flow. Can have different values. That is, in order to reduce the bitrate of the compressed image, an algorithm for searching for a block that is most suitable for extracting a motion vector during encoding of the compressed image may find an error using a constant function rather than finding the actual motion. A small block can be searched to determine the motion vector. Therefore, some motion vectors may have a direction that is independent of the actual motions without following the actual motions. Thus, when the motion vectors reconstructed from the compressed image are used as they are, deterioration of image quality occurs. In order to solve this problem, steps S270 to S290 subsequent to the above-described step S260 are performed.

Step S270 groups motion vectors having similar displacements among the motion vectors extracted in the aforementioned step S260 for each region. That is, by observing the displacement of the motion vector extracted from the compressed video, it is possible to group similar displacement vectors corresponding to the same object (object, background, etc.).

Next, in step S280, a motion vector having a completely different value when compared with the motion vector of the neighboring block is defined as noise, and the noise vector is corrected using the neighboring vectors to remove the noise error.

The motion vectors of each group classified in steps S270 and S280 and corrected for noise errors are assumed to be the approximate interior of the object, and in step S290 the boundary region of the object in the previous image frame stored in the frame buffer 150. By determining the data (contour of the object), finally the correct object is extracted (S300).

As such, when an object is constructed according to the magnitude of the motion using the motion vectors, the object data is composed of an aggregate of square shapes according to the size of the vector constituting the motion vector, not the object region. As described above, the incomplete object data constructed by using the motion vector has an accurate outline of the object by using the boundary region data of the 2D reconstructed image.

For example, in FIG. 2C, since the motion vectors (indicated by arrows) inside the objects A and B have similar vector values, the objects A and B may be determined by grouping vectors of similar displacements as described above. And, more preferably, by using the contour information of the object A and the object B, these objects can be extracted more accurately.

FIG. 3A illustrates a procedure of a method of determining a layer for each object following the method of FIG. 2B, and FIG. 3B is an exemplary diagram for describing perspective determination when objects overlap in the method of FIG. 3A.

Referring to FIG. 3A, step S300 extracts an object from an image according to the method of FIG. 2B, determines the movement of each extracted object (S310), and determines the motion of each object according to the motion parallax of the determined object. Perspective is determined (S320). In general, an image is composed of a background and a foreground, and in the case of a background image, a parallax of an image coming into a left and right view of a person is very small. Also, the closer the object is to the foreground, the greater the parallax and the less the parallax between the nearest foreground and the background. Therefore, using this property, it is possible to determine the perspective of each object according to the motion parallax of each object.

In operation S330, since the objects extracted in the above-described steps exist in the image, the layers are classified and classified according to depth information corresponding to the perspective of each object. On the other hand, if there is only a slight difference in the depth of the object is not easy to distinguish, if there are a variety of objects in the image does not constitute the depth of the object for each object, representative for the objects with a small difference in depth Depth can be assigned to a certain number of layers.

As shown in FIG. 3B, since the objects A and B exist at different positions in the previous frame and overlap each other in the current image frame, it is not easy to determine perspective using the conventional method. However, in the above-described method of FIG. 3B, since the objects are overlapped after the movement of each object occurs, the perspective of the object may be determined using the information moving backward or forward of each object from the movement of each object. have. In the case of FIG. 3B, it is determined that the object A is located ahead of the object B.

4A is a flowchart illustrating a method of synthesizing 3D image data for each layer subsequent to the method of FIG. 3A according to a preferred embodiment of the present invention. FIG. 4B is a view for explaining a layer configuration procedure in the method of FIG. 4A. It is an illustration.

As shown in FIG. 4A, first, a background screen is configured using background data and temporal data (S400). Due to the movement of the object, since no background data exists in the region where the object is previously located, a complete background screen may be configured by filling the image part using temporal data. Here, the temporal data means another frame that is not current in time. In the prior art, the entire frame is taken from a past frame in time, and according to the present invention, instead of obtaining the entire background data, only the vector from which the motion of the block originates is obtained by using the motion vector for the empty space part. Will fill.

Subsequently, the layers are sequentially synthesized from the lowest layer farthest in accordance with the above-described depth value (S410), and the superimposed three-dimensional images for each layer are synthesized (S420).

Referring to FIG. 4B, a background of a parallelogram is first constructed. Subsequently, a rectangular object B is formed on the background image, and the closest circular object A is formed. Thus, the closest synthesized object is displayed in the overlapping region of each object.

FIG. 5 is an exemplary diagram for describing a method of obtaining a motion vector using similarity of surrounding motion vectors according to a preferred embodiment of the present invention.

For example, in MPEG2, blocks may be configured in macroblock units, in MPEG4, blocks may be configured in 8 ㅧ 8 units and in H.264 in finer 4 H4 units. Construct a motion vector. When the motion vector is extracted, the blocks are reproduced from the raster scan direction, that is, from left to right and from top to bottom, so that the upper line block and the left block of the block from which the current motion vector is to be extracted are already reproduced. However, as described above, since motion vectors in the object have similar directionality and magnitude, when the motion vector of the current block is obtained by using the motion vector of the neighboring blocks already reproduced when constructing the compressed image, the correct parallax is calculated in the object. can do.

Referring to FIG. 5, an upper block T (meaning Top) of a block C to which the current motion vector is to be extracted, a block TR of the right side of the upper block (meaning Top Right), and a block ( The motion vectors of the left block L (meaning Left) of C) are represented by DV _T , DV _TR , and DV _L , respectively, to calculate disparity. At this time, these three blocks (T, TR, L) have obtained a motion vector before the reproduction of the current block (C) and have already been reproduced. The motion vectors of these neighboring blocks (T, TR, L) have similar directionality. Since it has the size and, it will be classified as the same object. Therefore, when the current blocks C are classified into the same object area as those surrounding blocks according to the above-described method, the parallax of the current blocks can be rapidly increased by using the motion vectors DV _T , DV _TR , and DV _L of these surrounding blocks. Can be determined.

As described above, in order to convert a 2D image into a 3D image, a motion vector is extracted from a conventional compression decoder, which may be provided in a conventional 2D video decoder, and thus 2D / 3D conversion according to the present invention. The function can be installed in the form of a software algorithm in the existing two-dimensional video player without adding additional hardware equipment. Therefore, the 2D / 3D conversion apparatus according to the present invention can be implemented as a device for converting a 3D image by connecting to a conventional 2D video player and, if necessary, according to the present invention in a conventional 2D video player. It can be implemented by adding a 2D / 3D conversion algorithm.

Although the preferred embodiment according to the present invention has been described above, this is merely exemplary and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the protection scope of the present invention should be defined by the following claims.

As described above, according to the present invention, since the present invention can be directly converted using the existing 2D video reproducing apparatus, it can be utilized in all conventional 2D video reproducing apparatuses.

Claims (18)

A method of converting a 2D image into a 3D image using a motion vector of a compressed video, Extracting a motion vector of each block from the compressed video; A second step of extracting an object by grouping similar motion vectors among the extracted motion vectors; A third step of determining the perspective of each object from the movement of each extracted object; Dividing the layer based on the perspective of each of the determined objects; A fifth step of forming a 3D image by sequentially overlapping the layers 3D image conversion method of a 2D image comprising a. The method of claim 1, wherein the first step Setting motion vectors of neighboring blocks already extracted as motion vectors of the current block 3D image conversion method of a 2D image comprising a. The method of claim 1, wherein the second step, Removing noise errors using the displacement of the motion vectors of neighboring blocks in the group of similar motion vectors; Setting a group of motion vectors from which the noise error is removed to an internal region of an object, and setting an outline of the object by using boundary region data of the 2D image 3D image conversion method of a 2D image comprising a. The method of claim 1, wherein the fourth step is Setting a depth value of each object corresponding to the determined perspective of each object; Dividing the extracted objects into respective layers corresponding to the set depth value 3D image conversion method of a 2D image comprising a. The method of claim 4, wherein the setting of the depth value provides a representative depth value. The method of claim 4, wherein the fifth step, Configuring the wallpaper using the background data; Sequentially superimposing the layers on the background screen according to a depth value 3D image conversion method of a 2D image comprising a. The method of claim 6, wherein the setting of the wallpaper is performed. Compensating the blank area of the background screen caused by the movement of the object by using the background data of the previous frame image 3D image conversion method of a 2D image comprising a. According to any one of claims 1 to 3, wherein the fifth step, Configuring the wallpaper using the background data; Sequentially superimposing the layers on the background screen according to a depth value 3D image conversion method of a 2D image comprising a. The method of claim 8, wherein the configuring of the wallpaper is performed. Compensating the blank area of the background screen caused by the movement of the object by using the background data of the previous frame image 3D image conversion method of a 2D image comprising a. An apparatus for converting a 2D image into a 3D image using a motion vector of a compressed video, A motion vector extractor which extracts a motion vector of each block from the compressed video; An object extractor which extracts an object by grouping similar motion vectors among the extracted motion vectors; Perspective determination unit for determining the perspective of each object from the movement of the extracted object, A layer determination unit that separates layers based on perspectives of the determined objects; An image synthesizer configured to sequentially overlap the layers to form a 3D image 3D image conversion apparatus of a 2D image comprising a. The method of claim 10, wherein the motion vector extractor 3D image conversion apparatus of a 2D image to set the motion vectors of the neighboring block already extracted as the motion vector of the current block. The method of claim 10, wherein the object extraction unit, Remove the noise error using the displacement of the motion vector of the neighboring block in the group of similar motion vectors, A group of motion vectors from which the noise error is removed is set as an internal region of an object. 3D image conversion apparatus for a two-dimensional image to set the contour of the object by using the boundary region data of the two-dimensional image. The method of claim 10, wherein the perspective determination unit, Setting a depth value of each object in correspondence with the determined perspective of each object, And dividing the extracted objects into respective layers corresponding to the set depth value. The 3D image conversion apparatus of claim 13, wherein the set depth value is a representative depth value. The method of claim 13, wherein the layer determination unit, Compose a wallpaper using background data, 3D image conversion apparatus for a two-dimensional image to sequentially overlap the layer on the background screen according to the depth value. The method of claim 15, wherein the layer determination unit, 3D image conversion apparatus for a 2D image by using the background data of the previous frame image, to compensate for the blank area of the background screen caused by the movement of the object. The method according to any one of claims 10 to 12, wherein the layer determining unit, Compose a wallpaper using background data, 3D image conversion apparatus for a two-dimensional image to sequentially overlap the layer on the background screen according to the depth value. The method of claim 17, wherein the layer determination unit, 3D image conversion apparatus for a 2D image by using the background data of the previous frame image, to compensate for the blank area of the background screen caused by the movement of the object.

Images