KR20070033863A - A parallax vector prediction method, a method and apparatus for encoding and decoding a multiview video using the method - Google Patents

Abstract

The present invention relates to a method and apparatus for encoding and decoding a multiview video. A method of predicting a parallax vector for encoding a multi-view video of the present invention includes: predicting a parallax vector for a predetermined number of pre-coded macroblocks; Calculating a disparity vector of macroblocks located adjacent to the encoded macroblocks using the predicted disparity vector. According to the present invention, it is possible to quickly encode a multiview video and to improve a compression ratio.

Multiview video, parallax vector, interpolation, prediction, macroblock

Description

Parallax vector prediction method, method and apparatus for encoding and decoding a multiview video using the method {method for interpolating disparity vector and method and apparatus for encoding and decoding multi-view video}

1 illustrates a stereo video encoder and decoder using an MPEG-2 multi-view profile.

FIG. 2 is a diagram illustrating prediction encoding considering only parallax using two parallax predictions for bidirectional prediction. FIG.

3 is a diagram illustrating predictive coding using parallax and motion vectors for bidirectional prediction.

4 is a diagram for explaining a direct mode of a B picture.

5 is a diagram for explaining a method of predicting a motion vector in a spatial domain.

6 is a block diagram illustrating an internal configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

7 shows a frame sequence photographed by a multiview camera.

8 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to an embodiment of the present invention.

FIG. 9 is a diagram illustrating a relationship between a parallax vector when the parallax vector is constant as shown in FIG. 8. FIG.

FIG. 10 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to another embodiment of the present invention. FIG.

FIG. 11 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to another embodiment of the present invention. FIG.

12 is a diagram for explaining a method for predicting a parallax vector, according to an embodiment of the present invention.

13 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to another embodiment of the present invention.

14 is a flowchart illustrating a multiview video encoding method according to another embodiment of the present invention.

15 is a block diagram illustrating an apparatus for decoding a multiview video according to an embodiment of the present invention.

The present invention relates to a method and apparatus for encoding and decoding a multiview video, comprising: a parallax vector prediction method for quickly performing multiview video encoding and improving a compression ratio, and a multiview video using a parallax vector prediction method. A method and apparatus for encoding / decoding.

One of the most ideal characteristics for realizing high quality information and communication services is reality. This can be accomplished by video communication based on three-dimensional images. 3D imaging systems can be used in many potential applications such as education, entertainment, medical surgery, video conferencing, and the like. In order to convey more vivid and accurate information about the scene at a remote location to multiple viewers, three or more cameras are placed at slightly different views to create a multiview sequence.

Interest in the field of 3D imaging has led many research groups to report on 3D image processing and display systems. In Europe, research on 3DTV has been initiated by several projects, such as DISTIMA, with the aim of developing a system for capturing, encoding, transmitting and displaying digital stereo image sequences. These projects led PANORAMA, another project aimed at improving visual information in communications with 3D telepresence. Currently, these projects have led another project ATTEST, in which technologies in various fields of 3D content acquisition, 3D compression and transmission, and 3D display systems have been studied. In this project, MPEG-2 and DVB (Digital Video Broadcasting) systems are specifically designed for time scalability, using a base layer for transmitting 2D content and an enhancement layer for transmitting 3D deep data. (Temporal Scalibility, TS) has been applied to transmit 3D content.

In MPEG-2, Multiview Profile (MVP) was defined in 1996 as a modification to the MPEG-2 standard. Important new components of MVP are the definition of the use of Time Extensibility (TS) mode for multi-camera sequences and the definition of acquisition camera parameters in MPEG-2 syntax.

Encoding an enhancement layer that can be used to encode a base layer stream representing a signal at a reduced frame rate and insert additional frames in the middle to allow playback at full frame rate if the two streams are valid. It is possible to define. An efficient way to encode an enhancement layer is to allow for each macroblock in an enhancement layer frame to make a decision regarding the best motion compensation prediction from a base layer frame or a recently reconstructed enhancement layer frame.

For such signals, it is simple to perform stereo and multi-view channel coding using temporal scalability syntax. For this purpose, a frame from one camera viewpoint (usually a frame in the left eye) is defined as an enhancement layer and a frame from the other is defined as an enhancement layer. The base layer represents a simultaneous monoscopic sequence. For the enhancement layer, disparity compensation prediction may fail in an occluded region, but may continue to maintain the image quality reconstructed by motion compensation prediction within the same channel. Since MPEG-2 MVP is mainly defined as a stereo sequence, it does not support a multiview sequence, and it is difficult to extend it to a multiview sequence inherently.

1 is a diagram illustrating an encoder and a decoder of an MPEG-2 multi-view profile.

Referring to FIG. 1, in the MPEG-2 multi-view profile (13818-2), a correlation between two images is found when encoding a 3D image using a left view image and a right view image. It can be seen that a 3D image is encoded and reproduced using a scalable codec that variably encodes a difference between two images according to a network state. In this case, the left image is encoded as a base layer video and the right image is encoded as an enhancement layer video. The base layer image is an image which can be encoded by itself, and the enhancement layer image is an image that is additionally encoded and sent when the network is in good condition in order to improve the quality of the base layer image. The encoding using both the base layer image and the enhancement layer image is called scalable coding.

The left view image is encoded by a first motion compensated DCT encoder 110. In the right view image, the difference between the two images calculated by the disparity estimator 122 and the disparity compensator 124 predicting the difference from the left view image is determined by the second motion compensation DCT encoder 126. Is encoded. When the first motion compensation DCT encoder 110 encoding a left view image is a base layer image encoder, a disparity predictor 122, a disparity compensator 124, and a second that encode a disparity between a right view image and a left view image. The motion compensation DCT encoder 126 may be referred to as an enhancement layer image encoder 120. The encoded base layer image and enhancement layer image are multiplexed by a system multiplexer 130 and transmitted to the decoder.

The multiplexed signal is divided into a left view image and a right view image by a system demultiplexer 140. The left view image is decoded by a first motion compensated DCT decoder 150. The parallax image is reconstructed into a right viewpoint image by a disparity compensator 162 and a second motion compensation DCT decoder 164 that compensate for parallax with a left viewpoint image. When the first motion compensation DCT decoder 150 for decoding a left view image is a base layer image decoder, a parallax compensator 162 for decoding a right view image by finding a parallax between a right view image and a left view image The two motion compensation DCT decoder 164 may be referred to as an enhancement layer image decoder 160.

FIG. 2 is a diagram illustrating prediction encoding considering only parallax using two parallax predictions for bidirectional prediction.

The left image is encoded using a non-scalable MPEG-2 encoder, and the right image is encoded using an MPEG-2 temporally located auxiliary view encoder based on the decoded left image.

That is, the right picture is encoded into a B picture using two reference pictures, for example, a prediction obtained from the left picture. In this case, one of the two reference images is the left image to be displayed at the same time, and the other is the left image to be next shown in time.

The two predictions have three prediction modes: forward, reverse, and bidirectional, similar to motion prediction / compensation. Here, the forward mode refers to the time difference predicted from the left image at the same time, and the reverse mode refers to the time difference predicted from the next left image. In this method, since the prediction of the right image is performed through the disparity vectors of the two left images, this type of prediction method is called predictive coding considering only the disparity vector. Therefore, the encoder predicts two parallax vectors for each frame of the right video, and the decoder decodes the right video from the left video using the two disparity vectors.

3 is a diagram illustrating prediction coding using disparity vectors and motion vectors for bidirectional prediction.

FIG. 3 uses the B picture with bi-prediction shown in FIG. 2, but bi-prediction uses one parallax prediction and one motion estimation. That is, the parallax prediction from the left image of one time zone and the motion prediction from the right image of the immediately preceding time are used.

Like predictive coding considering only parallax, bidirectional prediction includes three prediction modes called forward, reverse, and interpolated modes. Here, the forward mode means motion prediction from the decoded right image, and the reverse mode means disparity prediction from the decoded left image.

As mentioned above, the specification of the MPEG-2 multi-view profile itself does not take into consideration the encoder for multi-view video, and is not designed to be suitable for real stereo video, thereby providing three-dimensional and realism to many people at the same time. There is a need for an encoder that can efficiently provide multi-view video.

Recently, the new H.264 video coding standard has attracted attention due to the higher coding efficiency than the previous standard. This new standard not only takes into account common bidirectional (B) prediction slices, but also has variable block sizes ranging from 16x16 to 4x4, quadtree structure, multiple reference frames, intra for motion compensation in loop deblocking filters. It relies on several new features, such as considering prediction, context-adaptive entropy coding. Unlike standards such as MPEG-2 and MPEG-4 Part 2, B slices use multiple predictions coming from the same direction (forward or reverse), while they can be used with reference to other slices. However, the features described above require a fairly high rate of bits for motion information in the prediction mode for this standard.

To alleviate this problem, skip (SKIP) and direct (DIRECT) modes were introduced for the P slice and the B slice, respectively. These modes enable the prediction of the motion of any block of the picture to be currently encoded using previously encoded motion vector information. Thus, no additional motion data for the macroblock or block is encoded. The motion for these modes is obtained using the spatial (SKIP) or temporal (DIRECT) relevance of the motion of the adjacent MB or picture.

4 is a diagram for explaining a direct mode of a B picture.

In the direct mode, in order to predict the motion of an arbitrary block of a B picture to be currently encoded, a forward motion vector and a backward motion vector are obtained using a motion vector of a co-located block of a P picture, which is the next picture in time. To save.

Direct in reference list 1 picture 420, which is the next picture in time, to calculate the forward motion vector MV _LO and reverse motion vector MV _L1 of the direct mode block 402 whose motion is to be predicted in the B picture 410. A co-located block 404, which is a block at the same location as the mode block 402, finds a motion vector MV for the reference list 0 picture 430 referenced by the motion vector. Then, the forward motion vector MV _LO and the reverse motion vector MV _L1 of the direct mode block 402 of the B picture 410 are calculated by the following equation.

[Equation 1]

Here, MV represents a motion vector of the corresponding block 404 of the reference list 1 picture 420. TD _D indicates the distance between the reference list 0 picture 430 and the reference list 1 picture 420, and TD _B indicates the distance between the B picture 410 and the reference list picture 0 430.

5 is a diagram illustrating a method of predicting a motion vector in a spatial domain.

According to the H.264 standard used for encoding video data, a frame is divided into blocks having a predetermined size, and a motion search is performed to search for the most similar block by referring to an adjacent frame that has already been encoded. That is, among the motion vectors of the three macroblocks of the left (4), the top (2), and the top right (3) of the current macroblock (c), the intermediate value is determined as the predicted value of the motion vector. This motion vector prediction can be represented by Equation 2.

[Equation 2]

As described above, a method of encoding a video using spatial relevance as well as temporal relevance has been proposed, and a method of improving compression rate and processing speed is required for a multi-view video having more information than general video.

The technical problem to be achieved by the present invention, multiview video encoding to improve the compression ratio of the multiview video and to perform multiview video encoding quickly by using the relationship between the parallax vector for the multiview image captured by the multiview camera A method and apparatus are provided.

Another object of the present invention is to provide a multi-view video decoding method and apparatus for decoding a multi-view video encoded by using the correlation between the disparity vectors for the multi-view video.

In order to solve the above technical problem, a method for predicting a parallax vector for encoding a multi-view video according to an aspect of the present invention, comprising: predicting a parallax vector for a predetermined number of encoded macro blocks; Calculating a disparity vector of macroblocks located adjacent to the encoded macroblocks using the predicted disparity vector.

Advantageously, predicting the parallax vector comprises predicting a parallax vector for a predetermined number of encoded macroblocks by a full search method or a fast search method, and calculating the parallax vector comprises: Interpolating and calculating a parallax vector of macroblocks located adjacent to coded macroblocks using an interpolation equation of.

Preferably, the predetermined interpolation equation is selected according to the properties of the disparity vectors determined for the encoded macroblocks, the properties of the disparity vectors comprising whether the deviation of the predicted disparity vectors is constant.

In order to solve the above technical problem, a multi-view video encoding apparatus according to another aspect of the present invention receives a current frame and a reference frame, predicts a parallax vector for a predetermined number of encoded macroblocks, and predicts A direct mode performing unit configured to calculate parallax vectors of macroblocks located adjacent to macroblocks encoded using the parallax vector; An interpolation expression providing unit which provides an interpolation equation for calculating a parallax vector of macroblocks located adjacent to the encoded macroblocks to a direct mode performing unit; A search mode performing unit configured to receive a current frame and a reference frame and search for frames to predict a parallax vector; A cost calculator configured to calculate a cost according to the parallax vector determination method performed by the direct mode performer and the search mode performer, respectively; A comparison unit comparing the calculated cost; A mode selection unit for selecting an encoding mode according to the comparison result; And an encoder configured to encode a multiview video according to the selected encoding mode.

Preferably, the direct mode performing unit predicts a parallax vector for a predetermined number of coded macroblocks by a full search method or a fast search method, and positions adjacent to the coded macroblocks using a predetermined interpolation equation. Calculate by interpolating the parallax vector of the macroblocks.

Preferably, the interpolation provider stores at least one interpolation expression that can be selected according to the characteristics of the disparity vectors predicted for a predetermined number of encoded macroblocks.

Preferably, the cost calculator may include a direct mode cost calculator configured to calculate a direct mode cost by using the parallax vector DV_d calculated by the direct mode performer; And a search mode cost calculator configured to calculate a search mode cost by using the disparity vector DV_f predicted by the search mode performer.

Preferably, the cost calculator calculates the non-trace and / or PSNR for each of the direct mode and the seek mode.

Preferably, the comparison unit compares the difference between the direct mode cost and the search mode cost with a predetermined threshold value, and the mode selection unit sets a flag according to the comparison result and selects an encoding mode according to the set flag.

Preferably, when the difference between the direct mode cost and the search mode cost is greater than or equal to a predetermined threshold, the comparator determines a difference between the disparity vector DV_d determined according to the direct mode and the disparity vector DV_f determined according to the search mode. In comparison with, the mode selector sets a flag according to an encoding mode according to the comparison result, and selects an encoding mode according to the set flag.

Preferably, the encoding unit encodes the residual image when the selected encoding mode is the direct mode, the selected encoding mode is the search mode, and the disparity vector DV_d determined according to the direct mode and the disparity vector DV_f determined according to the search mode. Is smaller than a predetermined threshold, the difference between the parallax vector DV_d determined according to the residual image and the direct mode and the parallax vector DV_f determined according to the search mode is encoded, and the selected encoding mode is a search mode and direct When the difference between the parallax vector DV_d determined according to the mode and the parallax vector DV_f determined according to the search mode is greater than or equal to a predetermined threshold, the parallax vector according to the residual image and the search mode is encoded.

In order to solve the above technical problem, a multi-view video encoding method according to another aspect of the present invention, receiving a current frame and a reference frame; Performing a direct mode for predicting disparity vectors for a predetermined number of encoded macroblocks and calculating disparity vectors of macroblocks located adjacent to encoded macroblocks using the predicted disparity vector; Performing a search mode for searching a current frame and a reference frame to predict a parallax vector; Calculating a cost of performing a direct mode and a cost of performing a search mode, respectively; Comparing the calculated cost; Selecting an encoding mode according to the comparison result; And encoding the multiview video according to the selected encoding mode.

In order to solve the above technical problem, the multi-view video decoding apparatus according to another aspect of the present invention, the encoding mode check to confirm the encoding mode by checking the flag indicating the encoding mode included in the received multi-view video bitstream part; A parallax vector determiner for determining a parallax vector according to the identified encoding mode; A decoding unit that decodes a multiview video using the disparity determined in the identified encoding mode and the residual image data included in the multiview video bitstream, and the disparity vector determining unit includes the same parallax interpolation equation as that of the multiview video encoding apparatus Calculate the parallax vector using

Preferably, when the encoding mode confirmed by the mode confirming unit is the first mode in which the residual image is received, the parallax vector determiner calculates the parallax using a parallax interpolation equation, and the encoding mode confirmed by the mode confirming unit is the residual image. And a second mode in which a difference value between the parallax vector DV_d determined according to the direct mode and the parallax vector DV_f determined according to the search mode is transmitted, the parallax vector determiner calculated using the difference value and the parallax interpolation equation. Compute the parallax vector by adding the parallaxes.

In order to solve the above technical problem, a multi-view video decoding method according to another aspect of the present invention comprises: checking an encoding mode by checking flags indicating an encoding mode included in a received multi-view video bitstream; Determining a parallax vector according to the identified encoding mode; And decoding the multi-view video using the residual image data included in the disparity and multi-view video bitstream determined according to the identified encoding mode, and determining the disparity vector comprises: a multi-view video encoding apparatus; Calculating a parallax vector using the same parallax interpolation equation.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

6 is a block diagram illustrating an internal configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

A multiview video encoding apparatus according to an embodiment of the present invention includes a multiview image buffer 610, a predictor 620, a parallax / motion compensation unit 630, a difference image encoder 640, and an entropy encoder 650. ).

In FIG. 6, the proposed multi-view video encoding apparatus receives a multi-view video source that is typically obtained from multiple camera systems or other possible methods. The input multiview video is stored in the multiview image buffer 610. The multiview image buffer 610 provides the stored multiview video source data to the predictor 620 and the difference image encoder 640.

The predictor 620 includes a parallax predictor 622 and a motion predictor 624 to perform motion prediction and parallax prediction on the stored multi-view video source.

The parallax / motion compensation unit 630 performs the motion and parallax compensation using the motion vector and the parallax vector predicted by the parallax predictor 622 and the motion predictor 624. The parallax / motion compensation unit 630 provides an image reconstructed using the predicted motion and parallax vector to the difference image encoder 640.

The difference image encoder 640 may provide a difference image between the original image provided from the multiview image buffer 610 and the reconstructed image compensated by the parallax / motion compensation unit 630 to provide a better image quality and a three-dimensional effect. The encoding is performed and provided to the entropy encoder 650.

The entropy encoder 650 receives the information about the disparity vector and the motion vector generated by the predictor 620 and the residual image from the difference image encoder 640, and generates a bit stream for multi-view video source data.

In the disparity prediction unit 622 of FIG. 6, disparity information is used for disparity compensation, like motion information for motion prediction and motion compensation used in conventional codecs such as H.264 and MPEG 2/4. As an attempt is made to reduce motion information and increase coding efficiency in H.264, an attempt is made to reduce disparity information and increase coding efficiency in MVC.

To this end, the parallax prediction unit 622 predicts a parallax vector for a predetermined number of precoded macroblocks, and predicts a parallax vector of macroblocks located adjacent to the precoded macroblocks. Calculate using More specifically, the disparity prediction unit 622 performs disparity on a predetermined number of pre-coded macroblocks by a conventional disparity prediction method such as a full-search method or a fast-search method. A vector is predicted and calculated by interpolating a parallax vector of macroblocks located adjacent to previously encoded macroblocks using a predetermined interpolation equation.

Hereinafter, a method of predicting parallax according to an embodiment of the present invention will be described in detail with reference to FIGS. 7 to 12.

A predetermined number of pre-encoded macroblocks are referred to herein as "seed-macroblocks (hereinafter referred to as seed-MB)". The macroblocks located adjacent to the pre-encoded macroblocks in which the disparity vector is calculated using the predicted disparity vector are referred to as "non-seed macroblocks (hereinafter referred to as non-seed MBs)". In the present specification, the prediction of parallax by a conventional search method is called "disparity vector search", and the calculation of parallax according to the present invention is called "disparity vector interpolation" or "direct mode". . In addition, a conventional method of encoding a multiview video by a parallax prediction method is called "search mode coding", and a method of encoding a video by parallax vector interpolation of the present invention is called "direct mode coding".

7 is a diagram illustrating a frame sequence photographed by a multiview camera.

The horizontal axis represents the number of cameras for capturing a multiview image with the viewpoint axis. The vertical axis represents the number of time sequences on the time axis. If the multiview cameras are arranged in parallel, there will be a high association between the multiview frames in the same time sequence. Even if the multi-view cameras are not arranged in parallel, there will be relevance since the images captured from the multi-view cameras can be rectified to the same state as those taken by the multi-view cameras arranged in parallel.

When there is such a property that there is a high correlation between multi-view frames, like the direct mode used in H.264, the parallax vector is predicted by using "disparity vector interpolation" according to an embodiment of the present invention. Is effective. "Differential vector interpolation" is analogous to image interpolation and image scaling or image reconstruction used to increase resolution. In image interpolation, an image is obtained by interpolation from adjacent images by linear interpolation or cubic interpolation. Like image interpolation, parallax vectors can also be obtained by vector interpolation similar to the image interpolation method.

8 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to an embodiment of the present invention.

Vc and Vr in FIG. 8 are one line of MBs in any frames in the same time sequence. Vr is an already encoded reference view frame, and Vc is a frame predicted using Vr for current encoding. Macro blocks hatched by dashed lines represent " seed-MB " as already predicted blocks. Macroblocks denoted by diagonal lines represent non-seed MBs in which a parallax vector is predicted as seed-MB.

Using the seed-MB in the Vr frame, the parallax vector of the other MBs inside the two seed-MBs, that is, the non-seed MB, can be predicted by the parallax vector interpolation method of the present invention. Equation 3 below is used for disparity prediction of other MBs in two seed-MBs as shown in FIG. 8.

[Equation 3]

In Equation 3, D _V0 and DV _{N −1} indicate seed-MBs and are predicted by conventional methods such as a full-search method or a fast-search method.

As shown in FIG. 8, when the deviation of the parallax vector is constant, the relationship of the parallax vector may be represented by a graph as shown in FIG. 9. In other words, if the distance between MB is i, the ratio of the size of DV changed from DV _O to DV _i , and if the distance between MB is i to N-1, the size of DV changes from DV _i to DV _{N -1} Assuming equal to one ratio, equation (3) can be derived by the following equations (3-1) to (3-3).

[Equation 3-1]

[Equation 3-2]

[Equation 3-3]

FIG. 10 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to another embodiment of the present invention.

Equation 3 is useful when the change line of DVs, i.e., the deviations of DVs, is nearly constant as described above in FIG. However, when the deviation of the DVs is not constant, for example, when the change line of the DVs has a curved shape as shown in FIG. 11, the DV may be predicted using Equation 4.

[Equation 4]

As can be seen in FIG. 10 and Equation 4, two or more DVs are used to predict DV _i for the i-th MB. In Equation 4, DV _{i + h} is a non-seed MB and C _h is an interpolation coefficient for generating DV for a macroblock. An example of a process of calculating DV _i to understand Equation 4 will be described. For example, when i = 4 and N = 8, Equation 4 may be expressed as Equation 4-1.

[Equation 4-1]

In Equation 4-1, when DV ₁ , DV ₃ , DV ₅ , and DV ₇ are seed-MB, any DV that is not seed-MB, i.e., DV ₂ , DV ₄ , DV _6, and DV ₈ is 0. When set, DV ₄ can be determined.

FIG. 11 is a diagram for describing a method of predicting a disparity vector of adjacent macroblocks by a disparity vector DV of an already encoded macroblock according to another embodiment of the present invention.

FIG. 11 illustrates two-dimensional prediction of a parallax vector. A parallax vector may be predicted using Equation 5. FIG.

[Equation 5]

In Equation 5, C (x, y) is a two-dimensional coefficient for generating a DV for the macroblock. When DV (x + h, y + v) is not seed-MB in Equation 5, if DV (x + h, y + v) is set to 0, DV (x, y is the only value for seed-MB. ) Can be calculated. When the area that can be obtained by the parallax vector interpolation method of the present invention is large, the equation as shown in FIG. 5 can be used.

12 is a diagram for explaining a method for predicting a parallax vector, according to an embodiment of the present invention.

In order to predict the parallax vector as described above, first, a conventional parallax vector-search method is performed to obtain a parallax vector for the seed-MB. Seed-MB may be spaced at regular intervals to simplify the calculation. Parallax vector-search is performed between the current frame and the reference frame. Then, the parallax vectors for the MB located adjacent to the remaining seed-MB can be calculated and obtained by the parallax vector interpolation method of the present invention using the seed-MB.

In addition, a predetermined interpolation equation used in the parallax vector interpolation of the present invention may be selected according to the characteristics of the pre-coded macroblocks, that is, the disparity vectors determined for the seed-MB. Here, the characteristics of the parallax vectors may be whether or not the deviation of the predicted parallax vectors is constant, as described in Equations 3 and 4 above. Also, in the direct mode, equations such as Equations 3 to 5 may be determined at the design stage of the multi-view video encoding and decoding apparatus. The equations must be shared by the multi-view video encoding and decoding apparatus.

Hereinafter, a method of encoding a multiview video by disparity vector prediction according to another embodiment of the present invention will be described.

A parallax vector obtained by an interpolation method according to an embodiment of the present invention is a reliable value, but it may not be. Thus, to obtain high compression performance, both a parallax vector interpolation method according to the present invention and a parallax vector search method according to the conventional method, for example, a full or fast search method, can be performed.

That is, according to an embodiment of the present invention, the compression ratios for both the parallax vector interpolation and the parallax vector search are compared, and finally, when the compression ratio of the parallax vector interpolation is equal to or better than that of the parallax vector search, Use the result. When the result of the parallax vector interpolation is used, the result of the parallax vector interpolation can be reproduced if the receiving end only has information about how DV is generated without information about DV.

On the other hand, when the compression ratio for parallax vector search is higher and finally parallax vector search is used, DV must be coded and transmitted. In this case, when the difference between DV by parallax vector search and DV by parallax vector interpolation is small, the difference may be transmitted.

Therefore, when a frame is encoded, MB by disparity vector search and MB by direct mode according to an embodiment of the present invention should be distinguished. MB according to each method can be distinguished by defining a flag (flag_direct) for the direct mode as follows.

DV by parallax vector search: flag_direct = 0

DV by direct mode: flag_direct = 1

Of course, flag_direct of DV by parallax vector search can be set to 1, and flag_direct of DV by direct mode can be set to 0. In the decoding stage, if the flag for the MB-mode is the direct mode, DV may be calculated using a parallax vector interpolation equation such as Equations 3 to 5 below. Which of the equations (3) to (5) should be determined in advance at the encoding stage and the decoding stage.

On the other hand, a flag (flag_diff) can be defined and used to distinguish the difference between the DV for the parallax vector search and the DV for the direct mode from the DV for the basic parallax vector search. This is because even if a multiview video is encoded by search mode encoding, when the difference between the parallax vector DV_d by parallax vector interpolation and the parallax vector DV_f due to parallax vector search is small, encoding the difference value is effective to increase the compression ratio. .

If flag_diff = 0, DV by parallax vector search is transmitted.

If flag_diff = 1, the difference between DV for parallax vector search and DV by direct mode is transmitted.

Of course, you can set the flag_diff value in reverse.

Therefore, in the multi-view video encoding method of the present invention, three different modes can be defined.

1st MODE: (flag_direct = 1) & (flag_diff = 0 or 1)

Second MODE: (flag_direct = 0) & (flag_diff = 1)

Third MODE: (flag_direct = 0) & (flag_diff = 0)

Since the first mode is a direct mode, only the residual image may be encoded and transmitted to the decoder. In the second mode, the difference between the parallax vector in the residual image and the direct mode and the parallax vector in the parallax vector search mode may be encoded and transmitted. In the case of the third mode, a parallax vector obtained by searching for a residual image and a parallax vector may be encoded and transmitted.

13 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to another embodiment of the present invention.

In accordance with another embodiment of the present invention, a multi-view video encoding apparatus includes a direct mode performer 1310, an interpolation providing unit 1320, a search mode performer 1330, a direct mode cost calculator 1342, and a search mode cost. A cost calculator 1340 including a calculator 1343, a comparator 1350, a mode selector 1360, and an encoder 1370 are included.

As illustrated in FIG. 13, the direct mode performer 1310 receives a current frame and a reference frame. The direct mode performing unit 1310 encodes a predetermined number of macroblocks of a frame to be encoded in advance, and performs a conventional parallax vector search on the predetermined number of encoded macroblocks, that is, the disparity vector for the seed-MB. Predict.

Then, the direct mode performing unit 1310 predicts a parallax vector for a predetermined number of pre-coded macroblocks, and uses a predicted parallax vector to place macroblocks adjacent to the pre-coded macroblocks. The parallax vector is used to calculate the parallax interpolation method according to the present invention. That is, the parallax vector DV_d according to the direct mode is calculated by interpolating the parallax vector using a predetermined interpolation equation provided by the interpolation providing unit 1320.

In addition, the direct mode performing unit 1310 generates a frame compensated with the DV_d and the reference frame, and generates a residual image of the current frame by subtracting the original frame from the compensated frame. The direct mode performer 1310 transfers the generated DV_d and the residual image to the direct mode cost calculator 1342 of the cost calculator 1340.

The interpolation provider 1320 may generate at least one interpolation equation, for example, equations 3, 4, or 5, which may be selected according to the characteristics of the disparity vectors predicted with respect to a predetermined number of pre-coded macroblocks. Saving. Therefore, the direct mode performing unit 1310 may determine the parallax vector using the interpolation equation provided by the interpolation providing unit 1320.

The current frame and the reference frame are also input to the search mode performing unit 1330. In the search mode performing unit 1330, the parallax vector DV_f is determined by a conventional parallax searching method, and a residual image of the current frame makes a compensated frame using the DV_f and a reference frame, and subtracts the original frame from the compensated frame. To generate a residual image of the current frame. The search mode performer 1330 transfers the generated DV_f and the residual image to the search mode cost calculator 1343 of the cost calculator 1340.

The direct mode cost calculator 1342 calculates a cost of the direct mode execution result. That is, the direct mode cost calculator 1342 receives the DV_d and the residual image and calculates a cost between the non-seed MB and the MB compensated using the DV_d by parallax vector interpolation.

The search mode cost calculator 1343 calculates a cost of the search mode performance result. That is, the search mode cost calculator 1343 receives the DV_f and the residual image to calculate a cost between the non-seed MB and the MB compensated using the DV by the parallax search. The cost may be determined by calculating or predicting a peak signal to noise rate (PSNR) of the bitrate or compressed image. In this case, the cost may be determined using only the bit rate or may be determined using only the PSNR. Alternatively, the cost may be determined by considering both the bit rate and the PSNR according to a predetermined criterion.

The comparator 1350 compares the results calculated by the direct mode cost calculator 1341 and the search mode cost calculator 1343, and transfers the comparison result to the mode selector 1360. The comparator 1350 may compare the difference between the direct mode cost and the search mode cost with a predetermined threshold. As a result of the comparison, when the difference between the direct mode cost and the search mode cost is equal to or greater than a predetermined threshold value, the comparison unit 1350 determines a difference between the parallax vector by the parallax vector interpolation according to the present invention and the parallax vector by the conventional parallax vector search method. The comparison result is transmitted to the mode selection unit 1360 in comparison with the predetermined threshold value.

 The mode selector 1360 determines a flag based on the comparison result and determines an encoding mode according to the determined flag. When the mode selector 1360 receives a result from the comparison unit 1350 that the difference between the two cost functions is smaller than a predetermined threshold, the mode selector 1360 may set flag_direct to 1 to encode the direct mode according to the present invention. The mode selector 1360 may set flag_direct to 0 to determine a parallax vector by encoding a parallax vector according to a conventional parallax search when a difference between two cost functions is greater than or equal to a predetermined threshold. have.

When flag_direct is set to 0, the mode selector 1360 determines, from the comparison unit 1350, the difference between the parallax vector by the parallax vector interpolation of the present invention and the parallax vector by a conventional parallax vector search method with a predetermined threshold. The flag_diff is also determined by receiving the comparison result. The mode selector 1360 may set flag_diff to 1 when the difference is smaller than a predetermined threshold, and set flag_diff to 0 when the difference is greater than or equal to a predetermined threshold.

The mode selector 1360 sets the first mode when (flag_direct = 1) & (flag_diff = 0 or 1), sets the second mode when (flag_direct = 0) & (flag_diff = 1), and flag_direct = 0) & (flag_diff = 0), the third mode may be set.

The encoder 1370 encodes and transmits a multiview video according to the determined mode together with the determined flag information. When flag_direct = 1 (first mode), the encoder 1370 encodes only flag information and a residual image. When flag_direct = 0 and flag_diff = 1 (MODE 2), the difference between the parallax vector by the residual image and the direct mode and the parallax vector by the search mode is encoded and transmitted. When flag_direct = 0 and flag_diff = 0 (third MODE 2), the parallax vector according to the residual image and the search mode is encoded and transmitted.

14 is a flowchart illustrating a multiview video encoding method according to another embodiment of the present invention.

A disparity vector for a predetermined number of macroblocks, that is, seed-MB, pre-coded is determined by a conventional disparity vector prediction method (S 1410). The direct mode performing unit 1310 calculates a parallax vector by parallax vector interpolation using a predetermined interpolation equation (S 1420). The direct mode cost calculator 1341 receives the DV_d generated by the direct mode performer 1310 and the residual image, and calculates a cost between the non-seed MB and the MB compensated using DV_d by parallax vector interpolation. Calculate (S 1430).

On the other hand, the search mode execution unit 1330 determines the parallax vector DV_f by a conventional parallax searching method (S1440). The search mode cost calculator 1343 receives the DV_f generated by the search mode performer 1330 and the residual image, and calculates a cost between the non-seed MB and the MB compensated using the DV_f by the parallax vector search. Calculate (S 1450).

The comparison unit 1350 compares the results calculated by the direct mode cost calculator 1341 and the search mode cost calculator 1343, and transmits the comparison result to the mode selector 1360 (S 1460). The mode selector 1360 determines a flag based on the comparison result, and determines an encoding mode according to the determined flag (S 1470). The encoder 1370 encodes a multiview video according to an encoding mode together with the determined flag information (S 1480).

15 is a block diagram illustrating an apparatus for decoding a multiview video according to an embodiment of the present invention. A multi-view video decoding apparatus according to an embodiment of the present invention includes a mode checker 1510, a disparity vector determiner 1520, a disparity interpolation providing unit 1530, and a decoder 1540.

A flag indicating a multiview video encoding mode included in the multiview video bitstream received by the mode confirmation unit 1510, that is, flag_direct and flag_diff, is checked to confirm the encoding mode of the multiview video.

The parallax vector determiner 1520 determines a parallax vector according to the identified encoding mode. The parallax interpolation provider 1530 stores the same parallax interpolation equation as the multi-view video encoding apparatus, and provides the parallax interpolation equation to the parallax vector determiner 1520. The parallax vector determiner 1520 may calculate a parallax vector for a multiview video encoded according to a direct mode by using a parallax interpolation equation. The parallax interpolation providing unit 1530 may be included in the parallax vector determiner 1520.

When the parallax vector determiner 1520 determines the parallax vector in the first mode, that is, the direct mode, the residual image is transmitted, and thus the parallax vector may be calculated using the same interpolation equation as that of the multiview video encoding apparatus.

In the case of the second MODE, the difference value between the parallax vector DV_d in the residual image and the direct mode and the parallax vector DV_f in the search mode is transmitted. In the case of the second MODE, DV_f is the sum of the difference between the parallax vector in the direct mode and the parallax vector in the search mode, and DV_d, and the DV_d is the parallax using the parallax interpolation provided by the parallax interpolation providing unit 1530. Can be calculated, so that DV_f can be obtained.

In the third mode, since the parallax vector according to the residual image and the search mode is transmitted, the delivered parallax vector is used for decoding.

The decoder 1540 decodes the multiview video by using the disparity vector determined according to each mode and the residual image data included in the multiview video bitstream received and transmitted to the mode checker 1510.

The method according to the invention can be embodied as computer readable code on a computer readable recording medium. Codes and code segments implementing the above program can be easily deduced by computer programmers in the art. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, and the like, and may also include those implemented in the form of carrier waves (eg, transmission over the Internet). do. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

The above description is only one embodiment of the present invention, and those skilled in the art may implement the present invention in a modified form without departing from the essential characteristics of the present invention. Therefore, the scope of the present invention should not be limited to the above-described examples, but should be construed to include various embodiments within the scope equivalent to those described in the claims.

As described above, according to the present invention, when predicting the time difference in macroblock units, the time difference for the remaining macroblocks is calculated using the time difference predicted for a predetermined number of macroblocks without predicting the time difference for all macroblocks. By calculating, the coding speed of a multiview video can be improved.

In addition, according to the present invention, since the parallax vector for each macroblock does not have to be encoded, the compression rate of the multiview image can be improved.

According to the present invention, a method and apparatus for decoding a multiview video for decoding a multiview video encoded by using a correlation between disparity vectors with respect to a multiview image can be provided.

Claims (26)

In the method for predicting a parallax vector to encode a multi-view video, Predicting a parallax vector for a predetermined number of encoded macroblocks; Calculating a disparity vector of macroblocks located adjacent to the encoded macroblocks using the predicted disparity vector. The method of claim 1, Predicting the parallax vector comprises predicting a parallax vector for the predetermined number of encoded macroblocks by a full search method or a fast search method, The calculating of the parallax vector includes interpolating and calculating a parallax vector of macroblocks adjacent to the encoded macroblocks using a predetermined interpolation equation. The method of claim 2, The predetermined interpolation equation is selected according to the characteristics of the disparity vectors determined for the encoded macroblocks, and the characteristics of the disparity vectors include whether or not the deviation of the predicted disparity vectors is constant. Forecast method. Receive a current frame and a reference frame, predict a parallax vector for a predetermined number of encoded macroblocks, and use the predicted parallax vector to disparity vectors of macroblocks located adjacent to the encoded macroblocks Direct mode performing unit for calculating a; An interpolation expression providing unit which provides an interpolation equation for calculating a parallax vector of macroblocks located adjacent to the encoded macroblocks to the direct mode performing unit; A search mode performing unit configured to receive the current frame and the reference frame and to search the frames to predict a parallax vector; A cost calculator configured to calculate costs according to the parallax vector determination method performed by the direct mode performer and the search mode performer, respectively; A comparison unit comparing the calculated cost; A mode selection unit for selecting an encoding mode according to the comparison result; And And an encoder configured to encode a multiview video according to the selected encoding mode. The method of claim 4, wherein the direct mode performing unit, The parallax vector for the predetermined number of encoded macroblocks is predicted by a full search method or a fast search method, and a parallax vector of macroblocks located adjacent to the encoded macroblocks is determined using a predetermined interpolation equation. A multi-view video encoding device, characterized in that it is calculated by interpolation. The method of claim 5, And the interpolation expression providing unit stores at least one interpolation expression that can be selected according to the characteristics of the disparity vectors predicted with respect to the predetermined number of encoded macroblocks. The method of claim 4, wherein the cost calculation unit, A direct mode cost calculator configured to calculate a direct mode cost by using the parallax vector DV_d calculated by the direct mode performer; And And a search mode cost calculator configured to calculate a search mode cost by using the disparity vector DV_f predicted by the search mode performer. The method of claim 4, wherein And the cost calculator calculates a bit rate and / or a PSNR for each of the direct mode and the seek mode. The method of claim 7, wherein The comparison unit compares a difference between the direct mode cost and the search mode cost with a predetermined threshold value, And the mode selector sets a flag according to the comparison result, and selects an encoding mode according to the set flag. The method of claim 9, The comparison unit may determine a difference between the disparity vector DV_d determined according to the direct mode and the disparity vector DV_f determined according to the search mode when the difference between the direct mode cost and the search mode cost is greater than or equal to a predetermined threshold. Compared to the threshold, The mode selector sets a flag according to the encoding mode according to the comparison result, and selects an encoding mode according to the set flag. The method of claim 4, wherein the encoder, If the selected encoding mode is a direct mode, the residual image is encoded. If the selected encoding mode is a search mode and the difference between the parallax vector DV_d determined according to the direct mode and the parallax vector DV_f determined according to the search mode is smaller than a predetermined threshold, the residual image and the direct mode Encoding a difference between the determined disparity vector DV_d and the disparity vector DV_f determined according to the search mode, When the selected encoding mode is a search mode and a difference between the parallax vector DV_d determined according to the direct mode and the parallax vector DV_f determined according to the search mode is greater than or equal to a predetermined threshold, the residual image and the parallax due to the search mode A multi-view video encoding device, characterized in that the vector is encoded. Receiving a current frame and a reference frame; Performing a direct mode of predicting disparity vectors for a predetermined number of coded macroblocks and calculating disparity vectors of macroblocks adjacent to the coded macroblocks using the predicted disparity vector; Performing a search mode for predicting a parallax vector by searching the current frame and a reference frame; Calculating a cost of performing the direct mode and a cost of performing the search mode; Comparing the calculated costs; Selecting an encoding mode according to the comparison result; And Multi-view video encoding method according to the selected encoding mode. The method of claim 12, wherein performing the direct mode comprises: Predicting disparity vectors for the predetermined number of encoded macroblocks by a full search method or a fast search method; And And interpolating and calculating a parallax vector of macroblocks located adjacent to the encoded macroblocks using a predetermined interpolation equation. The method of claim 13, wherein the calculating by interpolating the parallax vector comprises: And calculating a disparity vector using one of at least one interpolation equation that can be selected according to the characteristics of the disparity vectors predicted with respect to the predetermined number of encoded macroblocks. The method of claim 12, wherein calculating the cost comprises: Calculating a direct mode cost by using the parallax vector DV_d calculated according to the direct mode; And And calculating a search mode cost using the disparity vector DV_f predicted according to the search mode. The method of claim 12, wherein calculating the cost comprises: And calculating a bit rate and / or a PSNR for each of the direct mode and the search mode. The method of claim 15, wherein comparing the costs comprises: Comparing a difference between the direct mode cost and the search mode cost and a predetermined threshold value; Selecting the encoding mode, And setting a flag according to the comparison result, and selecting an encoding mode according to the set flag. 18. The method of claim 17, wherein comparing the costs comprises: When the difference between the direct mode cost and the search mode cost is greater than or equal to a predetermined threshold, the difference between the disparity vector DV_d determined according to the direct mode and the disparity vector DV_f determined according to the search mode is compared with a predetermined threshold. , Selecting the encoding mode, And setting a flag according to the comparison result and selecting an encoding mode according to the set flag. The method of claim 12, wherein the encoding comprises: If the selected encoding mode is a direct mode, encoding a residual image, If the selected encoding mode is a search mode and a difference between the parallax vector DV_d determined according to the direct mode and the parallax vector DV_f determined according to the search mode is smaller than a predetermined threshold, the residual image and the direct mode Encoding the difference between the parallax vector DV_d determined according to the parallax vector DV_f according to the search mode, and When the selected encoding mode is a search mode and a difference between the parallax vector DV_d determined according to the direct mode and the parallax vector DV_f determined according to the search mode is greater than or equal to a predetermined threshold, the residual image and the parallax due to the search mode A multi-view video encoding method comprising the step of encoding a vector. An encoding mode checking unit for checking an encoding mode by checking flags indicating an encoding mode included in a received multiview video bitstream; A disparity vector determiner that determines a disparity vector according to the identified encoding mode; And a decoder configured to decode a multiview video using the disparity determined according to the identified encoding mode and the residual image data included in the multiview video bitstream. And the disparity vector determiner is configured to calculate a disparity vector using the same disparity interpolation equation as that of the multiview video encoding apparatus. The method of claim 20, When the encoding mode confirmed by the mode check unit is a first mode in which a residual image is received, the parallax vector determiner calculates a parallax using the parallax interpolation equation, If the encoding mode confirmed by the mode checking unit is a second mode in which a difference value between the parallax vector DV_d determined according to the residual image and the direct mode and the parallax vector DV_f determined according to the search mode is transmitted, the parallax The vector determining unit calculates a parallax vector by adding the difference and the parallax calculated using the parallax interpolation equation. Confirming an encoding mode by checking flags indicating an encoding mode included in the received multi-view video bitstream; Determining a parallax vector according to the identified encoding mode; And decoding the multi-view video using the disparity determined according to the identified encoding mode and the residual image data included in the multi-view video bitstream. Determining the parallax vector, Computing a parallax vector using the same parallax interpolation equation as the multi-view video encoding apparatus. The method of claim 22, wherein determining the parallax vector, Calculating the parallax using the parallax interpolation equation when the identified encoding mode is the first mode in which the residual image is received; When the identified encoding mode is a second mode in which a difference value between the parallax vector DV_d determined according to the residual image and the direct mode and the parallax vector DV_f determined according to the search mode is transmitted, the difference value and the parallax complement And calculating a disparity vector by adding the disparity calculated using a snack. A computer-readable recording medium having recorded thereon a program for implementing the method according to any one of claims 1 to 3. A computer-readable recording medium having recorded thereon a program for implementing the method of claim 12. A computer-readable recording medium having recorded thereon a program for implementing the method of claim 22 or 23.

Images