KR20080037593A - Multi-view video scalable coding and decoding - Google Patents

Abstract

A multi-view video scalable coding and decoding methods, and apparatuses thereof are provided to offer various video services to terminals of various environments through intra-estimation in other resolutions of peripheral videos. A multi-view video scalable coding apparatus comprises a base scalability video encoder(21) and plural enlarged scalability video encoders(22-25). The base scalability video encoder divides one base video into multi-layer resolutions through spatial filtering, and performs a scalable video coding for the divided low resolution image frames and at least one high resolution image frame through space time estimation. The plural enlarged scalability video encoders receive and divide self video and at least one other video at the same time into multi-layer resolutions through the spatial filtering. The enlarged scalability video encoders perform a scalable video coding for the divided low resolution image frames through time space estimation, and perform the scalable video coding for the at least one divided high resolution image frames through time space estimation.

Description

A scalable coding and decoding method of a multiview video, and a coding and decoding apparatus [Multi-view video scalable coding and decoding}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to scalable coding and decoding techniques of multiview video, and more particularly, to multi-view spatial and temporal scalable coding of multiview video in compression transmission of multiview video captured by multiple cameras. The present invention relates to a scalable coding and decoding method of multi-view video and to a coding and decoding apparatus for compressing transmission using technology, decoding at a receiving end, and providing two-dimensional or three-dimensional video services to various types of video terminals.

The present invention is derived from the research conducted as part of the IT new growth engine core technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management Number: 2005-S-403-02, Title: Intelligent Integrated Information Broadcasting (SmarTV) ) Technology Development].

The basic principle of compressing data is to eliminate spatial and temporal redundancy of the data. Spatial redundancy is the repetition of the same color or object in a video. And temporal overlap means that the adjacent picture is almost unchanged in the moving picture or the same sound is continuously repeated in the audio picture. In the general video coding method, temporal redundancy is eliminated by temporal filtering based on motion compensation, and spatial redundancy is removed by spatial transform.

In order to transmit the multimedia data generated after data duplication is removed, a transmission medium is required, and its performance is different for each transmission medium. There is a scalable video coding technique to support a variety of speed transmission media or to transmit multimedia data at a data rate suitable for a transmission environment.

The scalable video coding technique cuts a portion of the bitstream with respect to the already compressed bit-stream according to the peripheral conditions such as the transmission bit rate, transmission error rate, and system resources, so that the resolution, frame rate, and SNR (signal- to-noise ratio).

1 is a reference diagram for explaining a general scalable coding concept. Referring to FIG. 1, scalable video coding implements temporal scalability using temporal transform on input video and spatial scalability using 2D spatial transform. In addition, scalable video coding implements quality scalability by using texture coding. Motion Coding scalablely encodes motion information when implementing each spatial scalable. One bitstream is generated through such a coding algorithm.

Motion Compensated Temporal Filtering (MCTF) and Hierarchical B-pictures are two methods for providing temporal scalability and increasing compression efficiency in scalable video coding.

MCTF refers to performing a wavelet transform using motion information in a time direction in a video sequence. Wavelet transform is implemented using a lifting scheme. The lifting scheme consists of three operations: polyphase decomposition, prediction, and update.

Hierarchical B-pictures use various methods using the syntax of Memory Management Control Operation (MMCO) and Reference Picture List Reordering (RPLR), which manages a decoded picture buffer (DPB) that can store 16 pictures used in H.264. It can be implemented as.

Recently, due to technological developments and user demands, video information about a scene at various points of time is provided to the viewer, and the information sent from a broadcasting station for each viewer is edited by manipulating the scene that the viewer wants to see among the video information. Services that can be watched by people are being researched. In order to provide such a service, a video compression technique for multiview video is required.

Multi-view video compression is for simultaneously compressing, storing, and transmitting video input from multiple cameras providing multi-view video. When storing and transmitting a multiview video without compressing the data, the size of the data is very large. Therefore, a large transmission bandwidth is required to transmit data to a user through a broadcasting network or a wired / wireless internet in real time.

In general, multi-view video coding and decoding may be performed by independently coding and transmitting each video sequence and then decoding the video sequence. This can be easily implemented through existing MPEG-1 / 2/4 or H.261 / 263/264. However, this does not eliminate the redundancy between the videos caused by shooting the same object with multiple cameras.

A technique for removing redundancy between videos includes scalable video coding technology. In general, scalable coding technology of single-view video divides single-view video into a multi-layer resolution using a spatial filter on the spatial axis, and performs spatial and temporal scalability through hierarchical bidirectional motion prediction on the time axis. Here, motion, difference image, and intra prediction may be performed for effective prediction between layers at each resolution. In addition, scalability of an image quality perspective may be provided through entropy coding by hierarchical representation in transform encoding.

However, such a conventional scalable video coding technique is fixed to a single-view video spatial image quality, so that a terminal having various environments and various computing powers can reproduce a video in various three-dimensional or selective two-dimensional formats. There is a problem that a lot of overhead can be generated in terms of a video decoder.

Accordingly, the present invention has been proposed to solve the above-mentioned problems of the prior art, and is a multi-view as well as a motion prediction compensation that refers to the surrounding image in the time axis and the video (spatial) axis to compress and transmit the multi-view video. In order to provide scalability from the video to the spatial axis, multiview can provide more efficient video compression and provide various video services to terminals in various environments through motion and difference video and intra prediction of different resolutions of surrounding video. An object of the present invention is to provide a scalable coding method and apparatus for video.

Another object of the present invention is to provide a decoding method and apparatus for receiving and decoding a scalable coded signal for a multiview video.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

A scalable video coding apparatus according to the present invention for achieving the above object, in the scalable video coding apparatus for a multi-view video, separated into a multi-layer resolution through spatial filtering for one elementary video, separated low resolution An elementary scalability video encoder for performing scalable video coding through space-time prediction for each of the image frame and the at least one high resolution image frame; And receiving at least one other video as its own video and the reference video captured at the same time point, separating the multi-layer resolution through spatial filtering, and separating the separated low-resolution image frames in the same time axis as well as their adjacent frames. The scalable video coding is performed through the spatiotemporal prediction with reference to the other video frame, and the scaled through the spatiotemporal prediction with reference to the lower layer of the other video frame as well as its own lower layer for the separated at least one high resolution image frame. And a plurality of extended scalability video encoders for performing flexible video coding.

In addition, the scalable video coding method according to the present invention is a scalable video coding method for a multiview video, comprising: (a) splitting a single base video into multi-layer resolution and performing scalable video coding through space-time prediction; Performing; And (b) receiving at least one other video as the own video and the reference video captured at the same time point, separating the input videos into a multi-layer resolution, and performing scalable video coding through space-time prediction. In addition, the step (b) comprises the steps of: (c) performing scalable video coding through space-time prediction with reference to the other video frame on the same time axis as well as its adjacent frame, and (C) d) performing scalable video coding through spatiotemporal prediction on at least one high resolution image frame by referring not only to its lower layer but also to the lower layer of the other video frame.

Meanwhile, the decoding apparatus according to the present invention, in the scalable video decoding apparatus for a multiview video, receives a scalable coded bitstream for one elementary video, and restores it by inverse time transform and inverse spatial transform. An elementary scalability video decoder; And receiving a scalable coded bitstream through the spatiotemporal prediction of the own video and the reference video photographed at the same time, and inverse spatiotemporal space according to whether the sublayer of another video frame that is the reference video as well as its own lower layer is referenced. After reconstructing at least one high resolution image frame through prediction, and reconstructing the low resolution image frame through inverse spatiotemporal prediction according to whether the other video frame is referenced in the same time axis as well as its adjacent frame, the reconstructed high resolution image And a plurality of extended scalability video decoders for reconstructing the image through inverse spatial filtering on the frame and the reconstructed low resolution image frame.

Advantageously, said extended scalability video decoder comprises: demultiplexing means for demultiplexing a received bitstream; At least one of performing scalable decoding through inverse spatiotemporal motion prediction according to whether the lower layer of its own video as well as the lower layer of other video which is a reference video is referred to the high resolution image signal output by the demultiplexing means; Enhancement layer decoding means; A base layer decoding means for performing scalable decoding on the low resolution video signal output by the demultiplexing means through not only the inverse spatiotemporal motion prediction for its video frame but also the inverse motion prediction for the reference video frame on the time axis; And inverse spatial video filtering means for reconstructing the image through inverse spatial filtering on the high resolution image reconstructed by the enhancement layer decoding means and the low resolution image reconstructed by the base layer decoding means.

Preferably, the enhancement layer decoding means includes a flag indicating whether to use motion vector information of a lower layer, a flag indicating whether to use a reference index of a lower layer for another video as prediction information, and an intra block of the lower layer. A scalable decoding is performed by referring to a flag indicating whether a type of is used as prediction information, a flag indicating whether to use a difference image value of a lower layer, and an index of a reference view used for prediction.

In addition, the decoding method according to the present invention for achieving the above object, in the scalable video decoding method for a multi-view video, (a) performing scalable video decoding on one base video through inverse space-time prediction ; And (b) receiving a scalable coded bitstream with reference to at least one other video as its own video and reference video captured at the same time, and performing scalable video decoding through inverse space-time prediction and inverse spatial filtering. In addition, the step (b), (c) scalable video through the inverse spatiotemporal prediction according to the reference information of the sub-layer of the other video frame as well as its sub-layer for the demultiplexed high-resolution image frame Performing decoding; And (d) performing scalable video decoding on the demultiplexed low resolution video signal through inverse spatiotemporal prediction according to whether the other video frame is referenced in the same time axis as well as its adjacent frame.

Preferably, in step (c), a flag indicating whether to use motion vector information of a lower layer, a flag indicating whether a reference index of a lower layer for another video is used as prediction information, and a lower layer intra Scalable video decoding is performed using a flag indicating whether a block type is used as prediction information, a flag indicating whether a lower layer difference image value is used, and index information of a reference view used for prediction.

As described above, the present invention can effectively compress a multiview video by extending the space-time hierarchical structure of a general scalable coding technique between multiview videos. In addition, the present invention can provide a video service in a scalable manner to a two-dimensional or three-dimensional terminal of various forms by constructing a hierarchical structure on the time and space axis for the multi-view video.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is an exemplary diagram of a scalable coding apparatus of a multiview video according to the present invention. FIG. 2 illustrates a case in which each video is compressed by a scalable video encoder when video captured by five cameras from video 0 to video 4 is input.

The scalable coding apparatus for a multiview video according to the present invention performs a two-dimensional spatial transform and a temporal transform on video 0, which is a basic video, and a basic scalable video encoder for performing scalable coding through motion coding and texture coding. 21 and at least one neighboring video that is a reference video as well as its own video, respectively, are input and separated into multi-layer resolution through spatial filtering and temporal filtering, But also includes a plurality of extended scalability video encoders 22 to 25 that perform scalable coding with reference to the spatio-temporal layer image information and compression parameter of surrounding videos.

In the drawing, video 0 is defined as the base video, and the basic scalability video encoder 21 that performs scalable coding on video 0 has the same structure as the scalable coding apparatus of the conventional single-view video. Therefore, the basic video is a structure that can be compatible with existing scalable coding devices.

In order to scalablely compress not only the own video but also the reference video that is the reference video, video 1 through video 4 are compressed through the extended scalability video encoders 22 through 25. The extended scalability video encoders 22 to 25 receive their own video and the surrounding video, which is the reference video, like the conventional single-view video scalable coding device, and separate them into multi-layer resolution through spatial filtering. For each frame, scalable coding is performed by referring not only to the lower layer of its own frame but also the lower layer of the surrounding frame on the temporal and spatial axis. For example, the extended scalability video encoder 25 compressing video 4 performs scalable video coding on its own video 4 by predicting and referring to the multi-layer spatiotemporal resolution image information of the base video 0. In addition, the extended scalability video encoder 23 compressing the video 2 compresses its own video through bi-prediction using multi-layer spatiotemporal resolution image information of the base video 0 and the video 4.

Accordingly, when only the basic scalability video encoder 21 is used, the existing two-dimensional video service is possible, and the basic scalability video encoder 21 for the basic video 0 and the extended scalability video encoder 25 for the video 4 are provided. In this case, stereo video service is enabled. Optionally combining extended scalability video encoders 22 to 25 and elementary scalability video encoder 21 also enables three-view or five-view video services.

The structure and function of the extended scalability video encoder will be described in detail with reference to FIG. 3.

3 is a detailed functional block diagram of an extended scalability video encoder according to the present invention.

The extended scalability video encoder according to the present invention includes a spatial video filtering unit 31 which separates its own video and reference video into multi-layer resolution through spatial filtering, and output images of the spatial video filtering unit 31. Temporal video filtering unit 330 and 340 that separates the time by temporal filtering with respect to the temporal low frequency images output from the temporal video filtering unit 330, The base layer encoder 33 performs scalable coding through motion prediction for a reference video frame, and the reference video as well as a lower layer of its own video for temporal high frequency images output from the temporal video filter 340. Scalable coding by referring to the lower layer of other video At least one enhancement layer encoder 34 to perform, and a multiplexer 35 for outputting a bit stream by multiplexing the output of the base layer encoder and the output of the enhancement layer encoder.

The spatial video filtering unit 31 receives the own video captured by the camera and other surrounding video set as the reference video at predetermined time units, and performs multi-layer resolution through spatial filtering using the MCTF or the hierarchical B structure. To separate.

The base layer encoder 33 and the enhancement layer encoder 34 are temporal video filtering units 330 and 340, motion coding units 331 and 341, difference units 332 and 342, and spatial transform units 333, 343, quantizers 334 and 344, and entropy encoders 335 and 345. The base layer encoder 33 and the enhancement layer encoder 34 are similar in structure to a conventional scalable video encoder. Here, each functional unit will be briefly described.

The temporal video filtering unit 330 included in the base layer encoder 33 separates the low frequency images separated by the spatial filtering into the time axis through filtering using an MCTF or a hierarchical B structure. In addition, the temporal video filtering unit 340 included in the enhancement layer encoder 34 separates the high frequency images separated by the spatial filtering into the time axis through filtering using an MCTF or hierarchical B structure.

The motion coding units 331 and 341 generally consist of motion estimation and motion compensation. Motion estimation performs motion estimation of the current frame based on a reference frame among input video frames, and obtains a motion vector for forward prediction or bidirectional prediction. Here, in the present invention, not only its own frame but also surrounding frames may be used as reference frames in motion estimation. A widely used algorithm for such motion estimation is a block matching algorithm. That is, the displacement when the error is the lowest while moving the given motion block by the pixel unit within a specific search region of the reference frame is estimated as a motion vector. Motion data such as a motion vector, a motion block size, and a reference frame number obtained as a result of the motion estimation are provided to the entropy encoders 335 and 345. In addition, motion compensation generates a temporal prediction frame for the current frame by performing motion compensation on a forward reference frame, a reverse reference frame, or a bidirectional reference frame using a motion vector calculated by motion estimation.

Differences 332 and 342 remove temporal redundancy of the video by differentiating the current frame from the temporal predictive frame. The spatial transform units 333 and 343 remove spatial redundancy using a spatial transform method that supports spatial scalability for a frame from which temporal redundancy is removed by the difference units 332 and 342. As the spatial transform method, a discrete cosine transform (DCT), a wavelet transform, or the like are mainly used.

The quantization units 334 and 344 quantize the transform coefficients obtained by the spatial transform units 333 and 343. Quantization refers to an operation of dividing the transform coefficient represented by an arbitrary real value into a discrete value by dividing the transform coefficient into predetermined intervals, and matching them by a predetermined index.

The entropy encoders 335 and 345 losslessly encode the transform coefficients quantized by the quantizers 334 and 344 and the motion data provided by the motion estimation and generate an output bitstream. As such a lossless coding method, arithmetic coding, variable length coding, or the like may be used.

Meanwhile, in the present invention, intra prediction for an intra block may be used before spatial transform. To this end, the enhancement layer encoder may include a function of performing 2D spatial interpolation and an intra prediction function by receiving a reference frame reconstructed from a lower layer encoder.

In general, inter prediction searches for a block most similar to a block (current block) of a current frame from a reference frame, obtains a prediction block that best represents the current block, and then compares the current block with the prediction block. It is a way to quantize the difference. Inter prediction is bi-directional prediction using two " reference " frames, forward prediction using a previous < Desc / Clms Page number 7 > reference frame, and backward prediction using a subsequent reference frame. (backward prediction) and the like.

Meanwhile, intra prediction is a method of predicting a current block using a frame adjacent to the current block among neighboring blocks of the current block. Intra prediction differs from other prediction schemes in that it uses only information in the current frame and does not refer to another frame or a frame of another layer in the same layer.

Intra base prediction may be used when the current frame has a frame of a lower layer having the same temporal position. The macroblock of the current frame can be predicted efficiently from the macroblock of the corresponding base frame. That is, the difference between the macroblock of the current frame and the macroblock of the base frame corresponding to the macroblock is quantized. If the resolution of the lower layer and the resolution of the current layer are different, the macroblock of the base frame is upsampled to the resolution of the current layer before the difference is obtained.

Residual prediction is an extension of inter prediction in a single layer in the form of a multi-layer. Instead of directly quantizing the difference generated in the inter prediction process of the current layer, the residual prediction is generated in the inter prediction process of the difference and the lower layer. It is a technique to subtract again and quantize the result.

Meanwhile, in the present invention, when the enhancement layer encoder encodes a high resolution image frame, the enhancement layer encoder moves the base layer image (low resolution image) of its own video and the base layer image (low resolution image) of another video as a reference video. Use twice the vector.

In addition, in the present invention, the enhancement layer encoder encodes a base layer image (low resolution image) of its own video and a base layer image (low resolution image) of another video as a reference video for prediction of a difference image when encoding a high resolution image frame. After prediction, the residual image is interpolated to perform the difference image prediction.

In addition, in the present invention, the enhancement layer encoder intra-codes a base layer image (low resolution image) of its own video and a base layer image (low resolution image) of another video, which is a reference video, for intra prediction when encoding a high resolution image frame. Intra prediction is performed in the prediction mode.

4 is a diagram for describing a predictive reference structure in scalable video coding according to the present invention.

In the figure, P macroblock represents unidirectional prediction and B macroblock represents bidirectional prediction. The predictive reference according to the present invention is a structure for performing prediction in unidirectional prediction (P macroblock) and bidirectional prediction (B macroblock) in a plurality of resolution layers as well as a time axis and a space (view) axis.

4 illustrates only a two-layer structure consisting of a base layer and one enhancement layer as an embodiment, and more layers may be configured by repeatedly extending the enhancement layer.

4 to 41 show prediction references performed at the base layer encoder and the enhancement layer encoder in the base scalability video encoder (21 in FIG. 2), and 42 is an example of an extended scalability video encoder for video 1 shown in FIG. A prediction reference performed at the base layer encoder and the enhancement layer encoder in 22), and 43 is a prediction performed at the base layer encoder and the enhancement layer encoder in the extended scalability video encoder 23 for video 2 shown in FIG. Indicates a reference.

In other words, in FIG. 4, base layer 0 (L0) is the base layer encoder of base scalability video encoder 21, the base layer encoder of extended scalability video encoder 22 for video 1, and the extended layer for video 2. A prediction reference structure respectively performed in the base layer encoder of the scalability video encoder 23 is shown.

Likewise, in FIG. 4, enhancement layer 1 (L1) is an enhancement layer encoder of the elementary scalability video encoder 21, an enhancement layer encoder of the extended scalability video encoder 22 for video 1, and an extended scaler for video 2. A prediction reference structure respectively performed in the enhancement layer encoder of the capability video encoder 23 is shown.

Referring to FIG. 4, the base layer encoder of the base scalability video encoder 21 performs scalable video coding by predicting adjacent frames with respect to its low resolution image frames on the time axis as in the prior art. The base layer encoder of the extended scalability video encoder 22 for video 1 performs bidirectional prediction on its own frame using frames of video 0 and frames of video 2 which are reference peripheral video frames located on the same time axis. . In addition, the base layer encoder of the extended scalability video encoder 23 for video 2 performs unidirectional prediction with reference to base video 0 and performs bidirectional prediction using its own frame.

Meanwhile, the enhancement layer 1 (L1), which is an upper layer of the base layer, performs prediction by referring not only to spatial and temporal prediction of its own video frame, but also to the own frame of the base layer, which is a lower layer, and surrounding frames of the base layer. In the drawing, three marks represented by circles and scissors in each macroblock are used to indicate whether a lower layer is referenced. Here, the "circle" or "marker" in the middle of the three marks indicates whether the lower layer is referenced to its own video frame, and the "circle" or "marker" shown above and below indicates the lower layer of the surrounding video frame. Indicates whether is referenced.

Referring to FIG. 4, the enhancement layer encoder of the basic scalability video encoder 21 performs scalable video coding by referring to a frame of its lower layer as in the prior art. The enhancement layer encoder of the extended scalability video encoder 22 for video 1 not only refers to the frames of its lower layer but also refers to the lower layer frame of video 0, which is the surrounding video frame, and the lower layer frame of video 2. Perform bidirectional prediction on own frames. In addition, the enhancement layer encoder of the extended scalability video encoder 23 for video 2 not only refers to the frame of its lower layer but also refers to the lower layer frame of the base video 0 to perform prediction.

FIG. 5 shows the reference structure shown in FIG. 4 in a fixed time and in the spatial (view) hierarchy axis the reference structure as "circles" and "scissors". 5 shows the case of three hierarchies. In FIG. 5, 51 denotes a reference structure of video 0 which is a basic video, 52 denotes a reference structure of video 1, and 53 denotes a reference structure of video 2.

In FIG. 5, since the macroblock of enhancement layer 2 for video 0 51 is only “circle” in the middle and “scissors” on both sides, the existing scalable video coding (SVC) refers only to the lower layer for its video. Video coding is performed through motion, difference image, and intra prediction. On the other hand, the macroblocks in enhancement layer 2 for video 1 (52) are “circles,” both right and left, as well as in the middle, so that scalable video coding refers to the lower layers of the video as well as the lower layers of the surrounding video. Do this. In addition, since the middle block and the left side of the macroblock of the enhancement layer 2 for the video 2 53 are expressed as “circles”, scalable video coding is performed by referring to the lower layer of the video and the lower layer of the video 0.

In order to indicate whether the lower layer is referred to as described above, a scalable video syntax including information on the video of the reference layer is required according to whether the lower layer is referenced.

In the following syntax, ref_view_Idx represents a view number of a reference video of a lower layer. Here, base_mode_flag is a flag indicating whether motion vector information of a lower layer is to be used for motion prediction of the current block. If the value is 1, it indicates ref_view_Idx, which is a view number indicating which view of lower layer is used for motion vector information. Should give.

base_mode_refinement_flag is also a flag indicating whether motion vector information of a lower layer is used in motion vector prediction of a current block. The difference from base_mode_flag is that a reference index of a lower layer is also used as prediction information. Therefore, when the flag is 1, ref_view_Idx indicating which view motion vector and reference index information of a lower layer are used should be indicated. intra_base_flag is a flag indicating that the type of the intra block of the lower layer is used as prediction information of the block type of the current block. When 1, intra_base_flag uses information about the intra prediction mode of the lower layer in the current block. Therefore, ref_view_Idx, which is a view number for which view of the lower layer uses intra block type information, must be indicated.

residual_prediction_flag is a flag indicating whether the difference image value of the lower layer is used to predict the difference image of the current block. If residual_prediction_flag is 1, the residual_prediction_flag upsamples the difference image information of the lower layer. Again, ref_view_Idx, which is a view number for which view of the lower layer uses differential image information, must also be displayed. Table 1 shows the syntax of the scalable video as described above.

6 illustrates an embodiment of a reference structure in the case of an existing B frame structure that is not a hierarchical B structure in a basic scalable structure. In FIG. 6, 61 represents a reference structure for video 0, which is a basic video, 62 represents a reference structure for video 1, and 63 represents a reference structure for video 2. FIG.

FIG. 7 also shows the reference structure shown in FIG. 6 at fixed time and the reference structure in the spatial (view) hierarchy axis as "circles" and "scissors". Similarly, the case of three layers is shown in FIG. In FIG. 7, 71 denotes a reference structure of video 0 which is a basic video, 72 denotes a reference structure of video 1, and 73 denotes a reference structure of video 2.

6 and 7, video 0 (61, 71), which is the base video, performs scalable video coding by referring to only its lower layer frame. However, video 1 and video 2 not only refer to their lower layer frames, but also perform scalable video coding by referring to the lower layer of the surrounding video.

As described above, the prediction reference structure according to the present invention can be equally applied to a conventional P frame structure that is not a B frame prediction structure.

FIG. 8 is a block diagram of a scalable coding apparatus for a multiview video according to another embodiment of the present invention. FIG. 9 is a video 091 and a video 1 92 as a peripheral video and a peripheral video in FIG. 8. The reference structure for video 2 93 is shown.

8 and 9, the basic scalability video encoder 81 performing scalable coding for video 0, which is the basic video, refers to its lower layer in the same way as the scalable coding apparatus of the conventional single-view video. To perform scalable video coding. Therefore, the base video may be compatible with existing scalable coding devices.

And, from video 1 to video 4, scalable video coding is performed through the extended scalability video encoders 82 to 85. The extended scalability video encoders 82 to 85 separate images at a multi-layer resolution like the conventional single-view video scalable coding apparatus, and perform prediction in space-time for the separated image frames as well as the prediction of the surrounding video. Compression is performed by referring to the spatiotemporal layer image information and the compression parameter.

For example, as shown in FIG. 9, enhancement layer 1 92 of video 1 performs scalable video coding with reference to a lower layer of video 0, and enhancement layer 1 93 of video 2 performs video 1 encoding. Scalable video coding is performed by referring to a lower layer. In other words, as shown in FIG. 8, in another embodiment of the present invention, the scalable scalability video encoder sequentially performs reference to only one video next to the scalable video coding.

Accordingly, when only the basic scalability video encoder 81 is used, the existing two-dimensional video service is possible, and the basic scalability video encoder 81 for the basic video 0 and the extended scalability video encoder 82 for the video 1 are provided. In this case, stereo video service is enabled. In addition, three view service is possible by basic video 0, video 1, and video 2, and five view service is possible by basic video 0 through video 4.

The scalable coding technique for the multi-view video according to the present invention as described above is summarized as follows.

First, a single base video (e.g., video 0) is separated at multiple layers using a spatial filter on the spatial axis. The space-time scalable video coding is performed through hierarchical motion prediction on the separated low resolution image frame. Also, the space-time scalable video coding is performed through hierarchical motion prediction on the time axis with respect to the separated high resolution image frame. The coded low resolution image and the coded at least one high resolution image are multiplexed to generate a bit stream. Scalable video coding for such basic video is the same as the conventional method.

Next, we look at extended scalable video coding for multiview videos.

First, at least one peripheral video, which is a video of itself and a reference video, is input and separated into multi-layer resolution using a spatial filter on the spatial axis of the input video and the surrounding video. In addition, the spatio-temporal scalable video coding is performed through hierarchical motion prediction with respect to the separated low resolution image frame by referring to the neighboring frame as the reference frame as well as its adjacent frame on the time axis. In addition, scalable video coding is performed through hierarchical motion prediction with respect to the separated high resolution image frame by referring not only to a lower layer of its own video frame but also a lower layer of a neighboring video frame as a reference video. The coded low resolution image and the coded at least one high resolution image are multiplexed to generate a bit stream.

This scalable video coding differs from scalable video coding for single-view video using only its neighboring and lower layer frames as reference frames, as well as surrounding lower layer frames as well as its lower layer frames. use.

Meanwhile, the scalable video decoding apparatus for multiview video according to the present invention is performed in the reverse of the encoding apparatus as described above.

Referring to the decoding apparatus according to the present invention, the decoding apparatus includes: an elementary scalability video decoder for receiving a scalable coded bitstream for one elementary video and reconstructing it through inverse time transform and inverse spatial transform; And receiving a scalable coded bitstream through the spatiotemporal prediction of the own video and the reference video photographed at the same time, and inverse spatiotemporal space according to whether the sublayer of another video frame that is the reference video as well as its own lower layer is referenced. After reconstructing at least one high resolution image frame through prediction, and reconstructing the low resolution image frame through inverse spatiotemporal prediction according to whether the other video frame is referenced in the same time axis as well as its adjacent frame, the reconstructed high resolution image And a plurality of extended scalability video decoders for reconstructing the image through inverse spatial filtering on the frame and the reconstructed low resolution image frame.

In the decoding apparatus according to the present invention, the elementary scalability video decoder has the same structure as the existing scalable video decoder. Therefore, detailed description thereof will be omitted.

The extended scalability video decoder according to the present invention includes a demultiplexer for demultiplexing a received bitstream, and a reference video as well as a lower layer of its own video with respect to a high-resolution video signal output by the demultiplexer. At least one enhancement layer decoder that performs scalable decoding through inverse spatiotemporal motion prediction according to whether a lower layer is referred to the video, and an inverse spatiotemporal space for its video frame in a low resolution video signal output by the demultiplexer A base layer decoder that performs scalable decoding by performing inverse motion prediction on a reference video frame on a time axis as well as motion prediction, a high resolution image reconstructed by the enhancement layer decoder and a low resolution image reconstructed by the base layer decoder Inverse spatial filtering for It includes an inverse spatial video filtering unit for reconstructing the image through.

Here, the base layer decoder and the enhancement layer decoder perform inverse operations of the base layer encoder and the enhancement layer decoder described above, and a detailed structure and operation thereof will be omitted.

The enhancement layer decoder may include a flag indicating whether to use motion vector information of a lower layer and a reference index of a lower layer for another video as prediction information when decoding a high resolution video signal, and a lower layer. Scalable decoding is performed by referring to a flag indicating whether an intra block type of is used as prediction information, a flag indicating whether a difference image value of a lower layer is used, and an index of a reference view used for prediction.

On the other hand, the method of the present invention as described above can be written in a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. The recording medium may include any type of computer readable recording medium.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

1 is a diagram for explaining a general scalable video coding concept;

2 is a block diagram illustrating a scalable video coding apparatus for multiview video according to an embodiment of the present invention;

3 is a functional block diagram of an extended scalability video encoder according to the present invention;

4 is a diagram for explaining a frame reference structure for multi-layer video compression for multi-view video according to the present invention;

FIG. 5 is a diagram for explaining an inter-layer reference structure for FIG. 4; FIG.

FIG. 6 is a diagram for explaining a frame reference structure for a B frame motion prediction structure for multiview video according to the present invention; FIG.

FIG. 7 is a diagram for explaining an inter-layer reference structure of FIG. 6; FIG.

8 is a block diagram illustrating a scalable video coding apparatus for multiview video according to another embodiment of the present invention;

FIG. 9 illustrates a frame reference structure for multi-layer video compression of FIG. 8.

Claims (18)

In the scalable video coding apparatus for a multi-view video, An elementary scalability video encoder that separates one elementary video into multi-layer resolution through spatial filtering and performs scalable video coding through space-time prediction for each of the separated low resolution image frame and the at least one high resolution image frame; And Receives at least one other video as its own video and reference video captured at the same time point, and separates them into multi-layer resolution through spatial filtering, and separates the low-resolution video frames in the same time axis as well as their adjacent frames. Scalable video coding is performed through space-time prediction by referring to other video frames, and scalable through space-time prediction by referring not only to its own lower layer but also to the lower layer of the other video frame for at least one separated high resolution image frame. Multiple extended scalability video encoders to perform video coding Scalable video coding apparatus for a multi-view video comprising a. The method of claim 1, The extended scalability video encoder, Spatial video filtering means for separating the input video and other video which is the reference video into multi-layer resolution through spatial filtering; Base layer encoding for separating low-resolution images of the spatial video filtering means through temporal filtering and performing scalable coding through motion prediction for a reference video frame in the time axis as well as space-time motion prediction for its own video frame. Way; Separating high-resolution images of the spatial video filtering means through temporal filtering, and performing scalable coding through space-time motion prediction by referring to a lower layer of another video as well as a lower layer of its own video. At least one enhancement layer encoding means; And Multiplexing means for multiplexing the output of the base layer encoding means and the output of the enhancement layer encoding means and outputting one bitstream Scalable video coding apparatus for a multi-view video comprising a. The method of claim 2, The enhancement layer encoding means, As a result of the coding, a flag indicating whether motion vector information of a lower layer is used, a flag indicating whether a reference index of a lower layer of another video is used as prediction information, and a type of intra block of the lower layer are predicted information. And a flag indicating whether or not to use a flag, and a flag indicating whether to use a difference image value of a lower layer, respectively, and indicating an index of a used reference view. The method of claim 2, The enhancement layer encoding means, And two-dimensional spatial interpolation means for performing two-dimensional spatial interpolation on the reconstructed video frame for intra prediction for the intra block. The method of claim 2, The enhancement layer encoding means, A scalable video coding apparatus for a multiview video, characterized by encoding on the temporal and spatial axes through inter-frame motion, difference image, and intra prediction. The method of claim 5, wherein The enhancement layer encoding means, For motion prediction, motion prediction is performed using a value obtained by doubling a motion vector of a base layer image (low resolution image) of one's own video and a base layer image (low resolution image) of another video as a reference video. Scalable video coding apparatus for multiview video. The method of claim 5, wherein The enhancement layer encoding means, For predicting the difference image, the difference image prediction is performed by interpolating a residual layer after the prediction of the base layer image (low resolution image) of the video and the base layer image (low resolution image) of the other video as the reference video. Scalable video coding apparatus for multiview video. In the scalable video coding method for a multi-view video, (a) separating one base video into multilayer resolution and performing scalable video coding through space-time prediction; And (b) receiving at least one other video as the own video and the reference video captured at the same time point, and separating the input videos into multi-layer resolutions and performing scalable video coding through space-time prediction But In step (b), (c) performing scalable video coding through space-time prediction with reference to the other video frame on the same time axis as well as its adjacent frame for the low resolution image frame; (d) performing scalable video coding through spatiotemporal prediction on at least one high resolution image frame by referring to not only its own lower layer but also the lower layer of the other video frame Scalable video coding method for a multi-view video comprising a. The method of claim 8, As a result of performing step (d), a flag indicating whether to use motion vector information of a lower layer, a flag indicating whether a reference index of a lower layer for another video is used as prediction information, and an intra block of the lower layer And setting a flag indicating whether a type of is used as prediction information, a flag indicating whether to use a difference image value of a lower layer, and indicating an index of a used reference view. Scalable video coding method. The method of claim 8, Step (d) is. A scalable video coding method for a multiview video, characterized by encoding the video through inter-frame motion on the temporal and spatial axes, and difference and intra prediction. The method of claim 8, In step (d), A 2D spatial interpolation method for multi-view video, characterized in that to perform two-dimensional spatial interpolation on the reconstructed video frame for intra prediction for the intra block. The method of claim 8, In step (d), For motion prediction, motion prediction is performed using a value obtained by doubling a motion vector of a base layer image (low resolution image) of one's own video and a base layer image (low resolution image) of another video as a reference video. Scalable video coding method for multiview video. The method of claim 8, In step (d), For predicting the difference image, the difference image prediction is performed by interpolating a residual layer after the prediction of the base layer image (low resolution image) of the video and the base layer image (low resolution image) of the other video as the reference video. Scalable video coding method for multiview video. In the scalable video decoding apparatus for a multi-view video, An elementary scalability video decoder for receiving a scalable coded bitstream for one elementary video and reconstructing it through inverse time transform and inverse spatial transform; And Receives scalable coded bitstreams of own video and reference video captured at the same point in time, and inverse spatiotemporal prediction according to whether not only its own lower layer but also the lower layer of another video frame that is the reference video After reconstructing at least one high resolution image frame, and reconstructing the low resolution image frame through inverse spatiotemporal prediction according to whether the other video frame is referenced in the same time axis as well as its adjacent frame, the reconstructed high resolution image frame And a plurality of extended scalability video decoders for reconstructing the image through inverse spatial filtering on the reconstructed low resolution image frame Scalable video decoding apparatus for a multi-view video comprising a. The method of claim 14, The extended scalability video decoder, Demultiplexing means for demultiplexing the received bitstream; At least one of performing scalable decoding through inverse spatiotemporal motion prediction according to whether the lower layer of its own video as well as the lower layer of other video which is a reference video is referred to the high resolution image signal output by the demultiplexing means; Enhancement layer decoding means; A base layer decoding means for performing scalable decoding on the low resolution video signal output by the demultiplexing means through not only the inverse spatiotemporal motion prediction for its video frame but also the inverse motion prediction for the reference video frame on the time axis; Inverse spatial video filtering means for reconstructing the image through inverse spatial filtering on the high resolution image reconstructed by the enhancement layer decoding means and the low resolution image reconstructed by the base layer decoding means Scalable video decoding apparatus for a multi-view video comprising a. The method of claim 15, The enhancement layer decoding means, A flag indicating whether to use motion vector information of a lower layer, a flag indicating whether to use a reference index of a lower layer for another video as prediction information, and whether to use a type of an intra block of the lower layer as prediction information. A scalable video decoding apparatus for multiview video, characterized in that scalable decoding is performed by referring to an indicating flag, a flag indicating whether a difference image value of a lower layer is used, and an index of a reference view used for prediction. In the scalable video decoding method for a multi-view video, (a) performing scalable video decoding through inverse space-time prediction on one elementary video; And (b) receiving a scalable coded bitstream by referring to at least one other video as its own video and the reference video captured at the same time, and performing scalable video decoding through inverse space-time prediction and inverse spatial filtering Including, In step (b), (c) performing scalable video decoding on the demultiplexed high resolution image frame through inverse spatiotemporal prediction according to not only its own lower layer but also reference information of the lower layer of the other video frame; And (d) performing scalable video decoding on the demultiplexed low resolution video signal through inverse spatiotemporal prediction based on reference information of the other video frame on the same time axis as well as its adjacent frame; Scalable video decoding method for a multi-view video comprising a. The method of claim 17, In step (c), a flag indicating whether to use motion vector information of a lower layer, a flag indicating whether a reference index of a lower layer for another video is used as prediction information, and a type of an intra block of the lower layer Multi-view video, characterized in that scalable video decoding is performed using a flag indicating whether or not to use prediction information, a flag indicating whether a difference image value of a lower layer is used, and index information of a reference view used for prediction. Scalable video decoding method.

Images