JP5170786B2 - Multi-view video scalable coding and decoding method, and coding and decoding apparatus - Google Patents

Description

  The present invention relates to a multi-view video scalable coding and decoding technique, and more particularly, to compress multi-view video shot by a plurality of cameras and transmit the multi-view video to a multi-level spatial and temporal scalable coding. A multi-view video scalable coding and decoding method for providing 2D or 3D video services to various types of video terminals by compressing and transmitting using technology, and decoding at a receiving end, and coding and decoding It relates to a coding device.

  The basic principle of compressing data is to remove the spatial and temporal redundancy elements of the data. Spatial overlap refers to the same color or object being repeated in a video. In addition, temporal overlap refers to the fact that adjacent pictures in a moving picture are almost unchanged, or the same sound repeats in audio. In a typical video coding method, temporal overlap is removed by temporal filtering based on motion compensation, and spatial overlap is removed by spatial transformation.

  In order to transmit multimedia data generated after data duplication is removed, a transmission medium is required, but the performance varies depending on the transmission medium. There is a scalable video coding technique to support transmission media of various speeds or to transmit multimedia data at a transmission rate adapted to the transmission environment.

  The scalable video coding technique is a video resolution obtained by cutting out a part of a bit stream in accordance with peripheral conditions such as a transmission bit rate, a transmission error rate, and system resources with respect to an already compressed bit stream (bit-stream). It means a coding technique that enables adjustment of the frame rate, SNR (Signal-to-Noise Ratio), and the like.

  FIG. 1 is a reference diagram for explaining the concept of general scalable coding.

  As shown in the figure, scalable video coding implements temporal scalability using temporal transformation for an input video, and implements spatial scalable using two-dimensional spatial transformation. In addition, scalable video coding implements image quality scalability using texture coding. In motion coding, motion information when implementing each spatial scalable encoding is encoded so as to be scalable. One bit stream is generated through such a coding algorithm.

  In scalable video coding, there are a motion compensated temporal filter (MCTF) and a hierarchical B picture (hierarchical B-pictures) as methods for providing temporal scalability and improving compression efficiency.

  MCTF refers to performing wavelet transform using motion information in the time direction within a video sequence. The wavelet transform is implemented using a lifting scheme. The lifting scheme is composed of three types of operations: polyphase decomposition, prediction, and update.

  The hierarchical B picture is H.264. MMCO (Memory Management Control Operation) that manages DPB (Decoded Picture Buffer) that can store 16 pictures used in H.264, and RPLR (Reference Picture List Reordering) are implemented in various ways. be able to.

  Recently, due to technological developments and user requests, video information on scenes at various times is provided to viewers, and the corresponding video information is broadcast to each viewer through operations on the scenes that the viewer wants to see. Services that allow users to edit and view information sent from stations are being studied. In order to provide such a service, a video compression technique for multi-view video is necessary.

  Multi-view video compression is for simultaneously coding, storing and transmitting compressed video input from a plurality of cameras providing multi-view video. When storing and transmitting multi-view video without compression, the size of the data is so large that a large transmission bandwidth is needed to transmit the data to users over the broadcast network or wired / wireless Internet in real time. A width is required.

  In general, multi-view video coding and decoding may be performed by a method of decoding after each video sequence is independently coded and transmitted. This is because conventional MPEG-1 / 2/4 and H.264. It can be easily implemented through H.261 / 263/264. However, this does not remove the redundancy between videos that occurs because the same object is shot with multiple cameras.

  As a technique for removing redundancy between videos, there is a scalable video coding technique. In general, single-point video scalable coding technology separates single-point video with multi-layer resolution using spatial filter as spatial axis and hierarchical bi-directional motion prediction on temporal axis Spatiotemporal scalable via. Here, at each resolution, motion, difference images, and intra prediction can be performed for effective prediction between layers. Also, scalability in terms of image quality can be provided through entropy coding with hierarchical representation in transform encoding.

  However, the conventional scalable video coding technology is fixed as the video space image quality at a single point in time, and it can be used in terminals with various environments and various computing capabilities. When video is played back, there is a problem that many transmission rates and a lot of overhead may occur in terms of the video decoder.

  Accordingly, the present invention has been proposed to solve the above-described problems of the prior art, and the motion of referring to surrounding images on the time axis and the video (space) axis in order to compress the multi-view video. Compress video more efficiently via motion, difference images, and intra prediction at other resolutions of the surrounding video to provide scalability on the spatial axis in multi-view video as well as prediction compensation Another object of the present invention is to provide a multi-video scalable coding method and apparatus capable of providing various video services to terminals in various environments.

  It is another object of the present invention to provide a decoding method and apparatus for receiving and decoding a signal that is scalable coded for multi-video.

  Other objects and advantages of the present invention can be understood by the following description, and become more apparent from the embodiments of the present invention. Further, it should be easily understood that the objects and advantages of the present invention can be realized by the means and combinations shown in the claims.

Scalable video coding apparatus according to the present invention for achieving the above object, a scalable video coding apparatus for multiview video separates the foundation video image frames with multilayer resolutions through spatial filtering, and performing basic scalability video encoder scalable video coding via spatial prediction time for each of the separated low-resolution image frame and one or more high-resolution image frame, itself taken at the same time By the video and the reference video, at least one or more other videos are input and separated into image frames having multi-layer resolution via spatial filtering, and the adjacent frames to the separated low-resolution image frames As well as the first video and reference Is input and at least one or more video is a video separates the image frames with multilayer resolutions through spatial filtering, adjacent frames of the first video frame to the separated low-resolution image frame was And performing scalable video coding through spatio-temporal prediction with reference to a reference video frame on the same temporal axis, and sub-layering the first video with respect to the one or more separated high-resolution image frames, and comprising the scalability video encoders perform more expansion scalable video coding through a spatiotemporal prediction with reference to the lower layer of the reference video, a.

Also, scalable video coding method according to the present invention, there is provided a scalable video coding method for multi-view video, to separate the image frames with multilayer resolutions through spatial filtering (a) foundation video, the separation Performing scalable video coding via spatio-temporal prediction on each of the low-resolution image frame and one or more high-resolution image frames, and (b) at least one of a first video and a reference video is bi and Deo input described above, by separating the image frames with multilayer resolutions through spatial filtering, the separated first video frame to the low-resolution image frames in adjacent frames and the same time axis Scalable video via spatio-temporal prediction with reference to reference video frame Perform de-coding, and perform scalable video coding through spatio-temporal prediction with reference to the lower layer of the first video and the lower layer of the reference video for the one or more separated high-resolution image frames. Including the steps of :

On the other hand, the decoding apparatus according to the present invention, there is provided a scalable video decoding apparatus for multi-view video, and receives the scalable coded bit stream for foundation video, via the transform and inverse spatial transform inverse time and basic scalability video decoder to restore Te, through the spatial prediction when the first video and the reference video receives scalable coded bitstream, the first video of a lower layer and the reference video of a lower layer through inverse temporal and spatial prediction to restore one or more high-resolution image frame in accordance with the read permission information, the read permission information of the frame adjacent frame and the reference video with the same time axis of the frame included in the first video And after restoring the low-resolution image frame via reverse spatio-temporal prediction, Comprising a high resolution image frames reconstructed, and a plurality of extended scalability video decoder for reconstructing an image through the inverse spatial filtering for the low resolution image frames the restored.

Preferably, the enhanced scalability video decoder is configured to demultiplex the received bitstream, and to the high-resolution image signal output by the demultiplexing unit with respect to the first video. depending on the read permission information of a lower layer against the lower layer and the reference video, and one or more enhancement layer decoding means you perform scalable decoding through the inverse space-time motion estimation, by the demultiplexing means output, low-resolution image signal, performs a scalable decoding through the inverse motion prediction for a video frame of a reverse spatiotemporal motion prediction and the reference video on the time axis with respect to the video frame of the first video base layer decoding means A high-resolution image restored by the enhancement layer decoding means, and the Comprising the inverse spatial video filtering means for restoring the image through the inverse spatial filtering for the low-resolution image restored by the foundation hierarchical decoding means.

Preferably, the enhanced layer decoding unit, a flag representing a flag indicating whether to use the motion vector information of the lower layer, that the one of whether to use the reference index of a lower layer with respect to the reference video as prediction information, the a flag indicating whether the permission to use the type of intra block of a lower layer as a prediction information, the flag for the availability of the difference between image values of the lower layer, by referring to the index of the reference view were used to predict scalable Perform decoding.

Further, a scalable video decoding method for multi-view video, and restoring via (a) receives the scalable coded bit stream for foundation video, inverse time during transform and inverse spatial transform , (B) receiving a bitstream that is scalable-coded through spatio-temporal prediction with respect to the first video and the reference video , and reverse depending on whether or not the lower layer of the first video and the lower layer of the reference video are referred to One or more high-resolution image frames are restored through spatio-temporal prediction, and the inverse spatio-temporal prediction is performed according to whether or not the reference video frame can be referenced on the same time axis as the adjacent frame of the frame included in the first video. After restoring the low resolution image frame via the restored high resolution image frame and the restored And including the steps of reconstructing an image through the inverse spatial filtering on the low-resolution image frames.

Preferably, the (a) step, a flag indicating whether to use the motion vector information of the lower layer, the flag indicating that the one of the permission to use as prediction information reference index of a lower layer with respect to the reference video, the lower scalable by using the flag indicating that the one of the permission to use the type of intra block of the hierarchy as the prediction information, and flags for the availability of the difference between image values of the lower layer, and the index information of the reference view were used to predict Perform video decoding.

  The present invention as described above can effectively compress multi-view video by extending the space-time hierarchical structure of a general scalable coding technique between multi-view videos. In addition, the present invention provides a video service to be scalable to various types of 2D or 3D terminals by configuring a hierarchical structure on the time and space axes for multi-view video. it can.

It is a figure for demonstrating the concept of general scalable video coding. 1 is a block configuration diagram of a scalable video coding apparatus for multi-view video according to an embodiment of the present invention. FIG. FIG. 2 is a functional block configuration diagram of an extended scalability video encoder according to the present invention. FIG. 6 is a diagram illustrating a frame reference structure for multi-layer video compression for multi-view video according to the present invention. FIG. 5 is a diagram for explaining a reference structure between hierarchies with respect to FIG. 4. FIG. 6 is a diagram for explaining a frame reference structure for a B frame motion prediction structure for multi-view video according to the present invention. It is a figure for demonstrating the reference structure between the hierarchy with respect to FIG. FIG. 5 is a block diagram of a scalable video coding apparatus for multi-view video according to another embodiment of the present invention. FIG. 9 is a diagram illustrating a frame reference structure for multi-layer video compression with respect to FIG. 8.

  The detailed objects, features, and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs will The technical idea of can be easily implemented. Further, in describing the present invention, when it is determined that a specific description of a known technique related to the present invention may unnecessarily obscure the gist of the present invention, a detailed description thereof will be omitted. To do. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 2 is a diagram illustrating an embodiment of a multi-view video scalable coding apparatus according to the present invention.

  In the figure, when videos taken by five cameras from video 0 to video 4 are input, each video is compressed by a scalable video encoder.

  A scalable coding apparatus for multi-view video according to the present invention performs basic coding on video 0, which is a basic video, by performing two-dimensional spatial transformation and temporal transformation, and performs scalable coding through motion coding and texture coding. 21 and at least one other surrounding video that is a reference video, as well as its own corresponding video, are input and separated at multi-layer resolution through spatial filtering and temporal filtering, and the separation A plurality of extended scalability video encoders 22 to 25 that perform scalable coding with reference to space-time hierarchical image information and compression parameters of surrounding video as well as the video of itself to the generated frames. .

  In this figure, video 0 is defined as a basic video, and a basic scalability video encoder 21 that performs scalable coding on video 0 has the same structure as a conventional single-point video scalable coding apparatus. Therefore, the basic video has a structure that can be compatible with a conventional scalable coding apparatus.

  Also, in order to compress not only the video itself but also the surrounding video that is the reference video so as to be scalable, video 1 to video 4 are compressed via the extended scalability video encoders 22 to 25. Carry out. The extended scalability video encoders 22 to 25 receive their own video and the reference video, which is a peripheral video, and are separated at a multi-layer resolution through spatial filtering, like a conventional single-point video scalable coding device. Then, scalable coding is performed on each separated frame by referring not only to the lower layer for the own frame but also to the lower layers for the surrounding frames on the time and space axes. For example, the extended scalability video encoder 25 that compresses the video 4 performs scalable video coding on the video 4 by predicting and referring to the multi-layer space-time resolution image information of the base video 0. The extended scalability video encoder 23 that compresses the video 2 performs compression on its own video through bi-directional prediction using the multi-layer spatio-temporal resolution image information of the basic video 0 and the video 4.

  Accordingly, when only the basic scalability video encoder 21 is used, the conventional two-dimensional video service is possible, and the basic scalability video encoder 21 for the basic video 0 and the extended scalability video encoder 25 for the video 4 are used. Stereo video service will be possible. In addition, by selectively combining the extended scalability video encoders 22 to 25 and the basic scalability video encoder 21, a three-view or five-view video service becomes possible.

  The structure and function of such an extended scalability video encoder will be specifically described with reference to FIG.

  FIG. 3 is a detailed functional block configuration diagram of the extended scalability video encoder according to the present invention.

  The extended scalability video encoder according to the present invention includes a spatial video filtering unit 31 that separates a video and a reference video with multi-layer resolution through spatial filtering, and an output of the spatial video filtering unit 31. Temporal video filtering units 330 and 340 for temporally separating images through temporal filtering, and temporal and spatial motion of the video frames for temporal low frequency images output from the temporal video filtering unit 330 In addition to the prediction, the base layer encoder 33 that performs scalable coding through motion prediction with respect to the reference video frame on the time axis, and the temporal high-frequency image output from the temporal video filtering unit 340, Not only the lower hierarchy At least one enhancement layer encoder 34 that performs scalable coding with reference to a lower layer of other neighboring video that is a reference video, and an output of the base layer encoder and an output of the enhancement layer encoder are multiplexed into one And a multiplexing unit 35 that outputs a bit stream.

  The spatial video filtering unit 31 uses the MCTF or the hierarchical B structure by inputting a video of its own taken through its camera and other peripheral videos set as a reference video in a predetermined time unit. Separation with multi-layer resolution through spatial filtering.

  The base layer encoder 33 and the enhancement layer encoder 34 include temporal video filtering units 330 and 340, motion coding units 331 and 341, difference units 332 and 342, spatial transformation units 333 and 343, quantization units 334 and 344, and entropy. Encoders 335 and 345 may be provided. The basic layer encoder 33 and the enhancement layer encoder 34 are similar in structure to the conventional scalable video encoder, and each functional unit will be briefly described here.

  The temporal video filtering unit 330 provided in the base layer encoder 33 separates the low-frequency images separated by the spatial filtering on the time axis through filtering using MCTF or hierarchical B structure. Also, the temporal video filtering unit 340 included in the enhancement hierarchical encoder 34 separates the high-frequency image separated by the spatial filtering in time axis through filtering using MCTF or hierarchical B structure. .

  The motion coding units 331 and 341 are usually configured by motion estimation and motion compensation. In the motion estimation, a motion estimation of a current frame is performed based on a reference frame among input video frames, and a motion vector for forward prediction or bidirectional prediction is obtained. Here, the present invention can use not only its own frame but also a surrounding frame as a reference frame at the time of motion estimation. A widely used algorithm for such motion estimation is a block matching algorithm. That is, the motion coding units 331 and 341 estimate the displacement when the error is minimized while moving the given motion block in units of pixels within the specific search region of the reference frame as a motion vector. As a result of the motion estimation, motion data such as a calculated motion vector, a motion block size, and a reference frame number are provided to the entropy encoding units 335 and 345. In addition, the motion compensation is performed by performing motion compensation on a forward reference frame, a backward reference frame, or a bidirectional reference frame using a motion vector calculated by motion estimation. Generate a temporal prediction frame for the frame.

  The differentiators 332 and 342 remove video temporal redundancy by subtracting the current frame and the temporal prediction frame. Spatial transformation units 333 and 343 remove spatial redundancy using a spatial transformation method that supports spatial scalability for the frames from which temporal redundancy is removed by the differentiators 332 and 342. In such a spatial transformation method, DCT (Discrete Cosine Transform), wavelet transformation (wavelet transform), etc. are mainly used.

  The quantization units 334 and 344 quantize the transform coefficients obtained by the spatial transform units 333 and 343. Quantization means an operation of dividing the transform coefficient expressed by an arbitrary real value by a predetermined period and expressing it as a discrete value and matching it with a predetermined index.

  The entropy encoding units 335 and 345 generate lossless encoding of the transform coefficients quantized by the quantization units 334 and 344 and the motion data provided by the motion estimation to generate an output bitstream. As such a lossless coding method, arithmetic coding, variable length coding, or the like can be used.

  On the other hand, the present invention may use intra prediction for intra blocks prior to spatial transformation. For this, the enhancement layer encoder may include a function of performing a two-dimensional space interpolation by receiving a reference frame restored from a lower layer encoder and an intra prediction function.

  In general, inter prediction is performed by finding a block most similar to a block of the current frame (current block) from a reference frame, obtaining a prediction block that can best represent the current block, and then obtaining the current block. This is a method for quantizing the difference between a block and the prediction block. In the inter prediction, bi-directional prediction using two reference frames, forward prediction using a previous reference frame, and subsequent reference frames according to a method of referring to a reference frame. For example, there is a backward prediction.

  On the other hand, intra prediction is a method of predicting a current block using a frame adjacent to the current block among neighboring blocks of the current block. Intra prediction uses only information in the current frame and differs from other prediction methods in that it does not refer to other frames in the same layer or frames in other layers.

  Intra base prediction can be used when the current frame has lower-layer frames with the same temporal position. The macroblock of the current frame can be efficiently predicted from the macroblock of the corresponding base frame. That is, the difference between the macroblock of the current frame and the corresponding macroblock of the base frame is quantized. If the resolution of the lower layer and the resolution of the current layer are different from each other, the macroblock of the base frame is upsampled to the resolution of the current layer before obtaining the difference.

  The remaining prediction is an extension of inter prediction in a single layer to a multi-layer form, and instead of directly quantizing the difference generated in the inter prediction process of the current layer, the difference and the lower layer This is a technique of differentiating again the difference generated in the inter prediction process and quantizing the result.

  On the other hand, in the present invention, the enhancement layer encoder is a base layer image for its own video (low resolution image) and a basis for another video which is a reference video for motion prediction when encoding a high resolution image frame. A value obtained by doubling the motion vector of the hierarchical image (low-resolution image) is used.

  In the present invention, the enhancement layer encoder also encodes a base layer image (low-resolution image) for its own video and other video that is a reference video for prediction of a difference image when encoding a high-resolution image frame. After the base layer image (low resolution image) is predicted, the residual image is interpolated to predict the difference image.

  Also, in the present invention, the enhancement layer encoder is a base layer image for its own video (low-resolution image) and a base layer for another video that is a reference video for intra prediction when encoding a high-resolution image frame. Intra prediction is performed on the image (low resolution image) in the intra prediction mode.

  FIG. 4 is a diagram for explaining a prediction reference structure at the time of scalable video coding according to the present invention.

  In the figure, P macroblock represents unidirectional prediction, and B macroblock represents bidirectional prediction. The prediction reference according to the present invention refers to a structure in which unidirectional prediction (P macroblock) and bidirectional prediction (B macroblock) are predicted not only in the time axis and the space (view) axis but also in multiple resolution hierarchies. It is.

  The figure shows, as one embodiment, only a two-layer structure composed of a base layer and one enhancement layer, and more layers are configured by repeatedly expanding the enhancement layer. obtain.

  In FIG. 4, 41 represents a prediction reference performed by the base layer encoder and the enhancement layer encoder in the base scalability video encoder (21 in FIG. 2), and 42 is an extended scalability video for the video 1 shown in FIG. 2 represents a prediction reference performed at the base layer encoder and the enhancement layer encoder in the encoder 22, and 43 is performed at the base layer encoder and the enhancement layer encoder in the extended scalability video encoder 23 for the video 2 illustrated in FIG. Represents a prediction reference.

  In other words, in FIG. 4, the base layer 0 (L0) is the base layer encoder of the base scalability video encoder 21, the extended scalability for the video 1, the base layer encoder of the video encoder 22, and the extended scalability for the video 2. The prediction reference structure performed by each of the base layer encoders of the video encoder 23 is shown.

  Similarly, in FIG. 4, enhancement layer 1 (L1) is an enhancement layer encoder of basic scalability video encoder 21, enhanced scalability for video 1, enhancement layer encoder of video encoder 22, and enhanced scalability for video 2. The prediction reference structure performed by each of the enhancement layer encoders of the video encoder 23 is shown.

  As shown in FIG. 4, the base layer encoder of the base scalability video encoder 21 performs scalable video coding by predicting and referring to an adjacent frame with respect to its own low-resolution image frame on the time axis as in the prior art. In addition, the base layer encoder of the extended scalability video encoder 22 for video 1 uses a reference peripheral video frame in video 0 frame and a video 2 frame located on the same time axis for its own frame. Perform bi-directional prediction. In addition, the base layer encoder of the extended scalability video encoder 23 for video 2 performs unidirectional prediction with reference to the basic video 0 and simultaneously performs bi-directional prediction using its own frame.

  On the other hand, the enhancement layer 1 (L1), which is an upper layer of the basic layer, refers not only to the space and temporal prediction for its own video frame, but also to the lower layer's own frame and its surrounding frames. Carry out predictions. In the drawing, the three displays indicated by ◯ and X in each macro block are for indicating whether or not the lower layer can be referred to. Here, among the three displays, “○” and “×” in the middle indicate whether or not the lower layer can be referenced for the video frame of itself, and “○” and “×” displayed above and below indicate the surroundings. This indicates whether or not the lower layer can be referred to the video frame.

  As shown in FIG. 4, the enhancement layer encoder of the basic scalability video encoder 21 performs scalable video coding with reference to a frame in a lower layer of the conventional scalability encoder. Also, the enhanced hierarchical encoder of the extended scalability video encoder 22 for video 1 not only refers to its own lower layer frame, but also the lower layer frame of the surrounding video frame in video 0 and the lower layer of video 2 Referring to the frame, bi-directional prediction for the frame is performed. Also, the enhanced layer encoder of the extended scalability video encoder 23 for video 2 performs prediction not only by referring to its own lower layer frame but also by referring to the lower layer frame of the basic video 0.

  FIG. 5 shows the reference structure shown in FIG. 4. The reference structure is expressed by “◯” and “X” on the space (view) hierarchy axis with time fixed. In the figure, the case of three layers is shown. In this figure, 51 represents the reference structure of video 0, which is the basic video, 52 represents the reference structure of video 1, and 53 represents the reference structure of video 2.

  In FIG. 5, since the macroblock of the enhancement layer 2 for the video 0 51 is “◯” only in the middle and “x” on both sides, the conventional scalable video coding (see FIG. Coding is performed through motion, difference image, and intra prediction used in SVC (Scalable Video Coding). On the other hand, since the macro block of the enhancement layer 2 for the video 1 52 is not only in the middle but also “O” on the right side and the left side, not only the lower layer for the own video but also the lower layer for the surrounding video And perform scalable video coding. Further, since the macro block of the enhancement layer 2 for the video 253 is represented by “o” in the middle and the left side, the scalable video coding is performed by referring to the lower layer of the video and the lower layer of the video 0. To do.

As described above, in order to indicate whether or not the lower layer can be referred to, a scalable video syntax is required in which information on the reference layer video is added according to whether or not the lower layer can be referred to.
In the following syntax, ref_view_Idx represents the view number of the reference video in the lower layer. Here, base_mode_flag is a flag indicating whether the motion vector information of the lower layer is used for motion prediction of the current block. When the value is 1, the view indicating which view of the lower layer the motion vector information is used. It should represent the number ref_view_Idx.

  The base_mode_refinement_flag is also a flag that indicates whether the motion vector information of the lower layer is used for motion vector prediction of the current block. The difference from the base_mode_flag is that the reference index of the lower layer is also used as prediction information. Therefore, when the flag is 1, ref_view_Idx indicating which view's motion vector and reference index information in the lower hierarchy is used should be displayed. Intra_base_flag is a flag that represents the use of the intra block type of the lower layer as the prediction information of the block type of the current block. When 1, the information about the intra prediction mode of the lower layer is used in the current block. It becomes like this. Therefore, ref_view_Idx, which is a view number for which view of the intra-block type information of the lower hierarchy, is to be used.

  The residual_prediction_flag is a flag indicating whether or not the difference image value of the lower layer is used for prediction of the difference image of the current block. If 1, the difference image information of the lower layer is up-sampled and used. Again, at this time, ref_view_Idx, which is the view number corresponding to which view's difference image information to use, should be displayed. Table 1 shows the syntax of scalable video as described above.

  FIG. 6 represents one embodiment of a reference structure in the case of a conventional B frame structure that is not a hierarchical B structure in the basic scalable structure. In FIG. 6, 61 represents a reference structure for video 0, which is the basic video, 62 represents a reference structure for video 1, and 63 represents a reference structure for video 2.

  FIG. 7 shows the reference structure shown in FIG. 6. The reference structure is represented by “◯” and “X” on the spatial view hierarchy axis with time fixed. Similarly, FIG. 7 shows the case of three layers. In FIG. 7, reference numeral 71 represents a reference structure of video 0 which is a basic video, 72 represents a reference structure of video 1, and 73 represents a reference structure of video 2.

As shown in FIGS. 6 and 7, the videos 0 61 and 71, which are basic videos, perform scalable video coding with reference to only their lower layer frames. However, video 1 and video 2 perform scalable video coding not only by referring to their own lower layer frames but also by referring to lower layers of neighboring videos.
The prediction reference structure according to the present invention as described above can be similarly applied to a conventional P frame structure that is not a B frame prediction structure.

  FIG. 8 is a block diagram of a scalable coding apparatus for multi-view video according to another embodiment of the present invention. FIG. 9 is a video 0 91 which is a basic video and a video 1 92 which is a peripheral video in FIG. And a reference structure for video 293.

  As shown in FIGS. 8 and 9, a basic scalability video encoder 81 that performs scalable coding for video 0, which is the basic video, refers to its own lower layer, similar to a conventional single-point video scalable coding apparatus. Perform scalable video coding. Therefore, in the case of basic video, compatibility with a conventional scalable coding device or the like is possible.

  In addition, video 1 to video 4 are subjected to scale-love video coding via the extended scalability video encoders 82 to 85. The extended scalability video encoders 82 to 85 separate images at multi-layer resolution and perform space-time prediction on the separated image frames, as in a conventional single-point video scalable coding apparatus. In addition, compression is performed with reference to the spatio-temporal hierarchical image information and compression parameters of the surrounding video.

  For example, as illustrated in FIG. 9, the enhancement layer 1 192 of the video 1 performs scalable video coding with reference to the lower layer of the video 0, and the enhancement layer 193 of the video 2 refers to the lower layer of the video 1 And perform scalable video coding. In other words, as shown in FIG. 8, according to another embodiment of the present invention, an extended scalability video encoder performs scalable video coding by sequentially referring to only one video immediately next.

  Accordingly, when only the basic scalability video encoder 81 is used, a conventional two-dimensional video service is possible, and the basic scalability video encoder 81 for the basic video 0 and the extended scalability video encoder 82 for the video 1 are provided. When used, a stereo video service becomes possible. In addition, a three-view service is possible with basic video 0, video 1, and video 2, and a five-view service with basic video 0 to video 4 is possible.

  As described above, the scalable coding technique for multi-view video according to the present invention is summarized as follows.

  First, a basic video (eg, video 0) is separated at a multi-layer resolution using a spatial filter in the spatial axis. In addition, the separated low-resolution image frame is subjected to space-time scalable video coding through hierarchical motion prediction on a time axis. In addition, the separated high-resolution image frame is subjected to spatio-temporal scalable video coding through hierarchical motion prediction on the time axis with reference to a lower layer. Then, the coded low resolution image and at least one coded high resolution image are multiplexed to generate a bitstream. The scalable video coding for the basic video is the same as the conventional method.

  Next, extended scalable video coding for multi-view video will be described.

  First, the own video and at least one or more surrounding videos that are reference videos are input, and the input own video and the surrounding videos are separated with multi-layer resolution using a spatial filter in the spatial axis. . In addition, the separated low-resolution image frame is subjected to spatio-temporal scalable video coding through hierarchical motion prediction with reference to surrounding frames as reference frames as well as adjacent frames on the time axis. In addition, the separated high-resolution image frame is not limited to a lower layer relative to the video frame on the time axis, but is also referred to a lower layer of a peripheral video frame that is a reference video, through hierarchical motion prediction. Perform scalable video coding. Further, the coded low-resolution image and at least one coded high-resolution image are multiplexed to generate a bitstream.

  Thus, enhanced scalable video coding differs from scalable video coding for single point-in-time video that uses only its own neighboring and lower layer frames as reference frames, as well as its own lower layer frames as reference frames. , Use peripheral lower layer frames.

  Meanwhile, the scalable video decoding apparatus for multi-view video according to the present invention is performed in reverse to the coding apparatus as described above.

  Referring to a decoding apparatus according to the present invention, a decoding apparatus receives a scalable coded bitstream for one basic video and restores it through inverse temporal transform and inverse spatial transform; Receive a bitstream that is scalable coded via spatio-temporal prediction for its own video and reference video taken at the same time, and not only its own lower layer but also other video frames that are reference video The at least one high-resolution image frame is restored through inverse spatio-temporal prediction according to whether or not the lower layer of the image can be referenced, and whether or not the other video frame can be referenced on the same time axis as well as the adjacent frame of itself. Accordingly, after restoring the low resolution image frame via inverse spatio-temporal prediction, the restored A plurality of enhanced scalability video decoders for restoring an image through inverse spatial filtering with respect to the high-resolution image frame and the restored low-resolution image frame.

  In the decoding apparatus according to the present invention, the basic scalability video decoder has the same structure as a conventional scalable video decoder. Therefore, a detailed description thereof will be omitted.

  In the present invention, the extended scalability video decoder includes: a demultiplexing unit that demultiplexes a received bitstream; and a high-level image signal output by the demultiplexing unit with respect to a lower layer for its own video. In addition, at least one enhancement layer decoder that performs scalable decoding through inverse spatio-temporal motion prediction according to whether or not a lower layer reference to another video that is a reference video is possible, and the demultiplexing unit A base layer decoder that performs scalable decoding on the output low-resolution image signal not only by reverse spatio-temporal motion prediction for its own video frame but also by reverse motion prediction for a reference video frame in the time axis; The high-resolution image restored by coding and the restoration by the base layer decoder And an inverse spatial video filtering unit that restores the original low-resolution image through inverse spatial filtering.

  Here, the base layer decoder and the enhancement layer decoder perform the reverse operations of the base layer encoder and the enhancement layer decoder described above, and a description of the specific structure and operation thereof will be omitted.

  The enhancement layer decoder, when decoding a high-resolution image signal, a flag indicating whether or not lower layer motion vector information can be used and a flag indicating whether or not a lower layer reference index for other video is used as prediction information. Scalable with reference to the flag indicating whether or not the intra-block type of the lower layer is used as prediction information, the flag for the availability of the difference image value of the lower layer, and the index of the reference view used for prediction Perform decoding.

  On the other hand, the method of the present invention as described above can be created as a computer program. The code and code segment constituting the program can be easily inferred by a computer programmer in the corresponding field. In addition, the created program is stored in a recording medium information storage medium that can be read by a computer, and is read and executed by the computer to implement the method of the present invention. The recording medium includes any form of recording medium readable by a computer.

  The present invention described above can be variously replaced, modified, and changed by those who have ordinary knowledge in the technical field to which the present invention belongs without departing from the technical idea of the present invention. It is not limited by the form and attached drawings.

  The present invention as described above can effectively compress a multi-view video by extending a spatio-temporal hierarchical structure of a general scalable coding technique between multi-view videos. Also, the present invention can provide a video service in a scalable manner to various types of two-dimensional or three-dimensional terminals by configuring a hierarchical structure on the temporal and spatial axes for multi-view video.

Claims (18)

A scalable video coding device for multi-view video,
Separating the foundation video image frames with multilayer resolutions through spatial filtering, through the spatial prediction time for each of the separated low-resolution image frame and one or more high-resolution image frame A basic scalable video encoder that performs scalable video coding,
A first video, the at least one or more video is a reference video input, separates the image frames with multilayer resolutions through spatial filtering, the relative said separated low-resolution image frame was 1 with reference to the reference video frames in adjacent frames and the same time axis of the video frame by performing the scalable video coding through a spatiotemporal prediction, the first with respect to the separated one or more high-resolution image frame was and scalability video encoders perform more expansion scalable video coding through a spatiotemporal prediction with reference to the lower layer and the reference video of a lower layer of the video,
A scalable video coding apparatus for multi-view video. The extended scalability video encoder is
And the inputted first video, and the spatial video filtering means for separating the image frame having the reference video and the multilayer resolutions through spatial filtering on,
The first video and the reference video is separated into a low-resolution image frames through temporal filtering, scalable coding via motion prediction against the reference video in the space-time motion estimation and time axis for said first video frame A basic hierarchical encoding means for performing
The first video and the reference video, and separated into high-resolution image frame through temporal filtering, via the space-time motion prediction with reference to the lower layer against the lower layer and the reference video for the first video and at least one enhancement layer encoding means performs scalable coding,
And multiplexing means for outputting an output of the base layer encoding means, a bit stream by multiplexing the output of the enhancement layer encoding means,
The scalable video coding apparatus for multi-view video according to claim 1, comprising: The enhanced hierarchical encoding means comprises:
And flag the coding result representing the availability of the motion vector information of the lower layer, the flag indicating that the one of the permission to use as prediction information reference index of a lower layer with respect to the reference video, type of intra block of the lower layer A flag indicating whether or not to use the image as prediction information and a flag for determining whether or not to use the difference image value in the lower layer are set, respectively, and an index of the used reference view is displayed. 2. A scalable video coding apparatus for multi-view video according to 2. The enhanced hierarchical encoding means comprises:
The scalable multi-view video according to claim 2, further comprising two-dimensional spatial interpolation means for performing two-dimensional spatial interpolation on a video frame restored for intra prediction for an intra block. Video coding device. The enhanced hierarchical encoding means comprises:
The scalable video coding apparatus for multi-view video according to claim 2, wherein encoding is performed via motion between frames, difference images, and intra prediction on temporal and spatial axes. The enhanced hierarchical encoding means comprises:
Using said base layer image to the first video (low resolution image), and double the motion vector value of the base layer image against the reference video (low resolution image) for motion prediction performing a motion prediction The scalable video coding apparatus for multi-view video according to claim 5. The enhanced hierarchical encoding means comprises:
For prediction of the difference image, difference the underlying layer image to the first video (low resolution image), by interpolating the residual image after prediction of the base layer image (low resolution image) against the reference video 6. The scalable video coding apparatus for multi-view video according to claim 5, wherein image prediction is performed. A scalable video coding method for multi-view video,
(A) separating the foundation video image frames with multilayer resolutions through spatial filtering, the space-time prediction for each of the separated low-resolution image frame was and one or more high-resolution image frame Performing scalable video coding via:
(B) a first video, the at least one or more video is a reference video input, separates the image frames with multilayer resolutions through spatial filtering, the separated low-resolution image frame was And performing scalable video coding through spatio-temporal prediction with reference to an adjacent frame of the first video frame and a reference video frame on the same time axis, and for the one or more separated high-resolution image frames. Performing scalable video coding through spatio-temporal prediction with reference to a lower layer of the first video and a lower layer of the reference video, and a scalable video coding method for multi-view video. Step (b), a flag indicating whether to use the motion vector information of the lower layer, the flag indicating that the one of the permission to use as prediction information reference index of a lower layer with respect to the reference video, intra said lower layer a flag indicating that the one of the permission to use the type of block as the prediction information, the to each set a flag for the availability of the difference between image values of a lower layer, and Turkey to display the index of reference view were used The scalable video coding method for multi-view video according to claim 8. The step ( b )
Motion between frames in time and space axis, the difference image, and scalable video coding method for multi-view video according to claim 8, characterized in that the sign-of through intra prediction. The step ( b )
[9] The scalable video coding method of claim 8, wherein interpolation of a two-dimensional space is performed on a video frame restored for intra prediction for an intra block. The step ( b )
Using said base layer image to the first video (low resolution image), and double the motion vector value of the base layer image against the reference video (low resolution image) for motion prediction performing a motion prediction The scalable video coding method for multi-view video according to claim 8, wherein: The step ( b )
For the difference image prediction by interpolating the residual image after prediction of the base layer image to the first video (low resolution image), and the base layer image to the other video that is the reference video (low resolution image) 9. The scalable video coding method for multi-view video according to claim 8, wherein difference image prediction is performed. A scalable video decoding device for multi-view video,
Receives the scalable coded bit stream for foundation video, and basic scalability video decoder to be restored through the inverse time transform and inverse spatial transform,
Via spatial prediction when the first video and the reference video receives scalable coded bitstream, inverse space-time prediction in accordance with the read permission information of the first video of a lower layer and the reference video of a lower layer to restore one or more high-resolution image frame via the through inverse temporal and spatial prediction according to the reference whether frames of adjacent frames and the reference video with the same time axis of the frame included in the first video after restoring the low-resolution image frame, and a high-resolution image frame the restored, a plurality of extended scalability video decoder for reconstructing an image through the inverse spatial filtering for the low resolution image frames the restored ,
A scalable video decoding apparatus for multi-view video. The extended scalability video decoder is
Demultiplexing means for demultiplexing the received bitstream;
The high-resolution image signal outputted by the inverse multiplexing means, in response to the read permission information of the lower layer against the lower layer and the reference video for the first video, scalable de via inverse space-time motion estimation and one or more enhancement layer decoding means you perform coding,
The low-resolution image signal outputted by the demultiplexing means, perform scalable decoding through the inverse motion prediction for a video frame of the reference video in reverse at the spatial motion prediction and temporal axis for a video frame of the first video Base layer decoding means to perform,
An inverse spatial video filtering means for restoring an image through inverse spatial filtering with respect to a high resolution image restored by the enhancement layer decoding means and a low resolution image restored by the base layer decoding means;
15. The scalable video decoding apparatus for multi-view video according to claim 14, further comprising: The enhancement layer decoding means comprises:
A flag indicating whether to use the motion vector information of the lower layer, the flag indicating that the one of the permission to use as prediction information reference index of a lower layer with respect to the reference video, a type of intra block of the lower layer as prediction information according to the flag indicating whether the permission to use, the flag for the availability of the difference between image values of the lower layer, characterized in that to perform scalable decoding with reference to the index of the reference view were used to predict Item 16. A scalable video decoding apparatus for multiview video according to item 15. A scalable video decoding method for multi-view video,
(A) receiving a scalable coded bitstream with respect to foundation video, and restoring via the inverse time between conversion and inverse spatial transform,
(B) receiving a bitstream that is scalable-coded for the first video and the reference video through spatio-temporal prediction, and in reverse depending on whether or not the lower layer of the first video and the lower layer of the reference video can be referred to One or more high-resolution image frames are restored through spatial prediction , and inverse spatio-temporal prediction is performed according to whether or not the reference video frame can be referenced on the same time axis as an adjacent frame of the frame included in the first video after restoring the low-resolution image frames through, and restoring the image through a high resolution image frames the restored, the restored inverse spatial filtering on the low-resolution image frames,
A scalable video decoding method for multi-view video comprising: Step (a) is a flag indicating whether to use the motion vector information of the lower layer, the flag indicating that the one of the permission to use as prediction information reference index of a lower layer with respect to the reference video, intra said lower layer a flag indicating that the one of the permission to use the type of block as the prediction information, and flags for the availability of the difference between image values of the lower layer, by using the index information scalable video decoding reference view were used to predict The scalable video decoding method for multi-view video according to claim 17, wherein:

Images