KR100783722B1 - A Scalable Stereoscopic Video Coding method for Heterogeneous Environments and the apparatus theirof - Google Patents

Abstract

According to the present invention, in a stereo video encoding method considering various reception environments, in a method of encoding an image signal with a scalable stereoscopic video encoder, a stereo image input to express a spatial basic resolution and a full resolution is input. Down-sampling, the left image of spatial basic resolution is encoded using a non-scalable MPEG-2 encoder, and each frame is subjected to a B frame partition to express temporal resolution. By generating the base layer BL and the first extended layer EL1 bit string, the stereo image encoding technique using the space-time hierarchy technique for providing an efficient display of heterogeneous receiving terminals.

Furthermore, for efficient stereo image reproduction, one base layer (BL) and one or more enhancement layers (EL) are provided. In addition, technical features include ECSC (Enhanced Compatible Stereoscopic Coding) to improve stereo video encoding efficiency, and stereo video encoder using spatio-temporal hierarchy to provide flexible stereo image service. to provide.

 Stereo video, spatiotemporal hierarchy, scalable, coding, image

Description

A scalable stereoscopic video coding method for heterogeneous environments and the apparatus theirof}

1 is an overall configuration diagram of a scalable stereo video service according to the present invention.

FIG. 2 is a space-time hierarchy diagram of the present invention, and FIG. 2A shows a case in which three B frames exist between an I frame and a P frame (that is, M = 3), and FIG. 2B shows a space between an I frame and a P frame. In this case, 5 B frames exist (i.e., M = 5).

3 is a block diagram illustrating a scalable stereo video encoder according to the present invention.

4 is an explanatory diagram illustrating a prediction process of a B-frame using ECSC (Enhanced Compatible Stereoscopic Coding) according to the present invention.

The present invention relates to a stereo video encoding method and a device thereof in consideration of various receiving environments, and to a suitable spatiotemporal resolution in consideration of the allowable bandwidth of a receiving terminal having various heterogeneous network environments and display environments, a display system, and processing power thereof. A stereo video encoding method and apparatus for providing a stereo video service.

Recently, with the development of internet network and multimedia service, simultaneous collaboration environment has been made possible to provide high quality realistic media including 3D video or panoramic video among remote users.

However, a network infrastructure must be established on the basis of transmitting or using high-definition sensory media data of the 3D image, and display devices capable of visually displaying 3D image data must be accompanied. In addition, currently established network infrastructures and respective receiving apparatuses are heterogeneous with each other or have different amounts of data that can be transmitted or processed.

 Accordingly, in consideration of the above aspects, a 3D image data encoding technique considering various supportable ranges and capabilities of a reception apparatus to be considered in a network should be considered.

As such, many coding techniques have been studied to provide differentiated services according to the performance of various networks and receiving apparatuses. Moreover, in the future, for example, the research is being actively conducted in industries, schools, and research institutes with active research support for realistic broadcasting, which is currently being promoted in Korea, for the purpose of commercialization by 2015.

Currently, three-dimensional image data is the mainstream stereo image. It is required to efficiently encode stereo images and to provide differentiated services according to the bandwidth of the network or the receiving range of the receiving apparatus. In addition, the proposed stereo coding scheme considers the technical meaning of providing differentiated services based on scalability (scalability that can flexibly respond to an increase in the number of users). To have.

Background Art Conventional stereo image coding techniques approach problems in terms of efficiency of disparity estimation. Techniques to improve the speed of binocular car prediction and to improve the accuracy of binocular car prediction are being studied. However, these techniques can improve the overall coding efficiency, but are not suitable for the current network environment having various configurations. Not only is the bandwidth that each network environment can guarantee, but also variable, so it is difficult to cope with various changes. In addition, the devices for displaying three-dimensional image data are also different in performance. Therefore, when an excessively complicated method is applied, it is difficult to actually process the device in real time.

It is an object of the present invention to solve the problems inherent in the heterogeneous and adaptive aspects of a collaborative environment.

In addition, an object of the present invention is to provide a level of data processing corresponding to the situation of various networks, and to provide a level of stereo video service that can be supported by a receiving terminal.

Furthermore, the present invention is to provide a method and apparatus for stereo video encoding in consideration of various reception environments using a space-time hierarchy technique for providing an efficient display of heterogeneous receiving terminals.

In order to achieve the above object, the present invention provides a method for reproducing an efficient stereo image,

One base layer (BL) for providing a basic resolution of an input reference image (left image),

The present invention provides a stereo video encoding method considering various reception environments having one or more enhancement layers (ELs) for providing additional information for supporting a right image having a basic resolution and a left and right image having a high resolution.

Another object of the present invention is to provide a stereo video encoding apparatus considering various receiving environments including a 3D image encoding unit, a spatial hierarchy encoding unit, and a temporal hierarchy encoding unit.

Hereinafter, a stereo video encoding method and apparatus thereof in consideration of various reception environments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a stereo video service system using an encoding apparatus proposed by the present invention.

The encoder included in the server providing the video image to the terminal side performs encoding by dividing the stereo image into one base layer BL and a plurality of enhancement layers EL1 to EL5.

Differently, the base layer (BL) bit sequence or the base layer (BL) and the extension layer (EL) bit sequence are transmitted together according to the network bandwidth and processing environment of the different receiving terminals (Client A-D).

The receiving terminal (Client A-D) combines the received bit streams of various layers to display an image having a spatiotemporal resolution that can be supported by the receiving terminal (Client A-D).

For example, the first receiving terminal Client A needs only low resolution mono type video data and is provided with a video bit sequence corresponding to a base layer. On the contrary, the fourth receiving terminal Client D needs high-resolution 3D data, and in this case, a bit string of all layers is provided.

Therefore, each receiving terminal (Client AD) can be provided with a bit string suitable for its display and network environment, and mono or various stereo displays can be displayed for one 3D video source by the receiving terminal (Client AD) environment. .

The present invention provides an Enhanced Compatible Stereoscopic Coding (ECSC) method for improving stereo video encoding efficiency, and provides a stereo video encoder using space-time hierarchy to provide a stereo image service with flexibility. For this purpose, the reference image (left image) is encoded using the MPEG-2 standard, and the P-frame and B-frame of the right image are binocular vector (DV) and motion vectors in a compatible manner defined in the MPEG-2 multiview profile. Interpolation technique using (MV) was used.

2 is a space-time hierarchy configuration diagram of the present invention. FIG. 2A shows a case where three B frames exist between I and P frames (that is, M = 3), and one base layer BL is used for efficient three-dimensional image transmission between systems having heterogeneous and diverse display environments. In addition to the above), a plurality of extended layers EL are defined. The base layer BL is a bit sequence corresponding to the base resolution of the input reference image (left image), and the first extension layer EL1 is an additional bit sequence for providing a high resolution reference image in time. The third extension layer EL2 is an additional bit string for providing spatial high resolution with respect to the reference image. Meanwhile, the third extension layer EL3 is a bit string for representing the right image corresponding to the basic resolution. The fourth expansion layer EL4 and the fifth expansion layer EL5 generate a bit string for representing a high quality right image. The base layer BL removes temporal redundancy by performing encoding based on a motion compensation prediction (MCP) using a current frame and a neighboring frame. In the third extended layer EL3, the decoded left image of the base layer BL that matches the temporal layer BL in order to improve the compression efficiency while performing motion compensation prediction (MCP) from the images present in the adjacent r layer in time. It is generated by performing Disparity Compensation Prediction (DCP).

The first extension layer EL1 and the fourth extension layer EL4 are bit sequences for providing complete temporal resolution in addition to the base layer BL and the third extension layer EL3. Similarly, the second extension layer EL2 and the fifth extension layer EL5 include the base layer BL + the first extension layer EL1 and the third extension layer EL3 + the fourth extension layer EL4. It is a bit string to provide full spatial resolution in addition to the column.

FIG. 2B shows the modified embodiment of FIG. 2A as if there are five B frames (i.e., M = 5) between the I and P frames.

To this end, the stereo video encoding apparatus according to the present invention includes a spatial hierarchy encoding means, a three-dimensional image encoding means, and a temporal hierarchy encoding means for hierarchically encoding the layers.

3 illustrates a detailed configuration of a scalable stereoscopic video encoder according to the present invention.

According to FIG. 3, the input stereo image is downsampled in order to spatially represent the basic resolution and the full resolution. The left-sampled down-sampled spatial basic resolution is encoded using a conventional non-scalable MPEG-2 encoder (40, Non-Scalable MPEG-2 Encoder). Thereafter, in order to express various temporal resolutions through the temporal hierarchical encoding means 30, the base layer BL and the first extended layer EL1 bit strings are generated through a B frame partition.

In the 3D image encoding means 20, the left image having a spatial basic resolution that is temporally coincident with the current image is encoded to compress the right image. After selecting the higher transmission efficiency among the motion vector (MV) or the binocular difference vector (DV), the motion difference information (DFD; Displaced Frame Difference) or the binocular difference information (DCD) are encoded and transmitted accordingly. .

A method for encoding a right image having a spatial default resolution is as follows.

First, an I-frame in a right image is encoded using a block predicted using a binocular vector obtained from a temporally identical left image.

The P-frame of the right image is encoded using a binocular difference vector (DV) obtained by referring to a left image temporally identical to the motion vector (MV) obtained by performing forward prediction in the same sequence.

Next, the B-frame of the right image is encoded using a macroblock MB obtained through forward motion prediction and backward motion prediction in the same sequence as shown in FIG. 4 and a macroblock MB obtained through binocular prediction.

The additionally necessary bit strings EL2 and EL5 for full spatial resolution representation are generated by encoding a difference value from the original image that is temporally matched by up-sampling the decoded base resolution image to the original resolution. Application of the extension layer is used to provide high picture quality when the display of the receiving side is supported.

The spatial hierarchy encoding means 10 generates an additional bit string for supporting high resolution in a reference image having an existing resolution.

4 is a diagram illustrating a prediction method of enhanced compatible stereoscopic coding (ECSC) in the present invention. The predicted right image B-frame block is a sum of weights of I-frames and P-frames of the right image and blocks obtained by finding similar blocks in the B-frames of the left image. For a P-frame, the weights for the forward and binocular vectors are 0.5, for the B-frames the weights for the forward and reverse vectors are 0.25, respectively, and the binocular vectors are 0.5. In the prediction process of one block, the sum of all weights is one.

(Equation 1)

In the above equation, w represents a weight, v is a vector, and f, b, and d are forward, reverse, and binocular, respectively.

While the present invention has been shown and described with reference to certain preferred embodiments, the present invention is not limited to the above embodiments and those skilled in the art without departing from the spirit of the present invention. Various changes and modifications will be possible.

As described above, the present invention has the effect that it is possible to provide a stereo service suitable for a receiving environment to several heterogeneous receiving terminals at the same time with one encoding equipment. In particular, in case of stereo video encoded by the encoder using the present invention, re-encoding is not necessary according to the environment of the receiving terminal, and the service optimized for the display system and the network environment of each terminal when the environment of different receiving terminals is known in advance. Is possible. For example, when multicasting a stereo realistic media transmission service to multiple users, the transmission channel is wasted by sending a combination of bit strings suitable for a user's computing environment to a user who needs an encoded stereo video stream by extending to multiple layers. It can be used effectively without.

In addition, since the resolution of the image transmitted from the receiving terminal is already optimized for the display system of the receiving terminal, it is possible to display mono or stereo video without additional processing for adjusting the resolution, thereby saving terminal computing resources. It can improve the reaction speed, and can prevent the deterioration of image quality caused by the resolution adjustment.

Claims (8)

In the method for encoding a video signal with a scalable stereoscopic video encoder (Scalable Stereoscopic Video Encoder), One base layer (BL) for providing a basic resolution of an input reference image (left image), A stereo video encoding method considering various reception environments having at least one enhancement layer (EL) to provide additional information for supporting a right image of a native resolution and a left and right image of a high resolution. The method of claim 1, The extended layer (EL) is a stereo video encoding method in consideration of various reception environments, characterized in that it is composed of a spatial and temporal hierarchy consisting of a layer for providing a high resolution in space and a layer for providing a high resolution in time. The method according to claim 1 or 2, The right image of the native resolution is encoded by referring to the left image having a spatial native resolution that temporally matches the current image. After selecting a motion vector (MV) or a binocular vector (DV) having a high transmission efficiency, the motion difference information (DFD; Displaced Frame Difference) or the binocular difference information (DCD; Displacement Compensated Difference) ) Is a stereo video encoding method considering various receiving environments, characterized in that the transmission. The method according to claim 1 or 2,  The encoding of the right image having the spatial native resolution may include: (a) encoding an I-frame in a right image using a block predicted using a binocular vector obtained from a temporally identical left image; (b) encoding a P-frame of a right image using a binocular vector DV obtained by referring to a left image temporally identical to a motion vector MV obtained by performing forward prediction in the same sequence; (c) encoding the B-frame of the right image using a macroblock (MB) obtained through forward motion prediction and backward motion prediction in the same sequence and a macroblock (MB) obtained through binocular prediction; Stereo video encoding method considering the various receiving environment, characterized in that consisting of. The method of claim 4, wherein Stereo video encoding method considering various receiving environments, characterized in that the application of the spatiotemporal hierarchy composition consisting of an odd number of B-frames between the I-frame and the P-frame. The method of claim 4, wherein The predicted right image B-frame block is a sum of weighted blocks obtained by finding similar blocks in the I-frame and P-frame of the right image and the B-frame of the left image. Stereo video encoding method. The method of claim 4, wherein The P-frame is 0.5 as the weight for the forward and binocular vectors, the B-frame is 0.25 as the weight for the forward and reverse vectors, the binocular vector is 0.5 as the weight, and all weights in the prediction process of one block. The sum of 1 is obtained by the following equation, wherein the stereo video encoding method with extended compatibility. (Mathematical formula)

(W denotes a weight, v is a vector, and f, b, and d are forward, reverse, and binocular, respectively.) In the scalable stereo video encoding apparatus (Scalable Stereoscopic Video Encoder),  Three-dimensional image encoding means;  Spatial hierarchy encoding means;  Stereo video encoding apparatus considering various receiving environments including temporal hierarchy encoding means

Images