WO2014003240A1

WO2014003240A1 - Method and apparatus for processing three-dimensional image signals

Info

Publication number: WO2014003240A1
Application number: PCT/KR2012/007625
Authority: WO
Inventors: 이광순; 윤국진; 이진영; 김찬; 신홍창; 정원식; 허남호
Original assignee: 한국전자통신연구원
Priority date: 2012-06-27
Filing date: 2012-09-24
Publication date: 2014-01-03
Also published as: KR20140002853A

Abstract

The present invention relates to: a method for providing stereoscopic and/or multi-view images through a service compatible type three-dimensional TV broadcasting system; and an apparatus using the same. The method for processing three-dimensional image signals according to one embodiment of the present invention comprises the steps of: receiving three-dimensional image signals; extracting a first image stream and a second image stream by demultiplexing the three-dimensional image signals; obtaining a reference image by decoding the first image stream; obtaining a correspondence image corresponding to the reference image and an additional image by decoding the second image stream; and generating the three-dimensional images on the basis of the reference image, the corresponding image, and the additional image.

Description

Method and apparatus for processing 3D video signal

The present invention relates to a method for generating a 3D image using a left and right image and a depth image (or disparity image) and a device using the same. More specifically, the present invention relates to a stereoscopic method through a service compatible 3DTV broadcasting system. And / or a method for providing a multi-view image and an apparatus using the same.

Digital broadcasting service using 3D video is attracting attention as next generation broadcasting service following UDTV (Ultra Definition Television) service and HDTV (High Definition Television) service, and related technologies such as launching high definition stereoscopic display. With the development of 3D TV service, which can enjoy 3D video at home, it is expected to be provided within a few years.

Meanwhile, the 3DTV broadcasting system currently being used uses a stereoscopic video mainly composed of a left image and a right image. In a 3DTV broadcasting system using stereoscopic video, the transmitter transmits left and right video using a side-by-side, top-and-bottom format, and a receiver. Generates a multi-view image by synthesizing the images of several viewpoints. In this case, in order to provide an interactive service related to the depth of the image, it is preferable that a depth image or a shift image is provided along with the left image and the right image.

An object of the present invention is to provide a method and apparatus for processing a 3D video signal.

Another object of the present invention is to provide a method and apparatus for providing a multiview image through a 3DTV broadcasting system of a service compatible method.

Another object of the present invention is to provide a method and apparatus for transmitting a depth image or a disparity image required for synthesizing a 3D image together with a left image and a right image through a service compatible 3DTV broadcasting system.

Another object of the present invention is to provide a stereoscopic image to a 3DTV receiver of a general service compatible method, multi-view image or the interaction associated with the receiver capable of view-synthesis ) In providing services.

According to an embodiment of the present invention, a 3D image signal processing method is provided. The method may include generating a first video stream by encoding a reference video, generating a second video stream by encoding a corresponding video and an additional video corresponding to the reference video, and generating the first video stream and the second video stream. Multiplexing the video streams and transmitting them in a single stream.

The additional image may be a depth image or a disparity image.

The horizontal resolution of the corresponding image and the additional image may be 1/2 of the horizontal resolution of the reference image.

The additional image may include a first additional image that is an additional image with respect to the reference image, and a second additional image that is an additional image with respect to the corresponding image.

The horizontal resolution of the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image, and the vertical resolution of the first additional image and the second additional image is 1 of the vertical resolution of the reference image. May be / 2.

According to another embodiment of the present invention, a 3D image signal processing method is provided. The method may include receiving a 3D video signal, demultiplexing the 3D video signal, extracting a first video stream and a second video stream, decoding the first video stream, and obtaining a reference video. Decoding a second image stream to obtain a corresponding image and an additional image corresponding to the reference image, and generating a 3D image based on the reference image, the corresponding image, and the additional image.

The additional image may be a depth image or a disparity image.

The horizontal resolution of the corresponding image and the additional image may be 1/2 of the horizontal resolution of the reference image. The generating of the 3D image may include enlarging the corresponding image and the additional image by 2 times in a horizontal direction.

The vertical resolution of the corresponding image and the additional image may be 1/2 of the vertical resolution of the reference image. The generating of the 3D image may include enlarging the corresponding image and the additional image by 2 times in the vertical direction.

The horizontal resolution of the corresponding image, the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image, and the vertical resolution of the first additional image and the second additional image is determined by the reference image. It may be 1/2 of the vertical resolution. The generating of the 3D image may include enlarging the corresponding image, the first additional image and the second additional image in a horizontal direction by 2 times, and vertically expanding the first additional image and the second additional image. It may include a step 2 times.

The horizontal resolution of the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image, and the vertical resolution of the corresponding image, the first additional image and the second additional image is equal to that of the reference image. It may be 1/2 of the vertical resolution. The generating of the 3D image may include enlarging the first additional image and the second additional image in a horizontal direction twice, and vertically expanding the corresponding image, the first additional image, and the second additional image. It may include a step 2 times.

According to another embodiment of the present invention, a 3D image signal processing apparatus including a receiver, a demultiplexer, a first image decoder, a second image decoder, and an image synthesizer is provided. The receiver receives a 3D video signal. The demultiplexer extracts a first video stream and a second video stream by demultiplexing the 3D video signal. The first image decoder decodes the first image stream to obtain a reference image. The second video decoder decodes the second video stream to obtain a corresponding picture and an additional picture corresponding to the reference picture. The image synthesizer generates a 3D image based on the reference image, the corresponding image, and the additional image.

The additional image may be a depth image or a disparity image.

The horizontal resolution of the corresponding image and the additional image may be 1/2 of the horizontal resolution of the reference image. The apparatus for processing 3D video signals may include an image enlarger that enlarges the corresponding video and the additional video 2 times in a horizontal direction.

The horizontal resolution of the corresponding image, the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image, and the vertical resolution of the first additional image and the second additional image is determined by the reference image. It may be 1/2 of the vertical resolution. The 3D image signal processing apparatus may enlarge the corresponding image, the first additional image, and the second additional image in a horizontal direction, and double the first additional image and the second additional image in a vertical direction. It may include an image enlarger for enlarging.

According to the present invention, a depth image or a disparity image may be transmitted along with a left image and a right image through a dual stream service compatible 3DTV broadcasting system.

According to the present invention, a stereoscopic image is provided to a general service compatible 3DTV receiver, and a multi-view image or an interaction service related thereto is provided to a receiver capable of view-synthesis. Can provide.

1 is a conceptual diagram illustrating a typical dual stream service-compatible 3DTV broadcasting system.

2 illustrates a 3D image format in which the resolution in the horizontal direction or the vertical direction is reduced to 1/2.

3 illustrates a 3D image format according to an embodiment of the present invention.

4 illustrates a 3D image format according to another embodiment of the present invention.

5 shows an apparatus for processing a 3D video signal according to an embodiment of the present invention.

6 shows an apparatus for processing a 3D image signal according to another embodiment of the present invention.

FIG. 7 illustrates a method of processing a 3D video signal in a 3DTV receiver of a general service compatible method when providing a 3DTV broadcasting service based on a 3D video format according to an embodiment of the present invention.

8 is a flowchart illustrating a 3D image processing method (transmitter) according to an embodiment of the present invention.

9 is a flowchart illustrating a 3D image processing method (receiver) according to an embodiment of the present invention.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of this invention is described in detail with reference to drawings. However, in describing the embodiments of the present invention, when it is determined that the detailed description of the known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

When a component is described as being "connected" or "connected" to another component, it may be directly connected to or connected to another component, but another component may be in between. In addition, when the present invention is described as "includes" a specific component, rather than excluding components other than the component, it is understood that additional components may be included in the scope of the embodiments or technical spirit of the present invention. it means.

Terms such as "first" and "second" may be used to describe various components, but the components are not limited by the terms. In other words, the terms are used only for the purpose of distinguishing one component from another. Thus, the first component may be named the second component, and likewise the second component may be named the first component.

In addition, the components shown in the embodiments of the present invention are shown independently to indicate that they perform different characteristic functions, and do not mean that each component may not be implemented in one hardware or software. That is, each component is divided for convenience of description, and a plurality of components may be combined to operate as one component, or one component may be divided into and operate as a plurality of components, which does not depart from the essence of the present invention. Unless included in the scope of the present invention.

In addition, some components may be optional components for improving performance rather than essential components for performing essential functions of the present invention. The present invention may be implemented in a structure including only essential components except for optional components, and a structure including only essential components is also included in the scope of the present invention.

In FIG. 1, the reference image refers to an image compatible with a general 2D image. The corresponding image is an image paired to provide a stereoscopic image together with the reference image, and corresponds to the reference image. In this case, in a general dual stream service compatible 3DTV broadcasting system, a reference video is set as a left image and a corresponding video is set as a right image. However, the left image is not necessarily set as the reference image, and the right image may also be set as the reference image. Hereinafter, for convenience of description, the left image is set as a reference image, that is, a reference image.

In the transmitter 100, the first video encoder 111 is an encoder for encoding a reference video, and generates a first video stream. The second video encoder 112 is an encoder for encoding the corresponding video independently or cross-referenced with the reference video, and generates a second video stream.

The multiplexer 120 multiplexes the first video stream and the second video stream into a single stream, and the transmitter 130 transmits the single stream through a transmission network. In the above process, a single stream may go through channel encoding, modulation, and the like.

In the receiver 200, the receiver 210 and the demultiplexer 220 receive a 3D video signal transmitted through a transmission network, and perform a demodulation, channel decoding, and demultiplexing process to perform a first video stream and a second video. Extract the stream.

The extracted streams are decoded by the first video decoder 231 and the second video decoder 232, respectively, and sent to the 3D display.

Meanwhile, the signaling information generator 140 and the signaling information analyzer 240 signal information such as the type of 3DTV service, the format of the additional video, and whether the reference video is the left video or the right video.

On the other hand, the dual stream service compatible 3DTV broadcasting system supports a format in which the resolution in the horizontal or vertical direction is reduced to 1/2 to support various left and right compression bit rates. 2 illustrates a 3D image format in which the resolution in the horizontal direction or the vertical direction is reduced to 1/2.

In addition, the transmitter may generate a multi-view image by synthesizing the images of various viewpoints and provide an interactive service related to the depth of the image, so that the transmitter may transmit the depth ( It is efficient to separately transmit additional images such as depth images and disparity images. When the additional image is used together with the left image and the right image, a multiview image service based on depth based image rendering (DBIR) may be realized.

A transmitter in a dual stream based 3DTV broadcasting system according to an embodiment of the present invention may transmit a 3D video signal in a format as shown in FIG. 3. In the example of FIG. 3, a first image means an input of a first image encoder or an output of a first image decoder, that is, a reference image, and a second image means an input of a second image encoder or an output of a second image decoder. do.

Referring to FIG. 3, the horizontal resolution or vertical resolution of the right image R 'and the depth image D' is set to 1/2 of the horizontal resolution or vertical resolution of the left image L, and 1 of the second image is set. The / 2 area is used as it is for the transmission of the right image R ', but the remaining 1/2 area of the second image is used for the transmission of the additional image D'. As a result, since the resolution of the second image is maintained, the left image L, the right image R ', and the additional image D' may be provided using the general dual stream based 3DTV broadcasting system as it is. As described above, the additional image may mean a depth image or a disparity image.

Meanwhile, in generating a multiview image using the additional image, when a plurality of views and additional images are provided, a plurality of views and a high quality intermediate view may be generated.

A transmitter in a dual stream based 3DTV broadcasting system according to another embodiment of the present invention may transmit additional video for each of a left image and a right image to a receiver in the format as shown in FIG. 4. Similarly, in the example of FIG. 4, the first video refers to the input of the first video encoder or the output of the first video decoder, that is, the reference video, and the second video is the input of the second video encoder or the output of the second video decoder. Means.

Referring to FIG. 4, the horizontal resolution or vertical resolution of the right image R ′ is set to 1/2 of the horizontal resolution or vertical resolution of the left image L. FIG. In addition, the horizontal and vertical resolutions of the additional image D _L ′ for the left image are set to 1/2 of the horizontal and vertical resolutions of the left image L, respectively, and the additional image D _R ′ for the right image is set. ), The horizontal resolution and the vertical resolution are also set to 1/2 of the horizontal resolution and the vertical resolution of the left image, respectively. The half region of the second image is used for the transmission of the right image R ', but the remaining half region of the second image is used for the transmission of the additional images D _L ' and D _R '. do. In the example of FIG. 4, the first additional image refers to the additional image D _L ′ for the left image, and the second additional image refers to the additional image D _R ′ for the right image.

On the other hand, since additional images such as depth images and disparity images have a smaller amount of information than general color images, even if the resolution is reduced, the impact on the image quality of the entire image is minimal. In addition, the receiver may interpolate the additional image by using various methods by referring to the color image, and may be used for synthesizing the images and providing depth information for immersive interaction.

When the receiver 510 receives a 3D video signal transmitted through a dual stream 3DTV broadcasting system, the demultiplexer 520 demultiplexes the 3D video signal to extract a first video stream and a second video stream. . The first video stream and the second video stream are decoded by the first video decoder 531 and the second video decoder 532, respectively. That is, the first video decoder 531 decodes the first video stream to obtain the first video, that is, the reference video, and the second video decoder 532 decodes the second video stream to the second video, that is, the corresponding video. And additional images are acquired.

Meanwhile, when the 3D image signal is received in the format as shown in FIG. 3, since the corresponding image and the additional image are merged in the second image, the image separator 540 separates the corresponding image and the additional image, respectively. In addition, since the horizontal resolution or the vertical resolution of the corresponding video and the additional video is reduced to 1/2, the image enlarger 550 enlarges the corresponding video and the additional video twice in the horizontal direction or the vertical direction. The operation order of the image separator 540 and the image enlarger 550 may be changed according to an implementation method. That is, the operation of the image enlarger 550 may be performed first, and the operation of the image separator 540 may be performed later.

Finally, the corresponding image and the additional image reconstructed at the original resolution are used to generate the 3D image together with the reference image. That is, the image synthesizing unit 560 utilizes an additional image such as a depth image or a disparity image so as to synthesize a viewpoint image or a multiview image selected by the user and reproduce the image on the 3D display.

When the receiver 610 receives the 3D video signal transmitted through the dual stream 3DTV broadcasting system, the demultiplexer 620 demultiplexes the 3D video signal to extract the first video stream and the second video stream. . The first video stream and the second video stream are decoded by the first video decoder 631 and the second video decoder 632, respectively. That is, the first video decoder 631 decodes the first video stream to obtain the first video, that is, the reference video, and the second video decoder 632 decodes the second video stream, to the second video, that is, the corresponding video. The first additional image and the second additional image are obtained.

On the other hand, when the 3D video signal is received in the format as shown in FIG. 4, since the corresponding video, the first additional video and the second additional video are merged, the image separator 640 may correspond to the corresponding video and the first additional video. Separate the image and the second additional image, respectively. In addition, since the horizontal resolution or vertical resolution of the corresponding video is reduced to 1/2, the image enlarger 650 enlarges the corresponding video twice in the horizontal direction or the vertical direction. Similarly, since the horizontal resolution and the vertical resolution of the first additional image and the second additional image are reduced to 1/2, the image enlarger 650 divides the first additional image and the second additional image into two horizontal and vertical directions. 2x magnification. The operation order of the image separator 640 and the image enlarger 650 may be changed according to an implementation method. That is, the operation of the image enlarger 650 may be performed first, and the operation of the image separator 640 may be performed later.

Finally, the corresponding image reconstructed at the original resolution, the first additional image and the second additional image are used to generate the 3D image together with the reference image. That is, the image synthesizing unit 560 utilizes additional images, such as a depth image or a disparity image, to synthesize a viewpoint image or a multiview image selected by a user and reproduce the image on a 3D display.

When the 3DTV broadcasting service based on the 3D video format according to the embodiment of the present invention is provided as shown in the example of FIG. 7, information indicating that the resolution of the corresponding video is reduced may be provided to the receiver. For example, information on the resolution of the corresponding video and the additional video transmitted through the signaling information may be transmitted. If necessary, information on the resolution of the corresponding video and the additional video may be transmitted through information related to frame cropping and aspect ratio included in a sequence header in the encoding stream. . The receiver may magnify the reduced image based on the information.

The transmitter generates a first video stream by encoding the reference video (S810). As described above, the reference video is a video compatible with a general 2D video and is set as a left video in a general dual stream service compatible 3DTV broadcasting system.

The transmitter generates a second video stream by encoding the corresponding video and the additional video (S820). As described above, the corresponding video is a paired video for providing a stereoscopic video together with the reference video and is set as a right video in a general dual stream service compatible 3DTV broadcasting system. In addition, additional images such as depth images and disparity images may generate multi-view images by synthesizing images of various viewpoints and provide interactive services related to the depth of the images.

The transmitter generates a single stream by multiplexing the first video stream and the second video stream (S830), and transmits the single stream (S840).

Meanwhile, as described above, the horizontal resolution or vertical resolution of the corresponding image and the additional image may be set to 1/2 of the horizontal resolution or the vertical resolution of the reference image. Also, the additional image may include an additional image for the reference image in which both the horizontal resolution and the vertical resolution are reduced to 1/2 and the additional image for the corresponding image.

The receiver receives the 3D video signal (S910) and demultiplexes the 3D video signal to extract the first video stream and the second video stream (S920). In addition, in steps S910 to S920, the receiver may perform operations such as demodulation and channel decoding.

The receiver decodes the first video stream to obtain a reference video (S930). As described above, the reference video is a video compatible with a general 2D video and is set as a left video in a general dual stream service compatible 3DTV broadcasting system.

The receiver decodes the second video stream to obtain a corresponding picture and an additional picture (S940). As described above, the corresponding video is a paired video for providing a stereoscopic video together with the reference video and is set as a right video in a general dual stream service compatible 3DTV broadcasting system. In addition, additional images such as depth images and disparity images may generate multi-view images by synthesizing images of various viewpoints and provide interactive services related to the depth of the images.

The receiver generates a 3D image based on the reference image, the corresponding image, and the additional image (S950).

It is not limited to the order of steps described above, and some steps may occur in a different order or in parallel with other steps. In addition, one of ordinary skill in the art will appreciate that the steps shown in the flowcharts are not exclusive, that other steps may be included or some steps may be deleted.

In addition, the above-described embodiments include examples of various aspects. In order to represent various aspects, not all possible combinations may be described, but one of ordinary skill in the art will recognize that other combinations are possible. Accordingly, the invention is intended to embrace all other replacements, modifications and variations that fall within the scope of the following claims.

Claims

Generating a first video stream by encoding the reference video;

Generating a second video stream by encoding a corresponding video and an additional video corresponding to the reference video; And

And multiplexing the first video stream and the second video stream as a single stream.
The method of claim 1,

The additional image is a 3D image signal processing method, characterized in that the depth image or disparity image.
The method of claim 1,

The horizontal resolution of the corresponding video and the additional video is 1/2 of the horizontal resolution of the reference video, characterized in that the 3D video signal processing method.
The method of claim 1,

The additional image includes a first additional image that is an additional image for the reference image and a second additional image that is an additional image for the corresponding image.
The method of claim 4, wherein

The horizontal resolution of the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image,

And the vertical resolution of the first additional image and the second additional image is 1/2 of the vertical resolution of the reference image.
Receiving a 3D video signal;

Demultiplexing the 3D video signal to extract a first video stream and a second video stream;

Decoding the first video stream to obtain a reference video;

Decoding the second video stream to obtain a corresponding picture and an additional picture corresponding to the reference picture; And

And generating a 3D image based on the reference image, the corresponding image, and the additional image.
The method of claim 6,

The additional image is a 3D image signal processing method, characterized in that the depth image or disparity image.
The method of claim 6,

The horizontal resolution of the corresponding video and the additional video is 1/2 of the horizontal resolution of the reference video,

The generating of the 3D image may include expanding the corresponding image and the additional image by 2 times in a horizontal direction.
The method of claim 6,

The vertical resolution of the corresponding video and the additional video is 1/2 of the vertical resolution of the reference video,

The generating of the 3D image may include expanding the corresponding image and the additional image by 2 times in a vertical direction.
The method of claim 6,

The additional image includes a first additional image that is an additional image of the reference image and a second additional image that is an additional image of the corresponding image.
The method of claim 10,

The horizontal resolution of the corresponding image, the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image,

The vertical resolution of the first additional image and the second additional image is 1/2 of the vertical resolution of the reference image,

Generating the 3D image is

Enlarging the corresponding image, the first additional image and the second additional image two times in a horizontal direction; And

And expanding the first additional image and the second additional image two times in a vertical direction.
The method of claim 10,

The horizontal resolution of the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image,

The vertical resolution of the corresponding image, the first additional image and the second additional image is 1/2 of the vertical resolution of the reference image,

Generating the 3D image is

Enlarging the first additional image and the second additional image two times in a horizontal direction; And

And enlarging the corresponding image, the first additional image and the second additional image in a vertical direction by 2 times.
Receiving unit for receiving a 3D video signal;

A demultiplexer configured to demultiplex the 3D video signal to extract a first video stream and a second video stream;

A first video decoder to decode the first video stream to obtain a reference video;

A second video decoder configured to decode the second video stream to obtain a corresponding picture and an additional picture corresponding to the reference picture; And

And an image synthesizer configured to generate a 3D image based on the reference image, the corresponding image, and the additional image.
The method of claim 12,

3D image signal processing apparatus, characterized in that the additional image is a depth image or a shift image.
The method of claim 12,

The horizontal resolution of the corresponding video and the additional video is 1/2 of the horizontal resolution of the reference video,

And an image enlarger configured to enlarge the corresponding image and the additional image by 2 times in a horizontal direction.
The method of claim 12,

The additional image includes a first additional image that is an additional image for the reference image and a second additional image that is an additional image for the corresponding image.
The method of claim 16,

The horizontal resolution of the corresponding image, the first additional image and the second additional image is 1/2 of the horizontal resolution of the reference image,

The vertical resolution of the first additional image and the second additional image is 1/2 of the vertical resolution of the reference image,

And an image enlarger configured to enlarge the corresponding image, the first additional image, and the second additional image in a horizontal direction, and to enlarge the first additional image and the second additional image in a vertical direction. 3D video signal processing apparatus.