KR100913397B1 - Method of object description for three dimensional image service based on dmb, and method for receiving three dimensional image service and converting image format - Google Patents

Abstract

The present invention relates to a method for describing an object for a 3D stereoscopic image service based on a DMV, and a method for receiving and converting an image format according to the 3D stereoscopic image service. The present invention relates to a new 3D stereoscopic image service based on reference images and depth information. By providing an object descriptor structure to provide a, it is intended to minimize the bandwidth required for 3D stereoscopic image service without deteriorating image quality at the receiver side while maintaining compatibility with the existing DMB service.

To this end, the present invention, in the object description method for the 3D stereoscopic video service based on DMB, broadcasting which sets in the object descriptor for the reference video whether the broadcast service corresponds to a broadcast type of 2D broadcast or 3D broadcast Type setting step; And an object describing the reference image and depth information of the 3D stereoscopic image, which is the target of the broadcast service, using one object descriptor, wherein the reference image is set as an independent stream and the depth information is set as a dependent stream. Characterized in that it comprises a technical step.

DMB, 3D Broadcast, 3D Stereoscopic Image, MPEG-4, Reference Image, Depth Information, Content Type Descriptor, Depth Information Descriptor

Description

OBJECT DESCRIPTION FOR THREE DIMENSIONAL IMAGE SERVICE BASED ON DMB, AND METHOD FOR RECEIVING THREE DIMENSIONAL IMAGE SERVICE AND CONVERTING IMAGE FORMAT}

The present invention relates to a method for describing an object for a 3D stereoscopic image service based on DMV, and a method for receiving and transforming an image format according to the 3D stereoscopic image service, and more specifically, a new 3 based on reference image and depth information. DMB-based, which provides an object descriptor structure for providing 3D stereoscopic services, and minimizes the bandwidth required for 3D stereoscopic video services without deteriorating image quality at the receiver side while maintaining compatibility with existing DMB services. An object description method for a 3D stereoscopic image service, and a method of receiving a 3D stereoscopic image service and converting an image format accordingly.

The present invention is derived from the research conducted as part of the next generation core technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management Number: 2005-S-403-02, Title: Intelligent Integrated Information Broadcasting (SmaR TV) ) Technology development].

Since the conventional technology for 3D stereoscopic video service based on digital multimedia broadcasting (DMB) uses left / right image for 3D stereoscopic broadcasting service, image quality deterioration when supporting a variety of binocular 3D display devices There is also a problem that does not support at all for the multi-view three-dimensional display device. In addition, since the prior art independently codes the left and right images, there is a problem that the bandwidth required for coding is increased.

Next, a description will be given of a conventional object description method to the DMB service. In particular, Figure 1 shows the structure of an MPEG-4 object descriptor used to describe an audio / video stream in a conventional DMB service.

In DMB broadcasting, an object descriptor consists of an object descriptor for each audio / video and a plurality of object descriptors for additional data.

Audio streams and video streams are described through respective object descriptors 10 and 11 and elementary stream descriptors (ESD) 101 and 111, respectively.

That is, in case of a video stream, set to "ObjectTypeIndication = 0x21 (AVC) and StreamType = 0x04 (visual stream)" (102, 103), and in case of an audio stream, "ObjectTypeIndication = 0x40 (BSAC) and StreamType = 0x05 (audio). stream) "(112, 113).

On the other hand, the object descriptor identification information (OD_ID) for each audio / video stream has a unique value (for example, the audio stream is "10" and the video stream is "20"), which is described in the scene descriptor (BIFS). It is set equal to the positional information (for example, URL value) of each node. That is, to represent an audio stream and a video stream, the scene descriptor BIFS includes two nodes, each of which is associated with a corresponding object descriptor ID OD_ID.

However, since the object descriptor structure is focused on two-dimensional content, there is a limit to serving three-dimensional stereoscopic images of two or more viewpoints.

The prior art as described above is difficult to support a variety of three-dimensional display device (binocular / multi-view three-dimensional display device) without deterioration of image quality, and also to use a wide bandwidth to provide a multi-view three-dimensional image service There is a problem that the digital multimedia broadcasting (DMB) using a narrow bandwidth is inadequate, and to solve this problem is a problem of the present invention.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned above can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

In order to solve the above object, the present invention describes that the reference information and the depth information having a small amount of data information are transmitted in a dependency relationship using one object descriptor, and the receiving side uses the reference image and the depth information to transmit the depth information. It is characterized by composing various 3D stereoscopic images by generating images of viewpoints.

More specifically, the present invention, in the object description method for 3D stereoscopic video service based on DMB, broadcasting which sets in the object descriptor for the reference video whether the broadcast service corresponds to a broadcast type of 2D broadcast or 3D broadcast Type setting step; And an object describing the reference image and depth information of the 3D stereoscopic image, which is the target of the broadcast service, using one object descriptor, wherein the reference image is set as an independent stream and the depth information is set as a dependent stream. Includes technical steps.

The present invention also provides a DMB-based 3D stereoscopic video service receiving method, comprising: a decoding step of decoding a transport stream to obtain reference video, depth information, and broadcast type information; A three-dimensional image generation step of generating an image of a viewpoint different from the reference image by using the reference image and the depth information when it is confirmed as three-dimensional broadcasting through the broadcasting type information; And an image format conversion step of outputting a 3D stereoscopic image or outputting the reference image by combining the reference image and the other viewpoint image according to display control information.

The present invention also provides a video format conversion method in a 3D stereoscopic video service receiving system, comprising: detecting a vibration of the receiving system; And an image format conversion step of selecting and outputting any one of a two-dimensional reference image or a three-dimensional stereoscopic image using the reference image according to the sensed vibration intensity.

In the above-described invention, in providing a 3D stereoscopic image service based on DMB, the transmitting side transmits only the reference image and 'depth information having a small amount of data', and the receiving side uses various received reference images and depth information. Since various images (such as binocular images or multi-view images) for the display device are played back, it is possible to guarantee the image quality of the 3D stereoscopic image serviced to the user while maintaining compatibility with the existing DMB system and various 3D display devices. There is an effect that can provide a three-dimensional stereoscopic image service that can support.

In addition, since the present invention provides a 3D stereoscopic image service using depth information having a smaller amount of data than a multiview image, it is effective to minimize the bandwidth required for the 3D stereoscopic image service while preventing deterioration of image quality at the receiving side. have. That is, since the present invention uses depth information having a small amount of information, there is an effect of effectively providing stereoscopic image information even within a narrow bandwidth such as DMB.

In addition, the present invention can generate a variety of images (binocular image or multi-view image) using the reference image and depth information at the receiving side, and also the binocular or multi-view image of the same size (Size) generated in this way There is an effect of generating a three-dimensional stereoscopic image suitable for a variety of three-dimensional display device using.

In addition, the present invention, if the vibration (shaking) of the corresponding receiving terminal during the output of the three-dimensional stereoscopic image is replaced with the two-dimensional image, the output, thereby minimizing the user's visual fatigue.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 2 is a flowchart illustrating an object description method for a 3D stereoscopic image service based on DMB according to the present invention, and illustrates a method of expressing a reference image and depth information using one object descriptor. . That is, the present invention represents an MPEG-4 object descriptor structure in which a reference video stream and a depth information (depth map) stream output from a codec are represented in a master relationship using one object descriptor. 4 OD / BIFS section.

The present invention, unlike the existing technology that uses a video of a different point of view than the reference image to provide a three-dimensional stereoscopic image service, the 'side image' and 'depth information with a smaller amount of data than the image of the other view' on the transmitting side When transmitting, the receiver generates a binocular image or a multiview image suitable for various three-dimensional display apparatuses using the reference image and the depth information.

According to the present invention, two elementary stream descriptors (ESDs) 21 and 22 for the reference video stream and the depth information stream may be subordinated under one object descriptor 20 so as to provide a 3D stereoscopic image service. This dependency is defined to ensure compatibility with the single object descriptor structure assigned to the video stream in the DMB specification.

The 'reference video' is an independent stream 20, and is set to "ObjectTypeIndication = 0x21 (AVC) and StreamType = 0x04 (visual stream)" by using an advanced video coding (AVC) codec, which is an existing DMB standard.

Depth information (depth map) is a dependent stream, encoded using an existing codec or various codecs to be newly defined, such as "ObjectTypeIndication = 0xC0 (user private), StreamType = 0x04 (visual stream), StreamDependanceFlag = 1, dependsOn_ES_ID = reference video. ES_ID "or" ObjectTypeIndication = 0xC0 (user private), StreamType = 0x20 (user private), StreamDependanceFlag = 1, dependsOn_ES_ID = reference video ES_ID ". At this time, the depth information descriptor (Depth_descriptor) is included in "extDescr" of the elementary stream descriptor (ESD) 22 for representing the depth information stream, and through this, additional information required for generating a 3D stereoscopic image in the 3D receiving system. (See Figure 3).

In the present invention, the transmitter does not transmit a multi-view image, but transmits depth information indicating a depth value of each pixel, because the 'depth information' has a small amount of information and is suitable for transmission within a narrow bandwidth such as DMB. In addition, the receiver may generate a binocular 3D stereoscopic image or a multiview 3D stereoscopic image using the reference image and depth information.

3 is a structure diagram of an embodiment of a depth information descriptor (Depth_descriptor) according to the present invention, which may be extended according to various types of additional information required in the future.

The depth information descriptor (Depth_descriptor) includes information as shown in FIG. 3 in "extDescr" in the elementary stream descriptor (ESD) 22.

That is, in the present invention, the depth information descriptor (Depth_descriptor) includes "Tag" (221, 31), "Min_depth_infomation" (222, 33), "Max_depth_information" (223, 34), "Focal_length_information" (224, 35), and the like. It is characterized by having a structure to.

Here, "Tag" (221, 31) represents the identifier of the descriptor and is set to "0xC1 (user private)", "Min_depth_infomation" (222, 33) represents the minimum depth value of the scene set from the camera, "Max_depth_information" (223, 34) represents the maximum depth value of the scene set from the camera, "Focal_length_information" (224, 35) represents the focal length. Here, the "Focal_length_information" 224 and 35 may include various camera parameters as an example of the camera parameter.

In addition, the "Length Field" 32 indicates the length of the following information, and 8/16/24/32 is merely for convenience and in reality, only one of 8/16/24/32 bits is used. do.

4 is a structural diagram of an embodiment of a contents type descriptor (ContentsType_Descriptor) according to the present invention.

The "ContentsType_Descriptor" 201 defined in the "extDescr" of the object descriptor 20 for the reference video indicates a 'type of content' transmitted through the DMB, and in the case of an existing broadcast, the content type (Contents_type) 202. Is set to "2D" and in the case of 3D broadcasting, the content type (Contents_type) 202 is set to "3D". Then, the 3D receiving system (receiving terminal) can recognize the type of content (type of broadcast service) received through the contents type (Contents_type) 202.

In the present invention, the content type descriptor ContentsType_Descriptor is characterized by having a structure including "Tag" 201, 41, "Contents_type" 202, 43, and the like.

Here, "Tag" 201, 41 is an identifier of a descriptor and is set to "0xC0 (user private)", and "Contents_type" 202, 43 indicates a type of content to be received. In addition, the "Length Field" 42 indicates the length of information afterwards, and 8/16/24/32 is merely a representation of convenience and is actually used by only one of 8/16/24/32 bits. .

"Depth_descriptor" and "ContentsType_ Descriptor" described with reference to FIGS. 2 through 4 are defined in "extDescr" in the object descriptor (OD) 20 and the elementary stream descriptor (ESD) 22, and the value "Tag" is set to "user". private "can maintain compatibility with the existing DMB receiving terminal.

5 is a diagram illustrating an embodiment of a system for receiving a 3D stereoscopic image service based on DMB according to the present invention. Hereinafter, the 3D stereoscopic image service method performed in the DMB receiving system will also be described.

The 3D stereoscopic image service receiving system according to the present invention, as shown in FIG. 5, includes a decoder 50 and a 3D stereoscopic image reproducing unit 52, which will be described below. .

First, the decoder 50 restores the reference image, the depth information, the additional information necessary for generating the 3D stereoscopic image, and the broadcasting type information from the transport stream TS transmitted through the broadcasting network. OD) to recognize and decode the reference image and depth information separately, the demultiplexer 500, PSI analysis unit 501, IOD decoder 502, 14496 section de-packetizer (503) ), OD / BIFS decoder 504, PES depacketizer 505, SL (Sync Layer) depacketizer 506, AVC decoder 507, depth decoder 508, and BSAC decoder 509. It is made to include.

Among these, "500" to "506" receive a DMB signal and demultiplex and depacketize the MPEG-2 transport stream (TS) to perform the function of outputting each media encoded stream to a corresponding decoder. And "507" to "509" independently decode the reference video stream, the depth information stream, and the audio stream output from the system decoder.

The demultiplexer 500 analyzes the MPEG-2 transport stream (TS) and divides the program into a PSI section, a 14496 section, a reference picture packetized elementary stream (PES), a depth information PES, and an audio PES. The separated data is independently decoded as follows.

When the PSI analyzer 501 parses the PSI section data separated by the demultiplexer 500 and outputs an IOD (Initial Object Descriptor) stream included in a program map table (PMT), the IOD decoder 502 may output the IOD. Decode the stream.

When the 14496 section depacketizer 503 parses the 14496 section data separated by the demultiplexer 500 to output an OD (Object Descriptor) / BIFS (Baryary Format for Scenes) stream, the OD / BIFS decoder 504 Decode OD / BIFS streams. In particular, the OD / BIFS decoder 504 recognizes the reference video / depth information / audio stream through the object descriptor and controls the decoding of the corresponding decoders 507 to 509 according to the recognition result. In particular, the OD / BIFS decoder 504 recognizes the reference video stream and the depth information stream through the same object descriptor, and obtains the type information of the broadcast currently being serviced through the contents type descriptor (ContentsType_Descriptor). Depth_descriptor) obtains additional information required for generating a 3D stereoscopic image.

The PES depacketizer 505 depackets the reference video PES, the depth information PES, and the audio PES that are separated and output from the demultiplexer 500 to generate SL packets for each, and the SL depacketizer 506. Depacket each SL packet to generate an elementary stream (ES) for each.

Then, the Advanced Video Coding (AVC) decoder 507 decodes the input reference video elementary stream (ES) to restore the reference video, and the depth information decoder 508 decodes the input depth information elementary stream (ES). The depth information is restored, and the bit sliced arithmetic coding (BSAC) decoder 509 decodes the input audio elementary stream (ES) to restore the audio data.

In order to effectively provide a 3D stereoscopic video service, synchronization of a reference video and a depth information stream is required. In the present invention, a stream having a same CTS is transmitted to a corresponding decoder by comparing a time stamp (CTS), which is time information. Solve.

That is, in the present invention, synchronization is performed using the CTS of the SL packet output by depacketizing the packetized elementary stream (PES). First, buffering the reference video stream and the depth information stream in units of frames according to the value in the CTS. If the CTS of the reference video is larger than the CTS of the depth information, the CTS of the depth information is considered to have passed and the corresponding CTS is removed. Wait for a stream of depth information to have. On the other hand, if the CTS of the reference image is smaller than the CTS of the depth information, the frame of the reference image is removed and waits until a reference image having the same CTS comes in. In this way, the synchronization problem is solved by transmitting the reference video and the depth information stream having the same CTS to the corresponding decoder.

Next, the 3D stereoscopic image reproducing unit 52 will be described in detail.

The 3D stereoscopic image reproducing unit 52 according to the present invention generates and outputs a 3D stereoscopic image using the reference image, the depth information, the additional information, and the user input information according to the broadcasting type information, or outputs the reference image. The output unit includes a 3D image generator 521 and an image format converter 522.

The 3D image generator 521 may include the reference image and depth information (depth image) output from the decoder 50, broadcast type information and additional information (minimum depth value, maximum depth) output from the OD / BIFS decoder 504. Values, camera parameters, etc.), and images (binocular or multi-view images) of a viewpoint different from the reference image are generated based on the attribute information of the 3D display device, or the reference image is bypassed as it is.

In this case, the generated binocular or multiview image has the same image size to support various 3D display devices without deterioration of image quality when generating 3D stereoscopic images.

Here, the attribute information of the 3D display device may include view information (eg, whether a binocular 3D monitor or a multiview 3D monitor) that can be supported by the 3D display device, and the 3D display device. Contains disparity information. In the multi-view three-dimensional display device, the interval information between the viewpoints is additionally included. The attribute information of the 3D display apparatus is directly input by the user or input from the 3D display apparatus (that is, inputting a specification required by the 3D display apparatus to the 3D image generator 521). .

In other words, the 3D image generator 521 checks the broadcast type information and bypasses the reference image to the image format converting unit 522 if the 2D broadcast is generated, and generates an image at a different point of time than the reference image if the 3D broadcast is generated. do.

Meanwhile, the image format converter 522 combines binocular / multi-view images (including still images) output from the 3D image generator 521 according to user interaction information or vibration information (shake information), and then 3D. A stereoscopic image is generated and output, or a two-dimensional image (reference image) is simply output. Here, the user interaction information is user input information, which includes two-dimensional and three-dimensional reproduction mode information (display mode information), a three-dimensional adjustment mode, and the like.

That is, the image format converter 522 outputs one of the two-dimensional image (the reference image) or the three-dimensional stereoscopic image to the display device according to the two-dimensional and three-dimensional reproduction mode information input from the user. In addition, the image format converter 522 selectively outputs one of the two-dimensional image (the reference image) or the three-dimensional stereoscopic image according to the vibration information (shake information) of the 3D stereoscopic image service receiving system (see FIG. 6). ).

In addition, the image format converter 522 adjusts the depth of the 3D stereoscopic image reproduced according to the stereoscopic adjustment mode input by the user.

In short, the image format converter 522 simply transfers the reference image (2D image) bypassed by the 3D image generator 521 to the display device, or the binocular expression delivered from the 3D image generator 521. Combining multi-view images (including reference images) to create a 3D stereoscopic image and delivering it to the display device, depending on the display control information (2D / 3D playback mode information, stereoscopic adjustment information, vibration information). Adjust.

6 is a flowchart illustrating an image format conversion method using vibration information according to an exemplary embodiment of the present invention, and illustrates an image format conversion method performed by the image format conversion unit 522.

The image format converting unit 522 is a three-dimensional image generated by using a two-dimensional image (reference image) or a three-dimensional stereoscopic image (reference image) according to vibration information (shake information) of the three-dimensional stereoscopic image service receiving system (receiving terminal). Selectively output one of stereoscopic images). In particular, when a user shakes a lot while watching a 3D stereoscopic image, visual fatigue may be minimized by outputting the 2D image.

The image format converter 522 detects vibration (shake) of the 3D stereoscopic image receiving system (terminal) (600). Here, the vibration information may be detected through a motion sensor. The motion sensor may be included in the image format converter 522, and according to an embodiment, any position outside the image format converter 522 may be included. May be present in

Then, by comparing the detected intensity of the vibration with a predetermined threshold (602), if the intensity of the vibration is greater than the threshold (when the shaking is severe), the reference image (two-dimensional image) is selected to minimize the user's visual fatigue And output (604). That is, even though the output is a 3D stereoscopic image, the reference image is selected and output.

On the contrary, if the vibration intensity is smaller than the threshold value (when the shaking is weak), the 3D stereoscopic image generated by combining the image of another viewpoint with the reference image is selected and output (step 606).

 On the other hand, the method of the present invention as described above can be written in a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. The recording medium may include any type of computer readable recording medium.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

1 is a structural diagram of an MPEG-4 object descriptor used to describe an audio / video stream in a conventional DMB service;

2 is a flowchart illustrating an object description method for a 3D stereoscopic video service based on DMB according to the present invention;

3 is a structural diagram of an embodiment of a depth information descriptor (Depth_descriptor) according to the present invention;

4 is a structural diagram of an embodiment of a contents type descriptor (ContentsType_descriptor) according to the present invention;

5 is a configuration diagram of an embodiment of a DMB-based DMB-based 3D stereoscopic image service receiving system according to the present invention;

6 is a flowchart illustrating an embodiment of a video format conversion method using vibration information according to the present invention.

* Explanation of symbols on the main parts of the drawings

50: decoder 52 three-dimensional stereoscopic image reproduction unit

500: demultiplexer 501: PSI analyzer

502: IOD decoder 503: 14496 section inverse packetization unit

504: OD / BIFS decoder 505: PES reverse packetization unit

506: SL depacketization unit 507: AVC decoder

508: depth information decoder 521: 3D image generating unit

522: video format converter

Claims (10)

In the object description method for 3D stereoscopic video service based on DMB, A broadcast type setting step of setting in the object descriptor for the reference video whether the broadcast service corresponds to a broadcast type of 2D broadcast or 3D broadcast; And An object description for describing the reference image and depth information of the 3D stereoscopic image, which is the target of the broadcast service, using one object descriptor, wherein the reference image is set as an independent stream and the depth information is set as a dependent stream. step Object description method for a three-dimensional stereoscopic image service comprising a. The method of claim 1, The broadcast type setting step, An object description method for a 3D stereoscopic video service, characterized in that a broadcast type is defined using a content type descriptor. The method of claim 1, The object description step, An elementary stream descriptor (ESD) for the reference image and an elementary stream descriptor (ESD) for the depth information may be configured in a main or slave relationship using the same object descriptor (OD). Object description method for 3D stereoscopic image service. The method of claim 3, wherein The object description step, Under the depth information elementary stream descriptor (ESD), additional information required for generating a 3D stereoscopic image is described through a depth information descriptor. The method of claim 4, wherein The additional information, And at least one of a minimum and a maximum depth value or a camera parameter. In the method of receiving 3D stereoscopic image service based on DMB, A decoding step of decoding the transport stream to obtain reference video, depth information, and broadcast type information; A three-dimensional image generation step of generating an image of a viewpoint different from the reference image by using the reference image and the depth information when it is confirmed as three-dimensional broadcasting through the broadcasting type information; And An image format conversion step of outputting a 3D stereoscopic image or outputting the reference image by combining the reference image and the other viewpoint image according to display control information 3D stereoscopic video service receiving method comprising a. The method of claim 6, The decoding step, Recognizing and decoding reference image and depth information through an object descriptor, and receiving the broadcast type information through a content type descriptor defined in an object descriptor for the reference image . The method of claim 6, The decoding step, And receiving additional information necessary for generating the another view image through a depth information descriptor defined under an elementary stream descriptor (ESD) of the depth information. In the video format conversion method in the 3D stereoscopic video service receiving system, A sensing step of sensing vibration of the receiving system; And Outputting a two-dimensional reference image when the intensity of the detected vibration is greater than or equal to a predetermined threshold value, and outputting a three-dimensional stereoscopic image using the two-dimensional reference image when the intensity of the detected vibration is less than a predetermined threshold value. Video format conversion step Image format conversion method using vibration information comprising a. delete

Images