KR20130056684A - Supplementary data coding apparatus and method for color video and depth video - Google Patents

Abstract

PURPOSE: A device for encoding/decoding additional information about color images and depth images and a method thereof are provided to supply information about the encoding and decoding of the color images and the depth images by providing information about a view, an image type, a reference image, and focus depth. CONSTITUTION: A reference image confirming unit(140) confirms a color image and a depth image. A reference information generating unit(150) generates reference information including the number of color images, a list of the color images, the number of depth images and a list of the depth image. An encoding unit(180) encodes additional information including the reference information. [Reference numerals] (110) View confirming unit; (120) Image type determining unit; (130) View information generating unit; (140) Reference image confirming unit; (150) Reference information generating unit; (160) Focus depth receiving unit; (170) Encoding determining unit; (180) Encoding unit; (AA) Encoded additional information

Description

Device and method for encoding / decoding additional information about color image and depth image {Supplementary data Coding Apparatus and Method for color video and depth video}

The technical field relates to encoding and decoding of three-dimensional images. In particular, the present invention relates to an apparatus and method for efficiently encoding and decoding side information of a 3D image.

Recently, 3DVC (3D video coding), an international standard for 3D video image, is being prepared in MPEG, aiming at 3D image service using multiview color image and multiview depth image. In addition to compression on depth images, we are studying rendering techniques using them.

In order to provide a 3D image using the multi-view color image and the multi-view depth image, a relationship between the multi-view color image and the multi-view depth image and definition of additional information, which are information for generating a stereoscopic image, are required. do. There is also a need for an efficient encoding decoding apparatus and method for transmitting side information.

In one aspect, a reference image checking unit for identifying a virtual reference color image and a virtual reference depth image to encode an image, the number of the virtual reference color image, the list of the virtual reference color image, the number of the virtual reference depth image And a reference information generator for generating reference information including a list of the virtually referenced depth images, and an encoder for encoding additional information including the reference information.

In this case, the apparatus further includes a viewpoint checking unit for confirming the viewpoint of the image, an image type determination unit for confirming whether the image is a color image or a depth image, and a viewpoint information generation unit for generating viewpoint information including the image type. The encoder may encode additional information including the viewpoint information and the reference information.

In this case, the view point information generating unit generates the view point information by multiplying the view point of the checked image when the image is a color image by 2, and multiplies the view point of the checked image when the image is a depth image, and then adds 1 to the view point. View information may be generated.

In this case, the view point information generating unit generates the view point information by multiplying the view point of the checked image if the image is a depth image, and multiplies the view point of the checked image if the image is a color image, and then adds 1 to the view point. View information may be generated.

In this case, the viewpoint information generation unit may generate the viewpoint information including a viewpoint parameter indicating a viewpoint of the image, and flag information for distinguishing a color image and a depth image.

In this case, the apparatus further includes a focus depth receiver configured to receive focus depth information and an encoding determiner configured to determine that encoding is necessary when the focus depth information is updated, and the encoder includes additional information including the reference information and the updated focus depth information. Can be encoded.

In this case, when the first frame is transmitted, the encoding determiner determines that the focal depth information is required to be encoded, and when the first frame is transmitted, the encoder determines additional information including the reference information and the focal depth information. Can be encoded.

In this case, when the encoding determination unit transmits a frame having a predetermined interval to provide a random access service, it is determined that encoding of the depth of focus information is necessary, and when the encoding unit transmits the first frame, Additional information including the reference information and the depth of focus information may be encoded.

In one aspect, the decoding unit for decoding the encoded side information and the number of color images that are virtually referenced by analyzing the reference information included in the additional information, the list of the color reference color image, the number of the virtual reference depth image and the virtual reference depth There is provided an additional information decoding apparatus including a reference information analyzer which checks a list of images.

In this case, when the additional information includes viewpoint information, an image confirming point of time of the image, and when the additional information includes the viewpoint information, an image of determining whether the image is a color image or a depth image. The apparatus may further include a type checking unit.

In this case, if the viewpoint information is an even number, the viewpoint checking unit checks a value obtained by dividing the viewpoint information by two as the viewpoint of the image, and when the viewpoint information is odd, subtracts 1 from the viewpoint information by 2 and then divides the value by 2. It can be confirmed from the viewpoint of the image.

In this case, the image type checking unit may check the image type as a color image when the viewpoint information is even, and check the image type as a depth image when the viewpoint information is odd.

In this case, the image type checking unit may check the image type as the depth image when the viewpoint information is even, and check the image type as the color image when the viewpoint information is odd.

In this case, the viewpoint check unit checks the viewpoint parameters included in the viewpoint information and confirms the viewpoint of the image, and the image type confirmation unit checks the flag information for distinguishing the color image and the depth image included in the viewpoint information and the image. You can check the type.

In this case, if the storage unit storing the recent focus depth and the additional depth information includes the focus depth information, the focus depth of the display screen is checked using the focus depth information, and stored as the recent focus depth in the storage unit, If the depth information is not included in the additional information, the storage unit may further include a focus depth check unit that checks the recent depth of focus as the depth of focus of the image.

In an aspect, the method may further include: verifying a virtually referenced color image and a virtually referenced depth image for encoding an image, the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the virtual Generating reference information including a list of referenced depth images, identifying a viewpoint of the image, identifying a type of image whether the image is a color image or a depth image, and a viewpoint including the image type There is provided an additional information encoding method comprising generating information and encoding additional information including the viewpoint information and the reference information.

In this case, the method may further include receiving updated focus depth information, and encoding the additional information may encode additional information including the viewpoint information, the reference information, and the updated focus depth information.

In an aspect, the method may further include decoding a coded additional information, analyzing the reference information included in the additional information, the number of virtually referenced color images, the list of virtually referenced color images, the number of virtually referenced depth images, and the virtually referenced depth. Checking a list of images, if the additional information includes viewpoint information, and checking the viewpoint of the image and if the additional information includes the viewpoint information, whether the image is a color image or a depth image. There is provided a method of decoding additional information comprising the step of identifying the type.

In this case, if the depth information is included in the additional information, checking the focus depth of the display screen using the focus depth information and storing the latest focus depth, and if the additional information does not include the focus depth information, The method may further include checking the latest stored depth of focus as the depth of focus of the image.

The present invention provides an apparatus and method for encoding / decoding additional information about a color image and a depth image. The proposed encoding / decoding method of the additional information may provide necessary information for encoding and decoding the color image and the depth image by providing the viewpoint, the image type, the information of the reference image, and the depth of focus.

1 is a diagram illustrating a configuration of an encoding apparatus that encodes additional information about a color image and a depth image.
2 is a diagram illustrating a configuration of a decoding apparatus for decoding additional information about a color image and a depth image.
3 is a flowchart illustrating a process of encoding additional information about a color image and a depth image in an encoding apparatus.
4 is a flowchart illustrating a process of decoding additional information about a color image and a depth image in a decoding apparatus.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a configuration of an encoding apparatus that encodes additional information about a color image and a depth image.

Referring to FIG. 1, the encoding apparatus 100 may include a viewpoint identification unit 110, a warrant type determination unit 120, a viewpoint information generation unit 130, a reference image identification unit 140, a reference information generation unit 150, The apparatus may include a focus depth receiver 160, an encoding determiner 170, and an encoder 180.

The viewpoint checking unit 110 checks the viewpoint of the image.

The warrant type determiner 120 checks the image type whether the image is a color image or a depth image.

The viewpoint information generator 130 generates viewpoint information including an image type.

There are several methods for generating viewpoint information in the viewpoint information generation unit 130. First, let's take a look at how to express both viewpoint and image type using one parameter.

If the image is a color image, the viewpoint information generation unit 130 generates viewpoint information by multiplying the viewpoint of the checked image by 2, and if the image is a depth image, multiplies the viewpoint of the identified image by 2 and adds 1 to generate viewpoint information. have.

That is, the viewpoint information may be set as shown in Table 1 below.

Syntax Semantics nal_unit_header_mvc_extension () { view _ id If even, it is color view and it means view_id / 2nd viewpoint
If odd, it is depth view and means (view_id-1) / 2nd view }

In addition, if the image is a depth image, the viewpoint information generation unit 130 generates viewpoint information by multiplying the viewpoint of the checked image by two, and if the image is a color image, multiplies the viewpoint of the identified image by two, and then adds one to generate viewpoint information. can do.

In addition, the viewpoint information generator 130 may generate viewpoint information including a viewpoint parameter indicating a viewpoint of an image, and flag information as shown in Table 2 below to distinguish a color image and a depth image.

syntax Semantics nal_unit_header_mvc_extension () { color _ depth _ flag 0 means “color view” and “1” means depth view. }

The reference image checking unit 140 confirms the virtual image of the color image and the virtual image of the depth reference to encode the image.

In this case, the virtual reference picture is referred to as a virtual reference picture when the picture is used as ancillary data of a specific method when the picture is encoded or decoded through a specific method rather than an image directly referenced when encoding or decoding. In this case, as a specific technique, a virtual composite image based encoding technique may be used. The virtual synthesis image-based encoding technique is a technique of generating a reference image referred to when encoding through virtual synthesis. At this time, the image used for the virtual synthesis is the virtual reference image.

The reference information generator 150 generates reference information including the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images.

Reference information may be set in the format shown in Table 3 below.

syntax Semantics for (i = 1; i <= num_views_minus1; i ++) { num _ virtual _ color _ refs [i] Number of virtual color references at the i th time point for (j = 0; j <num_virtual_color_refs [i]; j ++) { virtual _ color _ refs [i] [j] reference index for the j-th virtual color reference at the i-th point } num _ virtual _ depth _ refs [i] Number of virtual depth references at the i th time point for (j = 0; j <num_virtual_depth_refs [i]; j ++) { virtual _ depth _ refs [i] [j] reference index to the j-th virtual depth reference at the i-th time point } }

The focus depth receiver 160 receives focus depth information. In this case, the focus depth information indicates a depth value of the screen when the 3D image frame is displayed. For example, when the depth range of a 3D image is expressed as a value of 1 to 10, if the depth of focus of the screen is 1, a three-dimensional effect is formed from the rear of the screen. Will be formed.

The depth of focus information may be set in the format of <Table 4> below.

syntax Semantics depth_of_focus_infor (payloadSize) { depth _ of _ focus Depth value on display screen of current frame }

The encoding determination unit 170 checks whether the focus depth information is updated and determines that encoding is necessary if the focus depth information is updated.

The encoding determiner 170 may determine that encoding of the depth depth information is necessary when transmitting the first frame or when transmitting a frame having a predetermined interval to provide a random access service.

The encoder 180 encodes additional information including view information, reference information, and updated focus depth information when encoding of the focus depth information is necessary as a result of the determination by the encoding determiner 250.

2 is a diagram illustrating a configuration of a decoding apparatus for decoding additional information about a color image and a depth image.

Referring to FIG. 2, the decoding apparatus 200 includes a decoder 210, a view point determiner 220, an image type checker 230, a reference information checker 240, a depth of focus checker 250, and storage. It may include a portion 260.

The decoder 210 decodes the encoded side information received from the encoding apparatus 100.

If the viewpoint information is included in the additional information, the viewpoint checking unit 220 checks the viewpoint of the image.

If the viewpoint information is even, the viewpoint checking unit 220 checks the value obtained by dividing the viewpoint information by two as the viewpoint of the image, and when the viewpoint information is odd, subtracts 1 from the viewpoint information by 2 and identifies the value divided by 2 as the viewpoint of the image. Can be.

Alternatively, the viewpoint checking unit 220 may confirm the viewpoint of the image by checking the viewpoint parameter included in the viewpoint information.

If the additional information includes viewpoint information, the image type checking unit 230 checks whether the image is a color image or a depth image.

The image type checking unit 230 may check whether the view information is even or odd to check the preset image type according to the even and the odd number.

Alternatively, the image type checking unit 230 may check the image type by checking flag information for distinguishing the color image and the depth image included in the viewpoint information.

The reference information checker 240 analyzes the reference information included in the additional information and checks the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images.

If the depth information includes the depth information, the depth checking unit 250 checks the focus depth of the display screen using the focus depth information, stores the focus depth in the storage unit 260 as the latest focus depth, and focuses on the additional information. If the depth information is not included, the storage unit 260 checks the latest focal depth as the focal depth of the image.

The storage unit 260 stores the latest focal depth.

Hereinafter, a method of encoding and decoding additional information about a color image and a depth image in the encoding / decoding apparatus according to the present invention configured as described above will be described with reference to the accompanying drawings.

3 is a flowchart illustrating a process of encoding additional information about a color image and a depth image in an encoding apparatus.

Referring to FIG. 3, in operation 310, the encoding apparatus 100 checks the viewpoint of an image.

In operation 312, the encoding apparatus 100 checks an image type whether the image is a color image or a depth image.

In operation 314, the encoding apparatus 100 generates view information including an image type. In this case, when the image is a color image, the view information may be generated by multiplying the viewpoint of the checked image by 2, and when the image is the depth image, the view information may be generated by multiplying the viewpoint of the checked image by 2 and then adding 1. In addition, the view information generation may generate view information by multiplying the viewpoint of the checked image by multiplying 2 when the image is a depth image, and generating the view information by multiplying 2 and then adding 1 to the view of the checked image when the image is a color image. In addition, the viewpoint information generation may generate viewpoint information including a viewpoint parameter indicating a viewpoint of an image and flag information for distinguishing a color image and a depth image.

In operation 316, the encoding apparatus 100 generates reference information including the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images.

In operation 318, the encoding apparatus 100 determines whether the updated depth depth information is received.

In operation 318, when the updated depth of focus information is received, the encoding apparatus 100 encodes additional information including view information, reference image information, and focus depth information in operation 320.

In operation 318, if the updated depth of focus information is not received, the encoding apparatus 100 encodes additional information including view information and reference information in operation 324.

4 is a flowchart illustrating a process of decoding additional information about a color image and a depth image in a decoding apparatus.

Referring to FIG. 4, in operation 410, the decoding apparatus 200 receives encoded additional information from an encoding apparatus.

In operation 412, the decoding apparatus 200 decodes the encoded side information.

In operation 414, the decoding apparatus 200 checks the viewpoint of the image when the viewpoint information is included in the additional information. If the viewpoint information is even, the view point of the image may be determined by dividing the viewpoint information by 2 as the viewpoint of the image, and when the viewpoint information is odd, the value obtained by subtracting 1 from the viewpoint information by 2 and dividing by 2 may be determined as the viewpoint of the image. . Alternatively, the viewpoint of the image may be confirmed by checking the viewpoint parameter included in the viewpoint information of the image.

In operation 416, if the additional information includes viewpoint information, the decoding apparatus 200 checks an image type whether the image is a color image or a depth image. In this case, the image type check may determine whether the view information is even or odd to check the preset image type according to the even and the odd number. Alternatively, the image type check may be performed by checking flag information for distinguishing the color image and the depth image included in the viewpoint information, and confirming the image type.

In operation 418, the decoding apparatus 200 analyzes the reference information included in the additional information and checks the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images. Shall.

In operation 420, the decoding apparatus 200 checks whether the depth information includes the focal depth information.

If the depth information is included in the additional information as a result of checking in step 420, the decoding apparatus 200 checks the focus depth of the display screen using the focus depth information in step 422 and stores it as the latest focus depth.

In operation 420, if the depth information is not included in the additional information, the decoding apparatus 200 checks the latest focal depth stored in operation 424 as the focal depth of the image.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

Claims (19)

A reference image checking unit which checks a color image and a virtual depth image referred to in encoding the image;
A reference information generation unit for generating reference information including the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images; And
And an encoder which encodes additional information including the reference information.
Additional information encoding apparatus.
The method of claim 1,
A view point checker to check a view point of the image;
An image type determination unit for determining whether an image is a color image or a depth image; And
The apparatus may further include a viewpoint information generation unit configured to generate viewpoint information including the image type.
Wherein the encoding unit comprises:
Encoding additional information including the viewpoint information and the reference information
Additional information encoding apparatus.
The method of claim 1,
The viewpoint information generation unit,
If the image is a color image, the viewpoint information is generated by multiplying the viewpoint of the identified image by two,
When the image is a depth image, the view information of the identified image is multiplied by 2 and then 1 is added to generate the view information.
Additional information encoding apparatus.
The method of claim 1,
The viewpoint information generation unit,
If the image is a depth image, the viewpoint information is generated by multiplying the viewpoint of the identified image by 2,
When the image is a color image, the view information of the identified image is multiplied by 2, and then 1 is added to generate the view information.
Additional information encoding apparatus.
The method of claim 1,
The viewpoint information generation unit,
Generating the viewpoint information including a viewpoint parameter indicating a viewpoint of the image and flag information for distinguishing a color image and a depth image;
Additional information encoding apparatus.
The method of claim 1,
A focus depth receiver configured to receive focus depth information; And
The apparatus may further include an encoding determiner configured to determine that encoding is necessary when the focus depth information is updated.
Wherein the encoding unit comprises:
Encoding additional information including the reference information and the updated focus depth information
Additional information encoding apparatus.
9. The method of claim 8,
The encoding determination unit,
When transmitting the first frame, it is determined that the encoding of the depth information is necessary,
Wherein the encoding unit comprises:
In case of transmitting the first frame, additional information including the reference information and the depth of focus information is encoded.
Additional information encoding apparatus.
The method according to claim 6,
The encoding determination unit,
When transmitting a frame of a predetermined interval to provide a random access (random access) service, it is determined that the encoding of the focus depth information is necessary,
Wherein the encoding unit comprises:
In case of transmitting the first frame, additional information including the reference information and the depth of focus information is encoded.
Additional information encoding apparatus.
A decoder which decodes the encoded side information; And
And a reference information analyzer for analyzing the reference information included in the additional information to identify the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images.
Side information decoding apparatus.
10. The method of claim 9,
A view point confirming unit for checking a view point when the view information is included in the additional information; And
If the additional information includes the viewpoint information, the image type confirmation unit for checking the image type whether the image is a color image or a depth image further comprises:
Side information decoding apparatus.
The method of claim 10,
The point of time confirmation unit,
When the viewpoint information is even, the value obtained by dividing the viewpoint information by two is determined as the viewpoint of the image.
If the viewpoint information is odd, the value obtained by subtracting 1 from the viewpoint information by 2 and dividing by 2 is determined as the viewpoint of the image.
Side information decoding apparatus.
The method of claim 11,
The image type checking unit,
If the viewpoint information is even, check the image type as a color image,
If the viewpoint information is odd, the image type is checked as the depth image.
Side information decoding apparatus.
The method of claim 11,
The image type checking unit,
If the viewpoint information is even, check the image type as the depth image,
When the viewpoint information is odd, the image type is checked as a color image.
Side information decoding apparatus.
The method of claim 10,
The point of time confirmation unit,
Confirming the viewpoint of the image by checking the viewpoint parameter included in the viewpoint information,
The image type checking unit,
Confirming the type of the image by checking the flag information for distinguishing the color image and the depth image included in the viewpoint information
Side information decoding apparatus.
10. The method of claim 9,
A storage unit for storing a recent focal depth; And
If the depth information includes the focus depth information, the focus depth of the display screen is checked using the focus depth information, and stored as the latest focus depth in the storage unit, and the depth information does not include the focus depth information. If not, the storage unit further includes a focal depth checking unit for checking the latest focal depth as the focal depth of the image.
Side information decoding apparatus.
Identifying a virtual image and a virtually referenced depth image for encoding the image;
Generating reference information including the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images;
Confirming the viewpoint of the image;
Checking an image type whether the image is a color image or a depth image;
Generating viewpoint information including the image type; And
Encoding additional information including the viewpoint information and the reference information.
Side information coding method.
17. The method of claim 16,
Receiving updated focal depth information;
The encoding of the additional information may include:
Encoding additional information including the viewpoint information, the reference information, and the updated focus depth information
Side information coding method.
Decoding the encoded side information;
Analyzing the reference information included in the additional information to identify the number of the virtually referenced color images, the list of the virtually referenced color images, the number of the virtually referenced depth images, and the list of the virtually referenced depth images;
If the additional information includes viewpoint information, checking a viewpoint of an image; And
If the additional information includes the viewpoint information, determining an image type whether the image is a color image or a depth image.
Side information decoding method.
19. The method of claim 18,
If the additional information includes focus depth information, checking a focus depth of a display screen using the focus depth information and storing the latest depth; And
If the additional information does not include focus depth information, the method may further include identifying the stored last focus depth as a focus depth of the image.
Side information decoding method.

Images