JP2020120322A - Distance image coding device and program of the same, and distance image decoding device and program of the same - Google Patents

Abstract

To provide a distance image coding device capable of hierarchically coding a distance image in each depth range.SOLUTION: A distance image coding device 1 comprises: basic layer coding means 12 of roughly coding a distance image to generate a basic layer coding stream; local decoding means 13 of generating a decoding image from a quantization coefficient; decoding difference image generation means 14 of generating a difference between the distance image and a decoding image as a decoding difference image; region division means 11 of dividing a region of the distance image in each depth range which has been previously set; individual depth difference image generation means 15 of generating each image in each depth range from the decoding difference image as an individual depth difference image; expansion layer coding means 16 of generating an expansion layer coding stream by finely coding the individual depth difference image in each depth range; and stream coupling means 17 of coupling the expansion layer coding stream in each basic layer coding stream and each depth range.SELECTED DRAWING: Figure 1

Description

The present invention relates to a distance image encoding device that encodes a distance image and a program thereof, and a distance image decoding device that decodes an encoded distance image and a program thereof.

At present, in transmitting a stereoscopic image (multi-viewpoint image), a method of transmitting a captured image and a distance image (depth map) indicating depth information from a shooting position (viewpoint position) together is being studied. In the international standard, a coding method such as 3D-AVC, 3D-HEVC in which a multi-view image and a distance image are coded and a decoding side generates an arbitrary viewpoint image is determined (Non-Patent Documents 1 and 2). reference).
Conventionally, the coding of a captured image based on motion compensation is basically applied as it is to the coding of a distance image. Further, while applying the encoding method of the captured image, focusing on the difference in the statistical property between the captured image and the distance image, for example, the captured image has more high-frequency components than the distance image, There is disclosed a method of encoding by changing a quantization parameter for an image (see Non-Patent Document 3) and a technique of suppressing compression of a distance image at a region boundary that affects image quality (see Non-Patent Document 4).
In addition, as a coding method of a captured image, there is a method of performing coding by hierarchically changing the compression efficiency depending on the specifications of a transmission line and a receiving device (decoding device) (Patent Documents 1 and 2, Non-Patent Documents) 5).

JP, 2004-350072, A JP, 2007-266748, A

Shimizu, "International Standardization Trend of 3D Video Coding Using Depth Map", IPSJ Research Report, Vol.2013-AVM-82, No.11, pp.1-6 (2013). Senoo, Yamamoto, Oi, Kurita, "Standardization activity of MPEG multi-view video coding", National Institute of Information and Communications Technology, Vol.56, Nos.1/2, pp.79-90 (2010). Saldanha, Sanchez, Marcon, Agostini, “Block-level fast coding scheme for depth maps in three-dimensional high efficiency video coding”, Journal of Electronic Imaging, Vol.27(1), 64,010502-1-4(2018) . Oh, Vetro, Ho, “Depth Coding Using a Boundary Reconstruction Filter for 3-D Video Systems”, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, Vol.21, No.3, pp.350-359 (2011). Tsukuba, Eikichi, Hanamura, Tominaga, "A Study on Spatial Scalable Coding Scheme with Independent Layers", IPSJ Research Report, Vol.2004-AVM-45, pp.29-34 (2004).

As a conventional distance image encoding method, the captured image encoding method is basically applied as it is. Therefore, in consideration of compression of encoded data due to restrictions on the bandwidth of the transmission path, performance of the receiving device, etc., it is necessary to hierarchically encode the captured image and also hierarchically encode the distance image. ..
However, if the conventional layering method is applied to the range image as it is, the range image is simply layered spatially regardless of the depth.
In addition, it is known that human depth sensitivity is high in a region close to the viewpoint and low in a region far from the viewpoint (references below).
(References) Nagata, "Visual Depth Distance Information and Its Depth Sensitivity", The Institute of Image Information and Television Engineers, Television, Vol.31, No.8, pp.649-655 (1977)
Therefore, if the distance image is simply hierarchically layered, when the bandwidth of the transmission path is narrow, when the performance of the receiving device is low, etc., in the reproduced image in which all the layers are not decoded, in the area close to the viewpoint. There is a problem that the deterioration of image quality becomes noticeable.

The present invention has been made in view of such a problem, and a distance image encoding device and a program thereof capable of hierarchically encoding/decoding a distance image for each depth range, and a distance image decoding device. It is an object to provide a device and its program.

In order to solve the above-mentioned problems, a distance image encoding device according to the present invention is a distance image encoding device that encodes a distance image indicating depth information of a subject, and includes a base layer encoding means and a local decoding means. The decoding differential image generating means, the area dividing means, the depth-based differential image generating means, the enhancement layer encoding means, and the stream combining means are provided.

In such a configuration, the distance image encoding device converts the distance image into frequency components by the base layer encoding means, quantizes the frequency components in the first quantization step, and performs variable length encoding of the quantized coefficient to thereby obtain the base layer code. Generate a stream. The first quantization step has a larger value than the second quantization step of the base layer coding means described later, so that the base layer coding means roughly quantizes the gray level of the range image. Encode.
Then, the distance image encoding device reproduces the decoded image of the distance image decoded on the decoding side by dequantizing the quantized coefficient and inversely transforming the frequency component by the local decoding means.

Further, the distance image coding device generates the difference between the distance image and the decoded image as the decoded difference image by the decoded difference image generating means.
Further, in the distance image coding device, the area dividing unit divides the area of the distance image for each preset depth range.
Then, the distance image encoding device uses the depth difference image generation means to generate an image for each area divided by the area division means from the decoded difference image as a depth difference image.

Then, in the distance image encoding device, the enhancement layer encoding means converts the corresponding depth-specific difference image into frequency components for each depth range, and quantizes them in a second quantization step smaller than the first quantization step. A variable length coding of the quantized coefficient generates an enhancement layer coded stream. Since the value of this second quantization step is smaller than that of the first quantization step of the base layer encoding means, the enhancement layer encoding means finely quantizes and encodes the gradation level of the depth difference image. be able to.

Then, the distance image encoding device combines the base layer encoded stream and the enhancement layer encoded stream for each depth range by the stream combining means to generate an encoded stream of the distance image.
By this, by the base layer encoded stream roughly encoded distance image, a plurality of enhancement layer encoded stream for each depth range finely encoded the difference between the coarsely encoded distance image and the original distance image, A hierarchically encoded stream is generated for each depth range.
The distance image encoding device can operate a computer with a program for causing each of the above-described means to function.

Further, in order to solve the above-mentioned problems, the distance image decoding device according to the present invention provides a base layer encoded stream obtained by encoding a distance image indicating depth information of a subject in a first quantization step, and the base layer encoded stream. Encoding by concatenating a plurality of enhancement layer coded streams for each depth range, in which the difference between the distance image coded as "1" and the original distance image is coded in the second quantization step smaller than the first quantization step A distance image decoding device for decoding a stream, comprising a stream separating means, a base layer decoding means, an enhancement layer decoding means, and an image synthesizing means.

In such a configuration, the distance image decoding device separates the encoded stream into the base layer encoded stream and the plurality of enhancement layer encoded streams by the stream separating means.

Then, the distance image decoding device decodes the base layer encoded stream by the base layer decoding means using the first quantization step having the same large value as that on the encoding side to generate a base layer decoded image. As a result, a range image having a rough gradation level is generated.
Further, the distance image decoding device uses the enhancement layer decoding means to decode the plurality of enhancement layer encoded streams using the same small second quantization step as that on the encoding side to generate a plurality of enhancement layer decoded images. As a result, the difference between the distance image encoded as the base layer encoded stream and the original distance image can be accurately reproduced up to a fine gradation level.

Then, the distance image decoding device synthesizes the base layer decoded image and the plurality of enhancement layer decoded images by the image synthesizing means to generate a distance image obtained by decoding the encoded stream.
Thereby, the distance image decoding device can hierarchically decode the encoded stream.
Note that the distance image decoding device can operate a computer by a program for causing each of the above-described means to function.

The present invention has the following excellent effects.
According to the present invention, a distance image can be hierarchically encoded/decoded for each depth range. As a result, the present invention enables coding in which the number of layers is limited according to the band of the transmission path, and the depth range is prioritized for decoding according to the performance of the distance image decoding device such as the CPU power. Streams can be created.

It is a block block diagram which shows the structure of the distance image coding apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating a distance image, (a) is a perspective view which shows arrangement|positioning of a camera (trinocular camera) and a to-be-photographed object, (b) shows a picked-up image, (c) shows a distance image. It is explanatory drawing for demonstrating the threshold value of a depth range. (A)-(c) is a figure which shows the mask data for every depth range. It is a flowchart which shows operation|movement of the distance image coding apparatus which concerns on embodiment of this invention. It is a block block diagram which shows the structure of the distance image decoding apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation|movement of the distance image decoding apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the other threshold value of a depth range.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of range image encoding device]
The configuration of the distance image encoding device 1 according to the embodiment of the present invention will be described with reference to FIG.

The distance image encoding device 1 hierarchically encodes a distance image (depth map) indicating depth information of a subject according to depth.
The distance image is an image in which the distance from the viewpoint position in the subject space where the subject is photographed to the subject is represented by pixel values assigned to a range from a depth minimum value to a depth maximum value that is predetermined for each pixel. If the captured image of the subject is a still image, the distance image is a single image. If the captured image is a moving image, the distance image will also be an image corresponding to the frame of the captured image.

Here, an example of the range image will be described with reference to FIG. 2.
As shown in FIG. 2A, the range image is two images (stereo image) of images obtained by photographing a subject (here, O _{1 to} O ₃ ) in the subject space with a camera (trinocular camera) C. It can be generated by allocating a pixel value for each pixel according to the parallax of FIG.
The camera C includes a left-eye camera C _L , a central camera C _C, and a right-eye camera C _R , which are arranged at equal intervals in the horizontal direction.
The central camera C _C is a camera that captures a subject as a captured image.
The left-eye camera C _L and the right-eye camera C _R are cameras that capture images for obtaining parallax.

FIG. 2 (b) shows a photographic image P taken by the center camera _{C C} in FIG. 2 (a). The distance image is matched with a photographed image (not shown) photographed by the left-eye camera C _L and a photographed image (not shown) photographed by the right-eye camera C _R , depending on the horizontal distance of the corresponding pixel. The pixel value corresponding to the parallax is assigned to the pixel position of the captured image P.
For example, the depth information corresponding to the captured image P in FIG. 2B is the distance image D represented by the gray image in FIG. 2C.
Here, the distance image D is displayed in white as it is closer to the viewpoint position and as black as it is farther from the viewpoint position. For example, when the gradation of the distance image is 256 gradations, the depth image has a minimum depth value of “0” and a maximum depth value of “255”.
Note that the distance image is generated here from the stereo image captured by the camera C, but it may be acquired by a general method such as a distance image sensor that measures the distance from the round trip time of the projected laser.

Returning to FIG. 1, the description of the configuration of the distance image encoding device 1 will be continued.
As shown in FIG. 1, the distance image encoding device 1 includes a threshold setting unit 10, a region dividing unit 11, a base layer encoding unit 12, a local decoding unit 13, a decoded difference image generating unit 14, and a depth. The differential image generation unit 15, the enhancement layer encoding unit 16, and the stream combining unit 17 are provided.

The threshold setting means 10 sets a threshold for hierarchically classifying the depth of the distance image. The threshold value is a depth value indicating the boundary of the depth layer in which the distance image is divided for each depth range. Note that, here, the number of depth layers is described as “4”, but it may be at least “2” or more.
The threshold value setting means 10 searches for the smallest depth value in the distance image, and sets the depth value obtained by adding a predetermined value to the depth value as the first threshold value T ₁ .
This makes it possible to specify the depth range that includes the subject whose distance from the viewpoint position (minimum depth) is closest.

Further, the threshold value setting means 10 equally divides the range from the threshold value T ₁ to the maximum depth value by the number obtained by subtracting “1” from the predetermined depth hierarchy number (here, “4”), and the second and later. Set the threshold of.
That is, the threshold value setting unit 10 sets the threshold value T ₁ and then sets the threshold value T _i (i is an integer of 2 or more and less than the number of depth layers) by the following equation (1). Note that D _max indicates the maximum depth value, and n indicates the number of depth layers.

As a result, as shown in FIG. 3, the depth range from the depth minimum value to the threshold T ₁ is the first layer L ₁ , the depth range from the threshold T ₁ to the threshold T ₂ is the second layer L ₂ , and the threshold T _{2 is} The depth image up to the threshold value T ₃ is the third layer L ₃ , and the depth range from the threshold value T ₃ to the maximum depth value is the fourth layer L ₄ , whereby the distance image can be hierarchically divided. At this time, the distance between the thresholds of the first layer L ₁ is d ₁ (=T ₁ ), and the distance between the thresholds of the second layer L ₂ to the fourth layer L ₄ is d _m (=(D _max − T ₁ )/3).
The threshold setting means 10 outputs the set thresholds (T _{1 to} T ₃ ) to the area dividing means 11.
When the input distance image is composed of continuous frames corresponding to the moving image, the threshold setting unit 10 may set the threshold for each frame, or set the threshold only for the first frame. It may be done.
In addition, if the distance to the closest object is known, the threshold setting unit 10 may input the threshold T ₁ from the outside.

The area dividing unit 11 divides the area of the distance image for each depth range specified by the set threshold.
The area dividing unit 11 sets, for each depth range, a set of pixels having a corresponding depth value in the distance image as area information indicating an area corresponding to the depth range.
Here, the area dividing unit 11 generates area information for each depth range as mask data. Specifically, the area dividing unit 11 generates mask data for each depth range by setting the pixel value of the pixel in which the pixel value of the distance image is included in the depth range to “1” and the other pixel values to “0”. To do. It should be noted that the threshold value that separates the two depth ranges is included in one of the depth ranges.

For example, as shown in FIG. 2, the distance image with respect to the objects O _{1 to} O ₃ is the distance image D of FIG. 2C, and as shown in FIG. 3, the objects O _{1 to} O ₃ have the first layer L ₁ and it was present in each of the depth range between the third layer L _3.
In this case, for the first layer L ₁ , the area dividing unit 11 sets “1” as the pixel value of the pixels in the depth range that is greater than or equal to the minimum depth value and less than the threshold value T ₁ in the distance image D, and sets the other pixel values as the pixel values. As “0”, the mask data M ₁ shown in FIG. 4A is generated.
In the distance image D, the area dividing unit 11 sets the pixel value of the pixels in the depth range equal to or more than the threshold value T _{1 and} less than the threshold value T ₂ to “1” for the second layer L ₂ , and sets the other pixel values to “1”. The mask data M ₂ shown in FIG. 4B is generated as 0″.
In the distance image D, the area dividing unit 11 sets the pixel values of the pixels in the depth range equal to or larger than the threshold T ₂ and smaller than the threshold T ₃ to “1” for the third layer L ₃ , and sets the other pixel values to “1”. The mask data M ₃ shown in FIG. 4C is generated as 0″.
In the distance image D, the area dividing unit 11 sets the pixel value of the pixels in the depth range equal to or larger than the threshold value T ₃ and equal to or smaller than the maximum depth value to “1” for the fourth layer L ₄ , and sets the other pixel values to “1”. Mask data (not shown) is generated as 0″.
As a result, the area dividing unit 11 can divide the area of the distance image for each depth range specified by the threshold value.

The area division unit 11 outputs the generated area information (mask data) to the depth-based difference image generation unit 15. Here, the area dividing unit 11 outputs the area information of each depth range to the depth difference image generating units 15A, 15B, 15C, and 15D in the order of increasing distance from the viewpoint position.

The base layer encoding means 12 generates a base layer coded stream (base layer coded stream) by converting a distance image into frequency components, quantizing them, and variable-length coding the quantized coefficients. ..
Here, the base layer indicates a layer in which the entire range image is an encoding target. Further, the enhancement layer described below indicates a layer (depth layer) in which the distance image is to be encoded for each depth range.
The base layer coding unit 12 performs quantization of the range image by roughening the gradation level in a quantization step (first quantization step) larger than that of the enhancement layer coding unit 16.
The base layer coding unit 12 includes an orthogonal transforming unit 120, a quantizing unit 121, and a variable length coding unit 122.

The orthogonal transformation unit 120 orthogonally transforms the distance image for each block (for example, 8×8 pixels) having a predetermined size, and transforms it into frequency components. This orthogonal transform is, for example, a discrete cosine transform (DCT). The orthogonal transforming unit 120 outputs the transform coefficient, which is the calculated frequency component, to the quantizing unit 121.

The quantizing means 121 quantizes the transform coefficient which is the frequency component calculated by the orthogonal transforming means 31 in a preset quantizing step (first quantizing step). This quantization step is a step size (quantization width) when the transform coefficient is discretized, and has a larger value than the quantization step used by the enhancement layer encoding means 16. The quantization step may be a preset value or a value externally designated by a quantization parameter.
The quantizing means 121 generates a quantized coefficient by dividing the transform coefficient by the size of the quantization step and rounding it to an integer value.
The quantizing means 121 outputs the quantized transform coefficient (quantized coefficient) to the variable length coding means 122 and the local decoding means 13.

The variable length coding unit 122 performs variable length coding on the quantized coefficient generated by the quantization unit 121 to generate stream data (base layer coded stream). The variable length coding in the variable length coding means 122 may be a general method such as Huffman coding or arithmetic coding.
The variable length coding means 122 outputs the generated base layer coded stream to the stream combining means 17.

The local decoding means 13 inversely quantizes the quantized coefficient generated by the base layer encoding means 12, and inversely transforms the frequency component to generate a decoded image obtained by decoding the distance image. This decoded image is an image that is locally reproduced on the encoding side (distance image encoding device), on the decoding side (moving image decoding device), as an image that is decoded as a distance image of the base layer.
The local decoding means 13 includes an inverse quantization means 130 and an inverse orthogonal transformation means 131.

The inverse quantization means 130 performs inverse quantization, which is the inverse processing of the processing performed by the quantization means 121, on the quantized coefficient quantized by the quantization means 121 of the base layer encoding means 12. is there. That is, the inverse quantization unit 130 multiplies the quantized coefficient by the size of the quantization step to generate a transform coefficient that is a frequency component.
The inverse quantization unit 130 outputs the inversely quantized transform coefficient to the inverse orthogonal transform unit 131.

The inverse orthogonal transform unit 131 performs an inverse orthogonal transform (for example, an inverse orthogonal transform) that is a process reverse to the process performed by the orthogonal transform unit 120 of the base layer coding unit 12 on the transform coefficient dequantized by the inverse quantization unit 130. Inverse Discrete Cosine Transform) is performed. An image (decoded image) that is decoded as a distance image of the base layer on the decoding side (moving image decoding device) can be reproduced by the image for each block converted by the inverse orthogonal transform unit 131.
The inverse orthogonal transformation unit 131 outputs the generated decoded image to the decoded difference image generation unit 14.

The decoded difference image generation means 14 generates a difference between the original distance image to be encoded and the decoded image decoded by the local decoding means 13 as a decoded difference image.
The decoded difference image generation unit 14 subtracts the decoded image from the distance image to generate a difference between the original distance image to be encoded and the distance image encoded in the base layer.
The decoded difference image generation means 14 outputs the generated decoded difference image to the depth-specific difference image generation means 15 (15A, 15B, 15C, 15D).

The depth-by-depth difference image generation means 15 generates an image for each area divided by the area division means 11 as a depth-by-depth difference image from the decoded difference image generated by the decoded difference image generation means 14.
Here, the depth-based difference image generation means 15 is configured by a plurality of depth-based difference image generation means 15A, 15B, 15C, and 15D corresponding to the number of enhancement layers (depth layers) for each depth range.
The depth-difference image generation unit 15A is configured to generate, as the depth-difference image, the region of the depth range having the shortest distance from the viewpoint position divided by the region division unit 11 in the decoded difference image.
The depth-difference image generation unit 15B is configured to generate, as the depth-difference image, a region in the depth range having the second closest distance from the viewpoint position divided by the region division unit 11 in the decoded difference image. ..
Similarly, the depth-by-depth difference image generation units 15C and 15D respectively determine the depth range regions whose distances from the viewpoint position divided by the region division unit 11 are the third and fourth distances from the decoded difference images. It is generated as another difference image.

It should be noted that the depth-specific difference image generation means 15A to 15D perform the same process in that the depth-specific difference images of the region of the predetermined depth range are generated from the decoded difference image.
Specifically, the depth-by-depth difference image generation units 15A to 15D each multiply the decoded difference image generated by the decoded difference image generation unit 14 by the mask data which is the region information output from the region classification unit 11. Thus, a depth-specific difference image for each depth range is generated.
The depth-specific difference image generation unit 15 outputs the generated depth-specific difference image for each depth range to the enhancement layer encoding unit 16.

The enhancement layer encoding unit 16 converts the depth-specific difference image generated by the depth-specific difference image generating unit 15 into a frequency component, and performs the quantization step (second quantization step) smaller than that of the base layer encoding unit 12. Quantization and variable length coding of the quantized coefficient generate an encoded stream of the enhancement layer (enhancement layer encoded stream).
Here, the enhancement layer coding means 16 is configured by a plurality of enhancement layer coding means 16A, 16B, 16C, 16D according to the number of enhancement layers for each depth range.

The enhancement layer encoding unit 16A encodes the depth-specific difference image in the depth range having the shortest distance from the viewpoint position generated by the depth-specific difference image generating unit 15 (15A).
The enhancement layer encoding unit 16B encodes the depth-specific difference image in the depth range having the second closest distance from the viewpoint position generated by the depth-specific difference image generation unit 15 (15B).
Similarly, the enhancement layer encoding units 16C and 16D respectively depth-difference images in the depth range whose distances from the viewpoint position generated by the depth-difference image generation unit 15 (15C and 15D) are the third and fourth closest, respectively. Is to be encoded.

Note that the enhancement layer coding units 16A to 16D perform the same processing in that the difference images for each depth are coded.
Since the enhancement layer coding means 16A to 16D have the same configuration, the configuration of the enhancement layer coding means 16A will be described here as an example.
The enhancement layer coding unit 16A includes an orthogonal transformation unit 160, a quantization unit 161, and a variable length coding unit 162.

The orthogonal transformation means 160 orthogonally transforms the depth difference image for each block of a predetermined size, and transforms it into frequency components. This orthogonal transform is, for example, a discrete cosine transform (DCT). The orthogonal transform means 160 outputs the transform coefficient, which is the calculated frequency component, to the quantizing means 161.

The quantizing unit 161 quantizes the transform coefficient, which is the frequency component calculated by the orthogonal transforming unit 160, in a preset quantizing step (second quantizing step). The quantizing means 161 finely quantizes the gradation level with a quantizing step having a smaller value than the quantizing step used in the base layer encoding means 12.
The quantizing means 161 outputs the quantized transform coefficient (quantized coefficient) to the variable length coding means 162.

The variable length coding unit 162 performs variable length coding on the quantized coefficient generated by the quantization unit 161 to generate stream data (enhancement layer coded stream).
The variable length coding unit 162 outputs the generated enhancement layer coded stream to the stream combining unit 17.
In this way, the enhancement layer encoding units 16A to 16D encode the depth-specific difference images for each depth range, and output the encoded enhancement layer encoded stream to the stream combining unit 17.

The stream combining means 17 combines the base layer coded stream generated by the base layer coding means 12 and the enhancement layer coded stream for each depth range generated by the enhancement layer coding means 16.
The stream combining unit 17 connects the base layer coded stream and the enhancement layer coded stream for each depth range. At this time, the stream combining unit 17 connects the enhancement layer coded stream next to the base layer coded stream. In addition, the stream combining unit 17 connects the plurality of enhancement layer coded streams earlier in the depth range closer to the viewpoint position. Here, the stream combining unit 17 connects the generated enhancement layer coded stream to the base layer coded stream in the order of the enhancement layer coding units 16A, 16B, 16C and 16D. As a result, the decoding side can decode the enhancement layer coded stream with the highest priority next to the base layer coded stream.

Furthermore, the stream combining unit 17 configures the layers (for example, the number of extension layers [depth layers]), the size of the distance image (the number of horizontal pixels and the number of vertical pixels), the data length of the base layer encoded stream, and each extension. Stream configuration information such as the data length of the layer encoded stream is generated as header information.
Then, the stream combining unit 17 adds header information to the stream obtained by concatenating the base layer coded stream and the enhancement layer coded stream for each depth range to generate a coded stream.
When the input range image is composed of continuous frames corresponding to the moving image, the stream combining unit 17 continuously outputs the encoded streams by the number of frames.

By configuring the distance image encoding device 1 as described above, the distance image encoding device 1 allows the distance image encoding device 1 to roughly quantize and encode the entire distance image and encode the distance image for each depth range. The distance image can be hierarchically encoded by the finely quantized encoded stream.

As a result, a range image decoding device that decodes an encoded stream decodes a range image hierarchically, such as decoding only the enhancement layer that is the closest to the base layer in distance from the viewpoint position, according to performance such as CPU power. can do.
Further, since the coded stream is hierarchically coded, it is possible to prevent transmission of a layer having a low priority according to the band on the transmission path. Even in this case, since the layer in the depth range close to the viewpoint position is the transmission target, the depth can be accurately reproduced in the decoding device.
It should be noted that the distance image encoding device 1 can operate a computer with a program (distance image encoding program) for causing each of the above-described means to function.

[Operation of range image encoding device]
Next, with reference to FIG. 5 (for the configuration, refer to FIG. 1 as needed), the operation of the distance image encoding device 1 according to the embodiment of the present invention will be described. Note that, here, the distance image is assumed to be composed of continuous frames corresponding to the moving image.
In step S1, the threshold setting means 10 sets a threshold for hierarchically dividing the depth of the distance image. With this threshold, the distance image can be hierarchized in the range from the minimum depth value to the maximum depth value for each depth range with the threshold as a boundary.
In step S2, the area dividing unit 11 divides the area of the distance image for each depth range specified by the threshold value set in step S1. Here, the area dividing unit 11 generates area information for each depth range as mask data.

In step S3, the base layer encoding means 12 encodes the entire distance image as a base layer. In this step S3, the base layer coding means 12 performs orthogonal transform (DCT transform) on the range image for each block of a predetermined size by the orthogonal transform means 120 to generate transform coefficients which are frequency components. Then, the base layer coding means 12 quantizes the transform coefficient by the quantizing means 121 in a quantization step larger than that of the enhancement layer, and generates a quantized coefficient. Then, the base layer coding means 12 performs variable length coding on the quantized coefficient by the variable length coding means 122 to generate stream data (base layer coded stream).

In step S4, the local decoding means 13 generates a decoded image obtained by decoding the distance image encoded by the base layer encoding means 12 from the quantized coefficient generated by the quantization in step S3. In step S4, the local decoding unit 13 dequantizes the quantized coefficient generated by the quantizing unit 121 by the dequantizing unit 130 to generate a transform coefficient that is a frequency component. Then, the local decoding unit 13 performs inverse orthogonal transform on the transform coefficient by the inverse orthogonal transform unit 131 to generate a decoded image.
In step S5, the decoded difference image generation unit 14 generates a difference between the original distance image to be encoded and the decoded image generated in step S4 as a decoded difference image.

In step S6, the depth-by-depth difference image generation unit 15 generates an image of a region for each depth range divided in step S2 as a depth-by-depth difference image from the decoded difference image generated in step S5. Here, the depth-specific difference image generation unit 15 multiplies the decoded difference image generated in step S5 by the mask data for each depth range generated in step S2 to generate the depth-specific difference image for each depth range. To generate.

In step S7, the enhancement layer encoding unit 16 encodes the depth-specific difference image generated in step S6 for each depth range as an enhancement layer. In step S7, the enhancement layer encoding unit 16 encodes the depth difference image by the same encoding process as in step S3 to generate stream data (enhancement layer encoded stream). The quantization in the encoding process of the enhancement layer encoding means 16 is performed with a smaller value of the quantization step than the quantization in the base layer encoding means 12 in step S3.

In step S8, the stream combining unit 17 combines the base layer coded stream generated in step S3 and the enhancement layer coded stream for each depth range generated in step S7 to generate a coded stream.
Here, when the next frame is further input (Yes in step S9), the distance image encoding device 1 returns to step S2 and continues the operation.
On the other hand, when the input is completed (No in step S9), the distance image encoding device 1 ends the operation. If the distance image is an image corresponding to the captured image of the still image, the operation of step S9 can be omitted.

[Configuration of range image decoding device]
Next, the configuration of the distance image decoding device 2 according to the embodiment of the present invention will be described with reference to FIG.

The distance image decoding device 2 decodes the encoded stream of the distance image encoded by the distance image encoding device 1 (FIG. 1).
As shown in FIG. 6, the distance image decoding device 2 includes a stream separating unit 20, a base layer decoding unit 21, an enhancement layer decoding unit 22, and an image synthesizing unit 23.

The stream separating means 20 separates the encoded stream into a base layer encoded stream and a plurality of enhancement layer encoded streams.
The stream separating means 20 refers to the header information of the coded stream and separates and extracts the base layer coded stream and the plurality of enhancement layer coded streams from the coded stream.
The stream separating means 20 outputs the separated base layer encoded stream to the base layer decoding means 21. Further, the stream separating means 20 outputs the separated enhancement layer encoded stream to the enhancement layer decoding means 22. Here, the stream separation unit 20 receives the plurality of enhancement layer encoded streams in the order of input, that is, in the order of high priority (closest distance from the viewpoint position), the enhancement layer decoding units 22A, 22B, 22C, 22D. Output to.

The base layer decoding means 21 decodes the base layer encoded stream separated by the stream separating means 20.
The base layer decoding unit 21 includes a variable length decoding unit 210, an inverse quantization unit 211, and an inverse orthogonal transformation unit 212.

The variable length decoding unit 210 performs variable length decoding on the base layer coded stream, which is the inverse transform of the variable length coding unit 122 (FIG. 1). By this variable length decoding, the quantized coefficient is decoded from the base layer encoded stream.
The variable length decoding means 210 outputs the decoded quantized coefficient to the inverse quantization means 211.

The inverse quantizer 211 performs inverse quantization, which is the reverse of the process performed by the quantizer 121 (FIG. 1), on the quantized coefficient decoded by the variable length decoder 210. That is, the inverse quantization unit 130 multiplies the quantized coefficient by the size of the same quantization step as the quantization unit 121 to generate a transform coefficient that is a frequency component.
The inverse quantization unit 211 outputs the inversely quantized transform coefficient to the inverse orthogonal transform unit 212.

The inverse orthogonal transform unit 212 performs an inverse orthogonal transform (for example, an inverse discrete cosine) that is the reverse process of the process performed by the orthogonal transform unit 120 (FIG. 1) on the transform coefficient inversely quantized by the inverse quantization unit 211. Conversion). An image obtained by decoding the base layer encoded stream (base layer decoded image) is generated by the image for each block converted by the inverse orthogonal transform unit 212. This base layer decoded image is an image that becomes a range image even by itself, but since it is an image that is roughly quantized and encoded/decoded, the depth accuracy is low.
The inverse orthogonal transform unit 212 outputs the generated base layer decoded image to the image synthesizing unit 23.
After generating the base layer decoded image, the inverse orthogonal transformation unit 212 instructs the enhancement layer decoding unit 22 (22A) to start decoding the enhancement layer encoded stream.

The enhancement layer decoding means 22 decodes the enhancement layer coded stream separated by the stream separating means 20.
Here, the enhancement layer decoding means 22 is composed of a plurality of enhancement layer decoding means 22A, 22B, 22C, 22D according to the number of enhancement layers for each depth range. Here, the enhancement layer decoding means 22 is composed of the same number of means as the enhancement layer coding means 16 of the distance image coding device 1 (FIG. 1), but the number of enhancement layer decoding means 22 is It may be "1" or more and less than the number of enhancement layers. However, in order to decode the distance image with high accuracy, it is preferable that the enhancement layer decoding means 22 is configured by the same number of means as the enhancement layer encoding means 16 of the distance image encoding device 1 (FIG. 1).

The enhancement layer decoding means 22A is for generating an enhancement layer decoded image (depth difference image) obtained by decoding the enhancement layer encoded stream having the highest priority separated by the stream separation means 20.
The enhancement layer decoding means 22B is for generating the enhancement layer decoded image (depth difference image) obtained by decoding the enhancement layer coded stream having the second highest priority separated by the stream separation means 20.
Similarly, the enhancement layer decoding means 22C and 22D respectively decode the enhancement layer decoded image (depth difference image) obtained by decoding the enhancement layer encoded streams separated by the stream separating means 20 and having the third and fourth highest priorities. Is generated.

Note that the enhancement layer decoding means 22A to 22D perform the same process in that the depth-specific difference image is decoded from the enhancement layer encoded stream.
Since the enhancement layer decoding means 22A to 22D have the same configuration, the configuration of the enhancement layer decoding means 22A will be described here as an example.
The enhancement layer decoding unit 22A includes a variable length decoding unit 220, an inverse quantization unit 221, and an inverse orthogonal transformation unit 222.

The variable length decoding unit 220, the dequantization unit 221, and the inverse orthogonal transformation unit 222 respectively include the variable length decoding unit 210, the dequantization unit 211, and the anti-orthogonal unit except that the decoding target is the enhancement layer coded stream. Since the same processing as that of the conversion means 212 is performed, detailed description will be omitted. However, the quantization step used by the inverse quantization means 221 has a smaller value than the quantization step used by the inverse quantization means 211, and is the same as the quantization step used by the quantization means 161 (FIG. 1). ..
The inverse quantization unit 221 outputs the decoded enhancement layer decoded image (depth-specific difference image) to the image synthesis unit 23.
The enhancement layer decoding means 22A starts decoding the enhancement layer coded stream at the stage when the base layer decoding means 21 (inverse orthogonal transformation means 212) instructs the start of decoding.

Further, the enhancement layer decoding means 22A decodes the enhancement layer coded stream having the first priority, and then starts decoding the enhancement layer coded stream having the second priority to the enhancement layer decoding means 22B having the low priority. Instruct.
In this way, the enhancement-layer decoding units 22A to 22D sequentially decode the enhancement-layer encoded streams with higher priorities, and output the decoded depth-specific difference images to the image synthesizing unit 23.

The image synthesizing unit 23 synthesizes the base layer decoded image decoded by the base layer decoding unit 21 and the plurality of enhancement layer decoded images (depth difference images) decoded by the enhancement layer decoding unit 22.
The image synthesizing unit 23 adds a plurality of enhancement layer decoded images to the base layer decoded image to generate a depth image with increased depth accuracy.

When decoding the range image corresponding to the captured image of the moving image, the image composition unit 23 needs to decode the range image within the cycle of one frame of the captured image. Therefore, the distance image decoding device 2 measures the time from the time point when the encoded stream for one frame is input by a timing unit such as a timer (not shown). Then, when the remaining time of the frame period is less than the decoding time of the enhancement frame for one layer, the image synthesizing means 23 determines the depth image difference between the depth image of the base layer and the enhancement layer decoded up to that point. The image is combined and output as a distance image for the encoded stream. It should be noted that the decoding time of the extension frame for one layer may be, for example, a fixed fixed time or the decoding time of the basic frame.

By configuring the distance image decoding device 2 as described above, the distance image decoding device 2 decodes the encoded stream in which the distance image encoding device 1 hierarchically encodes the distance image for each depth range. be able to.
Further, the distance image decoding device 2 can hierarchically decode the distance image, such as preferentially decoding only the enhancement layer having the closest distance from the base layer and the viewpoint position according to the performance such as CPU power. it can.

[Operation of range image decoding device]
Next, the operation of the distance image decoding device 2 according to the embodiment of the present invention will be described with reference to FIG. 7 (for the configuration, refer to FIG. 6 as appropriate). Note that, here, the distance image is assumed to be composed of continuous frames corresponding to the moving image.
In step S10, the stream separating means 20 separates the coded stream into a base layer coded stream and a plurality of enhancement layer coded streams for each frame. Note that, here, the distance image decoding device 2 measures the time from the time point when the encoded stream for one frame is input by the timing means (not shown).

In step S11, the base layer decoding means 21 decodes the base layer coded stream separated in step S10. In step S11, the base layer decoding means 21 generates a quantized coefficient by performing variable length decoding on the base layer encoded stream by the variable length decoding means 210. Then, the base layer decoding means 21 performs inverse quantization by the inverse quantization means 211 to generate transform coefficients which are frequency components. Then, the base layer decoding unit 21 performs an inverse orthogonal transform (for example, an inverse discrete cosine transform) on the transform coefficient by the inverse orthogonal transform unit 212 to obtain an image that is a decoding result of the base layer encoded stream (base layer decoded image). ) Is generated.

In step S12, the image synthesizing unit 23 determines whether the remaining time of the frame cycle is equal to or longer than the decoding time of the extended frame for one layer.
Here, when the remaining time of the frame period is less than the decoding time of the extension frame (No in step S12), the distance image decoding device 2 advances the operation to step S18.
On the other hand, when the remaining time of the frame cycle is equal to or longer than the decoding time of the enhancement frame (Yes in step S12), the enhancement layer decoding unit 22 initializes a variable i for identifying the enhancement layer (here, in step S13). Set to "1").

In step S14, the enhancement layer decoding means 22 decodes the i-th priority enhancement layer codestream separated in step S10. In this step S14, the enhancement layer decoding means 22 decodes the enhancement layer code stream by the same encoding process as in step S11 to generate an enhancement layer decoded image.
Here, when the remaining time of the frame period is less than the decoding time of the extension frame (No in step S15), the distance image decoding device 2 advances the operation to step S18.
On the other hand, when the remaining time of the frame period is equal to or longer than the decoding time of the enhancement frame (Yes in step S15), the enhancement layer decoding means 22 adds "1" to the variable i in step S16.

Here, if the variable i has not reached the number of enhancement layers (here, “4”) (No in step S17), the enhancement layer decoding unit 22 returns to step S14 and returns to the enhancement layer of the next layer. Decode the code stream.
On the other hand, when the variable i has reached the number of enhancement layers (Yes in step S17), the decoding process in the enhancement layer decoding means 22 ends, and the distance image decoding device 2 advances the operation to step S18.

In step S18, the image synthesizing unit 23 synthesizes the base layer decoded image decoded in step S11 and the enhancement layer decoded image decoded in the frame period in step S14 to generate a distance image.
Here, when the encoded stream of the next frame is further input (Yes in step S19), the distance image decoding device 2 returns to step S10 and continues the operation.
On the other hand, when the input is completed (No in step S19), the distance image decoding device 2 ends the operation.
If the distance image to be decoded is an image corresponding to the captured image of the still image, the operation of step S9 can be omitted. Also, the operations of steps S12 and S15 can be omitted.

[Modification]
Although the configurations and operations of the range image encoding device 1 and the range image decoding device 2 according to the embodiment of the present invention have been described above, the present invention is not limited to this embodiment.

(Modification 1)
Here, the threshold value setting means 10 of the distance image encoding device 1 sets the threshold value closest to the viewpoint position, and sets the threshold value so as to equally divide the depth value farther than that.
However, as shown in FIG. 8, the threshold value may be set such that the depth range becomes wider as the depth becomes deeper. In this case, the threshold value T ₁ closest to the viewpoint position is at least smaller than the value obtained by dividing the maximum depth value by the number of depth layers.
Specifically, the threshold setting means 10 sets the threshold T ₁ and then sets the threshold T _i (i is an integer of 2 or more and less than the number of depth layers) by the following equation (2). Here, d ₁ indicates the distance of the first depth range divided from the viewpoint position. That is, d ₁ =T ₁ . Further, r represents an integer larger than “1”.

When the depth maximum value is D _max , the depth range distance d _j has the relationship of the following expression (3). Note that n indicates the number of depth layers, and j indicates a number (1 or more and n or less) indicating the order of the depth range from the viewpoint position. Also, a=d ₁ =T ₁ .

Therefore, r of the equation (2) can be obtained as a solution of the equation of the following equation (4).

(Modification 2)
Here, the threshold value setting unit 10 of the distance image encoding device 1 sets the threshold value by being specified from the distance image or externally.
However, these thresholds may be set in advance in the internal memory of the distance image encoding device 1 or the like. In this case, the distance image encoding device 1 may omit the threshold setting means 10 from the configuration. Further, in this case, the area dividing unit 11 may divide the area of the distance image by a preset threshold value.

(Modification 3)
Here, the depth-difference image generation means 15 of the distance image encoding device 1 is configured to be operated in parallel by a plurality of depth-difference image generation means 15A, 15B, 15C, 15D according to the number of enhancement layers. Further, the enhancement layer coding means 16 is also configured to be operated in parallel by the plurality of enhancement layer coding means 16A, 16B, 16C, 16D according to the number of enhancement layers.
However, the depth-based difference image generation unit 15 and the enhancement layer encoding unit 16 do not necessarily have to be configured to operate in parallel.
That is, the depth-by-depth difference image generating unit 15 and the enhancement layer encoding unit 16 may each have a single configuration and operate sequentially for each enhancement layer.

(Modification 4)
Here, the enhancement layer decoding means 22 of the distance image decoding device 2 is configured to operate in order from the enhancement layer having the highest priority among the plurality of enhancement layer decoding means 22A, 22B, 22C, 22D according to the number of enhancement layers. ..
However, the distance image decoding device 2 may be configured to operate the enhancement layer decoding means 22A, 22B, 22C, 22D including the base layer decoding means 21 in parallel as long as there is a margin in performance such as CPU power.

(Modification 5)
Here, in the distance image encoding device 1, in the depth difference image generating means 15 and the enhancement layer encoding means 16, the depth range closer to the viewpoint position is assigned to the enhancement layer having higher priority and encoded. .. Then, in the distance image coding device 1, the stream combining means 17 connects the enhancement layer coded stream next to the base layer coded stream according to its priority.
However, the priority of the enhancement layer does not necessarily need to be set because the distance from the viewpoint position is short. For example, when displaying an image on a ray reproduction type three-dimensional display, it is preferable that the depth range displayed near the screen surface be assigned to an enhancement layer having a higher priority. Further, for example, when displaying an image in the integral stereoscopic method, it is preferable that the depth range displayed near the lens array surface is assigned to the enhancement layer having higher priority.
Thereby, it is possible to preferentially encode/decode the depth range desired to be displayed with high image quality.

DESCRIPTION OF SYMBOLS 1 Distance image coding apparatus 10 Threshold setting means 11 Area dividing means 12 Base layer coding means 120 Orthogonal transformation means 121 Quantization means 122 Variable length coding means 13 Local decoding means 130 Inverse quantization means 131 Inverse orthogonal transformation means 14 Decoding Difference image generating means 15, 15A, 15B, 15C, 15D Depth-by-depth difference image generating means 16, 16A, 16B, 16C, 16D Enhancement layer coding means 160 Orthogonal transformation means 161 Quantization means 162 Variable length coding means 17 Stream combination Means 2 Distance image decoding means 20 Stream separation means 21 Base layer decoding means 210 Variable length decoding means 211 Inverse quantization means 212 Inverse orthogonal transformation means 22, 22A, 22B, 22C, 22D Enhancement layer decoding means 23 Image combining means

Claims (8)

A distance image encoding device that encodes a distance image indicating depth information of a subject,
Base layer coding means for generating a base layer coded stream by converting the distance image into frequency components, quantized in the first quantization step, and variable length coding the quantized coefficients;
Dequantizing the quantized coefficient, local decoding means for generating a decoded image obtained by decoding the distance image by inversely transforming the frequency component,
A decoded difference image generation means for generating a difference between the distance image and the decoded image as a decoded difference image,
Area dividing means for dividing the area of the distance image for each preset depth range;
From the decoded difference image, depth-by-depth difference image generation means for generating an image for each area divided by the area division means as a depth-specific difference image,
For each depth range, the corresponding depth difference image is converted into frequency components, quantized in a second quantization step smaller than the first quantization step, and quantized coefficients are variable-length coded for expansion. Enhancement layer coding means for generating a layer coded stream,
Stream combining means for combining the base layer encoded stream and the enhancement layer encoded stream for each depth range to generate an encoded stream of the distance image,
A range image encoding device comprising: The distance image coding apparatus according to claim 1, further comprising a threshold setting unit that sets a threshold for dividing the depth range on a predetermined basis. The area dividing unit generates mask data for dividing the area of the distance image for each depth range,
The distance image encoding according to claim 1 or 2, wherein the depth-specific difference image generation means generates the depth-specific difference image by multiplying the decoded difference image by the mask data. apparatus. The enhancement layer coding means generates the enhancement layer coded stream from the depth difference image as a layer having a higher priority in a depth range closer to a predetermined depth,
4. The stream combining means connects the enhancement layer coded stream for each depth range to the base layer coded stream in the descending order of priority, according to any one of claims 1 to 3. The range image encoding device according to the item. A distance image coding program for causing a computer to function as the distance image coding device according to any one of claims 1 to 4. In the first quantization step, the difference between the base layer encoded stream obtained by encoding the distance image indicating the depth information of the subject and the distance image obtained by decoding the base layer encoded stream and the original distance image is obtained by the first quantization. A distance image decoding device for decoding a coded stream in which a plurality of enhancement layer coded streams for each depth range coded in a second quantization step smaller than the step is concatenated,
Stream separation means for separating the encoded stream into the base layer encoded stream and the plurality of enhancement layer encoded streams;
Base layer decoding means for decoding the base layer coded stream using the first quantization step to generate a base layer decoded image;
Enhancement layer decoding means for decoding the plurality of enhancement layer coded streams using the second quantization step to generate a plurality of enhancement layer decoded images;
An image synthesizing unit that synthesizes the base layer decoded image and the plurality of enhancement layer decoded images to generate a distance image by decoding the encoded stream,
A range image decoding device comprising: The encoded stream is a stream obtained by encoding a distance image corresponding to a captured image of a moving image,
The enhancement layer decoding means decodes the enhancement layer coded streams that are linked to the coded stream in a linking order,
7. The distance image decoding device according to claim 6, wherein the image synthesizing unit synthesizes the base layer decoded image and the enhancement layer decoded image decoded within a frame period. A distance image decoding program for causing a computer to function as the distance image decoding device according to claim 6.