JP2013131928A - Image encoding device and image encoding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image encoding device and an image encoding method that are capable of improving encoding efficiency of a display image using depth information.SOLUTION: An image encoding device comprises acquisition means, signal processing means, and output means. The acquisition means acquires depth information in each image area composed of an input image signal. The signal processing means performs processing for adjusting the data amount of an image signal in the image area depending on a depth value represented by the depth information acquired by the acquisition means. The output means outputs an image signal after the processing by the signal processing means.

Description

  Embodiments described herein relate generally to an image encoding device and an image encoding method.

  In the moving image encoding process, various adjustments of the quantization parameter are performed for the purpose of improving encoding efficiency and image quality. In recent years, since display devices that display stereoscopic video using depth information have been used, it is desired to improve the encoding efficiency of display image encoding processing using depth information. .

  As a conventional technique related to the adjustment of the quantization parameter, a technique for performing quantization so as to improve the encoding efficiency for each hierarchical data of the three-dimensional volume data is disclosed.

JP-A-6-113334

  However, the coding of the display image using the depth information has room for improvement in terms of efficiency, and a technique that can further improve the coding efficiency is desired.

  The present invention has been made in view of the above, and an object of the present invention is to provide an image encoding device and an image encoding method capable of improving the encoding efficiency of a display image using depth information.

  The image encoding apparatus according to the embodiment includes an acquisition unit, a signal processing unit, and an output unit. The acquisition unit acquires depth information of each image area configured by the input image signal. The signal processing unit performs a process of adjusting the data amount of the image signal in the image area according to the depth value represented by the depth information acquired by the acquisition unit. The output means outputs the image signal processed by the signal processing means.

FIG. 1 is a block diagram illustrating a configuration of an image encoding device according to the first embodiment. FIG. 2 is a diagram for explaining the filter processing. FIG. 3 is a flowchart illustrating the procedure of the filtering process performed by the prefilter unit. FIG. 4 is a block diagram illustrating a configuration of the encoding unit. FIG. 5 is a diagram illustrating an example of the relationship between the depth value and the quantization step width. FIG. 6 is a flowchart illustrating the procedure of the encoding process performed by the encoding unit. FIG. 7 is a block diagram illustrating a configuration of an image encoding device according to the second embodiment. FIG. 8 is a diagram illustrating an example of a relationship among a depth value, an activity, and a cutoff frequency used by the prefilter unit. FIG. 9 is a block diagram illustrating a configuration of the encoding unit. FIG. 10 is a diagram illustrating an example of the relationship among the depth value, the cutoff frequency, and the quantization step width set by the quantization control unit. FIG. 11 is a flowchart illustrating the procedure of the filtering process performed by the prefilter unit in the third embodiment. FIG. 12 is a block diagram illustrating a configuration of a modification of the encoding unit according to the first embodiment. FIG. 13 is a block diagram illustrating a configuration of a modification of the encoding unit according to the second embodiment. FIG. 14 is a block diagram illustrating a configuration of a modification of the encoding unit according to the second embodiment. FIG. 15 is a block diagram illustrating a configuration of a modification of the encoding unit according to the second embodiment.

(First embodiment)

  FIG. 1 is a block diagram illustrating a configuration of an image encoding device 10 according to the first embodiment. As shown in FIG. 1, the image encoding device 10 mainly includes a prefilter unit 11 and an encoding unit 12. An input image signal for a plurality of parallaxes and depth information of a subject image (image region) included in the input image signal are input to the image encoding device 10 in association with each other as a multi-parallax image. Although a case where a multi-parallax image is input as an input image signal will be described here, the input image signal is not limited to a multi-parallax image, and a two-dimensional image and depth information are input in association with each other. Also good.

  As the depth information, a depth value corresponding to the distance in the depth direction from the virtual point to the image area can be used. The depth information may be associated with the image area of the subject image or may be associated with each pixel. The pre-filter unit 11 and the encoding unit 12 each function as an acquisition unit that acquires depth information of each image area configured by an input image signal.

  Note that the method for acquiring depth information is not particularly limited. When depth information is generated when generating input image signals for a plurality of viewpoints, the generated depth information may be input to the image encoding device 10 as shown in FIG. Alternatively, as described later in other embodiments, the image encoding device 10 may acquire depth information by generating depth information from a two-dimensional image.

  The prefilter unit 11 performs a filter process for adjusting the definition of the input image signal in accordance with the depth value, and outputs the image signal after the filter process to the encoding unit 12. Further, the pre-filter unit 11 performs a filter process that further reduces the image definition as the image area has a larger depth value, that is, as the image area is displayed on the far side. As described above, the prefilter unit 11 adjusts the image definition according to the depth value, thereby adjusting the data amount of the image signal for each image region according to the depth value of the image region.

  The encoding unit 12 performs an encoding process for encoding the image signal after the filter process according to the depth value, and outputs the encoded data after the process. Also, the encoding unit 12 performs encoding so that an image area having a larger depth value has a larger quantization step width when the input image signal is quantized and further reduces the definition of the image in the image area. Process. That is, the encoding processing unit 12 adjusts the encoding amount according to the depth value, thereby adjusting the image definition, that is, the data amount of the image signal for each image region, according to the depth value of the image region.

  Thus, the prefilter unit 11 and the encoding unit 12 function as a signal processing unit that processes the input image signal and an output unit that outputs the image signal after the signal processing. Hereinafter, functions and operations of the prefilter unit 11 and the encoding unit 12 will be described in more detail.

  First, an example of filter processing performed by the prefilter unit 11 will be described with reference to FIG. FIG. 2 is a diagram for explaining the filter processing. In FIG. 2, a case where pixels arranged in the horizontal direction are schematically indicated by circles in an image of one frame and depth information is associated with each pixel will be described. As described above, when the depth information is associated with each image area such as each subject, the following process between pixels may be replaced with the process between the image areas. The type of filter processing performed by the prefilter unit 11 is not particularly limited, but an example of performing image blurring by performing low-pass filter processing will be described below.

尚、式（１）において関数ｇ₁は任意の関数を用いることができる。また、ｒ₁は係数であり、例えば画像符号化装置１０ごとに定められる係数が用いられる。 For the i-th pixel, the pre-filter unit 11 decreases the cut-off frequency f _i of the low-pass filter as the depth value d _i at that pixel increases. The following equation (1) shows an example of the relationship between the cutoff frequency f _i and the depth value d _i .

In equation (1), an arbitrary function can be used as the function g ₁ . R ₁ is a coefficient, and for example, a coefficient determined for each image encoding device 10 is used.

The larger the cut-off frequency f _i of the low-pass filter, the clearer the image, and the smaller the cut-off frequency f _i, the greater the blur of the image. Accordingly, by reducing the cutoff frequency as the depth value increases, it is possible to increase the blurring of the image in the image area displayed on the far side and reduce the data amount of the output signal. Further, by increasing the cutoff frequency as the depth value is smaller, it is possible to sharpen the image in the image area displayed on the near side and improve the definition of the image.

The pre-filter unit 11 receives the input signal values of the neighboring pixels having the number of sampling pixels (2N + 1) according to the cut-off frequency f _i with respect to the input signal value x _i of the i-th pixel (in FIG. 2, x _i−2 , X _i−1 , x _i , x _{i + 1} , x _{i + 2} ), the output signal value y _i for the i th pixel is obtained. Thereby, the output signal of the i-th pixel is blurred by the input signals of a plurality of neighboring pixels.

Here, (in FIG. 2, C _-2 -C ₂₎ input signal values of neighboring pixels _{(x i-2 ~x i +} 2) filter coefficients used for each of the C _n using Formula (1) Depending on the obtained cut-off frequency f _i , for example, an arbitrary function g ₂ is used to express the following equation (2).

Then, the pre-filter unit 11 obtains the output signal value y _i of the i-th pixel by the product-sum operation of the filter coefficient C _n and the sampled input signal value x _in of the pixel as shown in the following equation (3).

In the expression (3), the prefilter unit 11 uses neighboring pixels (x _i−2 , x _i−1 , x _i , x _{i + 1} , x _{i + 2 in} FIG. 2) arranged in the horizontal direction in the frame. The filtering process is performed in the same manner with respect to neighboring pixels arranged in the vertical direction in the frame.

  As described above, the pre-filter unit 11 performs filter processing with a greater blurring effect for an image region having a larger depth value. Therefore, although the image area displayed on the back side is blurred, the image area displayed on the near side is not blurred as far as the back side, and the definition can be maintained. Therefore, the amount of image signals after filtering can be reduced without reducing the subjective image quality, and the encoding efficiency can be improved.

Next, the operation of the prefilter unit 11 will be described. FIG. 3 is a flowchart illustrating the procedure of the filtering process performed by the prefilter unit 11. First, an input image signal of a multi-parallax image and depth information of the multi-parallax image are input to the pre-filter unit 11 in association with each frame (step S1). The pre-filter unit 11 performs a filter process for reducing the definition of the frame image based on the depth value (step S2). That is, as described above, the cut-off frequency f _i is calculated according to the depth value, and the output signal value y _i of the i-th pixel is calculated using 2N + 1 sampling pixels corresponding to the depth value. Then, the prefilter unit 11 outputs the output signal value after the filtering process to the encoding unit 12 for each frame (step S3).

In the above description, the example in which the cutoff frequency f _i and the filter coefficient C _n are calculated based on the depth value for each pixel has been described. However, the prefilter unit 11 is an image region (for example, a macro The cut-off frequency f _i and the filter coefficient C _n of the image area may be calculated based on depth information (for example, an average value of depth values in the image area) in the block unit image area.

  Next, the encoding unit 12 will be described. First, the configuration of the encoding unit 12 will be described in more detail with reference to FIG.

  FIG. 4 is a block diagram illustrating a configuration of the encoding unit 12. As shown in the figure, the encoding unit 12 includes a subtraction unit 101, an orthogonal transform unit 102, a quantization unit 103, an entropy encoding unit 104, a quantization control unit 105, an inverse quantization unit 106, an inverse orthogonal transform unit 107, An adder 108, a frame memory 109, and a predicted image generator 110 are included. Note that the processing up to encoding is performed in units of macroblocks such as 4 × 4 pixels and 8 × 8 pixels.

  The subtraction unit 101 receives the image signal (the moving image signal in units of frames) after the filtering process by the pre-filter unit 11 and the prediction image signal generated by the prediction image generation unit 110 described later. . The subtraction unit 101 generates a prediction error signal by taking the difference between the image signal and the prediction image signal, and outputs the prediction error signal to the orthogonal transformation unit 102.

  The orthogonal transform unit 102 generates orthogonal transform coefficient information by performing a known orthogonal transform process on the prediction error signal generated by the subtraction unit 101. For example, the orthogonal transform unit 102 performs orthogonal transform processing on the prediction error signal generated by the subtraction unit 101 in units of macro blocks, thereby generating orthogonal transform coefficient information that is concentrated on a small number of low frequency coefficients for each macro block. To do. Then, the orthogonal transform unit 102 outputs the generated orthogonal transform coefficient information. In this embodiment, discrete cosine transform (DCT) is used as the conversion method, but the present invention is not limited to this conversion method.

  The quantization unit 103 quantizes the orthogonal transform coefficient information input from the orthogonal transform unit 102 using a quantization step width input from the quantization control unit 105 described later. The quantization method is not particularly limited, and a known method can be used. In addition, the quantization unit 103 outputs the quantized orthogonal transform coefficient information to the entropy coding unit 104 and the inverse quantization unit 106 as quantized orthogonal transform coefficient information.

  The quantization control unit 105 calculates a quantization step width based on the depth value represented by the depth information for each macroblock, and outputs the calculated quantization step width to the quantization unit 103. Specifically, the quantization control unit 105 calculates the average value of the macroblock depth values, increases the quantization step width for macroblocks with a large average depth value, and sets the average value of the depth values. For small macroblocks, the quantization step width is reduced.

尚、式（４）において関数ｈ₁は任意の関数を用いることができる。また、ｒ₄は係数であり、例えば画像符号化装置１０ごとに定められる係数が用いられる。 Shows an example of the relationship between the average value of the depth value (depth value) d _j and the quantization step width s _j used in the macro block in the j-th macroblock in the following equation (4).

In Equation (4), an arbitrary function can be used as the function h ₁ . R ₄ is a coefficient, for example, a coefficient determined for each image encoding device 10 is used.

FIG. 5 is a diagram showing an example of the relationship between the depth value (average value of depth values) _dj and the quantization step width s _j, and is an example where r ₄ is a constant in the above equation (4). is there. As shown in FIG. 5, the quantization step width s _j may be set so as to change linearly with respect to the depth value d _j . Alternatively, the quantization step width s _j may be set to change nonlinearly (eg, exponentially) with respect to the depth value d _j .

  As described above, the quantization control unit 105 sets the quantization step width to be rougher for an image region having a larger depth value and displayed on the far side, and a quantization step for an image region having a smaller depth value and displayed on the near side. Set the width finely. As a result, while maintaining the image quality of the image area while maintaining the near-side code amount that is likely to be noticed, the coding amount can be reduced for the back-side image region that is difficult to attract attention. Therefore, it is possible to improve the encoding efficiency without reducing the subjective image quality.

Returning to FIG. 4, the inverse quantization unit 106 performs inverse quantization by multiplying the quantized orthogonal transform coefficient information output from the quantization unit 103 by the quantization step width s _j for each macroblock.

  The inverse orthogonal transform unit 107 further performs inverse orthogonal transform on the quantized orthogonal transform coefficient information inversely quantized by the inverse quantization unit 106, and outputs the inversely orthogonally transformed information to the addition unit 108.

  The adding unit 108 adds the predicted image signal to the information input from the inverse orthogonal transform unit 107 to generate a local decoded signal, and outputs the local decoded signal to the frame memory 109.

  The frame memory 109 stores the locally decoded image signal input from the input adding unit 108.

  The predicted image generation unit 110 performs matching (for example, block matching) between the input image signal input and the local decoded image signal stored in the frame memory 109 for each macroblock in the frame, and the difference is the smallest. And a predicted image signal is generated using a locally decoded image signal compensated by the motion vector. Then, the predicted image generation unit 110 outputs the motion vector information and the prediction mode information selected at the time of compensation together with the generated predicted image signal.

  The entropy encoding unit 104 compresses the quantized orthogonal transform coefficient information by the quantization unit 103 and the motion vector information and the prediction mode information input by the prediction image generation unit 110 by using the bias of the appearance probability of the input image signal. (Entropy encoding is performed). The compression method is not particularly limited, and for example, a variable length code method can be used. Then, the entropy encoding unit 104 outputs the encoded data generated by compression to a transmission system or a storage system (both not shown).

  Next, the operation of the encoding unit 12 will be described. FIG. 6 is a flowchart for explaining the procedure of the encoding process performed by the encoding unit 12.

  First, the image signal for each frame subjected to the filtering process by the prefilter unit 11 and the depth information corresponding to the frame are input to the encoding unit 12 (step S11). Then, the quantization control unit 105 of the encoding unit 12 calculates a quantization step width for each macroblock based on the depth information (step S12), and outputs the calculated quantization step width to the quantization unit 103. The quantization unit 103 quantizes the orthogonal transform coefficient information for each macroblock input from the orthogonal transform unit 102 using the input quantization step width (step S13). Then, the encoding unit 12 outputs the quantized and encoded data (encoded data) to an image display device such as a television receiver or a PC (step S14).

  As described above, according to the image encoding device 10 of the first embodiment, the image area with a larger depth value increases the amount of reduction in the definition of the image. While preferentially reducing, the image quality of the image area displayed on the near side is maintained. Therefore, the amount of image signal can be reduced without reducing the subjective image quality, and the coding efficiency of the display image using the depth information can be improved.

  Note that the encoding unit 12 converts the code amount that can be reduced by increasing the quantization step width to an image region with a small depth value, as described above, for an image region with a large depth value (for example, a macroblock). May be allocated. Further, the encoding unit 12 may use the code amount that can be reduced in one frame image in the subsequent frame images. Then, the quantization step width may be further reduced for an image region having a small depth value using the allocated code amount, and the image may be refined. As a result, the code amount can be reduced, and further refinement can be achieved with respect to the near-side image area that is likely to attract attention.

(Second Embodiment)
FIG. 7 is a block diagram illustrating a configuration of the image encoding device 20 according to the second embodiment. The image encoding device 20 further includes an activity calculation unit 23 in addition to the prefilter unit 21 and the encoding unit 22.

  The activity calculation unit 23 functions as a detection unit that detects definition information representing the definition of an image. The definition information is not particularly limited. For example, an activity indicating the complexity of the image is used. As another example of the definition information, an object in a frame may be detected and the boundary (edge) of the object may be used as the definition information. In this case, an image region with more object edges indicates higher image definition.

  Hereinafter, a case where an activity is used as definition information will be described. The activity calculation unit 23 divides the input image signal in units of frames into predetermined macroblock units, and calculates an activity indicating the complexity of the image of each macroblock. The activity calculation method is not particularly limited, and a known calculation method can be used.

  As shown in FIG. 7, the activity calculated by the activity calculation unit 23 is input to the prefilter unit 21 and the encoding unit 22 in association with the input image signal of each frame image. This is a region that requires higher definition of image quality as the activity is larger and the image is more complicated. Therefore, it can be said that it is preferable not to reduce the image quality for an image region having a large activity. Therefore, the prefilter unit 21 and the encoding unit 22 according to the present embodiment increase the image definition for an image region having a large depth value and a low definition (for example, an image region having a small activity or an image region having few edges). Process to reduce.

  The prefilter unit 21 is a macroblock in which the activity calculation unit 23 divides filter processing such as a low-pass filter for an input image signal based on definition information (for example, activity) representing the definition of the image in addition to the depth information. Apply in units.

  FIG. 8 is a diagram illustrating an example of the relationship between the depth value d, the activity act, and the cutoff frequency f used by the pre-filter unit 21. The prefilter unit 21 decreases the cutoff frequency f used in the macroblock as the average value (depth value d) of the depth values in the macroblock increases. In addition, as the activity act in the macroblock decreases, the cut-off frequency f used in the macroblock is decreased.

尚、式（５）において関数ｇ₃、ｇ₄は任意の関数を用いることができる。また、上記式（５）では右辺第１項と右辺第２項とを乗じるとしたが、これらの和を求めてカットオフ周波数ｆ_jを算出してもよい。また、ｒ₅₁、ｒ₅₂は係数であり、例えば画像符号化装置２０ごとに定められる係数が用いられる。 The following equation (5) shows an example of the relationship between the cutoff frequency f _j , the average value (depth value) d _j of the macro block depth value, and the activity act _j in the macro block for the _j-th macro block. .

In Equation (5), arbitrary functions can be used as the functions g ₃ and g ₄ . In the above formula (5), the first term on the right side and the second term on the right side are multiplied. However, the sum of these may be obtained to calculate the cutoff frequency f _j . R ₅₁ and r ₅₂ are coefficients, and for example, coefficients determined for each image encoding device 20 are used.

Further, the pre-filter unit 21 applies the cutoff frequency f _j obtained by Expression (5) to the pixels in the macroblock, and uses the above Expressions (2) and (3) to output the output signal value y of each pixel. _{Find i} . Then, the prefilter unit 21 outputs the input image signal after the filtering process to the encoding unit 12.

  In the above description, the example in which the cutoff frequency is changed based on the depth value and the activity has been described. However, other parameters related to the filter processing, such as the number of sampling pixels of the low-pass filter, may be changed.

  The encoding unit 22 determines a quantization step width based on definition information (for example, activity) representing the definition of the image in addition to the depth information, and quantizes the orthogonal transform coefficient information described above based on the quantization step width. Turn into.

  FIG. 9 is a block diagram illustrating a configuration of the encoding unit 22. The configuration of the encoding unit 22 is the same as the configuration of the encoding unit 12 of the first embodiment shown in FIG. The quantization control unit 105 of the encoding unit 22 receives the activity output from the activity calculation unit 23 in addition to the input image signal after the filtering process by the prefilter unit 21 and the depth information.

  The quantization control unit 105 calculates a quantization step width according to the activity for each macroblock in addition to the depth information, and outputs the calculated quantization step width to the quantization unit 103.

  FIG. 10 is a diagram illustrating an example of the relationship between the depth value d, the activity act, and the quantization step width s set by the quantization control unit 105. The quantization control unit 105 increases the quantization step width s used in the macroblock as the average value (depth value d) of the depth values in the macroblock increases. In addition, as the average depth value (depth value d) in the macroblock decreases, the quantization step width s used in the macroblock decreases. In addition, the quantization control unit 105 relatively decreases the quantization step width s of the macroblock as the macroblock activity act increases, and relatively decreases the quantization step width s as the macroblock activity act decreases. Make it bigger.

尚、式（６）において、関数ｈ₂、ｈ₃は任意の関数を用いることができる。また、上記式（６）では右辺第１項と右辺第２項とを乗じるとしたが、これらの和を求めて量子化ステップ幅を算出してもよい。また、ｒ₆₁、ｒ₆₂は係数であり、例えば画像符号化装置２０ごとに定められる係数が用いられる。 The following equation (6) shows an example of the relationship between the average value (depth value) d _j of the macro block depth value and the activity act _j in the macro block for the j-th macro block.

In Equation (6), functions h ₂ and h ₃ can be arbitrary functions. In the above equation (6), the first term on the right side and the second term on the right side are multiplied. However, the sum of these may be obtained to calculate the quantization step width. R ₆₁ and r ₆₂ are coefficients, and for example, coefficients determined for each image encoding device 20 are used.

  As described above, according to the image encoding device 20 of the second embodiment, even in an image area displayed on the back side, an image area having a high image definition (for example, activity) is an image definition. The reduction rate of the code amount is reduced as compared with an image area having a low image quality. Therefore, the definition of the image can be maintained even when the amount of encoding is reduced.

  That is, according to the present embodiment, the code amount is reduced for a region where the activity is small and the image is flat (smooth) and is not complicated, and the definition of the image is maintained for a region where the activity is large and the image is complex. be able to. Accordingly, it is possible to improve the coding efficiency by reducing the amount of code for an image region that is not easily noticed, and to prevent image quality deterioration for an image region that is easily noticed. Further, since the complex image area is displayed more clearly on the near side, it can be displayed so that the near side is in focus.

In the above description, the prefilter unit 11 sets the cut-off frequency so that the cut-off frequency has the relationship of FIG. 8 according to the depth value and the activity. It is not limited to examples. As another example, the cutoff frequency f may be set so as to satisfy the relationship of the following equation (7). D _min is the minimum value of the depth value d, d _max is the maximum value of the depth value d, act _min is the minimum value of the activity, and act _max is the maximum value of the activity.

In the above description, the encoding unit 12 sets the quantization step width so that the quantization step width has the relationship shown in FIG. 10 according to the depth value and the activity. It is not limited to the example. As another example, the quantization step width s may be set so as to satisfy the relationship of the following equation (8).

  By configuring the pre-filter unit 11 or the encoding unit 12 as in Expressions (7) and (8), the flat portion of the video displayed on the near side is displayed as a smoothed image that does not stand out from noise. Can do. Further, even a video displayed on the back side can be displayed without squashing a video of a complicated portion.

(Third embodiment)
As another embodiment, a predetermined threshold value may be provided for the depth value, and the filter process may be performed only on an image area that is equal to or greater than the threshold value.

FIG. 11 is a flowchart illustrating the procedure of the filtering process performed by the prefilter unit 11 in the third embodiment. First, an input image signal of a multi-parallax image and depth information of the multi-parallax image are input to the pre-filter unit 11 in association with each frame (step S20). The pre-filter unit 11 determines whether or not the average value (depth value d _j ) of the depth values of the macroblocks is greater than a predetermined threshold value θ (step S21). When the average depth value (depth value d _j ) of the macroblock is larger than the threshold θ (step S21: Yes), filtering processing using depth information is performed on the macroblock as described above, and the back side A filter process for reducing the definition of the image displayed on the screen is performed (step S22). On the other hand, if the average value of the macroblock depth values (depth value d _j ) is not larger than the threshold value θ (step S21: No), the filter processing that does not use the depth information is performed to refine the image displayed on the back side. Filter processing that does not decrease the degree is performed (step S23). Then, it is determined whether or not the filtering process for all macroblocks has been completed (step S24), and if not, the process proceeds to step S21. When the filter processing for all the macroblocks has been completed (step S24: Yes), the output signals after the filter processing for all the macroblocks are synthesized, and the output signal for one frame is sent to the encoding unit 12. Output (step S25).

  In step S23, the filtering process itself may not be performed on the macroblock.

  As described above, according to the third embodiment, the filtering process for reducing the definition of an image is performed only on an image area having a predetermined depth value or more. Accordingly, it is possible to reduce the amount of image signal to be filtered, and to improve the processing speed of the filtering process.

  As mentioned above, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

(Other embodiments)
For example, the device configuration and input information can be changed as appropriate. FIG. 12 is a block diagram illustrating a configuration of an encoding unit 12a that is a modification of the encoding unit 12 (see FIG. 4) described above in the first embodiment. As shown in FIG. 12, a depth information generation unit 180 may be provided in the encoding unit 12a to generate depth information from an input image signal input from the outside.
Note that the depth information generation method is not particularly limited. For example, the depth information may be generated by a motion stereo method for estimating the depth according to the degree of movement of the subject in the two-dimensional image.

  FIGS. 13 to 15 are block diagrams respectively showing configurations of encoding units 22a to 22c, which are modifications of the encoding unit 22 (see FIG. 9) described above in the second embodiment. As shown in FIG. 13, an activity calculating unit 190 may be provided in the encoding unit 22a to calculate an activity for each macroblock of each frame from an input image signal input from the outside. Further, as illustrated in FIG. 14, a depth information generation unit 180 may be provided in the encoding unit 22b. Further, by combining the configurations of FIGS. 13 and 14, as shown in FIG. 15, a depth information generation unit 180 and an activity calculation unit 190 may be provided in the encoding unit 22 c.

  Moreover, although the example which performs the encoding process by entropy encoding was demonstrated above, the kind of encoding process which the image encoding apparatuses 10 and 20 perform is not specifically limited, Other encodings, such as fixed length encoding, are carried out. Processing may be used.

  In addition, the use of data (encoded data) output after encoding is not particularly limited, and may be output to a three-dimensional image display device or may be output to a two-dimensional image display device.

  In addition, the filter processing performed by the prefilter units 11 and 21 is not particularly limited, and a one-dimensional filter, a two-dimensional filter that performs an operation on a two-dimensional pixel string in a frame, and pixels of frames at different times A spatio-temporal filter or the like that performs an operation using a column can also be used. The above is an example of a linear filter, but a non-linear filter may be used. As an example of nonlinear filter processing, if there is a large difference between the target pixel and the surrounding pixels, it is estimated that there is an edge, and a pixel determined to be on the opposite side of the edge is used instead of it. A process of multiplying the filter coefficient by the pixel on the same side can be used.

  As another example of the non-linear filter, an ε filter that performs smoothing only when the fluctuation range of the pixel output value is small with respect to the surrounding pixels can be used. In this regard, since smoothing is performed on the area determined to be noise, the value of ε for the image area displayed on the back side based on the depth information (or may be used together with the activity) The range of pixel output values to be smoothed may be controlled by increasing (threshold value determined not to be noise) and decreasing the value of ε for the image area displayed on the front side.

  In the above description, the image encoding devices 10 and 20 include the pre-filter units 11 and 21, but the image encoding devices 10 and 20 do not include the pre-filter units 11 and 21, and the encoding units 12 and 22 are included. It may be a form provided with only.

  Note that the program executed by the image encoding devices 10 and 20 of the present embodiment is provided by being incorporated in advance in a ROM or the like. A program executed by the image encoding devices 10 and 20 is a file in an installable format or an executable format, and is a computer such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). You may comprise so that it may record and provide on a readable recording medium.

  Furthermore, the program executed by the image encoding devices 10 and 20 of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. . The program executed by the image encoding devices 10 and 20 according to the present embodiment may be provided or distributed via a network such as the Internet.

  The program executed by the image encoding devices 10 and 20 of the present embodiment includes the above-described units (subtraction unit 101, orthogonal transform unit 102, quantization unit 103, entropy encoding unit 104, quantization control unit 105, inverse quantum). Module 106, inverse orthogonal transform unit 107, addition unit 108, and predicted image generation unit 110). As actual hardware, a CPU (processor) reads a program from the ROM and executes it. Are loaded onto the main storage device, the subtraction unit 101, the orthogonal transformation unit 102, the quantization unit 103, the entropy coding unit 104, the quantization control unit 105, the inverse quantization unit 106, the inverse orthogonal transformation unit 107, An addition unit 108 and a predicted image generation unit 110 are generated on the main storage device.

  As described above, according to each of the above-described embodiments, the data amount of the image signal of each image region is adjusted according to the depth value, so that the encoding efficiency of the display image using the depth information can be improved.

DESCRIPTION OF SYMBOLS 10, 20 ... Image coding apparatus 11, 21 ... Pre-filter part, 12, 12a, 22, 22a, 22b, 22c ... Coding part, 23 ... Activity calculation part, 101 ... Subtraction part, 102 ... Orthogonal transformation part, DESCRIPTION OF SYMBOLS 103 ... Quantization part 104 ... Entropy encoding part 105 ... Quantization control part 106 ... Inverse quantization part 107 ... Inverse orthogonal transformation part 108 ... Adder part 109 ... Frame memory 110 ... Predictive image generation part , 180 ... Depth information generation unit, 190 ... Activity calculation unit, C _n ... Filter coefficient, 2N + 1 ... Number of sampling pixels, d, d _i , d _j ... Depth value, f, f _i , f _j ... Cut-off frequency, g ₁ , g ₂ , g ₃ , g ₄ ... function, h ₁ , h ₂ , h ₃ , ... function, r ₁ , r ₄ , r ₅₁ , r ₅₂ , r ₆₁ , r ₆₂ ... coefficient, s, s _j ... Quantization step width, x _i ... input signal value, y _i : output signal value, act, act _j : activity, θ: threshold value.

Claims (10)

Acquisition means for acquiring depth information of each image area constituted by the input image signal;
Signal processing means for performing a process of adjusting the data amount of the image signal of the image area according to the depth value represented by the depth information acquired by the acquisition means;
Output means for outputting an image signal processed by the signal processing means;
An image encoding device comprising:   The image encoding device according to claim 1, wherein the signal processing unit increases an image definition reduction amount for an image region having a larger depth value, thereby reducing an image signal data amount for the image region.   The said signal processing means is further equipped with the encoding means which encodes by enlarging the quantization step width at the time of quantizing the said input image signal, so that the image area | region with the said depth value is large. Image encoding device.   4. The image encoding device according to claim 1, wherein the signal processing unit further includes a filter processing unit that performs a filter process for reducing an image definition in an image region having a larger depth value. 5. It further comprises detection means for detecting definition information representing the definition of the image,
5. The image encoding device according to claim 1, wherein the signal processing unit performs a process of reducing the definition of an image in an image region having a larger depth value and a lower definition. 6. The detection means detects an activity indicating the complexity of the image from the input image signal as the definition information,
The image encoding device according to claim 5, wherein the signal processing unit performs a process of reducing the definition of an image in an image region having a larger depth value and a smaller activity. The detection means detects an edge of an object displayed in an image area from the input image signal as the definition information,
The image encoding device according to claim 5, wherein the signal processing unit performs a process of reducing the definition of an image in an image region in which the depth value is large and the number of edges is small.   8. The filter processing unit according to claim 4, wherein the filter processing means includes a low-pass filter, and reduces an image area of the image area by decreasing a cutoff frequency of the low-pass filter in an image area having a larger depth value. The image coding apparatus according to one.   The image encoding device according to any one of claims 1 to 8, wherein the signal processing unit performs a process of reducing the definition of the image only in an image region in which the depth value is larger than a predetermined threshold. . In an image encoding device,
An acquisition step of acquiring depth information of each image region constituted by the input image signal;
A signal processing step of adjusting a data amount of an image signal of the image region according to a depth value represented by the depth information acquired by the acquisition unit;
An output step of outputting an image signal after processing by the signal processing step;
An image encoding method including:

Images