JP2018032913A - Video encoder, program and method, and video decoder, program and method, and video transmission system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce image deterioration and to increase encoding efficiency, when performing video encoding accompanying inter-layer prediction processing.SOLUTION: A video encoder performing hierarchical coding of images of multiple layers has: prediction means for generating a prediction image predicting an input image of a coding object layer by inter-layer prediction, with reference to at least a re-sample image performed filtering of the decoded image of other reference layer; filter factor determination means for determining an optimum filter factor set, with reference to the input image of the coded object layer, and the decoded image of the reference layer; re-sampling means for filtering the decoded image of the reference layer, by using the filter factor set; and encoding means for encoding the residual signal of the input image of the coded object layer and the prediction image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video encoding device, a program and a method, a video decoding device, a program and a method, and a video transmission system. For example, the present invention is applied to a system for compressing and encoding multi-layer video information and transmitting it as stream data. Can do.

  For example, H.M. H.264 / MPEG-4 AVC (Advanced Video Coding: hereinafter also referred to as “AVC”) and H.264 / MPEG-4 AVC. H.264 / MPEG-H HEVC (High Efficiency Video Coding: hereinafter also referred to as “HEVC”) and the like, video information compression and encoding processing is performed for each processing unit obtained by dividing an input target image. In addition, the transform coefficient obtained by performing spatial transformation such as discrete cosine transform on the prediction residual signal that is the difference between the predicted image that has been inter-predicted such as intra prediction and motion compensation prediction and the input target image is quantized. Thus, high-efficiency video compression is realized by entropy encoding this. Furthermore, in these video coding schemes, scalable coding extension can be used, and multiple video representations such as multiple spatial resolutions, frame rates, image quality, bit depth, etc. are encoded as a multi-layer video stream. be able to.

  FIG. 8 is a block diagram illustrating a configuration of a multi-layer video encoding apparatus. FIG. 9 is a block diagram illustrating a configuration of a multi-layer video decoding apparatus.

  In a multi-layer video encoding apparatus, an input video of layer 0 (base layer) is encoded using an encoder (layer 0 encoder) that does not use scalable encoding extension. Then, the video stream of only the base layer is decoded using a decoder (layer 0 decoder) that does not use scalable coding extension. In a multi-layer video encoding apparatus, encoders in layers 1 and higher (enhancement layers) are configured to be encoded with reference to decoded images of other layers including the base layer. An enhancement layer encoder performs inter-layer prediction using a decoded image (decoded image of a reference layer) generated by an encoder of another layer, thereby enabling more efficient encoding than when no inter-layer prediction is used. Do. The stream output from the encoder of each layer is multiplexed (MUX) and output from the multi-layer video encoding apparatus as a multi-layer stream.

  In the multi-layer decoding apparatus, the encoded stream of each layer is demultiplexed (DEMUX) from the multi-layer stream and input to the decoders (layer 0 to layer n decoder) of each layer. Then, the decoder of each layer generates a decoded image of the target layer and outputs it as a decoded video (layer 0 to layer n decoded video) of each layer. An enhancement layer (one or more layers) decoder performs decoding processing by performing inter-layer prediction using a decoded image (decoded image of a reference layer) generated by a decoder of another layer.

  FIG. 10 is a block diagram showing a configuration of an enhancement layer encoder in a conventional video encoding apparatus. In FIG. 10, for example, when an encoding technique such as HEVC is used, a layer i input video (an image of an encoding target layer) is input to the encoder. The input image of the encoding target layer is divided for each processing unit area such as an encoding unit, and is supplied to the difference processing unit 301. The input image divided for each processing unit region is predicted by the difference processing unit 301 by the inter prediction unit 308 with motion compensation or the prediction image by the intra prediction unit 309 that performs prediction based on the encoded pixels in the screen. A prediction residual signal that is a difference between the two is obtained. Then, the prediction residual signal is subjected to DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) by the transform quantization unit 302, and the obtained transform coefficient is quantized. In the entropy encoding unit 103, the quantized transform coefficient is entropy-encoded such as a variable length code or an arithmetic code, and is output as an encoded stream. Note that the encoded stream of the target layer is multiplexed with streams of other layers and output as a multi-layer stream.

  The quantized transform coefficient is inversely quantized and inversely transformed by the inverse quantization and inverse transform unit 304 and is added to the predicted image by the adder 305 to obtain a reconstructed image.

  Furthermore, an intra-loop filter unit 306 such as a deblocking filter that reduces block distortion is applied to the reconstructed image, and a decoded image (layer i decoded image) of the target layer generated on the decoding side is obtained. The decoded image of the target layer is held in the reference image buffer 307 as a reference image for inter prediction motion compensation at the time of encoding the subsequent input image, and is also a reference for inter-layer prediction in another enhancement layer A layer image is supplied to an encoder of another enhancement layer.

  In SHVC (Scalable High-efficiency Video Coding), which is a scalable extension of HEVC, a decoded image (layer j decoded image) from a reference layer is resampled to the spatial resolution and bit depth of the target image in the resampling unit 320. A reference image number is assigned in the same way as the reference image (layer i decoded image) of the target layer held in the reference image buffer 307. In the enhancement layer encoder, inter prediction is realized by generating a prediction image using inter prediction in the inter prediction unit 308. Note that the encoder of the base layer is equivalent to a configuration in which this inter-layer prediction part is omitted.

  FIG. 11 is a block diagram showing a configuration of an enhancement layer decoder in a conventional video decoding apparatus.

  The multi-layer stream generated by the video encoding device is demultiplexed, and the stream of each layer is input to the decoder of each layer.

  The input encoded stream (layer i stream) of the target layer is decoded by the entropy decoding unit 403 to obtain transform coefficients such as DCT, encoding mode information, and motion vector information. The obtained transform coefficient is inversely quantized and inversely transformed by the inverse quantization inverse transform unit 304 and then added to the prediction image generated by the inter prediction unit 408 or the intra prediction unit 409 by the addition unit 405. Thus, the same reconstructed image as that on the encoding side is generated.

  Further, the intra-loop filter unit 406 such as a deblocking filter is applied to the reconstructed image, and the reconstructed image is supplied as a decoded image of the target layer (layer i decoded image) to a decoder of another enhancement layer. Is stored in the reference image buffer 407 as a reference image.

  The decoded image (layer j decoded image) from the reference layer is resampled to the spatial resolution and bit depth of the target image by the resampling unit 420 and is stored in the reference image buffer 407 (layer i decoding). A reference image number is assigned in the same manner as (image). In the enhancement layer encoder, inter prediction is realized by generating a prediction image using inter prediction in the inter prediction unit 408. Note that the base layer decoder is equivalent to a configuration in which the inter-layer prediction portion is omitted.

  As described above, in HEVC, resampling processing for converting spatial resolution and bit depth is performed in order to perform inter-layer prediction using a decoded image of another layer whose spatial resolution or the like may be different as a reference image.

  12 and 13 are explanatory diagrams showing filter coefficients of a resampling filter used in HEVC encoding technology. FIG. 12 shows an 8-tap filter coefficient used for resampling luminance samples, and FIG. 13 shows a 4-tap filter coefficient used for resampling color difference samples.

  In HEVC, the pixel position in the reference layer corresponding to the pixel position of the target layer is calculated with 1/16 pixel accuracy, and the horizontal and vertical phases (integer pixel positions) with 1/16 pixel accuracy on the reference layer are calculated. The filter coefficient is selected according to the deviation from () and the resampling process is performed by performing the horizontal filter process and the vertical filter process.

  For example, the luminance filter uses the integer position (xR, yR) and phase (xP, yP) for the position (X, Y) of 1/16 pixel accuracy on the reference layer of the target pixel, and The resampled pixel value p at the position of the target pixel is obtained from the pixel value array r of the decoded image of the reference layer as expressed by equations (1-1) to (1-6).

ここで、ｆは、図１２のフィルタ係数（位相及びフィルタのタップ位置により定まる係数）である。ｓ_１、ｓ_２、ｏは画素値のビット深度に応じたシフト値と丸め値である。また、ｉとｊは、水平方向及垂直方向に用いるフィルタのタップ位置であり、図１２の例であれば、それぞれ０〜７の値が該当する。

Here, f is the filter coefficient of FIG. 12 (coefficient determined by the phase and the tap position of the filter). s ₁ , s ₂ , and o are a shift value and a rounded value corresponding to the bit depth of the pixel value. Further, i and j are the tap positions of the filters used in the horizontal direction and the vertical direction. In the example of FIG.

  Thus, in HEVC, a fixed coefficient value is used for each phase as a filter coefficient used in the resampling process. On the other hand, as a method for adapting filter coefficients for such resampling processing, for example, a method described in Patent Document 1 has been proposed.

Special table 2015-530805 gazette

  However, since the conventional resampling process uses a fixed value for the filter coefficient used in the resampling filter process, a filter process that is not necessarily optimal for the encoding target image has been performed.

  In addition, with regard to a method for making this an adaptive coefficient, there has been a problem that a specific implementation method including Patent Document 1 described above is not clear.

  Therefore, when performing video encoding processing and video decoding processing with inter-layer prediction processing (phosphorus sampling processing), a video encoding device, program, and method that can reduce image quality degradation and increase encoding efficiency In addition, a video decoding device, a program and method, and a video transmission system are desired.

  According to a first aspect of the present invention, in a video encoding apparatus that hierarchically encodes images of a plurality of layers, (1) a predicted image obtained by predicting an input image of an encoding target layer is at least a decoded image of another reference layer The resampled image that has been subjected to the filtering process in reference to the prediction unit that generates by inter-layer prediction, and (2) the input image of the encoding target layer and the decoded image of the reference layer Filter coefficient determining means for determining an appropriate filter coefficient set; (3) resampling means for performing filter processing on the decoded image of the reference layer using the filter coefficient set; and (4) the encoding target layer. And encoding means for encoding a residual signal between the input image and the prediction image.

  According to a second aspect of the present invention, in a video decoding apparatus for decoding hierarchically encoded data, (1) a predicted image obtained by predicting a target image of a decoding target layer is at least a decoded image of another reference layer (2) decoding means for decoding the filter coefficient set included in the encoded data that has been hierarchically encoded, (3) And resampling means for performing filter processing on the decoded image of the reference layer using the filter coefficient set decoded by the decoding means.

  A video encoding program according to a third aspect of the present invention provides a computer mounted in a video encoding apparatus that hierarchically encodes images of a plurality of layers, (1) a predicted image obtained by predicting an input image of an encoding target layer, at least Prediction means for generating by inter-layer prediction with reference to a resampled image obtained by filtering the decoded image of another reference layer, (2) an input image of the encoding target layer, and the reference layer And (3) resampling means for performing filter processing on the decoded image of the reference layer using the filter coefficient set. And (4) functioning as encoding means for encoding a residual signal between the input image of the encoding target layer and the predicted image.

  According to a fourth aspect of the present invention, there is provided a video decoding program comprising: a computer mounted in a video decoding apparatus that decodes hierarchically encoded data; (1) a prediction image obtained by predicting a target image of a decoding target layer; Prediction means for generating by inter-layer prediction with reference to a resampled image obtained by filtering the decoded image of another reference layer, and (2) a filter coefficient set included in hierarchically encoded data And (3) using the filter coefficient set decoded by the decoding means as a resampling means for performing a filtering process on the decoded image of the reference layer.

  According to a fifth aspect of the present invention, there is provided a video encoding method performed by a video encoding apparatus that hierarchically encodes images of a plurality of layers, comprising: prediction means, filter coefficient determination means, resampling means, and encoding means; (1) The prediction means generates a prediction image obtained by predicting an input image of the encoding target layer by inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer. (2) The filter coefficient determining means determines an optimum filter coefficient set with reference to the input image of the encoding target layer and the decoded image of the reference layer, and (3) the resampling means And (4) the encoding unit performs a filtering process on the decoded image of the reference layer using the filter coefficient set; Wherein the encoding the residual signal between the prediction image.

  According to a sixth aspect of the present invention, in the video decoding method performed by the video decoding apparatus for decoding the hierarchically encoded data, the video decoding device includes a prediction unit, a decoding unit, and a resampling unit. (1) A prediction image obtained by predicting a target image of the target layer is generated by inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer; (2) the decoding unit (3) The resampling means uses the filter coefficient set decoded by the decoding means to decode the decoded image of the reference layer. A filtering process is performed on.

  A video transmission system according to a seventh aspect of the present invention is a video comprising: a video encoding device that hierarchically encodes images of a plurality of layers; and a video decoding device that decodes encoded data hierarchically encoded by the video encoding device. In the transmission system, the video encoding device of the first invention is applied as the video encoding device, and the video decoding device of the second invention is applied as the video decoding device.

  ADVANTAGE OF THE INVENTION According to this invention, when performing the video encoding process and video decoding process accompanying an inter-layer prediction process (phosphorus sampling process), image quality degradation can be reduced and encoding efficiency can be improved.

It is a block diagram which shows the structure of the encoder of the enhancement layer in the video coding apparatus of embodiment. It is a block diagram which shows the structure of the decoder of the enhancement layer in the video decoding apparatus of embodiment. It is the block diagram shown about the whole structure of the video transmission system of embodiment. 5 is a flowchart illustrating determination processing of a filter coefficient set used in resampling processing in the enhancement layer encoder according to the embodiment. It is explanatory drawing which shows an example of the encoding object image (an inter-layer prediction area | region is used) used with the encoder of the enhancement layer of embodiment. It is a flowchart (the 1) which shows an example which calculates a filter coefficient set in the filter coefficient determination part of embodiment. It is a flowchart (the 2) which shows an example which calculates a filter coefficient set in the filter coefficient determination part of embodiment. It is a block diagram which shows the structure of a multi-layer video encoding apparatus. It is a block diagram which shows the structure of the video decoding apparatus of a multilayer. It is a block diagram which shows the structure of the encoder of the enhancement layer in the conventional video coding apparatus. It is a block diagram which shows the structure of the decoder of the enhancement layer in the conventional video decoding apparatus. It is explanatory drawing (the 1) which shows the filter coefficient of the resampling filter used with the encoding technique of HEVC. It is explanatory drawing (the 2) which shows the filter coefficient of the resampling filter used with the encoding technique of HEVC.

(A) Main Embodiments Hereinafter, embodiments of a video encoding device, a program and a method, a video decoding device, a program and a method, and a video transmission system according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of Embodiment (A-1-1) Overall Configuration FIG. 3 is a block diagram showing the overall configuration of the video transmission system 1 of this embodiment.

  In the video transmission system 1 shown in FIG. 3, the encoding target image (input image) constituting the encoding target video (input video) is encoded for each layer (layer), and the encoded stream for each layer is multiplexed. Video encoding apparatus 10 that outputs a multi-layer stream, and video decoding that returns the multi-layer stream to an encoded stream for each of a plurality of layers, decodes the encoded stream for each layer, and obtains a decoded image that constitutes the decoded video A device 20 is arranged.

  In this embodiment, the video encoding apparatus 10 employs an encoding method called hierarchical encoding, and includes a base layer encoder and one or more enhancement layer encoders. Similarly, the video decoding device 20 includes a base layer decoder and one or more enhancement layer decoders.

  In the video transmission system 1, the connection configuration between the video encoding device 10 and the video decoding device 20 is not limited. For example, the video transmission system 1 may transmit by communication via a network, or offline data You may make it supply (for example, supply with data recording media, such as DVD and HDD). In addition, among the multi-layer streams encoded by the video encoding device 10, only a partial layer stream that can be decoded up to a layer having the video quality (video expression) required by the video decoding device 20 is transmitted. It may be.

(A-1-2) Detailed Configuration of Video Encoding Device (Enhancement Layer Encoder) FIG. 1 is a block diagram showing a configuration of an enhancement layer encoder in the video encoding device. The enhancement layer encoder of this embodiment may be configured as hardware such as a dedicated IC chip on which the components shown in FIG. 1 are mounted, or may be software centered on a CPU and a program executed by the CPU. Although functionally configured, it can be functionally represented in FIG.

  In FIG. 1, an enhancement layer encoder 100 includes a difference processing unit 101, a transform quantization unit 102, an entropy coding unit 103, an inverse quantization inverse transform unit 104, an addition unit 105, an in-loop filter unit 106, and a reference image buffer 107. , An inter prediction unit 108, an intra prediction unit 109, a switching unit 110, a resampling unit 120, a filter coefficient storage unit 130, and a filter coefficient determination unit 131.

  The enhancement layer encoder 100 encodes video input to layer i (layer i input video) using a predetermined encoding method, and outputs an encoded stream of layer i (layer i stream). Also, the enhancement layer encoder 100 receives the input encoding target layer image (layer i input image) in the same manner as the enhancement layer encoder (encoder of FIG. 10 described above) of the conventional video encoding apparatus. The processing is performed for each area (hereinafter referred to as “processing unit area”) of a predetermined unit (for example, a block of a predetermined size) such as an encoding unit. In this embodiment, it is described that the input images of layer i are individually input. However, from these original input images, the image representation corresponding to each layer by image processing such as down-sampling is the layer i. It can be configured to be generated as an input image.

  In this embodiment, the encoding method is H.264. The case where the encoding method is expanded based on the standardization technology of H.265 / MPEG-H HEVC and the like is illustrated, but the present invention is not limited to this, and the present invention can be applied to various similar encoding methods.

  The difference processing unit 101 calculates a difference between an input image for each predetermined processing unit region and a predicted image corresponding to the processing unit region from the inter prediction unit 108 or the intra prediction unit 109 in order to obtain a prediction residual signal. And the difference is given to the transform quantization unit 102 as a prediction residual signal.

  The transform quantization unit 102 transforms the input prediction residual signal into transform coefficients by DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), etc., and quantizes the obtained transform coefficients It is.

  The entropy coding unit 103 entropy-encodes the quantized transform coefficients (including coding mode information, motion vector information, etc.) from the transform quantization unit 102 in order to compress the bias of the appearance probability of the code. Thus, it is output as a code stream of the layer. Note that the filter coefficient determined by the filter coefficient determining unit 131 (filter coefficient stored in the filter coefficient storage unit 130), which will be described later, is also entropy-coded and multiplexed into a multi-layer stream in the same manner as the quantized transform coefficient. The

  The encoded stream (layer i stream) output from the entropy encoding unit 103 is multiplexed with an encoded stream of another layer and output as a multi-layer stream that is an encoding processing result of the video encoding device 10. The In addition, the filter coefficient determined by the filter coefficient determination unit 131 (the filter coefficient stored in the filter coefficient storage unit 130) is also encoded and multiplexed into the encoded multi-layer stream and output.

  The inverse quantization inverse transform unit 104 performs inverse quantization and inverse transform on the quantized transform coefficient from the transform quantization unit 102 in order to restore a residual signal (residual image) from the encoded signal, This is given to the adder 112.

  The adding unit 105 adds the prediction image from the inter prediction unit 108 or the intra prediction unit 109 to the restored residual signal from the inverse quantization and inverse transform unit 104 via the switching unit 110, thereby obtaining a reconstructed image. It is what you want. The adding unit 112 provides the reconstructed image to the in-loop filter unit 106 and the intra prediction unit 109.

  The in-loop filter unit 106 filters the reconstructed image from the adder unit 105 as a decoded image in order to reduce coding distortion (for example, block distortion, ringing distortion, etc.) caused by quantization processing in the coding loop. It is. This embodiment exemplifies a case where HEVC or the like is applied, and reduces coding distortion using a deblocking filter or the like. The decoded image (layer i decoded image) output from the in-loop filter unit 106 is held in the reference image buffer 107 as a reference image for motion compensation or the like in the inter prediction unit 108, and is also an encoder of another enhancement layer. To be supplied.

  The reference image buffer 107 holds the image output from the in-loop filter unit 106 as a reference image. The output image from the in-loop filter unit 106 becomes a reference image for inter prediction motion compensation at the time of encoding the subsequent input image.

  The inter prediction unit 108 performs inter prediction with reference to an image held in the reference image buffer 107.

  The inter prediction unit 108 also performs inter prediction (inter-layer prediction) with reference to the decoded image of the reference layer resampled by the resampling unit 120 together with the image held in the reference image buffer 107. . Since inter-layer prediction can refer to an image at the same time in another layer (a resampled decoded image of a reference layer) as a predicted image, the encoding efficiency can be further improved. As a modification, the inter prediction unit 108 may perform inter prediction (interlayer prediction) with reference to only the decoded image of the reference layer resampled by the resampling unit 120.

  The intra prediction unit 109 performs intra prediction using the image (encoded reconstruction pixel or the like) output from the addition unit 105.

  The switching unit 110 switches the output (predicted image) of the inter prediction unit 108 or the intra prediction unit 109 according to the coding mode (inter prediction mode, inter-layer prediction mode, or intra prediction mode).

  The filter coefficient determination unit 131 determines a filter coefficient group (hereinafter referred to as “filter coefficient set”) used in the resampling unit 120. Details of the filter coefficient determination unit 131 will be described in the operation section. The filter coefficient set is composed of a plurality of filter coefficients.

  The filter coefficient storage unit 130 stores the filter coefficient set determined by the filter coefficient determination unit 131.

  The resampling unit 120 resamples (filters) the reference layer decoded image (layer j decoded image) using the filter coefficient set stored in the filter coefficient storage unit 130. Details of the resampling unit 120 will be described in the section of operation.

  Here, the difference between the enhancement layer encoder 100 of this embodiment and the conventional enhancement layer encoder described above with reference to FIG. 10 will be briefly described. The resampling unit 120 of the enhancement layer encoder 100 is different from the conventional resampling process in that the resampling process is performed using a filter coefficient set determined appropriately instead of a fixed filter coefficient. Moreover, the point which provided the filter coefficient determination part 131 and the filter coefficient memory | storage part 130 which determine the filter coefficient set used by this resampling process differs from the encoder of the conventional enhancement layer. The filter coefficient set determined by the filter coefficient determination unit 131 is encoded by the entropy encoding unit 103, and is multiplexed and output in the multi-layer stream of the encoding result, which is different from the conventional enhancement layer encoder.

  The base layer encoder 150 applied to the video encoding device 10 is the same as that obtained by omitting the components related to the inter-layer prediction processing from the enhancement layer encoder 100 as in the prior art.

(A-1-3) Detailed Configuration of Video Decoding Device (Enhancement Layer Decoder) FIG. 2 is a block diagram illustrating a configuration of an enhancement layer decoder in the video decoding device of the embodiment. The enhancement layer decoder of this embodiment may be configured as hardware such as a dedicated IC chip on which the components shown in FIG. 2 are mounted, or software centered on a CPU and a program executed by the CPU. Although it may be configured in terms of function, it can be functionally represented in FIG.

  In FIG. 2, the enhancement layer decoder 200 includes an entropy decoding unit 203, an inverse quantization inverse transformation unit 204, an addition unit 205, an in-loop filter unit 206, a reference image buffer 207, an inter prediction unit 208, an intra prediction unit 209, and a switching. Unit 210, resampling unit 220, and filter coefficient storage unit 230.

  The enhancement layer decoder 200 decodes the enhancement layer encoded stream encoded by the enhancement layer encoder 100 to obtain an enhancement layer decoded image. The enhancement layer decoder 200 receives an encoded stream (layer i stream) for each layer obtained by demultiplexing the multi-layer stream input to the video decoding device 20.

  The entropy decoding unit 203 entropy decodes the input encoded stream to obtain decoded data. The decoded data includes quantized transform coefficients (data obtained by transforming the prediction residual signal by DCT or the like and further quantizing it). Also, the encoded stream includes other additional information as described above. The additional information includes coding mode information, motion vector information, a filter coefficient set used for resampling processing (a filter coefficient set determined by the filter coefficient determination unit 131 of the enhancement layer encoder 100), and the like. . The entropy decoding unit 203 supplies the quantized transform coefficient obtained by decoding to the inverse quantization inverse transform unit 204, and supplies the filter coefficient set used in the resampling process to the filter coefficient storage unit 230.

  The inverse quantization inverse transform unit 204 performs inverse quantization and inverse transform on the quantized transform coefficient from the entropy decoding unit 203 in order to restore a residual signal (residual image) from the encoded signal, and adds This is given to the unit 205.

  The adding unit 205 adds the predicted image from the inter prediction unit 208 or the intra prediction unit 209 to the reconstructed residual signal from the inverse quantization and inverse transform unit 204 via the switching unit 210 to obtain a reconstructed image. It is what you want. The adding unit 205 provides the reconstructed image to the in-loop filter unit 206 and the intra prediction unit 209.

  The in-loop filter unit 206 filters the reconstructed image from the adder unit 205 as a decoded image in order to reduce coding distortion (for example, block distortion, ringing distortion, etc.) caused by quantization processing in the coding loop. It is. This embodiment exemplifies a case where HEVC or the like is applied, and reduces coding distortion using a deblocking filter or the like. A decoded image (layer i decoded image) output as a decoding result from the in-loop filter unit 206 is held in the reference image buffer 207 as a reference image for motion compensation or the like in the inter prediction unit 208, and other extension Supplied to the layer decoder. In this embodiment, each layer i decoded image is described as being output as a decoding result. Of these, only a decoded image of a layer having an image quality (image representation) required as a decoding process is a video. You may comprise so that it may output as a decoding result of the decoding apparatus 20. FIG.

  The reference image buffer 207 holds the image output from the in-loop filter unit 206 as a reference image.

  The inter prediction unit 208 performs inter prediction with reference to the image held in the reference image buffer 207 and the decoded image of the reference layer resampled by the resampling unit 220.

  The intra prediction unit 209 performs intra prediction using an image (such as a reconstructed pixel in the screen) output from the addition unit 205.

  The switching unit 210 switches the output (predicted image) of the inter prediction unit 208 or the intra prediction unit 209 according to the encoding mode information restored from the encoded stream by the entropy decoding unit 203.

  The filter coefficient storage unit 230 stores the filter coefficient set supplied from the entropy decoding unit 203.

  The resampling unit 220 resamples the reference layer decoded image (layer j decoded image) using the filter coefficient set stored in the filter coefficient storage unit 230.

  Here, the difference between the enhancement layer decoder 200 of this embodiment and the conventional enhancement layer decoder described with reference to FIG. 11 will be briefly described. The resampling unit 220 of the enhancement layer decoder 200 uses the filter coefficient set obtained by decoding the stream from the video encoding device 10 by the entropy decoding unit 203 instead of the fixed filter coefficient, and the resampling unit 220. The resampling process is different from the conventional resampling process. Moreover, the point which provided the filter coefficient memory | storage part 230 which memorize | stores the filter coefficient set decoded by the entropy decoding part 203 used by this resampling process differs from the decoder of the conventional enhancement layer.

(A-2) Operation of Embodiment Next, the operation (video encoding method and video decoding method of the embodiment) of the video transmission system 1 in this embodiment having the above-described configuration will be described.

  The following description focuses on processing related to resampling (resampling processing and filter coefficient determination processing) of the enhancement layer encoder 100 (enhancement layer decoder 200), which is a characteristic part of the present embodiment.

(A-2-1) Resampling Process As described above, in the resampling unit 120 (resampling unit 220), the filter coefficient used for the filter process is not a fixed value but a variable value (a filter coefficient determination unit of the enhancement layer encoder 100). The filter coefficient set determined in 131).

  In other words, the resampling unit 120 (resampling unit 220) uses the filter coefficient set stored in the filter coefficient storage unit 130 (filter coefficient storage unit 230) to perform the processing on the reference layer corresponding to the pixel position on the target layer. For example, using the phase (xP, yP) that is a deviation from the integer position (xR, yR) and the integer position with respect to the position (X, Y) of the target pixel with 1/16 pixel accuracy at As in equations (2-1) to (2-6), the position of the target pixel is resampled from the pixel value array r of the reference layer decoded image (layer j decoded image in the examples of FIGS. 1 and 2). The obtained pixel value p is obtained.

上記の（２−１）〜（２−６）式おいて、ａ（ａ［ｘＰ，ｉ］）、ｂ、ｃ（ｃ［ｙＰ，ｊ］）、ｄは、フィルタ係数記憶部１３０（フィルタ係数記憶部２３０）に記憶されるフィルタ係数セットを構成するフィルタ係数である。なお、ｂ及びｄは、定数項である（この実施形態では、定数項であるｂ及びｄもフィルタ係数と呼ぶものとする）。また、上記の（２−１）〜（２−６）式において、ｓ_１，ｓ_２は、画素値のビット深度に応じたシフト値であり、ｏは丸め値である。さらに、Ｎはフィルタタップ数である。また、ｉ及びｊは、水平方向及び垂直方向に用いるフィルタのタップ位置を示す値である。なお、この実施形態では、参照レイヤ上での画素位置を１／１６画素精度で表現し、位相としては１６通りのフィルタ係数を用いる例を示したが、１／１６画素精度とは異なる精度（１／ｎ画素）で表現しても良い。

In the above equations (2-1) to (2-6), a (a [xP, i]), b, c (c [yP, j]), and d are filter coefficient storage units 130 (filter coefficients). Filter coefficients constituting a filter coefficient set stored in the storage unit 230). Note that b and d are constant terms (in this embodiment, constant terms b and d are also called filter coefficients). In the above equations (2-1) to (2-6), s ₁ and s ₂ are shift values according to the bit depth of the pixel value, and o is a rounded value. Further, N is the number of filter taps. I and j are values indicating the tap positions of the filters used in the horizontal and vertical directions. In this embodiment, an example in which the pixel position on the reference layer is expressed with 1/16 pixel accuracy and 16 filter coefficients are used as the phase is shown. However, an accuracy different from the 1/16 pixel accuracy ( (1 / n pixel) may be used.

  The filter tap number N may be a different value for each color component, for example, 8 for a luminance component (luminance filter) and 4 for a color difference component (color difference filter). In addition, the number of filter taps N may be specified with the upper syntax.

  The above equations (2-1) to (2-6) are equations having constant terms such as b and d. It should be noted that b and d may be restricted (b = 0, d = 0) from being used depending on the profile used. Further, the filter coefficients a and b used in the horizontal direction and the filter coefficients c and d used in the vertical direction can be set independently. However, the same coefficient may be used (a = c, b = d). Due to these limitations, the number of filter coefficients to be encoded can be reduced.

  In the video encoding device 10 (enhancement layer encoder 100), the filter coefficient determination unit 131 designs an optimum filter coefficient set for resampling and uses the obtained filter coefficient in the resampling unit 120. While being stored in the filter coefficient storage unit 130 as a set, the entropy encoding unit 103 multiplexes it into an encoded stream according to various encoding methods.

  In the video decoding device 20 (enhancement layer decoder 200), the entropy decoding unit 203 decodes the filter coefficient set used in the resampling process multiplexed in the encoded stream, and filters as the filter coefficient set used in the resampling unit 220. Stored in the coefficient storage unit 230.

  As a multiplexing method of the filter coefficient set in the entropy encoding unit 103, for example, encoding may be performed using higher syntax such as a parameter set such as a slice header or a picture parameter set.

  For example, predetermined filter coefficients as shown in FIGS. 12 and 13 are determined in advance, and the entropy encoding unit 103 displays only a flag indicating no change in the upper syntax when there is no change from the default filter coefficient. And so on. Signaling is a signal for specifying the desired information in the transmission signal (encoded stream) or a signal indicating the information itself so that the receiving side (decoding side) can obtain the desired information. Is meant to be included.

  When there is a change, the entropy coding unit 103 provides a flag for each phase, and signals a flag indicating a phase including a filter coefficient different from a predetermined filter coefficient. Thereby, for example, in the case of resampling that expands the spatial resolution by 2 × 2, there are times when only the filters of 0 and 8 are used as the phase, but the entropy coding unit 103 Can efficiently encode only these specific phase filter coefficients. Further, the entropy encoding unit 103 is configured to encode the difference value between the predetermined filter coefficient value corresponding to the filter coefficient value and the changed filter coefficient value, thereby encoding the filter coefficient value. It may be configured to reduce the amount of code required for conversion.

  In the video decoding device 20 (enhancement layer decoder 200), the entropy decoding unit 203 decodes the signaled filter coefficient set as described above. Further, the phase instructed to use the predetermined filter coefficient is stored (stored) in the filter coefficient storage unit 230 together with the predetermined default filter coefficient as shown in FIGS. To do. Then, the resampling unit 220 uses the filter coefficient set stored in the filter coefficient storage unit 230 to perform the resampling process based on the above equations (2-1) to (2-6), so that the target layer Can be decrypted.

(A-2-2) Design (Selection) of Optimal Filter Coefficient Set Used for Resampling Process Next, the filter coefficient set design process in the filter coefficient determination unit 131 of the enhancement layer encoder 100 will be described.

  Assuming that the decoded image of the reference layer is r, the input image of the encoding target layer is q, and the resampled image by the filter coefficient set (filter coefficients a to d) is p, the resampled image p is (2-1) described above. ) To (2-6). The filter coefficient determination unit 131 designs filter coefficients a to d that minimize the error of the input image q.

  For example, the filter coefficient determination unit 131 uses the least-squares method for the filter coefficients a to d that minimize the square sum E of errors between the resampled image p and the input image q, as shown in the following equation (3). Use to find.

ここで、上記の（３）式における誤差の二乗和Ｅは、対象レイヤの画像の画面全体のすべての画素位置に関する和として求める。ｐ及びｑは対象レイヤ上の画素位置ごとの値である。

Here, the square sum E of errors in the above equation (3) is obtained as the sum of all pixel positions of the entire screen of the image of the target layer. p and q are values for each pixel position on the target layer.

  In addition, for example, when the filter coefficient determination unit 131 has a part of an area (interlayer prediction area) to be encoded using inter-layer prediction for the entire screen of the encoding target image as illustrated in FIG. 5. Alternatively, the filter coefficient may be determined by a procedure as shown in FIG.

  FIG. 4 is a flowchart showing determination processing of filter coefficients used in resampling processing in the encoder 100 of the enhancement layer.

  The filter coefficient determination unit 131 performs a resampling process on the decoded image of the reference layer using a predetermined filter coefficient (for example, the filter coefficient illustrated in FIGS. 12 and 13), and temporarily refers to a temporary layer. Create an image. Then, the enhancement layer encoder 100 uses the inter-layer reference image and the reference image of the target layer of the reference image buffer, and the encoding mode (intra prediction, inter prediction, inter-layer prediction) of each processing unit region, A reference image to be referred to is selected (S11).

  The filter coefficient determination unit 131 extracts a region in which inter-layer prediction is selected (for example, the inter-layer prediction region in FIG. 5) from the encoding target image (S12).

  The filter coefficient determination unit 131 obtains filter coefficients a to d that minimize the square sum E of errors obtained by the above-described equation (3) only for the region where inter-layer prediction is selected (S13). That is, the filter coefficient is obtained on the basis of an equation in which the sum of squares of errors is limited to the pixel position in the region where inter-layer prediction is selected. The calculated filter coefficients a to d are stored in the filter coefficient storage unit 130.

  Thereafter, in the resampling unit 120, filter processing is executed using the filter coefficient determined by the filter coefficient determination unit 131 (filter coefficient stored in the filter coefficient storage unit 130) (S14).

  In the above-described step S11, the case where the temporary inter-layer reference image using the re-sampling process using the predetermined filter coefficient has been described. However, the filter coefficient determination unit 131 uses the entire screen pixel to temporarily filter coefficients. May be configured to generate a temporary inter-layer reference image by performing resampling processing using the filter coefficient.

  Next, details of the least square method for filter design will be described.

  When obtaining the filter coefficients a to d that minimize the square sum E of the error as in the above-described equation (3), the filter coefficient determination unit 131 uses an equation obtained by partially differentiating the square sum E of the errors by a to d. The filter coefficients a to d that become 0 are obtained by solving simultaneous equations such as the following equation (4), for example.

上記の（４）式は、ａ〜ｄに関する１次式とはならないが、既定のフィルタ係数値を始点とするような解析的な処理方法等によりフィルタ係数ａ〜ｄを求めることができる。

The above expression (4) is not a linear expression related to a to d, but the filter coefficients a to d can be obtained by an analytical processing method or the like using a predetermined filter coefficient value as a starting point.

  FIG. 6 is a flowchart illustrating an example in which the filter coefficient determination unit approximately obtains the filter coefficient using only the solution of the linear equation.

  The filter coefficient determination unit 131 substitutes default filter coefficient values for c and d on the assumption that c and d in the above equation (3) are the same values as the default filter coefficients. Then, the filter coefficient determination unit 131 obtains filter coefficients a and b that minimize the square sum E of the error in the substituted equation (3) (S21). If the filter coefficient determination unit 131 regards c and d as constants, the partial differentials (4-1) and (4-2) of E in equation (3) by a and b are linear expressions relating to a and b. Thus, a and b can be obtained using a simple linear equation solving method.

  The filter coefficient determination unit 131 considers the coefficients c and d that minimize E from the expression (3) in which a and b obtained in step S21 described above are substituted as constants (4-3), ( It is obtained from equation 4-4 (S22). The filter coefficient determination unit 131 can perform processing by solving a linear equation as in step S21 described above.

  Note that the filter coefficient determination unit 131 may first determine the coefficients c and d in step S21, and then determine the coefficients a and b in step S22.

  Also, for example, in the case of providing a constraint (a = c, b = d) that uses the same filter coefficient in the horizontal direction and the vertical direction, the filter coefficient determination unit 131 assumes that one filter coefficient is a constant and the other You may perform the process (FIG. 7) which calculates | requires a filter coefficient by repeating the process which calculates | requires a filter coefficient.

  The filter coefficient determination unit 131 initializes the filter coefficients c and d with predetermined filter coefficients (S31).

  The filter coefficient determination unit 131 considers the filter coefficients c and d as constants, and performs a determination process of the filter coefficients a and b so as to minimize the error sum of squares E in the above equation (3) (S32). .

  The filter coefficient determination unit 131 regards the coefficients a and b obtained in step S32 described above as constants, and similarly determines the coefficients c and d so as to minimize the square sum E of the error in the equation (3). (S33).

  The filter coefficient determination unit 131 determines whether or not to use the obtained filter coefficients c and d. If not, the determination process is performed again from step S32 (S34). On the other hand, when using the obtained filter coefficients c and d (when it is determined that the coefficient values have converged), the filter coefficient determination unit 131 cancels the determination process.

  As described above, the filter coefficient determination unit 131 obtains a filter coefficient set that is optimally designed for the inter-layer prediction region, and supplies the filter coefficient set to the resampling unit 120 (filter coefficient storage unit 130) and the video decoding device 20 side.

(A-3) Effects of Embodiment According to this embodiment, the following effects can be achieved.

  In this embodiment, filter coefficient sets used in the resampling unit 120 of the enhancement layer encoder 100 and the resampling unit 220 of the enhancement layer decoder 200 can be set to appropriate values (the filter coefficient set can be arbitrarily changed). It was. Thereby, in the video encoding device 10 (enhancement layer encoder 100) of this embodiment, it is possible to perform the optimum filter processing according to the property of the image to be encoded, and the image quality deterioration can be reduced, and more An encoded stream with high encoding efficiency can be generated.

  Further, since the filter coefficient determination unit 131 is configured to encode only filter coefficients different from the predetermined resampling filter coefficients, it is possible to reduce the amount of code required for encoding the filter coefficients.

  Furthermore, since the enhancement layer encoder 100 (enhancement layer decoder 200) can also specify a filter coefficient of phase 0, an inter-layer reference that does not involve spatial resolution conversion (the reference layer and the target layer have the same spatial resolution). Or in the case of inter-view reference prediction (inter-view prediction) in the case of encoding using multi-view encoding extension, such as stereo 3D image encoding, the resampling unit 120 (resampling unit 220) is It is also possible to function as an image quality improvement filter for the reference image from the reference layer.

(B) Other Embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments as exemplified below.

  In the above embodiment, an example in which the present invention is applied to a video transmission system (video encoding device and video decoding device) has been described. However, the present invention can be used for various other encoding processes. For example, the prediction process and transform quantization process related to encoding other than the phosphorus sampling process of the above embodiment are not limited to the configuration described in the above embodiment, and various encoding tools can be used. The present invention can also be used for a combination of encoding processes.

DESCRIPTION OF SYMBOLS 1 ... Video transmission system, 10 ... Video encoding apparatus, 20 ... Video decoding apparatus, 100 ... Encoder of an enhancement layer, 101 ... Difference processing part, 102 ... Transform quantization part, 103 ... Entropy encoding part, 104 ... Inverse quantum Inverse conversion unit 105 ... Adding unit 106 ... In-loop filter unit 107 ... Reference image buffer 108 ... Inter prediction unit 109 ... Intra prediction unit 110 ... Switching unit 112 ... Adding unit 120 ... Resampling unit , 130 ... filter coefficient storage unit, 131 ... filter coefficient determination unit, 200 ... enhancement layer decoder, 203 ... entropy decoding unit, 204 ... inverse quantization inverse transform unit, 205 ... addition unit, 206 ... in-loop filter unit, 207 Reference image buffer 208 Inter-prediction unit 209 Intra prediction unit 210 Switching unit 220 Resan Ring portion, 230 ... filter coefficient storage unit.

Claims (16)

In a video encoding device that hierarchically encodes images of a plurality of layers,
A prediction unit that generates a prediction image obtained by predicting an input image of an encoding target layer by performing inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer;
Filter coefficient determining means for determining an optimum filter coefficient set with reference to the input image of the encoding target layer and the decoded image of the reference layer;
Resampling means for performing a filtering process on the decoded image of the reference layer using the filter coefficient set;
A video encoding device comprising: encoding means for encoding a residual signal between an input image of the encoding target layer and the predicted image. The re-sampling means has an integer position (xR, yR) with respect to the position (X, Y) of the target pixel of the decoded image of the reference layer corresponding to the pixel position of the input image of the encoding target layer. The pixel value p resampled from the pixel value array r of the decoded image of the reference layer according to the following expressions (A1) and (A2) using the phase (xP, yP) that is a deviation from the integer position: The video encoding apparatus according to claim 1, wherein the video encoding apparatus is obtained.
t _j = (Σ _i a [xP, i] * r [xR + i−N / 2 + 1, yR + j−N / 2 + 1] + b) >> s ₁ (A1)
p = (Σ _j c [yP, j] * t _j + d + o) >> s ₂ (A2)
In the expressions (A1) and (A2), a, b, c, d are filter coefficients constituting the filter coefficient set, and s ₁ , s ₂ , o are shift values according to the bit depth of the pixel value. And rounded value. In equations (A1) and (A2), N is the number of filter taps, and i and j are filter tap positions used in the horizontal and vertical directions. The resampling means expresses the position (X, Y) of the target pixel of the decoded image of the reference layer with 1/16 pixel accuracy, and the integer position (xR, yR) of the target pixel of the decoded image of the reference layer 3. The video according to claim 2, wherein the filter coefficient set is set as a filter coefficient with respect to 16 phases by calculating the phase and the phase (xP, yP) according to the following expressions (B1) to (B4): Encoding device.
xR = X >> 4 (B1)
xP = X% 16 (B2)
yR = Y >> 4 (B3)
yP = Y% 16 (B4)   4. The video encoding according to claim 1, wherein the encoding means outputs an encoded stream obtained by multiplexing and encoding the filter coefficient set determined by the filter coefficient determining means. apparatus.   The filter coefficient determining means is provided with a restriction that the same value is used for the horizontal filter coefficients a and b used in the resampling means and the filter coefficients c and d of the vertical filter. 5. The video encoding apparatus according to claim 2, wherein a coefficient set is determined.   The filter coefficient determining means determines the filter coefficient set after providing a constraint that the filter coefficients b and d, which are constant terms used in the resampling means, are not used. Item 5. The video encoding device according to any one of Items 2 to 4. A predetermined filter coefficient for each phase of the target pixel position of the decoded image of the reference layer used in the resampling means;
The encoding means reduces the amount of data to be encoded by signaling a flag indicating whether or not the filter coefficient determination means has determined a filter coefficient having a value different from the predetermined filter coefficient. The video encoding device according to claim 2. The encoding unit reduces a data amount to be encoded by encoding a difference value between the filter coefficient determined by the encoding unit and the corresponding predetermined filter coefficient. 8. The video encoding device according to 7.   The video encoding apparatus according to claim 2, wherein the filter coefficient determination unit determines the filter coefficient set using a least square method. The filter coefficient determining unit generates a temporary inter-layer reference image by performing a filtering process on the decoded image of the reference layer using the resampling unit and the predetermined filter coefficient,
The filter coefficient determination means performs inter-layer prediction among input images of the encoding target layer with reference to the temporary inter-layer reference image and an inter prediction reference image referred to for inter prediction. Determine the inter-layer prediction region,
The filter coefficient set determination process is performed so as to minimize a sum of squares of errors between the resampled image and the input image of the encoding target layer relating to only the inter-layer prediction region. The video encoding device described. In a video decoding device that decodes hierarchically encoded data,
A prediction unit that generates a predicted image obtained by predicting a target image of a decoding target layer by performing inter-layer prediction with reference to at least a resampled image obtained by performing filtering on the decoded image of another reference layer;
Decoding means for decoding a filter coefficient set included in the hierarchically encoded data;
A video decoding device comprising: a resampling unit that performs a filtering process on a decoded image of the reference layer using the filter coefficient set decoded by the decoding unit. A computer mounted on a video encoding device that hierarchically encodes images of multiple layers,
A prediction unit that generates a prediction image obtained by predicting an input image of an encoding target layer by performing inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer;
Filter coefficient determining means for determining an optimum filter coefficient set with reference to the input image of the encoding target layer and the decoded image of the reference layer;
Resampling means for performing a filtering process on the decoded image of the reference layer using the filter coefficient set;
A video encoding program that functions as an encoding unit that encodes a residual signal between an input image of the encoding target layer and the predicted image. A computer mounted on a video decoding device that decodes hierarchically encoded data,
A prediction unit that generates a predicted image obtained by predicting a target image of a decoding target layer by performing inter-layer prediction with reference to at least a resampled image obtained by performing filtering on the decoded image of another reference layer;
Decoding means for decoding a filter coefficient set included in the hierarchically encoded data;
A video decoding program that functions as resampling means for performing filter processing on a decoded image of the reference layer, using the filter coefficient set decoded by the decoding means. In a video encoding method performed by a video encoding device that hierarchically encodes images of a plurality of layers,
A prediction unit, a filter coefficient determination unit, a resampling unit, and an encoding unit;
The prediction means generates a prediction image obtained by predicting an input image of the encoding target layer by inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer. ,
The filter coefficient determination means determines an optimal filter coefficient set with reference to the input image of the encoding target layer and the decoded image of the reference layer,
The resampling means performs a filtering process on the decoded image of the reference layer using the filter coefficient set,
The video encoding method, wherein the encoding means encodes a residual signal between an input image of the encoding target layer and the predicted image. In a video decoding method performed by a video decoding device that decodes hierarchically encoded data,
A prediction unit, a decoding unit, and a resampling unit;
The prediction unit generates a prediction image obtained by predicting a target image of a decoding target layer by performing inter-layer prediction with reference to at least a resampled image obtained by performing filtering on a decoded image of another reference layer,
The decoding means decodes a filter coefficient set included in the hierarchically encoded data,
The video decoding method, wherein the resampling means performs a filtering process on the decoded image of the reference layer using the filter coefficient set decoded by the decoding means. In a video transmission system comprising a video encoding device that hierarchically encodes images of a plurality of layers and a video decoding device that decodes encoded data hierarchically encoded by the video encoding device, the video encoding device is claimed as the video encoding device. 12. A video transmission system, wherein the video encoding device according to claim 1 is applied, and the video decoding device according to claim 11 is applied as the video decoding device.

Images