JP2013110502A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a more efficient use of memory resources while reducing influence of noise when calculating prediction values of color difference components in the LM mode.SOLUTION: An image processing apparatus comprises: a filter that generates an input value for a brightness component to be substituted into a prediction function that is for predicting the value of a first color difference component, by filtering the value of a filter tap comprising one or more brightness components at a pixel position that is common with the first color difference component, and one or more brightness components at a pixel position that is not common with the first color difference component; and a prediction unit that predicts the value of the first color difference component, by substituting, into the prediction function, the input value generated by the filter for the brightness component. The filter of this image processing apparatus is a one-dimensional filter.

Description

  The present disclosure relates to an image processing apparatus and an image processing method.

  One of the important techniques in an image encoding method for compressing the data size of a digital image is intra-screen prediction, that is, intra prediction. Intra prediction is a technique for reducing the amount of encoded information by using the correlation between adjacent blocks in an image and predicting the pixel values in a block from the pixel values of other adjacent blocks. . In the image coding system before MPEG4, only the direct current component and low frequency component of the orthogonal transform coefficient are targeted for intra prediction. In contrast, H.H. In H.264 / AVC (Advanced Video Coding), intra prediction is possible for all components. By using intra prediction, for example, an image with a gradual change in pixel value, such as an image of a blue sky, is expected to greatly improve the compression ratio.

  In intra prediction, usually, an optimal prediction mode for predicting a pixel value of a prediction target block is selected from a plurality of prediction modes. The prediction mode can typically be distinguished by the prediction direction from the reference pixel to the prediction target pixel. For example, H.M. In H.264 / AVC, when predicting a color difference component, four prediction modes of average value prediction, horizontal prediction, vertical prediction, and plane prediction can be selected. Further, H.C. In HEVC, which is being standardized as a next-generation image encoding method following H.264 / AVC, a linear model that predicts a pixel value of a color difference component using a linear function of a luminance component that is dynamically constructed as a prediction function An additional prediction mode called (LM: Linear Model) mode is expected to be employed (see Non-Patent Document 1 below).

  The density of the color difference component of the image is lower than the density of the luminance component when the chroma format is 4: 2: 0. Therefore, in the method described in Non-Patent Document 1 below, when a value of a certain color difference component is predicted in the LM mode, the prediction is performed by down-sampling two luminance component values at pixel positions common to the color difference component. An input value of the luminance component to be substituted into the function is generated. On the other hand, Non-Patent Document 2 listed below also includes the luminance component value at a pixel position that is not in common with the chrominance component in order to reduce the influence of noise and increase the accuracy of the prediction, and the luminance component to the prediction function It is proposed to generate an input value of 2 by a two-dimensional filter of 3 × 2 taps. Non-Patent Document 3 below proposes to perform downsampling of the reference pixel of the adjacent block on the left when determining the coefficient of the prediction function in the LM mode with a 4-tap filter instead of 2-tap.

Jianle Chen, et al. “CE6.a.4: Chroma intra prediction by reconstructed luma samples” (JCTVC-E266, March 2011) Akira Minezawa, et al. “An improvement to chroma intra prediction from luma” (JCTVC-F173, July 2011) Yongbing Lin, et al. “Modifications to intra frame coding” (JCTVC-G119, November 2011)

  However, in the method proposed by Non-Patent Document 2, the luminance component referred to when generating one input value to the prediction function exists over a plurality of rows and a plurality of columns. For this reason, in order to calculate the prediction value of the color difference component in the LM mode, it is required to temporarily store the luminance component value of the prediction target block in a plurality of line memories. On the other hand, in HEVC, it is assumed that the inverse orthogonal transform provided in the previous stage of intra prediction is implemented by a method of sequentially executing a one-dimensional horizontal transformation process and a one-dimensional vertical transformation process. Has been.

  Therefore, in calculating the predicted value of the color difference component in the LM mode, if only a luminance component value stored in a single line memory is referred to, a sequential process in units of lines is adopted to save memory resources. It can be used more efficiently.

  According to the present disclosure, the value of a filter tap including one or more luminance components at a pixel position common to the first color difference component and one or more luminance components at a pixel position not common to the first color difference component is obtained. A filter that generates an input value of a luminance component to be substituted into a prediction function for predicting a value of the first color difference component by filtering, and the input value of the luminance component generated by the filter A prediction unit that predicts the value of the first chrominance component by substituting it into a prediction function, and the filter is a one-dimensional filter.

  The above-described image processing apparatus can be applied to both an image encoding apparatus and an image decoding apparatus.

  Further, according to the present disclosure, a filter tap including one or more luminance components at a pixel position common to the first color difference component and one or more luminance components at a pixel position not common to the first color difference component. By filtering the value with a one-dimensional filter, an input value of a luminance component to be substituted into a prediction function for predicting the value of the first color difference component is generated, and generated by the one-dimensional filter An image processing method including predicting a value of the first color difference component by substituting the input value of the luminance component into the prediction function is provided.

  According to the technology according to the present disclosure, it is possible to more efficiently use memory resources while reducing the influence of noise when calculating the predicted value of the color difference component in the LM mode.

It is a block diagram which shows an example of a structure of the image coding apparatus which concerns on one Embodiment. It is a block diagram which shows an example of a detailed structure of the intra estimation part of the image coding apparatus which concerns on one Embodiment. It is explanatory drawing for demonstrating the candidate of the prediction direction which can be selected according to the angle intra prediction method of HEVC. It is explanatory drawing for demonstrating calculation of the reference pixel value in the angle intra prediction method of HEVC. It is explanatory drawing for demonstrating LM mode. It is explanatory drawing for demonstrating the existing 1st method of the downsampling of the luminance component in LM mode. It is explanatory drawing for demonstrating the existing 2nd method of the downsampling of the luminance component in LM mode. It is explanatory drawing for demonstrating the order of the sequential inverse orthogonal transformation in an inverse orthogonal transformation part. It is explanatory drawing for demonstrating the 1st method of the downsampling of the luminance component which concerns on one Embodiment. It is explanatory drawing for demonstrating the 2nd method of the downsampling of the luminance component which concerns on one Embodiment. It is explanatory drawing for demonstrating the 1st example of switching of the number of filter taps according to the angle of the prediction direction of a luminance component. It is explanatory drawing for demonstrating the 2nd example of switching of the number of filter taps according to the angle of the prediction direction of a luminance component. It is a flowchart which shows an example of the flow of the intra prediction process which concerns on one Embodiment. It is a flowchart which shows an example of the detailed flow of the LM mode prediction process shown in FIG. It is a flowchart which shows the 1st example of the detailed flow of the downsampling process shown in FIG. It is a flowchart which shows the 2nd example of the detailed flow of the downsampling process shown in FIG. 12 is a flowchart illustrating a third example of a detailed flow of the downsampling process illustrated in FIG. 11. It is a block diagram which shows an example of a structure of the image decoding apparatus which concerns on one Embodiment. It is a block diagram which shows an example of a detailed structure of the intra estimation part of the image decoding apparatus which concerns on one Embodiment. It is a flowchart which shows an example of the flow of the intra prediction process at the time of the decoding which concerns on one Embodiment. It is a block diagram which shows an example of a detailed structure of the intra estimation part which concerns on a 1st modification. It is explanatory drawing for demonstrating LM mode in case a chroma format is 4: 2: 2. It is explanatory drawing for demonstrating LM mode in case a chroma format is 4: 4: 4. It is a block diagram which shows an example of a schematic structure of a television apparatus. It is a block diagram which shows an example of a schematic structure of a mobile telephone. It is a block diagram which shows an example of a schematic structure of a recording / reproducing apparatus. It is a block diagram which shows an example of a schematic structure of an imaging device.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be given in the following order.
1. 1. Configuration example of image encoding device according to one embodiment 2. Processing flow during encoding according to an embodiment 3. Configuration example of image decoding apparatus according to one embodiment 4. Process flow during decoding according to one embodiment Modification 6 Application example 7. Summary

<1. Configuration Example of Image Encoding Device According to One Embodiment>
[1-1. Overall configuration example]
FIG. 1 is a block diagram illustrating an example of a configuration of an image encoding device 10 according to an embodiment. Referring to FIG. 1, an image encoding device 10 includes an A / D (Analogue to Digital) conversion unit 11, a rearrangement buffer 12, a subtraction unit 13, an orthogonal transformation unit 14, a quantization unit 15, a lossless encoding unit 16, The accumulation buffer 17, rate control unit 18, inverse quantization unit 21, inverse orthogonal transform unit 22, addition unit 23, deblock filter 24, frame memory 25, selectors 26 and 27, motion search unit 30, and intra prediction unit 40 Prepare.

  The A / D conversion unit 11 converts an image signal input in analog format into image data in digital format, and outputs a series of digital image data to the rearrangement buffer 12.

  The rearrangement buffer 12 rearranges images included in a series of image data input from the A / D conversion unit 11. The rearrangement buffer 12 rearranges the images according to the GOP (Group of Pictures) structure related to the encoding process, and then outputs the rearranged image data to the subtraction unit 13, the motion search unit 30, and the intra prediction unit 40. To do.

  Image data input from the rearrangement buffer 12 and predicted image data input from the motion search unit 30 or the intra prediction unit 40 described later are supplied to the subtraction unit 13. The subtraction unit 13 calculates prediction error data that is the difference between the image data input from the rearrangement buffer 12 and the predicted image data, and outputs the calculated prediction error data to the orthogonal transform unit 14.

  The orthogonal transform unit 14 performs orthogonal transform on the prediction error data input from the subtraction unit 13. In HEVC, the orthogonal transform by the orthogonal transform unit 14 is a two-dimensional discrete cosine transform (DCT) for each transform unit (TU). The two-dimensional DCT can be performed for each line in a method in which one of the vertical one-dimensional DCT and the horizontal one-dimensional DCT is sequentially executed first and the other one after the other sequentially. . In the present embodiment, it is assumed that one-dimensional DCT in the vertical direction is performed first. The orthogonal transform unit 14 outputs transform coefficient data acquired by the orthogonal transform process to the quantization unit 15.

  The quantization unit 15 is supplied with transform coefficient data input from the orthogonal transform unit 14 and a rate control signal from a rate control unit 18 described later. The quantizing unit 15 quantizes the transform coefficient data and outputs the quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless encoding unit 16 and the inverse quantization unit 21. Further, the quantization unit 15 changes the bit rate of the quantized data input to the lossless encoding unit 16 by switching the quantization parameter (quantization scale) based on the rate control signal from the rate control unit 18. Let

  The lossless encoding unit 16 performs a lossless encoding process on the quantized data input from the quantization unit 15 to generate an encoded stream. The lossless encoding by the lossless encoding unit 16 may be variable length encoding or arithmetic encoding, for example. In addition, the lossless encoding unit 16 multiplexes information related to intra prediction or information related to inter prediction input from the switch 27 in the header area of the encoded stream. Then, the lossless encoding unit 16 outputs the generated encoded stream to the accumulation buffer 17.

  The accumulation buffer 17 temporarily accumulates the encoded stream input from the lossless encoding unit 16 using a storage medium such as a semiconductor memory. Then, the accumulation buffer 17 outputs the accumulated encoded stream to a transmission unit (not shown) (for example, a communication interface or a connection interface with a peripheral device) at a rate corresponding to the bandwidth of the transmission path.

  The rate control unit 18 monitors the free capacity of the accumulation buffer 17. Then, the rate control unit 18 generates a rate control signal according to the free capacity of the accumulation buffer 17 and outputs the generated rate control signal to the quantization unit 15. For example, the rate control unit 18 generates a rate control signal for reducing the bit rate of the quantized data when the free capacity of the storage buffer 17 is small. For example, when the free capacity of the accumulation buffer 17 is sufficiently large, the rate control unit 18 generates a rate control signal for increasing the bit rate of the quantized data.

  The inverse quantization unit 21 performs an inverse quantization process on the quantized data input from the quantization unit 15. Then, the inverse quantization unit 21 outputs transform coefficient data acquired by the inverse quantization process to the inverse orthogonal transform unit 22.

  The inverse orthogonal transform unit 22 restores the prediction error data by performing an inverse orthogonal transform process on the transform coefficient data input from the inverse quantization unit 21. The inverse orthogonal transform by the inverse orthogonal transform unit 22 is a two-dimensional inverse discrete cosine transform (IDCT) for each TU. The two-dimensional IDCT can be substantially performed for each line by a method of sequentially executing the one-dimensional IDCT in the vertical direction and the one-dimensional IDCT in the horizontal direction in the reverse order of the orthogonal transformation. . In the present embodiment, it is assumed that horizontal one-dimensional IDCT is performed first. Then, the inverse orthogonal transform unit 22 outputs the restored prediction error data to the addition unit 23.

  The adding unit 23 reconstructs decoded image data by adding the restored prediction error data input from the inverse orthogonal transform unit 22 and the predicted image data input from the motion search unit 30 or the intra prediction unit 40. (Reconstruct). Then, the addition unit 23 outputs the reconstructed decoded image data to the deblock filter 24 and the frame memory 25. Further, the adding unit 23 outputs the decoded image data of the luminance component used when predicting the color difference component in the LM mode to the intra prediction unit 40.

  The deblocking filter 24 performs a filtering process for reducing block distortion that occurs when an image is encoded. The deblocking filter 24 removes block distortion by filtering the decoded image data input from the adding unit 23, and outputs the decoded image data after filtering to the frame memory 25.

  The frame memory 25 stores the decoded image data input from the adder 23 and the decoded image data after filtering input from the deblock filter 24 using a storage medium.

  The selector 26 reads out the decoded image data after filtering used for inter prediction from the frame memory 25 and supplies the read out decoded image data to the motion search unit 30 as reference image data. The selector 26 reads out decoded image data before filtering used for intra prediction from the frame memory 25 and supplies the read decoded image data to the intra prediction unit 40 as reference image data.

  In the inter prediction mode, the selector 27 outputs predicted image data as a result of the inter prediction output from the motion search unit 30 to the subtraction unit 13 and outputs information related to the inter prediction to the lossless encoding unit 16. Further, in the intra prediction mode, the selector 27 outputs predicted image data as a result of the intra prediction output from the intra prediction unit 40 to the subtraction unit 13 and outputs information related to the intra prediction to the lossless encoding unit 16. . The selector 27 switches between the inter prediction mode and the intra prediction mode according to the size of the cost function value output from the motion search unit 30 and the intra prediction unit 40.

  The motion search unit 30 performs inter prediction processing (interframe prediction processing) based on the image data to be encoded (original image data) input from the rearrangement buffer 12 and the decoded image data supplied via the selector 26. )I do. For example, the motion search unit 30 evaluates the prediction result in each prediction mode using a predetermined cost function. Next, the motion search unit 30 selects the prediction mode with the smallest cost function value, that is, the prediction mode with the highest compression rate, as the optimum prediction mode. In addition, the motion search unit 30 generates predicted image data according to the optimal prediction mode. Then, the motion search unit 30 outputs prediction mode information indicating the selected optimal prediction mode, information on inter prediction including motion vector information and reference image information, a cost function value, and predicted image data to the selector 27.

  The intra prediction unit 40 performs an intra prediction process for each block set in the image based on the original image data input from the rearrangement buffer 12 and the decoded image data as reference image data supplied from the frame memory 25. I do. Then, the intra prediction unit 40 outputs information related to intra prediction including prediction mode information representing the optimal prediction mode, a cost function value, and predicted image data to the selector 27. The prediction modes selectable by the intra prediction unit 40 include a linear model (LM) mode for color difference components in addition to the existing intra prediction modes. Unlike the LM mode described in Non-Patent Document 2, the LM mode in the present embodiment has a feature that a one-dimensional filter is used for downsampling of a luminance component. Such intra prediction processing by the intra prediction unit 40 will be described in detail later.

[1-2. Configuration example of intra prediction unit]
FIG. 2 is a block diagram illustrating an example of a detailed configuration of the intra prediction unit 40 of the image encoding device 10 illustrated in FIG. 1. Referring to FIG. 2, the intra prediction unit 40 includes a prediction control unit 41, a coefficient calculation unit 42, a luminance component memory 43, a filter 44, a prediction unit 45, and a mode determination unit 46.

  The prediction control unit 41 controls intra prediction processing in the intra prediction unit 40. For example, the prediction control unit 41 executes an intra prediction process for the luminance component (Y) and an intra prediction process for the color difference components (Cb, Cr) for each coding unit (Coding Unit: CU). In the intra prediction process for the luminance component, the prediction control unit 41 causes the prediction unit 45 to generate a prediction pixel value of each pixel in a plurality of prediction modes, and causes the mode determination unit 46 to determine the optimal prediction mode of the luminance component. The arrangement of prediction units (PUs) in the CU is also determined. In the intra prediction process for the color difference component, the prediction control unit 41 causes the prediction unit 45 to generate a prediction pixel value of each pixel in a plurality of prediction modes, and causes the mode determination unit 46 to determine an optimal prediction mode for the color difference component.

  The prediction mode candidates for the luminance component may include prediction modes associated with various prediction directions. HEVC uses the Angular Intra Prediction method described in “Description of video coding technology proposal by Tandberg, Nokia, Ericsson” (Kemal Ugur, et al., JCTVC-A119, April 2010). Is done. FIG. 3 is an explanatory diagram for explaining prediction direction candidates that can be selected according to the angle intra prediction method. The pixel P1 illustrated in FIG. 3 is a prediction target pixel. The shaded pixels around the block to which the pixel P1 belongs are reference pixels. When the block size is 4 × 4 pixels, 17 types of prediction directions (corresponding to prediction modes) connecting the reference pixel and the prediction target pixel, which are indicated by solid lines (both thick and thin lines) in the figure, Can be selected (in addition to DC prediction). When the block size is 8 × 8 pixels, 16 × 16 pixels, or 32 × 32 pixels, prediction types corresponding to 33 types of prediction directions (shown by dotted lines and solid lines (both thick lines and thin lines)) are shown. Mode) can be selected (in addition to DC prediction and planar prediction). When the block size is 64 × 64 pixels, two types of prediction directions (corresponding to prediction modes) indicated by bold lines in the figure can be selected (in addition to DC prediction). In the angle intra prediction method, the angle resolution in the prediction direction is high. For example, in the case of 8 × 8 pixels, the angle difference between adjacent prediction directions is 180 degrees / 32 = 5.625 degrees. Therefore, a reference pixel value with 1/8 pixel accuracy as shown in FIG. 4 is calculated as the reference pixel value and can be used for calculation of the predicted pixel value.

  Prediction mode candidates for the color difference component may also include prediction modes associated with various prediction directions. Further, the prediction mode candidates for the color difference components include the LM mode described above. The LM mode is a prediction mode in which a pixel value of a color difference component is predicted using a linear function of a luminance component that is dynamically constructed as a prediction function. The prediction function used in the LM mode may be a linear linear function described in Non-Patent Document 1 as follows:

In Expression (1), Re _L ′ (x, y) represents a value after down-sampling of the luminance component of the decoded image (so-called reconstructed image). The downsampling of the luminance component can be performed when the density of the color difference component is different from the density of the luminance component depending on the chroma format.

For example, referring to FIG. 5, when the chroma format is 4: 2: 0, the luminance component (Luma) and the corresponding color difference component (Chroma) in one PU having a size of 16 × 16 pixels are rounded. It is conceptually indicated by a sign. The density of the luminance component is twice the density of the color difference component in each of the horizontal direction and the vertical direction. Circles located around the PU and filled in the figure indicate reference pixels that are referred to when calculating the coefficients α and β of the prediction function. The circles shaded with diagonal lines on the right in the figure indicate the downsampled luminance components. By substituting the value of the luminance component down-sampled in this way into Re _L ′ (x, y) on the right side of the prediction function, the predicted value of the color difference component at the common pixel position is calculated. Similarly, the reference pixels can be down-sampled.

  When the chroma format is 4: 2: 0, one luminance component input value (a value substituted into the prediction function) is generated by downsampling for every 2 × 2 luminance components as in the example of FIG. Is done. According to Non-Patent Document 2, downsampling includes one or more luminance components at pixel positions that are common to individual color difference components, and one or more luminance components at pixel positions that are not common to the color difference components. This is done by filtering the value of the filter tap using a two-dimensional filter. Here, the luminance component at a pixel position “in common” with a certain color difference component is the pixel position (2x, y) of the luminance component with respect to the pixel position (x, y) of the color difference component when the chroma format is 4: 2: 0. 2y), (2x + 1, 2y), (2x, 2y + 1) and (2x + 1, 2y + 1).

FIG. 6A is an explanatory diagram for explaining the existing first method of downsampling described in Non-Patent Document 2. In the upper left of FIG. 6A, the color difference component Cr _{1,1 at} the pixel position (1,1) is shown. The predicted value of the color difference component Cr _1,1 is calculated by substituting the luminance component input value IL _1,1 into the prediction function. The luminance component input value IL _1,1 includes luminance components Lu _2,2 , Lu _3,2 , Lu _2,3 , Lu _3,3 and color difference components Cr _1,3 at pixel positions common to the color difference components Cr _{1,1 .} It can be generated by filtering the value of a 3 × 2 filter tap including the luminance components Lu ₁ , ₂ , Lu ₁ , ₃ at pixel positions not common to ₁ . The numbers shown in the circles of the filter taps in the figure are filter coefficients that are multiplied by each filter tap. Thus, by including the luminance component around the pixel position common to each color difference component in the filter tap at the time of downsampling, the influence of noise of the luminance component is reduced, and the accuracy of intra prediction in the LM mode is reduced. Is increased.

FIG. 6B is an explanatory diagram for describing an existing second method of downsampling. In the second method, the input value IL _1,1 of the luminance component is the same as the luminance components Lu _3,2 , Lu _3,3 and the color difference component Cr _1,1 at the same pixel position as the color difference component Cr _1,1. It can be generated by filtering the values of 2 × 2 filter taps including the luminance components Lu ₁ , ₂ , Lu ₁ , ₃ at the non-pixel positions. The filter coefficients are as shown in the figure.

  By the way, as described above, the inverse orthogonal transform unit 22 performs the inverse orthogonal transform for each block after the one-dimensional IDCT in the horizontal direction and the one-dimensional IDCT in the vertical direction first. . For example, for the 8 × 8 pixel block shown in FIG. 7, the inverse orthogonal transform unit 22 sequentially performs one-dimensional IDCT for the horizontal lines R0 to R7, and then sequentially performs 1 for the vertical lines C0 to C7. Perform dimension IDCT. Accordingly, the decoded image data is input to the intra prediction unit 40 via the addition unit 23 in units of lines in the vertical direction. Therefore, in the method using the two-dimensional filter as described with reference to FIGS. 6A and 6B, it is required to store the decoded image data (reconstructed pixel value) of the luminance component in a memory over a plurality of lines. However, in the intra prediction of the color difference component, the value of the luminance component is not referred to in a prediction mode other than the LM mode. Therefore, consuming a large amount of line memory only for the LM mode is not desirable from the viewpoint of efficient use of memory resources. Therefore, in this embodiment, downsampling when generating an input value of a luminance component in the LM mode is performed using a one-dimensional filter.

FIG. 8A is an explanatory diagram for describing a first method of downsampling of luminance components according to the present embodiment. In the upper left of FIG. 8A, the color difference component Cr _{1,1 at} the pixel position (1,1) is shown. The predicted value of the color difference component Cr _1,1 is calculated by substituting the luminance component input value IL _1,1 into the prediction function. Luminance component of the input values _{IL 1, 1} of the luminance component, the luminance component _Lu 2, 2 of the pixel positions in common with the color difference components _{Cr 1, 1,} pixel position not common and _{Lu 2,3} and chrominance components _{Cr 1, 1} It can be generated by filtering the values of four filter taps including Lu _2,1 and Lu _2,4 with a one-dimensional filter. The numbers shown in the circles of the filter taps in the figure are filter coefficients that are multiplied by each filter tap.

FIG. 8B is an explanatory diagram for explaining a second method of downsampling of the luminance component according to the present embodiment. In the example of FIG. 8B, the number of filter taps is expanded to six. Luminance component of the input values _{IL 1, 1} of the luminance component, the luminance component _Lu 2, 2 of the pixel positions in common with the color difference components _{Cr 1, 1,} pixel position not common and _{Lu 2,3} and chrominance components _{Cr 1, 1} It can be generated by filtering the values of six filter taps including Lu _2,0 , Lu _2,1 , Lu _2,4 , Lu _2,5 with a one-dimensional filter.

  In this way, by adopting a configuration in which the input value of the luminance component is generated using a one-dimensional filter, only the decoded image data of the luminance component for one line or less is stored in the memory, and the LM mode can be used. Intra prediction of color difference components is possible. Note that the combination of the number of taps and the filter coefficient of the one-dimensional filter is not limited to the example illustrated in FIGS. 8A and 8B, and any other combination may be used.

  Returning to FIG. 2, the description of the configuration of the intra prediction unit 40 is continued.

  The coefficient calculation unit 42, the luminance component memory 43, and the filter 44 are provided for realizing the LM mode using the one-dimensional filter as described above.

  The coefficient calculation unit 42 calculates the coefficient of the prediction function used by the prediction unit 45 in the LM mode with reference to pixels around the PU to which the prediction target pixel belongs, that is, reference pixels. The coefficient α of the prediction function shown in the equation (1) is calculated according to the following equation (2). The coefficient β is calculated according to the following equation (3).

  In Expressions (2) and (3), I represents the number of reference pixels. For example, if the block size of the color difference component is 8 × 8 pixels and the left eight reference pixels and the upper eight reference pixels are both available, I = 8 + 8 = 16.

  Decoded image data including luminance component values reconstructed using the prediction error data is input from the adding unit 23 to the luminance component memory 43 in units of lines. The line here is a line along a direction to be executed later in the IDCT in the horizontal direction and the vertical direction in the inverse orthogonal transform unit 22, and corresponds to a column of blocks in the present embodiment. The luminance component memory 43 temporarily stores the decoded image data of the luminance component in units of lines.

The filter 44 extracts filter taps as illustrated in FIG. 8A or FIG. 8B from the luminance component memory 43 under the control of the prediction control unit 41. Then, the filter 44 filters the extracted filter tap value to input an input value to the prediction function for predicting the value of the color difference component (for example, the input value IL _1,1 illustrated in FIGS. 8A and 8B). ) Is generated. The filter 44 sequentially generates the input value of each pixel position (x, y) in this way, and outputs the generated input value to the prediction unit 45.

  The prediction control unit 41 may switch the number of filter taps of the filter 44 according to the prediction direction of the intra prediction selected as the optimal prediction mode for the luminance component at the pixel position common to each color difference component. More specifically, in the first example, the prediction control unit 41 determines the number of filter taps when the angle between the prediction direction and the direction along the filter tap (hereinafter referred to as tap direction) is smaller. It can be set to a larger number. In the second example, the prediction control unit 41 sets the number of filter taps to the first number when the prediction direction overlaps with the tap direction, and filters when the prediction direction does not overlap with the tap direction. The number of taps may be set to a second number that is smaller than the first number.

  FIG. 9A is an explanatory diagram for describing a first example of switching the number of filter taps according to the angle of the prediction direction of the luminance component. Here, a threshold Th1 = 45 [degrees] is defined as an angle threshold for switching the number of filter taps. Referring to FIG. 9A, seven types of prediction directions D1 to D7 for the pixel P1 are illustrated. The angle formed with respect to the horizontal direction of the prediction directions D2, D5, and D6 is 45 degrees and is equal to the threshold value Th1. The angle formed with respect to the horizontal direction of the prediction directions D1 and D7 exceeds the threshold value Th1. The angle formed with respect to the horizontal direction of the prediction directions D3 and D4 is less than the threshold value Th1. In this case, for example, the prediction control unit 41 has six (or four) filter taps for the prediction directions D1, D2, D5, D6, and D7, and four filter taps for the prediction directions D3 and D4 ( Or two). Here, an example has been described in which the number of filter taps is switched between two types of filter tap number candidates. However, the present invention is not limited to this example, and the number of filter taps may be switched between three or more types of filter number candidates using two or more angle thresholds. For example, the number of filter taps may be switched between 2, 4, and 6 using two angle thresholds Th1 = 45 [degrees] and Th2 = 67.5 [degrees].

  FIG. 9B is an explanatory diagram for describing a second example of switching the number of filter taps according to the angle of the prediction direction of the luminance component. Here, it is assumed that the number of filter taps is switched depending on whether or not the angle formed with respect to the horizontal direction of the prediction direction is 90 degrees. Referring to FIG. 9B, three types of prediction directions D0 to D2 for the pixel P1 are illustrated. The angle formed with respect to the horizontal direction of the prediction direction D0 is 90 degrees. In this case, the prediction control unit 41 can set the number of filter taps of the filter 44 for the pixel P1 to 6 (or 4). On the other hand, when the angle is not 90 degrees as in the prediction directions D1 and D2, the prediction control unit 41 can set the number of filter taps of the filter 44 for the pixel P1 to four (or two). .

  By adaptively switching the number of filter taps, the tap length is increased when the correlation of the image along the tap direction is strong, and the tap length is shortened when the correlation of the image along the tap direction is weak. be able to. Thereby, the accuracy of prediction can be improved by changing the input value to the prediction function of the LM mode in accordance with the correlation of the images and enhancing the robustness against noise.

  The filter coefficient of the filter 44 may be the same filter coefficient as the interpolation filter used in motion compensation in the motion search unit 30. By sharing the filter coefficient in this way, it is possible to reduce the amount of memory resources required to store the filter coefficient.

  The prediction unit 45 predicts the pixel value of the luminance component and the pixel value of the color difference component of the pixel to be predicted according to various prediction mode candidates under the control of the prediction control unit 41. The luminance component prediction mode candidates may include prediction modes associated with various prediction directions in the angle intra prediction method described above. The color difference component prediction mode candidates include the LM mode described above. In the LM mode, the prediction unit 45 predicts the value of each color difference component by substituting the input value of the luminance component generated by the filter 44 into a prediction function having a coefficient calculated by the coefficient calculation unit 42. Intra prediction by the prediction unit 45 in other prediction modes may be performed in the same manner as an existing method. The prediction unit 45 outputs predicted image data generated as a result of prediction to the mode determination unit 46 for each prediction mode.

  The mode determination unit 46 calculates a cost function value for each prediction mode based on the original image data input from the rearrangement buffer 12 and the predicted image data input from the prediction unit 45. Then, the mode determination unit 46 determines an optimal prediction mode for each color component based on the calculated cost function value. Then, the mode determination unit 46 outputs information related to intra prediction including prediction mode information indicating the determined optimal prediction mode, cost function values, and predicted image data including predicted pixel values of each color component to the selector 27.

<2. Flow of processing during encoding according to one embodiment>
Next, the flow of processing during encoding will be described with reference to FIGS.

[2-1. Intra prediction process]
FIG. 10 is a flowchart illustrating an example of the flow of an intra prediction process at the time of encoding by the intra prediction unit 40 having the configuration illustrated in FIG. The intra prediction process shown in FIG. 10 is executed for each block in the image as a processing target.

  First, the prediction control unit 41 determines whether or not the color component to be processed is a luminance component (step S110). If the color component to be processed is a luminance component, the process proceeds to step S120. On the other hand, if the color component to be processed is a color difference component, the process proceeds to step S150.

  In step S120, the prediction control unit 41 causes the prediction unit 45 to generate a prediction pixel value of the luminance component according to a plurality of prediction mode candidates, and causes the mode determination unit 46 to determine the optimal prediction mode of the luminance component (step S120). . As a result, the optimal prediction mode of the luminance component for each PU is determined, and predicted image data is generated. Next, the prediction control unit 41 stores decoded image data, which is the sum of the predicted image data of the luminance component and the prediction error data output from the inverse orthogonal transform unit 22, in the luminance component memory 43 in units of lines (step) S130).

  Thus, after the intra prediction about a luminance component is complete | finished, the intra prediction about the color difference component of the same block is performed.

  In step S150, the prediction control unit 41 causes the prediction unit 45 to perform prediction processing in the LM mode, which is one of prediction mode candidates (step S150). The LM mode prediction process by the prediction unit 45 here will be further described later. Next, the prediction control unit 41 causes the prediction unit 45 to generate a prediction pixel value of the color difference component according to a plurality of prediction mode candidates other than the LM mode (step S170). And the prediction control part 41 makes the mode determination part 46 determine the optimal prediction mode of a color difference component (step S180). As a result, the optimum prediction mode of the color difference component for each PU is determined.

  When the intra prediction is performed for all the color components of the processing target block, the intra prediction process illustrated in FIG. 10 ends (step S190).

[2-2. LM mode prediction process]
FIG. 11 is a flowchart illustrating an example of a detailed flow of the LM mode prediction process illustrated in FIG. 10. The LM mode prediction process shown in FIG. 11 is executed for each line in each block.

  Referring to FIG. 11, first, the coefficient calculation unit 42 uses the coefficients α and β of the prediction function used by the prediction unit 45 in the LM mode as the reference pixel values included in the reference image data input from the frame memory 25. (Step S151).

  Further, the filter 44 performs a downsampling process using the values of the plurality of luminance components stored in the luminance component memory 43, and generates an input value to the prediction function (step S152). The downsampling process by the filter 44 here is further demonstrated later.

  Next, the predicting unit 45 substitutes the input value of the luminance component generated as a result of the downsampling process into a prediction function having the coefficient calculated by the coefficient calculating unit 42, thereby calculating the predicted value of each color difference component. Generate (step S153).

  The processes in steps S151 to S153 can be repeated until prediction is completed for all pixels in the processing target line (step S154).

[2-3. Downsampling process]
FIG. 12A is a flowchart showing a first example of a detailed flow of the downsampling process shown in FIG. In the first example, the number of filter taps is fixedly set.

  Referring to FIG. 12A, first, the filter 44 extracts a filter tap of a one-dimensional filter as illustrated in FIG. 8A or FIG. 8B from the luminance component memory 43 (step S161). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S162).

  FIG. 12B is a flowchart illustrating a second example of a detailed flow of the downsampling process illustrated in FIG. 11. In the second example, the number of filter taps is adaptively switched between two values.

  Referring to FIG. 12B, first, the prediction control unit 41 sets an angle formed with respect to the horizontal direction of the prediction direction of intra prediction selected for the luminance component at the pixel position common to the color difference component to be processed as a predetermined threshold value. Compare (step S160). Here, for example, if the angle is equal to or greater than the threshold, the process proceeds to step S161. On the other hand, if the angle is below the threshold, the process proceeds to step S164.

  In step S161, the filter 44 extracts filter taps of the one-dimensional filter having the tap number N1 from the luminance component memory 43 (step S161). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S162).

  In step S164, the filter 44 extracts the filter taps of the one-dimensional filter having the tap number N2 (N2 <N1) from the luminance component memory 43 (step S164). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S165).

  FIG. 12C is a flowchart illustrating a third example of a detailed flow of the downsampling process illustrated in FIG. 11. In the third example, the number of filter taps is adaptively switched between three values.

  Referring to FIG. 12C, first, the prediction control unit 41 sets the angle formed with respect to the horizontal direction of the prediction direction of intra prediction selected for the luminance component at the pixel position common to the color difference component to be processed as the first threshold value. Compare with Th1 (step S160). Here, for example, when the angle is equal to or greater than the first threshold Th1, the process proceeds to step S161. On the other hand, if the angle is less than the first threshold Th1, the process proceeds to step S163.

  In step S161, the filter 44 extracts filter taps of the one-dimensional filter having the tap number N1 from the luminance component memory 43 (step S161). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S162).

  In step S163, the prediction control unit 41 further compares the angle with a second threshold Th2 (Th2 <Th1) (step S163). Here, for example, when the angle is equal to or larger than the second threshold Th2, the process proceeds to step S164. On the other hand, if the angle is less than the second threshold Th2, the process proceeds to step S166.

  In step S164, the filter 44 extracts the filter taps of the one-dimensional filter having the tap number N2 (N2 <N1) from the luminance component memory 43 (step S164). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S165).

  In step S166, the filter 44 extracts filter taps of the one-dimensional filter having the tap number N3 (N3 <N2) from the luminance component memory 43 (step S166). Next, the filter 44 filters the extracted filter tap with a predetermined filter coefficient to generate an input value of the luminance component to the prediction function (step S167).

  By the encoding process as described in this section, the memory size of the luminance component memory 43 provided for the introduction of the LM mode can be reduced, and the memory resources of the image encoding device 10 can be efficiently used. It becomes. Such memory resource saving is also achieved in the image decoding device 60 described in the following sections.

<3. Configuration Example of Image Decoding Device According to One Embodiment>
FIG. 13 is a block diagram illustrating an example of the configuration of the image decoding device 60 according to an embodiment. Referring to FIG. 13, the image decoding device 60 includes an accumulation buffer 61, a lossless decoding unit 62, an inverse quantization unit 63, an inverse orthogonal transform unit 64, an addition unit 65, a deblock filter 66, a rearrangement buffer 67, a D / A A (Digital to Analogue) conversion unit 68, a frame memory 69, selectors 70 and 71, a motion compensation unit 80, and an intra prediction unit 90 are provided.

  The accumulation buffer 61 temporarily accumulates the encoded stream input via the transmission path using a storage medium.

  The lossless decoding unit 62 decodes the encoded stream input from the accumulation buffer 61 according to the encoding method used at the time of encoding. In addition, the lossless decoding unit 62 decodes information multiplexed in the header area of the encoded stream. The information multiplexed in the header area of the encoded stream may include the above-described information related to inter prediction and information related to intra prediction. The lossless decoding unit 62 outputs information related to inter prediction to the motion compensation unit 80. Further, the lossless decoding unit 62 outputs information related to intra prediction to the intra prediction unit 90.

  The inverse quantization unit 63 inversely quantizes the quantized data decoded by the lossless decoding unit 62. The inverse orthogonal transform unit 64 restores the prediction error data by performing inverse orthogonal transform on the transform coefficient data input from the inverse quantization unit 63 according to the orthogonal transform method used at the time of encoding. The inverse orthogonal transform by the inverse orthogonal transform unit 64 is a two-dimensional IDCT for each TU. The two-dimensional IDCT can be substantially performed for each line by a method of sequentially executing the one-dimensional IDCT in the vertical direction and the one-dimensional IDCT in the horizontal direction in the reverse order of the orthogonal transformation. . In the present embodiment, it is assumed that horizontal one-dimensional IDCT is performed first. Then, the inverse orthogonal transform unit 64 outputs the generated prediction error data to the addition unit 65.

  The adding unit 65 reconstructs decoded image data by adding the prediction error data input from the inverse orthogonal transform unit 64 and the predicted image data input from the selector 71. Then, the adding unit 65 outputs the reconstructed decoded image data to the deblock filter 66 and the frame memory 69. Further, the adding unit 65 outputs the decoded image data of the luminance component used when predicting the color difference component in the LM mode to the intra prediction unit 90.

  The deblock filter 66 removes block distortion by filtering the decoded image data input from the adder 65 and outputs the decoded image data after filtering to the rearrangement buffer 67 and the frame memory 69.

  The rearrangement buffer 67 generates a series of time-series image data by rearranging the images input from the deblocking filter 66. Then, the rearrangement buffer 67 outputs the generated image data to the D / A conversion unit 68.

  The D / A converter 68 converts the digital image data input from the rearrangement buffer 67 into an analog image signal. Then, the D / A conversion unit 68 displays an image by outputting an analog image signal to a display (not shown) connected to the image decoding device 60, for example.

  The frame memory 69 stores the decoded image data before filtering input from the adding unit 65 and the decoded image data after filtering input from the deblocking filter 66 using a storage medium.

  The selector 70 switches the output destination of the image data from the frame memory 69 between the motion compensation unit 80 and the intra prediction unit 90 for each block in the image according to the mode information acquired by the lossless decoding unit 62. . For example, when the inter prediction mode is designated, the selector 70 outputs the decoded image data after filtering supplied from the frame memory 69 to the motion compensation unit 80 as reference image data. Further, when the intra prediction mode is designated, the selector 70 outputs the decoded image data before filtering supplied from the frame memory 69 to the intra prediction unit 90 as reference image data.

  The selector 71 switches the output source of the predicted image data to be supplied to the addition unit 65 between the motion compensation unit 80 and the intra prediction unit 90 according to the mode information acquired by the lossless decoding unit 62. For example, when the inter prediction mode is designated, the selector 71 supplies the predicted image data output from the motion compensation unit 80 to the adding unit 65. In addition, when the intra prediction mode is designated, the selector 71 supplies the predicted image data output from the intra prediction unit 90 to the adding unit 65.

  The motion compensation unit 80 performs motion compensation processing based on the inter prediction information input from the lossless decoding unit 62 and the reference image data from the frame memory 69 to generate predicted image data. Then, the motion compensation unit 80 outputs the generated predicted image data to the selector 71.

  The intra prediction unit 90 performs intra prediction processing based on the information related to intra prediction input from the lossless decoding unit 62 and the reference image data from the frame memory 69, and generates predicted image data. Then, the intra prediction unit 90 outputs the generated predicted image data to the selector 71. Such intra prediction processing by the intra prediction unit 90 will be described in detail later.

[3-2. Configuration example of intra prediction unit]
FIG. 14 is a block diagram illustrating an example of a detailed configuration of the intra prediction unit 90 of the image decoding device 60 illustrated in FIG. 13. Referring to FIG. 14, the intra prediction unit 90 includes a prediction control unit 91, a coefficient calculation unit 92, a luminance component memory 93, a filter 94, and a prediction unit 95.

  The prediction control unit 91 controls the intra prediction process in the intra prediction unit 90 according to the information related to intra prediction input from the lossless decoding unit 62. For example, the prediction control unit 91 executes an intra prediction process for the luminance component (Y) and an intra prediction process for the color difference components (Cb, Cr) for each CU. In the intra prediction process for the luminance component, the prediction control unit 91 causes the prediction unit 95 to generate a predicted pixel value of each luminance component in accordance with the designated luminance component prediction mode. The prediction mode for the luminance component may be any one of prediction modes associated with various prediction directions in the angle intra prediction method described with reference to FIG. Further, the prediction control unit 91 causes the prediction unit 95 to generate a prediction pixel value of each color difference component in accordance with the designated color difference component prediction mode. The prediction mode candidates for the color difference component include the LM mode described above.

  The coefficient calculation unit 92, the luminance component memory 93, and the filter 94 are provided for realizing the LM mode using the one-dimensional filter as described above.

  The coefficient calculation unit 92 refers to the reference pixels around the PU to which the pixel to be predicted belongs, and calculates the coefficients α and β of the prediction function used by the prediction unit 95 in the LM mode using the above-described equations (2) and (2). Calculate according to (3).

  Decoded image data including the value of the luminance component reconstructed using the prediction error data is input from the adding unit 65 to the luminance component memory 93 in units of lines. The line here is a line along a direction to be executed later in the IDCT in the horizontal direction and the vertical direction in the inverse orthogonal transform unit 22, and corresponds to a column of blocks in the present embodiment. The luminance component memory 93 temporarily stores the decoded image data of the luminance component in units of lines.

  The filter 94 extracts filter taps as illustrated in FIG. 8A or 8B from the luminance component memory 93 under the control of the prediction control unit 91. Then, the filter 94 filters the extracted filter tap value to generate an input value to a prediction function for predicting the value of the color difference component, and outputs the generated input value to the prediction unit 95.

  The prediction control unit 91 may switch the number of filter taps of the filter 94 according to the prediction direction indicated by the prediction mode specified for the luminance component at the pixel position common to each color difference component. More specifically, in the first example, the prediction control unit 91 may set the number of filter taps to a larger number when the angle between the prediction direction and the tap direction is smaller. Further, in the second example, the prediction control unit 91 sets the number of filter taps to the first number when the prediction direction overlaps the tap direction, and filters when the prediction direction does not overlap the tap direction. The number of taps may be set to a second number that is smaller than the first number. Thereby, when the correlation of the image along the tap direction is strong, the tap length can be lengthened, and when the correlation of the image along the tap direction is weak, the tap length can be shortened. Note that the filter coefficient of the filter 94 may be the same filter coefficient as the interpolation filter used in the motion compensation in the motion compensation unit 80.

  Under the control of the prediction control unit 91, the prediction unit 95 predicts the pixel value of the luminance component and the pixel value of the color difference component of the prediction target pixel according to the designated prediction mode. The color difference component prediction mode candidates include the LM mode described above. In the LM mode, the prediction unit 95 predicts the value of each color difference component by substituting the input value of the luminance component generated by the filter 94 into a prediction function having a coefficient calculated by the coefficient calculation unit 92. Intra prediction by the prediction unit 95 in other prediction modes may be performed in the same manner as the existing method. The prediction unit 95 outputs predicted image data generated as a result of prediction to the addition unit 65.

<4. Flow of processing at the time of decoding according to an embodiment>
Next, the flow of processing during decoding will be described with reference to FIG.

  FIG. 15 is a flowchart illustrating an example of the flow of intra prediction processing at the time of decoding by the intra prediction unit 90 having the configuration illustrated in FIG. 14. The intra prediction process shown in FIG. 15 is executed for each block in the image as a processing target.

  First, the prediction control unit 91 determines whether or not the color component to be processed is a luminance component (step S210). If the color component to be processed is a luminance component, the process proceeds to step S220. On the other hand, if the color component to be processed is a color difference component, the process proceeds to step S250.

  In step S220, the prediction control unit 91 causes the prediction unit 95 to perform intra prediction of the luminance component according to the designated prediction mode, and generate predicted image data (step S220). Next, the prediction control unit 91 stores decoded image data that is the sum of the predicted image data of the luminance component and the prediction error data output from the inverse orthogonal transform unit 64 in the luminance component memory 93 in units of lines (step) S230).

  Thus, after the intra prediction about a luminance component is complete | finished, the intra prediction about the color difference component of the same block is performed.

  In step S250, the prediction control unit 91 determines whether or not the LM mode is specified for the color difference component (step S250). Here, when the LM mode is designated, the prediction control unit 91 causes the prediction unit 95 to perform intra prediction of the color difference component according to the LM mode (step S260). The LM mode prediction processing by the prediction unit 95 here may be the same as the processing at the time of encoding described with reference to FIGS. When a prediction mode other than the LM mode is designated, the prediction control unit 91 causes the prediction unit 95 to perform intra prediction of the color difference component according to the designated prediction mode (step S270). As a result, predicted image data of the color difference component is generated.

  When the intra prediction is performed for all the color components of the processing target block, the intra prediction process illustrated in FIG. 15 ends (step S280).

<5. Modification>
[5-1. First Modification]
In the embodiment described above, the decoded image data of the luminance component is temporarily stored in the memory in units of lines. Then, at the time of intra prediction in the LM mode, downsampling of the luminance component is executed using a filter tap read from the memory. On the other hand, as in the first modification described in this section, the luminance component is down-sampled at an earlier stage, and the input value to the prediction function obtained as a result of the down-sampling is stored in the memory. It is also possible.

  FIG. 16 is a block diagram illustrating an example of a detailed configuration of the intra prediction unit 90 according to the first modification. Referring to FIG. 16, the intra prediction unit 90 includes a prediction control unit 91, a coefficient calculation unit 92, a prediction unit 95, a filter 96, and a luminance component memory 97.

  The filter 96 and the luminance component memory 97 are provided for realizing an LM mode using a one-dimensional filter.

  The filter 96 acquires decoded image data including luminance component values reconstructed using the prediction error data from the adding unit 65 in units of lines, and forms filter taps as illustrated in FIG. 8A or 8B. And the filter 96 produces | generates the input value of the luminance component to a prediction function by filtering the value of the formed filter tap.

  The luminance component memory 97 temporarily stores the input value of the luminance component after downsampling generated by the filter 96 in units of lines. The input value of the luminance component to the prediction function stored in the luminance component memory 97 is read out by the prediction unit 95 when the LM mode is designated for the color difference component.

  The prediction control unit 91 may switch the number of filter taps of the filter 96 according to the prediction direction indicated by the prediction mode designated for the luminance component at the pixel position common to each color difference component.

[5-2. Second Modification]
In the above-described embodiment, the case where the chroma format is 4: 2: 0 is mainly described. Even when the chroma format is 4: 2: 2, the density of the chrominance component is lower than the density of the luminance component. Therefore, when the chrominance component is predicted in the LM mode, May be downsampled. Even when the chroma format is 4: 4: 4, the luminance component value may be filtered using a one-dimensional filter when predicting the color difference component in the LM mode (in this case, the color difference). This is not "downsampling" because the density of the component is equal to the density of the luminance component However, when the chroma format is 4: 2: 2 or 4: 4: 4, there is a luminance component whose pixel position matches each color difference component. Therefore, when the chroma format is 4: 2: 2 or 4: 4: 4, the prediction unit 45 of the image encoding device 10 and the prediction unit 95 of the image decoding device 60 perform pixel positions that match each color difference component. May be substituted into the prediction function. In this case, the one-dimensional filter can be disabled.

  For example, referring to FIG. 17, when the chroma format is 4: 2: 2, the luminance component (Luma) and the corresponding color difference component (Chroma) in one PU having a size of 16 × 16 pixels are rounded. It is conceptually indicated by a sign. The density of the luminance component is twice the density of the color difference component only in the horizontal direction. The circles shaded with diagonal lines on the right in the figure indicate the luminance components that are substituted into the prediction function when the corresponding color difference components are predicted.

  Referring to FIG. 18, when the chroma format is 4: 4: 4, the luminance component (Luma) and the corresponding color difference component (Chroma) in one PU having a size of 16 × 16 pixels are circled. It is conceptually indicated by a sign. The density of the luminance component is equal to the density of the color difference component. The circles shaded with diagonal lines on the right in the figure indicate the luminance components that are substituted into the prediction function when the corresponding color difference components are predicted.

  As described above, by filtering the luminance component only when the chroma format is 4: 2: 0, it is possible to reduce the complexity of processing and to reduce the cost of the implementation accompanying the introduction of the LM mode.

<6. Application example>
The image encoding device 10 and the image decoding device 60 according to the above-described embodiments are a transmitter or a receiver in satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication, The present invention can be applied to various electronic devices such as a recording device that records an image on a medium such as an optical disk, a magnetic disk, and a flash memory, or a playback device that reproduces an image from these storage media. Hereinafter, four application examples will be described.

[6-1. First application example]
FIG. 19 shows an example of a schematic configuration of a television apparatus to which the above-described embodiment is applied. The television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, And a bus 912.

  The tuner 902 extracts a signal of a desired channel from a broadcast signal received via the antenna 901, and demodulates the extracted signal. Then, the tuner 902 outputs the encoded bit stream obtained by the demodulation to the demultiplexer 903. In other words, the tuner 902 serves as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

  The demultiplexer 903 separates the video stream and audio stream of the viewing target program from the encoded bit stream, and outputs each separated stream to the decoder 904. Further, the demultiplexer 903 extracts auxiliary data such as EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control unit 910. Note that the demultiplexer 903 may perform descrambling when the encoded bit stream is scrambled.

  The decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. Then, the decoder 904 outputs the video data generated by the decoding process to the video signal processing unit 905. In addition, the decoder 904 outputs audio data generated by the decoding process to the audio signal processing unit 907.

  The video signal processing unit 905 reproduces the video data input from the decoder 904 and causes the display unit 906 to display the video. In addition, the video signal processing unit 905 may cause the display unit 906 to display an application screen supplied via a network. Further, the video signal processing unit 905 may perform additional processing such as noise removal on the video data according to the setting. Further, the video signal processing unit 905 may generate a GUI (Graphical User Interface) image such as a menu, a button, or a cursor, and superimpose the generated image on the output image.

  The display unit 906 is driven by a drive signal supplied from the video signal processing unit 905, and displays a video or an image on a video screen of a display device (for example, a liquid crystal display, a plasma display, or an OLED).

  The audio signal processing unit 907 performs reproduction processing such as D / A conversion and amplification on the audio data input from the decoder 904 and outputs audio from the speaker 908. The audio signal processing unit 907 may perform additional processing such as noise removal on the audio data.

  The external interface 909 is an interface for connecting the television device 900 to an external device or a network. For example, a video stream or an audio stream received via the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also has a role as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

  The control unit 910 includes a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory stores a program executed by the CPU, program data, EPG data, data acquired via a network, and the like. The program stored in the memory is read and executed by the CPU when the television device 900 is activated, for example. The CPU controls the operation of the television device 900 according to an operation signal input from the user interface 911, for example, by executing the program.

  The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches for the user to operate the television device 900, a remote control signal receiving unit, and the like. The user interface 911 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 910.

  The bus 912 connects the tuner 902, the demultiplexer 903, the decoder 904, the video signal processing unit 905, the audio signal processing unit 907, the external interface 909, and the control unit 910 to each other.

  In the television device 900 configured as described above, the decoder 904 has the function of the image decoding device 60 according to the above-described embodiment. Thus, memory resources can be used more efficiently when the television device 900 decodes an image.

[6-2. Second application example]
FIG. 20 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied. A cellular phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, a control unit 931, an operation A portion 932 and a bus 933.

  The antenna 921 is connected to the communication unit 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 933 connects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the demultiplexing unit 928, the recording / reproducing unit 929, the display unit 930, and the control unit 931 to each other.

  The mobile phone 920 has various operation modes including a voice call mode, a data communication mode, a shooting mode, and a videophone mode, and is used for sending and receiving voice signals, sending and receiving e-mail or image data, taking images, and recording data. Perform the action.

  In the voice call mode, an analog voice signal generated by the microphone 925 is supplied to the voice codec 923. The audio codec 923 converts an analog audio signal into audio data, A / D converts the compressed audio data, and compresses it. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 expands the audio data and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  Further, in the data communication mode, for example, the control unit 931 generates character data that constitutes an e-mail in response to an operation by the user via the operation unit 932. In addition, the control unit 931 causes the display unit 930 to display characters. In addition, the control unit 931 generates e-mail data in response to a transmission instruction from the user via the operation unit 932, and outputs the generated e-mail data to the communication unit 922. The communication unit 922 encodes and modulates email data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to reconstruct email data, and outputs the reconstructed email data to the control unit 931. The control unit 931 displays the content of the electronic mail on the display unit 930 and stores the electronic mail data in the storage medium of the recording / reproducing unit 929.

  The recording / reproducing unit 929 has an arbitrary readable / writable storage medium. For example, the storage medium may be a built-in storage medium such as a RAM or a flash memory, or an externally mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card. May be.

  In the shooting mode, for example, the camera unit 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the recording / playback unit 929.

  Further, in the videophone mode, for example, the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and the multiplexed stream is the communication unit 922. Output to. The communication unit 922 encodes and modulates the stream and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. These transmission signal and reception signal may include an encoded bit stream. Then, the communication unit 922 demodulates and decodes the received signal to reconstruct the stream, and outputs the reconstructed stream to the demultiplexing unit 928. The demultiplexing unit 928 separates the video stream and the audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, and a series of images is displayed on the display unit 930. The audio codec 923 decompresses the audio stream and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  In the mobile phone 920 configured as described above, the image processing unit 927 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Accordingly, memory resources can be used more efficiently when the mobile phone 920 encodes and decodes images.

[6-3. Third application example]
FIG. 21 shows an example of a schematic configuration of a recording / reproducing apparatus to which the above-described embodiment is applied. For example, the recording / reproducing device 940 encodes audio data and video data of a received broadcast program and records the encoded data on a recording medium. In addition, the recording / reproducing device 940 may encode audio data and video data acquired from another device and record them on a recording medium, for example. In addition, the recording / reproducing device 940 reproduces data recorded on the recording medium on a monitor and a speaker, for example, in accordance with a user instruction. At this time, the recording / reproducing device 940 decodes the audio data and the video data.

  The recording / reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface. 950.

  The tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Then, the tuner 941 outputs the encoded bit stream obtained by the demodulation to the selector 946. That is, the tuner 941 has a role as a transmission unit in the recording / reproducing apparatus 940.

  The external interface 942 is an interface for connecting the recording / reproducing apparatus 940 to an external device or a network. The external interface 942 may be, for example, an IEEE 1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission unit in the recording / reproducing device 940.

  The encoder 943 encodes video data and audio data when the video data and audio data input from the external interface 942 are not encoded. Then, the encoder 943 outputs the encoded bit stream to the selector 946.

  The HDD 944 records an encoded bit stream in which content data such as video and audio is compressed, various programs, and other data on an internal hard disk. Also, the HDD 944 reads out these data from the hard disk when playing back video and audio.

  The disk drive 945 performs recording and reading of data with respect to the mounted recording medium. The recording medium mounted on the disk drive 945 may be, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.) or a Blu-ray (registered trademark) disk. .

  The selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video and audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 during video and audio reproduction.

  The decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. The decoder 904 outputs the generated audio data to an external speaker.

  The OSD 948 reproduces the video data input from the decoder 947 and displays the video. Further, the OSD 948 may superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.

  The control unit 949 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the recording / reproducing apparatus 940 is activated, for example. The CPU controls the operation of the recording / reproducing device 940 according to an operation signal input from the user interface 950, for example, by executing the program.

  The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for the user to operate the recording / reproducing device 940, a remote control signal receiving unit, and the like. The user interface 950 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.

  In the recording / reproducing apparatus 940 configured as described above, the encoder 943 has the function of the image encoding apparatus 10 according to the above-described embodiment. The decoder 947 has the function of the image decoding device 60 according to the above-described embodiment. Thereby, the memory resource can be used more efficiently when the recording / reproducing apparatus 940 encodes and decodes an image.

[6-4. Fourth application example]
FIG. 22 shows an example of a schematic configuration of an imaging apparatus to which the above-described embodiment is applied. The imaging device 960 images a subject to generate an image, encodes the image data, and records it on a recording medium.

  The imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a bus. 972.

  The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 972 connects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970 to each other.

  The optical block 961 includes a focus lens and a diaphragm mechanism. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 includes an image sensor such as a CCD or a CMOS, and converts an optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. Then, the imaging unit 962 outputs the image signal to the signal processing unit 963.

  The signal processing unit 963 performs various camera signal processes such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data after the camera signal processing to the image processing unit 964.

  The image processing unit 964 encodes the image data input from the signal processing unit 963, and generates encoded data. Then, the image processing unit 964 outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes encoded data input from the external interface 966 or the media drive 968 to generate image data. Then, the image processing unit 964 outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may display the image by outputting the image data input from the signal processing unit 963 to the display unit 965. Further, the image processing unit 964 may superimpose display data acquired from the OSD 969 on an image output to the display unit 965.

  The OSD 969 generates a GUI image such as a menu, a button, or a cursor, and outputs the generated image to the image processing unit 964.

  The external interface 966 is configured as a USB input / output terminal, for example. The external interface 966 connects the imaging device 960 and a printer, for example, when printing an image. Further, a drive is connected to the external interface 966 as necessary. For example, a removable medium such as a magnetic disk or an optical disk is attached to the drive, and a program read from the removable medium can be installed in the imaging device 960. Further, the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has a role as a transmission unit in the imaging device 960.

  The recording medium mounted on the media drive 968 may be any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. Further, a recording medium may be fixedly attached to the media drive 968, and a non-portable storage unit such as a built-in hard disk drive or an SSD (Solid State Drive) may be configured.

  The control unit 970 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the imaging device 960 is activated, for example. The CPU controls the operation of the imaging device 960 according to an operation signal input from the user interface 971, for example, by executing the program.

  The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960. The user interface 971 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 970.

  In the imaging device 960 configured as described above, the image processing unit 964 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Accordingly, memory resources can be used more efficiently when the image capturing apparatus 960 encodes and decodes an image.

<7. Summary>
So far, the image encoding device 10 and the image decoding device 60 according to an embodiment have been described in detail with reference to FIGS. According to the above-described embodiment, when the LM mode using the function of the value of the luminance component is adopted for intra prediction of the color difference component in the encoding and decoding of the image, the luminance component is down-sampled using the one-dimensional filter. Generates an input value to the prediction function. Thereby, it is possible to use memory resources more efficiently by adopting sequential processing in units of lines while reducing the influence of noise.

  Further, according to the above-described embodiment, the filter taps of the one-dimensional filter are arranged along the direction to be executed later among the inverse orthogonal transforms in the horizontal direction and the vertical direction when the decoded image is reconstructed. Thereby, the size of the memory for storing the decoded image of the luminance component, which is provided between the output of the inverse orthogonal transform and the input of the intra prediction, can be reduced.

  Further, according to the above-described embodiment, the number of taps of the filter taps of the one-dimensional filter may be adaptively controlled according to the prediction direction of the intra prediction for the luminance component at the pixel position common to each color difference component. . Thereby, when the correlation of the image along a filter tap is strong, tap length can be lengthened and the robustness with respect to noise can be improved.

  In the present specification, an example in which information related to intra prediction and information related to inter prediction is multiplexed on the header of the encoded stream and transmitted from the encoding side to the decoding side has been mainly described. However, the method for transmitting such information is not limited to such an example. For example, these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream). Information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

The following configurations also belong to the technical scope of the present disclosure.
(1)
Filtering the value of a filter tap including one or more luminance components at a pixel position common to the first chrominance component and one or more luminance components at a pixel position not common to the first chrominance component; A filter that generates an input value of a luminance component to be substituted into a prediction function for predicting a value of the first color difference component;
A prediction unit that predicts the value of the first color difference component by substituting the input value of the luminance component generated by the filter into the prediction function;
With
The filter is a one-dimensional filter;
Image processing device.
(2)
The image processing according to (1), wherein the filter tap of the filter is arranged along a first direction that is executed later among inverse orthogonal transforms in a horizontal direction and a vertical direction when an image is reconstructed. apparatus.
(3)
The image processing apparatus further includes a control unit that controls the number of taps of the filter tap according to a prediction direction of intra prediction selected for a luminance component at a pixel position common to the first color difference component, The image processing apparatus according to (2).
(4)
The said control part is an image processing apparatus as described in said (3) which sets the said tap number to a larger number, when the angle between the said prediction direction and the said 1st direction is smaller.
(5)
The control unit sets the number of taps to a first number when the prediction direction overlaps the first direction, and sets the number of taps when the prediction direction does not overlap the first direction. The image processing apparatus according to (3), wherein the second number is set to be a second number smaller than the first number.
(6)
When the chroma format is not 4: 2: 0, the prediction unit substitutes a value of a luminance component at a pixel position that matches the first chrominance component into the prediction function, (1) to (5) The image processing apparatus according to any one of the above.
(7)
The image processing apparatus according to any one of (1) to (6), wherein the filter has the same filter coefficient as an interpolation filter used in motion compensation.
(8)
The filter tap value including one or more luminance components at a pixel position common to the first color difference component and one or more luminance components at a pixel position not common to the first color difference component is filtered by a one-dimensional filter. Generating an input value of a luminance component to be substituted into a prediction function for predicting a value of the first color difference component;
Predicting a value of the first color difference component by substituting the input value of the luminance component generated by the one-dimensional filter into the prediction function;
An image processing method including:

10 Image encoding device (image processing device)
DESCRIPTION OF SYMBOLS 41 Prediction control part 42 Coefficient calculation part 43 Luminance component memory 44 Filter 45 Prediction part 60 Image decoding apparatus (image processing apparatus)
91 Prediction control unit 92 Coefficient calculation unit 93 Luminance component memory 94 Filter 95 Prediction unit

Claims (8)

Filtering the value of a filter tap including one or more luminance components at a pixel position common to the first chrominance component and one or more luminance components at a pixel position not common to the first chrominance component; A filter that generates an input value of a luminance component to be substituted into a prediction function for predicting a value of the first color difference component;
A prediction unit that predicts the value of the first color difference component by substituting the input value of the luminance component generated by the filter into the prediction function;
With
The filter is a one-dimensional filter;
Image processing device.   The image processing apparatus according to claim 1, wherein the filter tap of the filter is arranged along a first direction that is executed later among inverse orthogonal transforms in a horizontal direction and a vertical direction when an image is reconstructed. .   The image processing apparatus further includes a control unit that controls the number of taps of the filter tap according to a prediction direction of intra prediction selected for a luminance component at a pixel position common to the first color difference component. Item 3. The image processing apparatus according to Item 2.   The image processing apparatus according to claim 3, wherein the control unit sets the number of taps to a larger number when an angle between the predicted direction and the first direction is smaller.   The control unit sets the number of taps to a first number when the prediction direction overlaps the first direction, and sets the number of taps when the prediction direction does not overlap the first direction. The image processing apparatus according to claim 3, wherein the image processing apparatus is set to a second number smaller than the first number.   The image processing according to claim 1, wherein when the chroma format is not 4: 2: 0, the prediction unit substitutes a value of a luminance component at a pixel position that matches the first chrominance component into the prediction function. apparatus.   The image processing apparatus according to claim 1, wherein the filter has the same filter coefficient as an interpolation filter used in motion compensation. The filter tap value including one or more luminance components at a pixel position common to the first color difference component and one or more luminance components at a pixel position not common to the first color difference component is filtered by a one-dimensional filter. Generating an input value of a luminance component to be substituted into a prediction function for predicting a value of the first color difference component;
Predicting a value of the first color difference component by substituting the input value of the luminance component generated by the one-dimensional filter into the prediction function;
An image processing method including:

Images