JP2023046435A - Moving image decoder and moving image encoder - Google Patents

Abstract

To provide a moving image decoder and a moving image encoder which can perform suitable intra prediction while reducing a data amount of a matrix intra prediction mode.SOLUTION: In a moving image decoder, a matrix intra prediction technique (MIP) unit comprises: a matrix reference pixel derivation part which derives an image obtained by down-sampling images adjacent to the upper side and the left side of an object block as a reference image; a weighting matrix derivation part which derives a matrix of a weighting coefficient according to the intra prediction mode and the object block size; a matrix prediction image derivation part which derives a prediction image with a product of an element of the reference image and an element of the matrix of the weighting coefficient; and a matrix prediction image interpolation part which derives the prediction image or an image obtained by interpolating the prediction image as the prediction image. The weighting matrix derivation part derives the matrix in the size equal to or less than the width and equal to or less than the height of the object block size.SELECTED DRAWING: Figure 10

Description

TECHNICAL FIELD Embodiments of the present invention relate to a moving image decoding device and a moving image encoding device.

A moving image encoding device that generates encoded data by encoding a moving image and a moving image that generates a decoded image by decoding the encoded data in order to efficiently transmit or record the moving image An image decoding device is used.

Examples of specific moving image coding methods include methods proposed in H.264/AVC and HEVC (High-Efficiency Video Coding).

In such a video coding method, the images (pictures) that make up the video are divided into slices obtained by dividing the image, and coding tree units (CTU: Coding Tree Units) obtained by dividing the slices. ), a coding unit obtained by dividing the coding tree unit (also called a coding unit (CU)), and a transform unit obtained by dividing the coding unit (TU: Transform Unit), and encoded/decoded for each CU.

Further, in such a video encoding method, a predicted image is normally generated based on a locally decoded image obtained by encoding/decoding an input image, and the predicted image is generated from the input image (original image). The prediction error obtained by subtraction (sometimes called the "difference image" or "residual image") is encoded. Inter-prediction and intra-prediction are methods for generating predicted images.

In addition, Non-Patent Document 1 can be cited as a technique of video encoding and decoding in recent years. Non-Patent Document 1 discloses a matrix-based intra prediction (MIP) technique for deriving a predicted image by multiply-adding a reference image derived from an adjacent image and a weighting matrix.

ITU-T Rec. H.266

In matrix intra prediction as in Non-Patent Document 1, an appropriate matrix is selected from a plurality of predefined matrices to generate a predicted image, so the encoded data for selecting the matrix, that is, the amount of data in matrix intra prediction mode increases. There is a problem of

An object of the present invention is to perform suitable matrix intra prediction while reducing the amount of data in matrix intra prediction mode.

In order to solve the above problems, a video decoding device according to one aspect of the present invention includes:
a matrix reference pixel deriving unit that derives an image obtained by down-sampling images adjacent to the upper and left sides of a target block as a reference image;
A mode derivation unit that derives a prediction mode candidate list used in the target block according to the reference image and the target block size;
A prediction processing parameter derivation unit that derives parameters used in calculations for deriving a predicted image according to the candidate list, the matrix intra prediction mode indicator, and the target block size;
a matrix predicted image derivation unit that derives a predicted image based on the elements of the reference image and the prediction processing parameters;
a matrix predicted image interpolation unit that derives the predicted image or an image obtained by interpolating the predicted image as a predicted image;
, wherein the mode derivation unit derives the candidate list with a number of elements less than the total number of prediction modes defined for the target block size.

ADVANTAGE OF THE INVENTION According to 1 aspect of this invention, suitable intra prediction can be performed, reducing the data amount of matrix intra prediction mode.

1 is a schematic diagram showing the configuration of an image transmission system according to this embodiment; FIG. 1 is a diagram showing the configuration of a transmitting device equipped with a moving image encoding device and a receiving device equipped with a moving image decoding device according to an embodiment; FIG. PROD_A indicates a transmitting device equipped with a video encoding device, and PROD_B indicates a receiving device equipped with a video decoding device. 1 is a diagram showing configurations of a recording device equipped with a moving image encoding device and a reproducing device equipped with a moving image decoding device according to an embodiment; FIG. PROD_C indicates a recording device equipped with a moving image encoding device, and PROD_D indicates a reproducing device equipped with a moving image decoding device. FIG. 3 is a diagram showing the hierarchical structure of data in an encoded stream; FIG. 4 is a diagram showing an example of CTU division; FIG. 3 is a schematic diagram showing types (mode numbers) of intra prediction modes; 1 is a schematic diagram showing the configuration of a video decoding device; FIG. FIG. 4 is a diagram showing reference regions used for intra prediction; It is a figure which shows the structure of an intra prediction image production|generation part. FIG. 4 is a diagram showing details of the MIP unit; FIG. 10 is a diagram showing an example of MIP processing; FIG. 10 is a diagram showing an example of MIP processing; 1 is a block diagram showing the configuration of a video encoding device; FIG.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

The image transmission system 1 is a system that transmits an encoded stream obtained by encoding an encoding target image, decodes the transmitted encoded stream, and displays the image. The image transmission system 1 includes a moving image coding device (image coding device) 11, a network 21, a moving image decoding device (image decoding device) 31, and a moving image display device (image display device) 41. .

An image T is input to the moving image encoding device 11 .

The network 21 transmits the encoded stream Te generated by the video encoding device 11 to the video decoding device 31 . Network 21 includes the Internet (Internet), wide area network (
Wide Area Network (WAN), Local Area Network (LAN), or a combination thereof. The network 21 is not necessarily a two-way communication network, and may be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Also, the network 21 may be replaced by a storage medium such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark) that records the encoded stream Te.

The moving image decoding device 31 decodes each of the encoded streams Te transmitted by the network 21 and generates one or more decoded images Td.

The moving image display device 41 displays all or part of one or more decoded images Td generated by the moving image decoding device 31 . The moving image display device 41 includes, for example, a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The form of the display includes stationary, mobile, HMD, and the like. In addition, when the moving image decoding device 31 has high processing power, it displays an image with high image quality, and when it has only lower processing power, it displays an image that does not require high processing power and display power. .

<operator>
The operators used in this specification are described below.

>> is right bit shift, << is left bit shift, & is bitwise AND, | is bitwise OR
, |= is an OR assignment operator, && indicates a logical product (AND), and || indicates a logical sum (OR).

x?y:z is a ternary operator that takes y if x is true (not 0) and z if x is false (0).

Clip3(a,b,c) is a function that clips c to a value greater than or equal to a and less than or equal to b, if c<a returns a, if c>b returns b, otherwise is a function that returns c (where a<=b).

Clip1Y(c) is an operator set to a=0 and b=(1<<BitDepthY)-1 in Clip3(a,b,c). BitDepthY is the luminance bit depth.

abs(a) is a function that returns the absolute value of a.

Int(a) is a function that returns the integer value of a.

Floor(a) is a function that returns the largest integer less than or equal to a.

Ceil(a) is a function that returns the smallest integer greater than or equal to a.

a/d represents division of a by d (truncated after the decimal point).

Min(a,b) is a function that returns the smaller of a and b.

<Structure of encoded stream Te>
Prior to a detailed description of the video encoding device 11 and the video decoding device 31 according to the present embodiment, data of the encoded stream Te generated by the video encoding device 11 and decoded by the video decoding device 31 I will explain the structure.

FIG. 4 is a diagram showing the hierarchical structure of data in the encoded stream Te. The encoded stream Te illustratively includes a sequence and a plurality of pictures that constitute the sequence. FIG. 4 shows a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, and coded slice data that defines the slice data. A diagram showing the included coding tree unit and the coding units included in the coding tree unit is shown.

(encoded video sequence)
The encoded video sequence defines a set of data that the video decoding device 31 refers to in order to decode the sequence SEQ to be processed. The sequence SEQ includes a video parameter set, a sequence parameter set SPS, a picture parameter set PPS, a picture PICT, and a picture PICT, as shown in the encoded video sequence of FIG. It contains supplemental enhancement information SEI (Supplemental Enhancement Information).

A video parameter set VPS is a set of coding parameters common to multiple video images, multiple layers included in the video image, and coding parameters related to individual layers. Sets are defined.

The sequence parameter set SPS defines a set of coding parameters that the video decoding device 31 refers to in order to decode the target sequence. For example, the width and height of the picture are defined. A plurality of SPSs may exist. In that case, one of a plurality of SPSs is selected from the PPS.

The picture parameter set PPS defines a set of coding parameters that the video decoding device 31 refers to in order to decode each picture in the target sequence. For example, it includes a quantization width reference value (pic_init_qp_minus26) used for picture decoding and a flag (weighted_pred_flag) indicating application of weighted prediction. A plurality of PPSs may exist. In that case, one of a plurality of PPSs is selected from each picture in the target sequence.

(coded picture)
The encoded picture defines a set of data that the video decoding device 31 refers to in order to decode the picture PICT to be processed. A picture PICT includes slice 0 to slice NS-1 (NS is the total number of slices included in the picture PICT), as shown in the coded picture in FIG.
.

In the following description, if there is no need to distinguish between slices 0 to NS-1, the suffixes may be omitted. The same applies to other data with subscripts that are included in the encoded stream Te described below.

(coded slice)
The encoded slice defines a set of data that the video decoding device 31 refers to in order to decode the slice S to be processed. A slice is represented by the coded slice in Figure 4.
Contains slice header and slice data.

The slice header includes a coding parameter group that the video decoding device 31 refers to in order to determine the decoding method for the target slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in a slice header.

Slice types that can be specified by the slice type specifying information include (1) I slices that use only intra prediction during encoding, (2) P slices that use unidirectional prediction or intra prediction during encoding, (3) B slices using uni-prediction, bi-prediction, or intra-prediction during encoding. Note that inter prediction is not limited to uni-prediction and bi-prediction, and a predicted image may be generated using more reference pictures. Below, P
, B slices refer to slices containing blocks for which inter prediction can be used.

Note that the slice header may include a reference (pic_parameter_set_id) to the picture parameter set PPS.

(encoded slice data)
The encoded slice data defines a set of data that the video decoding device 31 refers to in order to decode slice data to be processed. The slice data contains CTU, as shown in the encoded slice header in FIG. A CTU is a fixed-size (for example, 64x64) block that forms a slice, and is also called a largest coding unit (LCU).

(encoding tree unit)
The coding tree unit in FIG. 4 defines a set of data that the video decoding device 31 refers to in order to decode the CTU to be processed. CTU is a recursive quad tree partition (QT (Quad Tree) partition), a binary tree partition (BT (Binary Tree) partition) or a ternary tree partition (TT (Ternary Tree)
) division) into coding units CU, which are basic units for coding processing. BT partitioning and TT partitioning are collectively called multi-tree partitioning (MT (Multi Tree) partitioning). A node of a tree structure obtained by recursive quadtree partitioning is called a coding node. 4
The intermediate nodes of branch trees, binary trees, and ternary trees are coding nodes, and the CTU itself is defined as the top-level coding node.

CT has a QT split flag (cu_split_flag) indicating whether or not to perform QT splitting, MT
It includes an MT split flag (split_mt_flag) indicating the presence/absence of splitting, an MT split direction (split_mt_dir) indicating the split direction of MT split, and an MT split type (split_mt_type) indicating the split type of MT split. cu_split_flag, split_mt_flag, split_mt_dir and split_mt_type are transmitted for each encoding node.

If cu_split_flag is 1, the coding node is split into 4 coding nodes (QT in Figure 5).

When cu_split_flag is 0 and split_mt_flag is 0, the encoding node is not split and has one CU as a node (no split in FIG. 5). A CU is a terminal node of an encoding node and is not split any further. A CU is a basic unit of encoding processing.

When split_mt_flag is 1, the coding node is MT-split as follows. When split_mt_type is 0, the encoding node is horizontally split into two encoding nodes when split_mt_dir is 1 (BT (horizontal split) in FIG. 5), and when split_mt_dir is 0, the encoding node is split into two encoding nodes. It is vertically split into nodes (BT (vertical split) in Fig. 5). Also, when split_mt_type is 1 and split_mt_dir is 1, the encoding node is horizontally split into 3 encoding nodes (TT (horizontal split) in FIG. 5).
, the encoding node is vertically split into three encoding nodes when split_mt_dir is 0 (Fig. 5
TT (vertical split)). These are shown in the CT information in FIG.

If the CTU size is 64x64 pixels, the CU size is 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels. , 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels. .

(encoding unit)
As shown in the encoding unit in FIG. 4, a set of data that the video decoding device 31 refers to in order to decode the encoding unit to be processed is defined. Specifically, a CU is composed of a CU header CUH, prediction parameters, transform parameters, quantized transform coefficients, and the like. A prediction mode and the like are defined in the CU header.

Prediction processing may be performed in units of CUs or may be performed in units of sub-CUs obtained by further dividing a CU. If the CU and sub-CU sizes are equal, there is one sub-CU in the CU. If the CU is larger than the sub-CU size, the CU is split into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into 4 sub-CUs consisting of 2 horizontal divisions and 2 vertical divisions.

There are two types of prediction (prediction mode): intra prediction and inter prediction. Intra prediction is prediction within the same picture, and inter prediction is prediction processing performed between different pictures (for example, between display times, between layer images).

Transformation/quantization processing is performed in units of CUs, but the quantized transform coefficients may be entropy-encoded in units of subblocks such as 4x4.

(prediction parameter)
A predicted image is derived from the prediction parameters associated with the block. The prediction parameters include prediction parameters for intra prediction and inter prediction.

The prediction parameters for intra prediction will be described below. The intra prediction parameters are composed of a luminance prediction mode IntraPredModeY and a color difference prediction mode IntraPredModeC. FIG. 6 is a schematic diagram showing types of intra prediction modes (mode numbers). As shown in the figure, there are, for example, 67 types (0 to 66) of intra prediction modes. For example, planar prediction (0), DC prediction (1), Angular prediction (2-66). Furthermore, LM mode may be added for color difference.

Syntax elements for deriving intra prediction parameters include, for example, intra_luma_mpm_flag, intra_luma_mpm_idx, and intra_luma_mpm_remainder.

(MPM)
intra_luma_mpm_flag is a flag indicating whether or not IntraPredModeY and MPM (Most Probable Mode) of the target block match. MPM is a prediction mode included in the MPM candidate list mpmCandList[]. The MPM candidate list is a list storing candidates that are estimated to have a high probability of being applied to the target block from the intra prediction modes of neighboring blocks and the predetermined intra prediction mode. When intra_luma_mpm_flag is 1, the MPM candidate list and index intra_luma_mpm_idx are used to derive IntraPredModeY of the target block.

IntraPredModeY = mpmCandList[intra_luma_mpm_idx]
(REM)
When intra_luma_mpm_flag is 0, an intra prediction mode is selected from RemIntraPredMode remaining modes excluding intra prediction modes included in the MPM candidate list from all intra prediction modes. Intra-prediction modes selectable as RemIntraPredMode are called "non-MPM" or "REM." RemIntraPredMode is derived using intra_luma_mpm_remainder.

(Configuration of video decoding device)
The configuration of the video decoding device 31 (FIG. 7) according to this embodiment will be described.

The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse It includes a quantization/inverse transformation unit 311 and an addition unit 312 . Note that the moving image decoding device 31 may have a configuration in which the loop filter 305 is not included in accordance with the moving image encoding device 11 described later.

Also, parameter decoding section 302 includes inter prediction parameter decoding section 303 and intra prediction parameter decoding section 304 (not shown). The predicted image generator 308 includes an inter predicted image generator 309 and an intra predicted image generator 310 .

Further, although an example using CTU and CU as units of processing will be described below, the processing is not limited to this example, and processing may be performed in units of sub-CUs. Alternatively, CTU and CU may be read as blocks, sub-CUs as sub-blocks, and processing may be performed in units of blocks or sub-blocks.

The entropy decoding unit 301 performs entropy decoding on an externally input encoded stream Te to separate and decode individual codes (syntax elements). For entropy coding, a method of variable-length coding syntax elements using a context (probability model) adaptively selected according to the type of syntax elements and surrounding circumstances, a predetermined table, or There is a method of variable-length coding syntax elements using a formula. The former CABAC (Context Adaptive Binary Arithmetic Coding) stores in memory an updated probability model for each coded or decoded picture (slice). and P picture,
Alternatively, as the initial state of the B-picture context, a picture probability model using the same slice type and the same slice level quantization parameter is set from among the probability models stored in the memory. This initial state is used for encoding and decoding. The separated codes include prediction information for generating a prediction image, prediction error for generating a difference image, and the like.

Entropy decoding section 301 outputs the separated code to parameter decoding section 302 . Control of which code is to be decoded is performed based on an instruction from parameter decoding section 302 .

(Configuration of intra prediction parameter decoding unit 304)
Based on the code input from the entropy decoding unit 301, the intra prediction parameter decoding unit 304 refers to the prediction parameters stored in the prediction parameter memory 307 and decodes the intra prediction parameters, for example, the intra prediction mode IntraPredMode. The intra prediction parameter decoding unit 304 outputs the decoded intra prediction parameters to the prediction image generation unit 308 and stores them in the prediction parameter memory 307 . The intra prediction parameter decoding unit 304 may derive different intra prediction modes for luminance and color difference.

The intra prediction parameter decoding unit 304 includes a MIP parameter decoding unit 3041, a luminance intra prediction parameter decoding unit 3042, and a chrominance intra prediction parameter decoding unit 3043.

The MIP parameter decoding unit 3041 decodes intra_mip_flag from encoded data. If intra_mip_flag is 0 and intra_luma_mpm_flag is 1, decode intra_luma_mpm_idx. Also, if intra_luma_mpm_flag is 0, intra_luma_mpm_remainder is decoded. Then, refer to mpmCandList[ ], intra_luma_mpm_idx, and intra_luma_mpm_remainder to derive IntraPredModeY and output it to the intra prediction image generation unit 310 .

Also, the MIP parameter decoding unit 3041 derives IntraPredModeC from the syntax element of the intra prediction parameter of chrominance, and outputs it to the intra prediction image generation unit 310 .

A loop filter 305 is a filter provided in the encoding loop, and is a filter that removes block distortion and ringing distortion to improve image quality. A loop filter 305 applies filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the addition unit 312 .

The reference picture memory 306 stores the decoded image of the CU generated by the adding unit 312 in a predetermined position for each target picture and target CU.

The prediction parameter memory 307 stores prediction parameters in predetermined positions for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores the parameters decoded by the parameter decoding unit 302, the prediction mode predMode separated by the entropy decoding unit 301, and the like.

A prediction mode predMode, a prediction parameter, and the like are input to the prediction image generation unit 308 . Also, the predicted image generation unit 308 reads a reference picture from the reference picture memory 306 . The predicted image generating unit 308 generates a predicted image of a block or sub-block using the prediction parameter and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, a reference picture block is a set of pixels on a reference picture (usually rectangular and therefore called a block), and is an area referred to for generating a prediction image.

(Intra prediction image generator 310)
When the prediction mode predMode indicates the intra prediction mode, the intra prediction image generation unit 310 performs intra prediction using the intra prediction parameters input from the intra prediction parameter decoding unit 304 and the reference pixels read from the reference picture memory 306.

Specifically, the intra-prediction image generator 310 reads from the reference picture memory 306 adjacent blocks within a predetermined range from the current block on the current picture. The predetermined range is adjacent blocks on the left, upper left, above, and upper right of the target block, and the area referred to differs depending on the intra prediction mode.

The intra-predicted image generation unit 310 generates a predicted image of the target block by referring to the read decoded pixel values and the prediction mode indicated by IntraPredMode. The intra prediction image generation unit 310 outputs the generated block prediction image to the addition unit 312 .

Generation of predicted images based on intra-prediction modes is described below. In Planar prediction, DC prediction, and Angular prediction, a decoded peripheral region adjacent (adjacent) to a block to be predicted is set as a reference region R. Then, a predicted image is generated by extrapolating the pixels on the reference region R in a specific direction. For example, the reference region R is an L-shaped region including the left and top (or further, upper left, upper right, and lower left) of the block to be predicted (for example, pixels marked with hatched circles in reference region example 1 in FIG. 8). area shown).

(Details of predictive image generator)
Next, the details of the configuration of the intra prediction image generation unit 310 will be described using FIG. The intra prediction image generation unit 310 includes a reference sample filter unit 3103 (second reference image setting unit), a prediction unit 3104
, and a predicted image correction unit 3105 (predicted image correction unit, filter switching unit, weight coefficient changing unit).

Based on each reference pixel (reference image) on the reference region R, the filtered reference image generated by applying the reference pixel filter (first filter), and the intra prediction mode, the prediction unit 3104 performs provisional prediction of the prediction target block. An image (pre-correction predicted image) is generated and output to the predicted image correction unit 3105 . The predicted image correcting unit 3105 corrects the provisional predicted image according to the intra prediction mode, generates a predicted image (corrected predicted image), and outputs it.

Each unit included in the intra prediction image generation unit 310 will be described below.

(Reference sample filter unit 3103)
The reference sample filter unit 3103 derives reference samples s[x][y] at each position (x, y) on the reference region R by referring to the reference image. In addition, the reference sample filter unit 3103 applies a reference pixel filter (first filter) to the reference sample s[x][y] according to the intra prediction mode, and obtains each position (x, y) update reference samples s[x][y] (derivate filtered reference image s[x][y]). Specifically, a low-pass filter is applied to the reference image at the position (x, y) and its surroundings to derive a filtered reference image (reference region example 2 in FIG. 8). Note that it is not always necessary to apply a low-pass filter to all intra prediction modes, and a low-pass filter may be applied to some intra prediction modes. Note that while the filter applied to the reference image on the reference region R in the reference sample filter unit 3103 is referred to as a "reference pixel filter (first filter)", the predicted image correction unit 3105 described later applies a tentative predicted image. A filter for correction is called a "position-dependent filter (second filter)".

(Configuration of intra prediction unit 3104)
The intra prediction unit 3104 generates a provisional prediction image (provisional prediction pixel value, pre-correction prediction image) of the prediction target block based on the intra prediction mode, the reference image, and the filtered reference pixel value, and sends it to the prediction image correction unit 3105. Output. The prediction unit 3104 includes a planar prediction unit 31041, DC prediction unit 31042, Angular prediction unit 31043, LM prediction unit 31044 and MIP unit 31045 inside. A prediction unit 3104 selects a specific prediction unit according to the intra prediction mode, and inputs a reference image and a filtered reference image. The relationship between the intra prediction modes and the corresponding predictors is as follows.
・Planar prediction ・・・Planar prediction unit 31041
・DC prediction: DC prediction unit 31042
・Angular prediction ・・・Angular prediction unit 31043
・LM prediction ・・・LM prediction unit 31044
・Matrix intra prediction MIP unit 31045
(Planar prediction)
The planar prediction unit 31041 linearly adds the reference samples s[x][y] according to the distance between the prediction target pixel position and the reference pixel position to generate a provisional prediction image, and outputs the provisional prediction image to the prediction image correction unit 3105 .

(DC prediction)
The DC prediction unit 31042 derives a DC prediction value corresponding to the average value of the reference samples s[x][y], and outputs a provisional prediction image q[x][y] having the DC prediction value as the pixel value.

(Angular prediction)
The Angular prediction unit 31043 generates a provisional prediction image q[x][y] using the reference sample s[x][y] in the prediction direction (reference direction) indicated by the intra prediction mode, and sends it to the prediction image correction unit 3105 Output.

(LM prediction)
The LM prediction unit 31044 predicts the chrominance pixel value based on the luminance pixel value. Specifically, it is a method of generating a prediction image of a color difference image (Cb, Cr) using a linear model based on a decoded luminance image. CCLM (Cross-Component Linear Model prediction
) there is a prediction. CCLM prediction is a prediction scheme that uses a linear model to predict color difference from luminance for a block.

(Configuration of predicted image correction unit 3105)
A predicted image correction unit 3105 corrects the provisional predicted image output from the prediction unit 3104 according to the intra prediction mode. Specifically, the predicted image correction unit 3105 derives a position-dependent weighting factor for each pixel of the provisional predicted image according to the positions of the reference region R and the target predicted pixel. Then, a predicted image (corrected predicted image) Pred[][] obtained by correcting the provisional predicted image is derived by weighted addition (weighted average) of the reference sample s[][] and the provisional predicted image. Note that in some intra prediction modes, the predicted image correction unit 3105 may not correct the provisional predicted image, and the output of the prediction unit 3104 may be directly used as the predicted image.

(MIP Example 1)
The MIP parameter decoding unit 3041 decodes intra_mip_flag from encoded data. When intra_mip_flag is 1, MIP parameter decoding section 3041 decodes intra_mip_transposed_flag and matrix intra prediction mode indicator intra_mip_mode_idx. intra_mip_mode_idx is a value from 0 to NumMipModes-1, and may be decoded using TB (Truncated Binary) code of cMax=NumMipModes-1. NumMipModes is the number of MIPs available in the target block. For example, depending on the target block size (nTbW, nTnH), cMax may be derived as follows.

cMax = (nTbW==4 && nTbH==4) ? NumMipModes_SizeId0-1 : (nTbW==4 || nTbH==4) || (nTbW==8 && nTbH==8) ? NumMipModes_SizeId1-1 : NumMipModes_SizeId2- 1
For example, but not limited to, NumMipModes_SizeId0=16, NumMipModes_SizeId1=8, NumMipModes_SizeId2=6.

(Example of TB code)
A TB code may be derived as follows.

n = cMax + 1
k = Floor(Log2(n))
u = (1 << (k + 1)) - n
When the value of the syntax element synVal (merge_gpm_partition_idx here) is less than u, the parameter decoding unit 302 uses Fixed Length Binary (hereinafter FL binary) using cMax = (1<<k)-1 to generate TB code. derive Otherwise (synVal is greater than or equal to u), set cMax = (1<<(k+1))-1.

In deriving the FL binary, the parameter decoding unit 302 may derive the BIN length fixedLength of the syntax element, and may derive synVal by binary representation with fixedLength bits.

fixedLength = Ceil(Log2(cMax + 1))
Also, the parameter decoding unit 302 may perform binarization of mpm_merge_gpm_partition_idx using Truncated Rice (TR) code in which cMax is determined and the Rice parameter is set to 0. In the example where mpm_merge_gpm_partition_idx takes a value from 0 to 5 and one is selected from 6 candidates, the value of mpm_merge_gpm_partition_idx is encoded as a maximum 5-bit bit string (binary values: 0, 10, 110, 1110, 11110, 11111). become.

An example of MIP processing (Matrix-based intra prediction) executed by the MIP unit 31045 will be described below. MIP is a technique for deriving a predicted image by multiply-adding a reference image derived from an adjacent image and a weight matrix. FIG. 10 shows the configuration of the MIP section 31045 in this embodiment. The MIP unit 31045 is composed of a matrix reference pixel derivation unit 4501, a matrix prediction image derivation unit 4502, a mode derivation unit 4503, a prediction processing parameter derivation unit 4504, and a matrix prediction image interpolation unit 4505.

(1) Boundary reference pixel derivation
The MIP unit 31045 derives a variable sizeId regarding the size of the target block using the following formula.

sizeId = (nTbW==4 && nTbH==4) ? 0 : (nTbW==4 || nTbH==4) || (nTbW==8 && nTbH==8) ? 1 : 2 (MIP-1)
Next, the MIP unit 31045 uses sizeId to determine the total number of MIP modes numTotalMipModes, the reference area redT[] after downsampling, the size boundarySize of redL[], the intermediate prediction image predMip[][]
Deriving the width and height of predSize, predSize.

numTotalMipModes = (sizeId==0) ? 32 : (sizeId==1) ? 16 : 12 (MIP-2)
boundarySize = (sizeId==0) ? 2 : 4
predSize = (sizeId<=1) ? 4 : 8
Also, the number of reference pixels used for prediction by the weight matrix mWeight is derived by the following formula.

inSize = 2*boundarySize - ( (sizeId==2) ? 1 : 0 )
The weight matrix mWeight is a matrix whose size is represented by mWeight[predSize*predSize][inSize]. For sizeId=0 and sizeId=1, predSize*predSize=16, inSize=4, and for sizeId=2, predSize*predSize=64, inSize=7.

The matrix reference pixel derivation unit 4501 extracts the pixel values predSamples[x][-1] (x=0..nTbW-1) of the blocks above the target block into the first reference region refT[x](x= 0..nTbW-1). Also, the pixel values predSamples[-1][y](y=0..nTbH-1) of the block adjacent to the left of the target block are stored in the first reference region refL[y](y=0..nTbH-1 ). Next, the matrix reference pixel deriving unit 4501 down-samples the first reference regions refT[x] and refL[y] to obtain the second reference regions redT[x](x=0..boundarySize-1), Derive redL[y](y=0..boundarySize-1). Since downsampling is performed for refT[] and refL[] in the same way, refS[i](i=0..nTbX-1), redS[i](i=0..boundarySize-1 ).

Matrix reference pixel derivation section 4501 performs the following MIP boundary downsampling processing with refT[] as refS[] and nTbs=bTbH to derive redT (=redS[]).

Matrix reference pixel derivation section 4501 performs the following MIP boundary downsampling processing with refL[] as refS[] and nTbs=bTbW to derive redL(=redS[]).

(MIP boundary downsampling processing)
if (boundarySize<nTbS) {
bDwn = nTbS/boundarySize (MIP-3)
for (x=0; x<boundarySize; x++)
redS[x] = (ΣrefS[x*bDwn+i]+(1<<(Log2(bDwn)-1)))>>Log2(bDwn)
}
else
for (x=0; x<boundarySize; x++)
redS[x] = refS[x]
where Σ is the sum from i=0 to i=bDwn-1.

Next, the matrix reference pixel derivation unit 4501 combines the second reference regions redL[] and redT[] to derive p[i] (i=0..2*boundarySize-1). isTransposed sets the value of intra_mip_transposed_flag in the target block.

if (isTransposed==1) (MIP-4)
for (i=0;i<boundarySize;i++) {
pTemp[i] = redL[i]
pTemp[i+boundarySize] = redT[i]
}
else
for (i=0;i<boundarySize;i++) {
pTemp[i] = redT[i]
pTemp[i+boundarySize] = redL[i]
}
if (sizeId==2)
for (i=0;i<inSize;i++)
p[i] = pTemp[i+1]-pTemp[0]
else {
p[0] = pTemp[0] - (1<<(BitDepthY-1))
for (i=1;i<inSize;i++)
p[i] = pTemp[i]-pTemp[0]
}
bitDepthY is the bit depth of luminance and may be 10 bits, for example.

It should be noted that when the above reference pixels cannot be referred to, values of available reference pixels are used as in conventional intra prediction. If all reference pixels cannot be referenced, 1<<(bitDepthY-1) is used as the pixel value. isTransposed indicates whether or not the prediction direction is close to the vertical prediction, so switching between storing redL and redT in the first half of p[] by isTransposed reduces the pattern of mWeight[][] in half. can do.

(2) Prediction mode derivation 4503
MIP section 31045 uses mode derivation section 4503 to derive intra prediction mode modeId used in matrix intra prediction (MIP).

The mode derivation unit 4503 of the MIP unit 31045 derives a prediction method candidate list for the MIP mode to be used in the target block using information on neighboring blocks in the target block. For example, the mode derivation unit 4503 may derive a number mip_set_id indicating the candidate list. Here, if the number of candidate lists is NumMipSet, then mip_set_id = 0..NumMipSet-1. Let NumMipModes be the number of prediction modes in the candidate list, and if different candidate lists do not contain the same prediction mode, the total NumTotalMipModes of MIP prediction modes at a sizeId is NumMipSet * NumMipModes. Note that different candidate lists may contain the same prediction mode.

Here, all lists included in the MIP are referred to as the overall MIP list. It can also be said that the mode derivation unit 4503 derives a subset of the overall MIP list as the target block candidate list.

(Example of mip_set_id derivation)
A process of deriving mip_set_id by the mode derivation unit 4503 is illustrated. The mode derivation unit 4503 derives the value of mip_set_id depending on, for example, whether the following conditions are satisfied.
a) The size of a specific element of p[] and the size relationship between elements
b) Neighboring pixel region activity derived from p[]
c) Features such as mean values derived from the elements of p[]
d) Absolute difference between adjacent pixel values of p[]
e) Quantization parameter QP of target block
The following formula may be used as an example.
a) p[0] < p[3],
b) p[0] + p[1] < p[2] + p[3]
c)(p[0] + … + p[inSize-1])>>log2(insize) >= th_avg,
d) abs(p[1]-p[0]) + … + abs(p[inSize-1] - p[inSize-2]) < th_sad
e) QP < th_qp
where th_avg, th_sad, and th_qp are predetermined constants. Alternatively, it may be derived from a table without using branching.
a) mip_set_id=tbl_grad[p[0] - p[3])]
b) mip_set_id=tbl_act_[(p[0] + p[1]) - (p[2] + p[3])]
c) mip_set_id=tbl_avg[(p[0] + … + p[inSize-1])>>log2(insize)]
d) mip_set_id=tbl_sad[abs(p[1]-p[0]) + … + abs(p[inSize-1] - p[inSize-2])]
e) mip_set_id=tbl_qp[QP]
where tbl_avg, th_act, tbl_avg, tbl_sad, tbl_qp are tables respectively.

(Derivation example 1 of MIP prediction method)
The MIP unit 31045 may be configured such that the mode derivation unit 4503 derives mip_set_id from the surroundings, and the prediction processing parameter derivation unit 4504 derives mWeight from modeId and sizeId obtained from intra_mip_mode_idx of encoded data. A modeId is derived using intra_mip_mode_idx decoded from the encoded data.
modeId = intra_mip_mode_idx
The MIP unit 31045 derives mWeight from mip_set_id, modeId (intra_mip_mode_idx), and sizeId.

For example, the MIP unit 31045 may derive a matrix from mip_set_id, modeId, and sizeId by referring to a table as follows.

mWeight = mWeightTable[sizeId][mip_set_id][modeId]
Alternatively, a particular table may be selected by branching as follows.

if (sizeId == 0 && mip_set_id == 0 && modeId==0)
mWeight = mWeightTable[0][0][0]
else if (sizeId == 0 && mip_set_id == 0 && modeId==1)
mWeight = mWeightTable[0][0][1]
else if (sizeId == 0 && mip_set_id == 1 && modeId==0)
mWeight = mWeightTable[0][1][0]
else if (sizeId == 0 && mip_set_id == 1 && modeId==1)
mWeight = mWeightTable[0][1][1]
else if (sizeId == 1 && mip_set_id == 0 && modeId==0)
mWeight = mWeightTable[1][0][0]
else if (sizeId == 1 && mip_set_id == 0 && modeId==1)
mWeight = mWeightTable[1][0][1]
…
else if (sizeId == 2 && mip_set_id == NumMipSet-1 && modeId==0)
mWeight = mWeightTable[2][NumMipSet-1][0]
else if (sizeId == 2 && mip_set_id == NumMipSet-1 && modeId==1)
mWeight = mWeightTable[2][NumMipSet-1][1]
(Derivation example 2 of MIP prediction method)
The mode derivation unit 4503 may be configured to derive a candidate list indicated by mip_set_id and derive a prediction mode from the candidate list and intra_mip_mode_idx. MIP selected by mode derivation unit 4503
A candidate list (subset) of may be:
・List of MIP prediction mode numbers (Structure 1)
・List of matrices used for MIP (Composition 2)
・List of neural network parameters used for MIP (Composition 3)
(Specific example of configuration 1)
The mode derivation unit 4503 derives a selectable modeId candidate list modeIdCandListSet from the sizeId and mip_set_id, and the MIP modeId candidate list set modeIdCandListSet.

modeIdCandList = modeIdCandListSet[mip_set_id]
Here, modeIdCandList[] is a list whose elements are modeId.
For example, the following may be used.

modeIdCandListSet[0][] = {0, 1, 2, 3}
modeIdCandListSet[1][] = {0, 1, 4, 5}
modeIdCandListSet[2][] = {0, 1, 6, 7}
…
In this example, if mip_set_id=0, modeIdCandList[] = {0, 1, 2, 3}.
The mode derivation unit 4503 derives modeId from modeIdCandList and intra_mip_mode_idx.

modeId = modeIdCandList[intra_mip_mode_idx]
Also, modeIdCandListSet may be a set of candidate lists by sizeId as described below.

modeIdCandList = modeIdCandListSet[sizeId][mip_set_id]
modeIdCandListSet[sizeId][0][] = {0, 1, 2, 3}
modeIdCandListSet[sizeId][1][] = {0, 1, 4, 5}
modeIdCandListSet[sizeId][2][] = {0, 1, 6, 7}
…
That is, modeId = modeIdCandListSet[sizeId][mip_set_id][intra_mip_mode_idx].

The prediction processing parameter derivation unit 4504 selects a weight matrix mWeight[predSize*predSize][inSize] from the set of matrices by referring to sizeId and modeId.

When sizeId=0, the prediction process parameter derivation unit 4504 selects mWeight[16][4] from the array WeightS0[16][16][4] storing the weight matrix by referring to modeId. When sizeId=1, select mWeight[16][8] from the array WeightS1[8][16][8] that stores the weight matrix by referring to modeId. When sizeId=2, select mWeight[64][7] from the array WeightS2[6][64][7] that stores the weight matrix by referring to modeId. These are represented by the following formulas.

if (sizeId==0) (MIP-5)
mWeight[i][j] = WeightS0[modeId][i][j] (i=0..15, j=0..3)
else if (sizeId==1)
mWeight[i][j] = WeightS1[modeId][i][j] (i=0..15, j=0..7)
else // sizeId=2
mWeight[i][j] = WeightS2[modeId][i][j] (i=0..63, j=0..6)
(Specific example of configuration 2)
The mode derivation unit 4503 of the MIP unit 31045 derives a candidate list matrixCandList of selectable matrices from sizeId and mip_set_id and all candidate lists matrixCandListSet of matrices of selectable MIPs as follows.

matrixCandList = matrixCandListSet[sizeId][mip_set_id]
Here, matrixCandList[] is a list whose elements are weight matrices mWeightX corresponding to one prediction mode. mWeightX is a matrix of size (predSize*predSize, inSize) respectively.
For example, the following may be used.

matrixCandListSet[sizeId][0][] = {mWeight0, mWeight1, mWeight2, mWeight3}
matrixCandListSet[sizeId][1][] = {mWeight0, mWeight1, mWeight4, mWeight5}
matrixCandListSet[sizeId][2][] = {mWeight0, mWeight1, mWeight6, mWeight7}
…
The prediction processing parameter derivation unit 4504 of the MIP unit 31045 derives mWeight from matrixCandList and intra_mip_mode_idx.

mWeight = matrixCandList[intra_mip_mode_idx]
The whole is shown below.

mWeight = matrixCandListSet[sizeId][mip_set_id][intra_mip_mode_idx]
In addition, already explained (Derivation example 1 of the MIP prediction method) formally performs the same operation as follows.

mWeight = mWeightTable[sizeId][mip_set_id][intra_mip_mode_idx]
(Specific example of configuration 3)
The mode derivation unit 4503 of the MIP unit 31045 derives a candidate list modelCandList of selectable neural network models from mip_set_id and all candidate list modelCandListSet of selectable MIP neural network models as follows.

modelCandList = modelCandListSet[sizeId][mip_set_id]
Here, modelCandList[] is a list whose elements are neural networks NNX corresponding to one prediction mode. NNX is a parameter representing a neural network model that inputs input data p[] of length inSize and outputs an intermediate predicted image of (predSize*predSize).
For example, the following may be used.

modelCandListSet[0][] = {NN0, NN1, NN2, NN3}
modelCandListSet[1][] = {NN0, NN1, NN4, NN5}
modelCandListSet[2][] = {NN0, NN1, NN6, NN7}
…
The prediction processing parameter deriving unit 4504 of the MIP unit 31045 uses modelCandList and intra_mip_mode_idx
derive the neural network NN used for prediction from

NN = modelCandList[intra_mip_mode_idx]
A neural network NN is represented by a network structure and parameters (weight and bias values). Alternatively, it may be an index or parameter for indirectly specifying such information. For example, a neural network with inSize input layers and predSize*predSize output layers fully connected has inSize*predSize*predSize weight parameters.

In this way, by specifying a mode to be used from a candidate list smaller than the total number of modes, the amount of syntax data required for selection can be reduced, and coding efficiency can be improved. For example, consider a case where the total number of modes of MIP at a certain sizeId is 2L, and a candidate list containing L elements (modes) is derived for the target block. At this time, the matrix intra prediction mode indicator intra_mip_mode_idx can specify a mode with a data amount that is 1 bit smaller than when one mode is selected from all matrix intra prediction modes. That is, by reducing the size of one candidate list to half or less of the total number of modes, it is possible to select a prediction mode with a small amount of data. Candidate modes are preferably defined such that each prediction mode belongs to one of the candidate modes. As another example, if the total number of modes is 4L and a candidate list with L elements is derived, the mode can be specified with 2 bits less data.

(Another specific example of candidate list derivation)
The mode derivation unit 4503 of the MIP unit 31045 may derive the prediction mode candidate list for the target block each time based on p[ ] and other values. For example:

Initialize the candidate list candList. In the example below, the initial state is empty, but may contain one or more elements. Elements of candList may be the intra prediction mode modeId, the weight matrix mWeight, or the neural network NN, as described above. MIP unit 31045 adds an element to candList if a predetermined condition is satisfied.

Specifically, the conditions used here include the evaluation formula based on the following.
a) The size of a specific element of p[] and the size relationship between elements
b) Features such as mean values derived from the elements of p[]
c) absolute difference between adjacent pixel values of p[]
d) Neighboring pixel region activity derived from p[]
e) Quantization parameter QP of target block
If exemplified by a formula, it can be expressed as follows.
a) p[0] < p[3],
b) p[0] + p[1] < p[2] + p[3]
c) (p[0] + … + p[inSize-1])/insize >= th_avg,
d) abs(p[1]-p[0]) + … + abs(p[inSize-1] - p[inSize-2]) < th_sad
e) QP < th_qp
Alternatively, other information than p[] may be used to evaluate the condition. For example:
a) Whether the quantization parameter (QP) of the target block or adjacent block satisfies a predetermined range and magnitude relationship
b) the size of the neighboring block, whether the intra prediction mode of the neighboring block is of a particular type (Planar prediction, Angular prediction, matrix intra prediction, etc.)
c) Whether or not the mode number is equal to a particular mode, or whether it is a complex condition based on the logic operation of these conditions may be used.

In this way, the mode derivation unit 4503 of the MIP unit 31045 derives the final candidate list candList based on cond[sizeId][X] and addList[sizeId][X]. If candList is a list of mode IDs derived according to configuration example 1, the MIP unit 31045 derives mode modeId used for prediction from the candidate list candList and intra_mip_mode_idx.

modeId = candList[intra_mip_mode_idx]
The method by which the prediction process parameter derivation unit 4504 derives mWeight from this is as explained in the specific example of configuration 1. FIG.

If candList is a weight matrix list derived by configuration example 2, the prediction processing parameter deriving unit 4504 uses candList and intra_mip_mode_idx to derive mWeight as follows.

mWeight = candList[intra_mip_mode_idx]
If candList is a neural network list derived according to configuration example 3, the prediction processing parameter derivation unit 4504 derives NN as follows using candList and intra_mip_mode_idx.

NN = candList[intra_mip_mode_idx]
(3) Predicted pixel derivation (matrix operation)
In the case of configuration example 1 and configuration example 2, the MIP unit 31045 performs STEP 3 predicted pixel derivation (matrix operation) in FIG.
, an intermediate predicted image predMip[][] of size predSize*predSize is derived by matrix operation on p[]. First, an example of using the weighting matrix mWeight[][] derived in Configuration Example 1 and Configuration Example 2 to derive the intermediate predicted image predMip[][] will be described.

The matrix prediction image deriving unit 4502 of the MIP unit 31045 derives predMip[][] with a size of predSize*predSize by performing a (MIP-7) sum-of-products operation on p[]. Here, the elements of the weight matrix mWeight[][] are referenced for each corresponding position of predMip[][] to derive the intermediate predicted image.

oW = 32 - 32*Σp[i]
for (x=0; x<predSize; x++) (MIP-7)
for (y=0; y<predSize; y++) {
predMip[x][y] = (((ΣmWeight[i][y*predSize+x]*p[i])+ oW)>>6) + pTemp[0]
predMip[x][y] = Clip1Y(predMip[x][y])
}
Σ is the sum from i=0 to i=inSize-1.

When isTransposed=1, the input p[] to the sum-of-products operation stores the positions of the upper reference pixel and the left-side reference pixel interchanged. ) before outputting for processing.

if (isTransposed==1) { (MIP-8)
for (x=0; x<predSize; x++)
for (y=0; y<predSize; y++)
tmpPred[x][y] = predMip[y][x]
for (x=0; x<predSize; x++)
for (y=0; y<predSize; y++)
predMip[x][y] = tmp Pred[x][y]
}
The matrix predicted image deriving unit 4502 can also use the neural network NN derived in configuration example 3 instead of the two-dimensional weight matrix mWeight[][] in deriving the intermediate predicted image predMip[][]. A neural network NN is a model (network structure) in which an input layer receives one-dimensional data with the number of elements inSize and an output layer outputs two-dimensional data with predSize x predSize. When the function func_NN for conversion by this network and the input data p[] are input, the predicted image predMip of predSize x predSize is expressed by the following equation.

predMip = func_NN(p) (MIP-9)
Here, the process of transposing the output when isTransposed=1 is as described above (MIP-8).

In addition, the neural network NN may receive parameters other than p[] derived from adjacent pixel values. For example, the prediction modes IntraPredModeT and IntraPredModeL of the upper and left neighboring blocks, the QP value of the target block or the neighboring block, and the like. Based on these additional parameters in addition to p[ ], it is possible to derive a predicted image that takes into account coding information around the target block.

The number and structure of intermediate layers (hidden layers) of the neural network NN may be arbitrarily configured. However, since the amount of calculation increases according to the complexity of the network, a simple configuration such as one or two layers is desirable. Moreover, it is not preferable that the amount of calculation of the network varies greatly depending on the model. Therefore, for prediction modes belonging to the same sizeId, it is desirable to keep the amount of calculation constant by, for example, using the same model and changing parameters.

(4) Prediction pixel derivation (linear interpolation)
When nTbW=predSize and nTbH=predSize, the matrix prediction image interpolation unit 4505 of the MIP unit 31045 copies predMip[][] to predsamples[][].

for (x=0; x<nTbW; x++)
for (y=0; y<nTbH; y++)
predSamples[x][y] = predMip[x][y]
Otherwise (nTbW>predSize or nTbH>predSize), the matrix prediction image interpolation unit 4505 performs prediction image predSamples of size nTbW*nTbH in STEP 4 Predicted pixel derivation (linear interpolation) 4-1 in FIG. Store predMip[][] in [][]. When predSize is different from nTbW and nTbH, the predicted pixel value is interpolated in 4-2.

(4-1) The matrix prediction image interpolation unit 4505 stores predMip[][] in predSamples[][]. That is, in the pre-interpolation image of FIG. 12, predMip[][] is stored in the shaded pixel position in the upper right and lower left direction.

upHor = nTbW/predSize (MIP-10)
upVer = nTbH/predSize
for (x=0; x<predSize; x++)
for (y=0; y<predSize; y++)
predSamples[(x+1)*upHor-1][(y+1)*upVer-1] = predMip[x][y]
(4-2) If nTbH>nTbW, the pixels not stored in (4-1) are interpolated using the pixel values of adjacent blocks in the horizontal and vertical directions to generate a predicted image.

Perform horizontal interpolation, and use predSamples[xHor][yHor] and predSamples[xHor+upHor][yHor] (shaded pixels in the image after horizontal interpolation in the figure) to determine the pixel values at the positions indicated by "○". derive

for (m=0; m<predSize; m++) (MIP-11)
for (n=1; n<=predSize; n++)
for (dX=1; dX<upHor; dX++) {
xHor = m*upHor-1
yHor = n*upVer-1
sum = (upHor-dX)*predSamples[xHor][yHor]+dX*predSamples[xHor+upHor][yHor]
predSamples[xHor+dX][yHor] = (sum+upHor/2)/upHor
}
After horizontal interpolation, use predSamples[xVer][yVer] and predSamples[xVer][yVer+upVer] (shaded pixels in the image after vertical interpolation in the figure) to derive the pixel value at the position indicated by "○" .

for (m=0; m<nTbW; m++) (MIP-12)
for (n=0; n<predSize; n++)
for (dY=1; dY<upVer; dY++) {
xVer = m
yVer = n*upVer-1
sum = (upVer-dY)*predSamples[xVer][yVer]+dY*predSamples[xVer][yVer+upVer]
predSamples[xVer][yVer+dY] = (sum+upVer/2)/upVer
}
When nTbH<=nTbW, interpolation is performed using pixel values of adjacent blocks in the order of vertical and horizontal directions to generate a predicted image. The vertical and horizontal interpolation process is the same as for nTbH>nTbW.

The inverse quantization/inverse transform unit 311 inversely quantizes the quantized transform coefficients input from the entropy decoding unit 301 to obtain transform coefficients. This quantized transform coefficient is obtained by quantizing the prediction error by performing frequency transform such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) in the encoding process. is the coefficient. The inverse quantization/inverse transform unit 311 performs inverse frequency transform such as inverse DCT and inverse DST on the obtained transform coefficients to calculate prediction errors. Inverse quantization/inverse transform section 311 outputs the prediction error to addition section 312 .

The addition unit 312 adds the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization/inverse transform unit 311 for each pixel to generate a decoded image of the block.
The adder 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305 .

(Configuration of video encoding device)
Next, the configuration of the video encoding device 11 according to this embodiment will be described. FIG. 13 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a predicted image generation unit 101, a subtraction unit 102, a transformation/quantization unit 103, an inverse quantization/inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit , frame memory) 108 , reference picture memory (reference image storage unit, frame memory) 109 , coding parameter determination unit 110 , parameter coding unit 111 and entropy coding unit 104 .

The predicted image generation unit 101 generates a predicted image for each CU, which is an area obtained by dividing each picture of the image T. FIG. The operation of the predicted image generation unit 101 is the same as that of the predicted image generation unit 308 already described, and the description thereof is omitted.

The subtraction unit 102 subtracts the pixel values of the predicted image of the block input from the predicted image generation unit 101 from the pixel values of the image T to generate a prediction error. Subtraction section 102 outputs the prediction error to transform/quantization section 103 .

Transform/quantization section 103 calculates transform coefficients by frequency transforming the prediction error input from subtraction section 102, and derives quantized transform coefficients by quantization. The transform/quantization unit 103 is
The quantized transform coefficients are output to entropy coding section 104 and inverse quantization/inverse transform section 105 .

The inverse quantization/inverse transform unit 105 is an inverse quantization/inverse transform unit 311 (FIG. 7) in the video decoding device 31.
is the same as , and the description is omitted. The calculated prediction error is output to addition section 106 .

Entropy coding section 104 receives the quantized transform coefficients from transform/quantization section 103 and the coding parameters from parameter coding section 111 . The entropy encoding unit 104 entropy-encodes the division information, prediction parameters, quantized transform coefficients, and the like to generate and output an encoded stream Te.

Parameter coding section 111 includes header coding section 1110, CT information coding section 1111, CU
An encoding unit 1112 (prediction mode encoding unit), an inter prediction parameter encoding unit 112 and an intra prediction parameter encoding unit 113 are provided. CU encoding section 1112 further comprises TU encoding section 1114 .

(Configuration of intra prediction parameter encoding section 113)
Intra prediction parameter encoding section 113 derives a format for encoding (for example, intra_luma_mpm_idx, intra_luma_mpm_remmainder, etc.) from IntraPredMode input from encoding parameter determination section 110 . Intra prediction parameter encoding section 113 includes a configuration that is partly the same as the configuration in which intra prediction parameter decoding section 304 derives intra prediction parameters.

The addition unit 106 adds pixel values of the predicted image of the block input from the predicted image generation unit 101 and prediction errors input from the inverse quantization/inverse transform unit 105 for each pixel to generate a decoded image. The addition unit 106 stores the generated decoded image in the reference picture memory 109 .

A loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the adder 106 . Note that the loop filter 107 does not necessarily include the three types of filters described above, and may be configured with only a deblocking filter, for example.

The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in predetermined positions for each current picture and CU.

The reference picture memory 109 stores the decoded image generated by the loop filter 107 in a predetermined position for each target picture and CU.

Coding parameter determination section 110 selects one set from a plurality of sets of coding parameters. The coding parameter is the above-described QT, BT or TT division information, prediction parameters, or parameters to be coded generated in relation to these. The predicted image generator 101 uses these coding parameters to generate predicted images.

Coding parameter determination section 110 calculates an RD cost value indicating the amount of information and the coding error for each of the plurality of sets. Coding parameter determination section 110 selects a set of coding parameters that minimizes the calculated cost value. As a result, entropy encoding section 104 outputs the selected set of encoding parameters as encoded stream Te. Coding parameter determination section 110 stores the determined coding parameters in prediction parameter memory 108 .

Note that a part of the video encoding device 11 and the video decoding device 31 in the above-described embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the prediction image generation unit 308, the inverse quantization/inverse Transformation unit 311, addition unit 312, prediction image generation unit 101, subtraction unit 102, transformation/quantization unit 103, entropy coding unit 104, inverse quantization/inverse transformation unit 105, loop filter 107, coding parameter determination unit 110 , the parameter encoding unit 111 may be realized by a computer. In that case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. The "computer system" here is a computer system built in either the moving image encoding device 11 or the moving image decoding device 31, and includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically stores a program for a short period of time, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. In that case, it may also include a memory that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Also, part or all of the moving image encoding device 11 and the moving image decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the moving image encoding device 11 and the moving image decoding device 31 may be processorized individually, or may be partially or wholly integrated and processorized. Also, the method of circuit integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc., can be made without departing from the gist of the present invention. It is possible to

[Application example]
The moving image encoding device 11 and the moving image decoding device 31 described above can be used by being installed in various devices for transmitting, receiving, recording, and reproducing moving images. The moving image may be a natural moving image captured by a camera or the like, or may be an artificial moving image (including CG and GUI) generated by a computer or the like.

First, it will be described with reference to FIG. 2 that the moving image encoding device 11 and the moving image decoding device 31 described above can be used for transmitting and receiving moving images.

FIG. 2 shows a block diagram showing the configuration of the transmission device PROD_A equipped with the video encoding device 11. As shown in FIG. As shown in the figure, the transmission device PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding a moving image, and a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1. and a transmitter PROD_A3 for transmitting the modulated signal obtained by the modulator PROD_A2. The video encoding device 11 described above is used as this encoding unit PROD_A1.

The transmission device PROD_A uses a camera PROD_A4 for capturing a moving image, a recording medium PROD_A5 for recording the moving image, an input terminal PROD_A6 for externally inputting the moving image, and , and an image processing unit A7 for generating or processing an image. In the drawing, the configuration in which the transmitter PROD_A has all of these is illustrated, but some of them may be omitted.

The recording medium PROD_A5 may record an unencoded moving image, or record a moving image encoded by an encoding scheme for recording different from the encoding scheme for transmission. can be anything. In the latter case, a decoding unit (
(not shown) may be interposed.

Also, FIG. 2 shows a block diagram showing the configuration of the receiving device PROD_B equipped with the video decoding device 31. As shown in FIG. As shown in the figure, the receiver PROD_B includes a receiver PROD_B1 that receives a modulated signal, a demodulator PROD_B2 that obtains encoded data by demodulating the modulated signal received by the receiver PROD_B1, and a demodulator PROD_B2 that obtains encoded data. and a decoding unit PROD_B3 that obtains a moving image by decoding the encoded data. The video decoding device 31 described above is used as this decoding unit PROD_B3.

The receiving device PROD_B supplies the moving image output from the decoding unit PROD_B3 to a display PROD_B4 for displaying the moving image, a recording medium PROD_B5 for recording the moving image, and an output terminal for outputting the moving image to the outside. PROD_B6 may also be provided. In the drawing, the configuration in which the receiving device PROD_B has all of these is illustrated, but some of them may be omitted.

The recording medium PROD_B5 may be used for recording unencoded moving images, or may be encoded by an encoding scheme for recording that is different from the encoding scheme for transmission. may In the latter case, an encoding unit (not shown) may be interposed between the decoding unit PROD_B3 and the recording medium PROD_B5 to encode the moving image acquired from the decoding unit PROD_B3 according to the recording encoding method.

A transmission medium for transmitting the modulated signal may be wireless or wired. Further, the transmission mode for transmitting the modulated signal may be broadcasting (here, transmission mode in which the destination is not specified in advance), or communication (here, transmission mode in which the destination is specified in advance). aspect) may be used. That is, transmission of the modulated signal may be realized by any of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

For example, a digital terrestrial broadcasting station (broadcasting equipment, etc.)/receiving station (television receiver, etc.) is an example of a transmitting device PROD_A/receiving device PROD_B that transmits and receives a modulated signal by radio broadcasting. A broadcasting station (broadcasting equipment, etc.)/receiving station (television receiver, etc.) of cable television broadcasting is an example of a transmitting device PROD_A/receiving device PROD_B that transmits/receives a modulated signal by cable broadcasting.

In addition, servers (workstations, etc.) and clients (television receivers, personal computers, smartphones, etc.) for VOD (Video On Demand) services and video sharing services using the Internet are transmission devices that transmit and receive modulated signals through communication. This is an example of PROD_A/receiving device PROD_B (usually, in LAN, either wireless or wired transmission medium is used, and in WAN, wired transmission medium is used). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multifunctional mobile phone terminals.

A client of the video sharing service has a function of decoding encoded data downloaded from a server and displaying the data on a display, as well as a function of encoding a video captured by a camera and uploading it to the server. That is, the client of the video sharing service functions as both the transmitting device PROD_A and the receiving device PROD_B.

Next, it will be described with reference to FIG. 3 that the moving image encoding device 11 and the moving image decoding device 31 described above can be used for recording and reproducing moving images.

FIG. 3 shows a block diagram showing the configuration of a recording device PROD_C equipped with the moving image encoding device 11 described above. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding a moving image, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 to the recording medium PROD_M. and have. Video encoding device 11 described above
is used as this encoder PROD_C1.

The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), or (2) an SD memory. Storage devices PROD_ such as cards and USB (Universal Serial Bus) flash memory
(3) DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark) built into the recording device PROD_C It may be loaded in a drive device (not shown).

In addition, the recording device PROD_C includes a camera PROD_C3 for capturing the moving image, an input terminal PROD_C4 for inputting the moving image from the outside, and a receiving terminal for receiving the moving image as a supply source of the moving image to be input to the encoding unit PROD_C1. It may further include a unit PROD_C5 and an image processing unit PROD_C6 that generates or processes an image. In the drawing, the configuration in which the recording device PROD_C includes all of these is exemplified, but some of them may be omitted.

Note that the receiving unit PROD_C5 may receive an unencoded moving image, or receive encoded data encoded by an encoding scheme for transmission that is different from the encoding scheme for recording. It may be something to do. In the latter case, it is preferable to interpose a decoding unit for transmission (not shown) that decodes encoded data encoded by an encoding method for transmission between the receiving unit PROD_C5 and the encoding unit PROD_C1.

Such a recording device PROD_C includes, for example, a DVD recorder, a BD recorder, an HDD (Hard Disk Drive) recorder, etc. (In this case, the input terminal PROD_C4 or the receiver PROD_C5 is the main supply source of moving images.) . In addition, a camcorder (in this case, the camera PROD_C3 is the main source of moving images), a personal computer (in this case, the receiver PROD_C5 or the image processing unit C6 is the main source of moving images), a smartphone (in this case, the camera PROD_C3 is the main source of moving images) In this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main supply source of moving images) is also an example of such a recording device PROD_C.

Further, FIG. 3 shows a block diagram showing the configuration of a playback device PROD_D equipped with the moving image decoding device 31 described above. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads encoded data written to the recording medium PROD_M, and a decoding unit PROD_D2 that obtains a moving image by decoding the encoded data read by the reading unit PROD_D1. , is equipped with The video decoding device 31 described above is used as this decoding unit PROD_D2.

Note that the recording medium PROD_M may be (1) a type built into the playback device PROD_D, such as an HDD or SSD, or (2) an SD memory card, USB flash memory, or the like. It may be of a type that is connected to the playback device PROD_D, or (3) like a DVD or BD, it may be loaded into a drive device (not shown) built into the playback device PROD_D. good.

Further, the playback device PROD_D includes a display PROD_D3 for displaying the moving image, an output terminal PROD_D4 for outputting the moving image to the outside, and a transmitting unit for transmitting the moving image, as destinations to which the moving image output by the decoding unit PROD_D2 is supplied. PROD_D5 may also be provided. In the drawing, the configuration in which the playback device PROD_D includes all of these is illustrated, but some of them may be omitted.

The transmission unit PROD_D5 may transmit an unencoded moving image, or transmit encoded data encoded by an encoding scheme for transmission different from the encoding scheme for recording. It may be something to do. In the latter case, it is preferable to interpose an encoding unit (not shown) that encodes a moving image using an encoding method for transmission between the decoding unit PROD_D2 and the transmitting unit PROD_D5.

Such a playback device PROD_D includes, for example, a DVD player, a BD player, an HDD player, etc. (In this case, the output terminal PROD_D4 to which a television receiver or the like is connected is the main supply destination of moving images.) . In addition, television receivers (in this case, display PROD_D3 is the main supply of moving images), digital signage (also called electronic billboards, electronic bulletin boards, etc.), and display PROD_D3 or transmission unit PROD_D5 is the main supply of moving images. first), desktop PC (in this case, output terminal PROD_D4 or transmitter PROD_D5 is the main destination of the video), laptop or tablet PC (in this case, display PROD_D3 or transmitter PROD_D5 An example of such a playback device PROD_D is a smartphone (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main destination of moving images).

(Hardware realization and software realization)
Further, each block of the moving image decoding device 31 and the moving image encoding device 11 described above may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip).
(Central Processing Unit) may be used for software implementation.

In the latter case, each of the above devices has a CPU that executes the instructions of the program that implements each function, a ROM (Read Only Memory) that stores the above program, and a RAM (Random Memory) that expands the above program.
Access Memory), and a storage device (recording medium) such as a memory for storing the above program and various data. An object of the embodiments of the present invention is a computer-readable record of the program code (executable program, intermediate code program, source program) of the control program for each of the above devices, which is software for realizing the above functions. It can also be achieved by supplying a medium to each of the devices described above and causing the computer (or CPU or MPU) to read and execute the program code recorded on the recording medium.

Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks/hard disks, CD-ROMs (Compact Disc Read-Only Memory)/MO disks (Magneto-Optical discs). ) / MD (Mini Disc) / DVD (Digital Versatile Disc: registered trademark) / CD-R (CD Recordable) / Blu-ray Disc (Blu-ray Disc: registered trademark) and other optical discs, IC cards (memory cards ) / cards such as optical cards, mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark) / semiconductor memories such as flash ROM, or PLD ( Logic circuits such as Programmable logic device) and FPGA (Field Programmable Gate Array) can be used.

Further, each device may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. This communication network is not particularly limited as long as it can transmit the program code. For example, Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television/Cable Television) communication network, Virtual Private Network Network), telephone line network, mobile communication network, satellite communication network, etc. can be used. Also, the transmission medium constituting this communication network is not limited to a specific configuration or type as long as it can transmit the program code. For example, even wired such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA (Infrared Data Association) and remote control , Bluetooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone networks, satellite circuits, terrestrial digital broadcasting networks, etc. Also available wirelessly. Embodiments of the invention may also be implemented in the form of a computer data signal embedded in a carrier wave, with the program code embodied in electronic transmission.

The embodiments of the present invention are not limited to the embodiments described above, and various modifications are possible within the scope of the claims. That is, the technical scope of the present invention also includes embodiments obtained by combining technical means appropriately modified within the scope of the claims.

INDUSTRIAL APPLICABILITY Embodiments of the present invention are preferably applied to a moving image decoding device that decodes encoded image data and a moving image encoding device that generates encoded image data. be able to. Also, the present invention can be preferably applied to the data structure of encoded data generated by a video encoding device and referenced by a video decoding device.

31 Image decoder
301 Entropy Decoder
302 Parameter decoder
303 Inter Prediction Parameter Decoding Unit
304 Intra prediction parameter decoder
308 Predictive image generator
309 Inter prediction image generator
310 Intra prediction image generator
311 Inverse Quantization/Inverse Transform Unit
312 adder
31045 MIP section
4501 Matrix reference pixel derivation unit
4502 Matrix prediction image derivation unit
4503 mode derivation unit
4504 Prediction processing parameter derivation unit
4505 Matrix prediction image interpolation unit
11 Image encoding device
101 Predictive image generator
102 Subtractor
103 Transform/Quantization Unit
104 Entropy Encoder
105 Inverse Quantization/Inverse Transform Unit
107 loop filter
110 Encoding parameter determination unit
111 Parameter encoder
112 Inter prediction parameter coding unit
113 Intra prediction parameter encoder
1110 Header encoder
1111 CT information encoder
1112 CU encoder (prediction mode encoder)
1114 TU encoder

Claims (5)

a matrix reference pixel deriving unit that derives an image obtained by down-sampling images adjacent to the upper and left sides of a target block as a reference image;
A mode derivation unit that derives a prediction mode candidate list used in the target block according to the reference image and the target block size;
A prediction processing parameter derivation unit that derives parameters used for deriving a predicted image according to the candidate list, the matrix intra prediction mode indicator, and the target block size;
a matrix predicted image derivation unit that derives a predicted image based on the elements of the reference image and the prediction processing parameters;
a matrix predicted image interpolation unit that derives the predicted image or an image obtained by interpolating the predicted image as a predicted image;
wherein the mode derivation unit derives a candidate list with a number of elements less than half of the total number of prediction modes defined for the target block size. Image decoder. 2. The moving image decoding according to claim 1, wherein the mode derivation unit derives the candidate list based on a magnitude relation between pixel values included in the reference image or between a pixel value and a threshold. Device. The mode derivation unit, based on the magnitude relationship between the feature amount and the feature amount, or between the feature amount and the threshold value, derived from the pixel values included in the reference image using any one of the average, difference, and absolute value calculations. 2. The moving picture decoding device according to claim 1, wherein the candidate list is derived by 2. The moving image decoding device according to claim 1, wherein the mode derivation unit derives the candidate list using prediction modes of adjacent blocks or quantization parameters. a matrix reference pixel deriving unit that derives an image obtained by down-sampling images adjacent to the upper and left sides of a target block as a reference image;
A mode derivation unit that derives a prediction mode candidate list used in the target block according to the reference image and the target block size;
A prediction processing parameter derivation unit that derives parameters used for deriving a predicted image according to the candidate list, the intra prediction mode, and the target block size;
a matrix predicted image derivation unit that derives a predicted image based on the elements of the reference image and the prediction processing parameters;
a matrix predicted image interpolation unit that derives the predicted image or an image obtained by interpolating the predicted image as a predicted image;
wherein the mode derivation unit derives a candidate list with a number of elements less than half of the total number of prediction modes defined for the target block size. Image encoding device.

Images