JP2023181459A - Moving-image decoding device, moving-image encoding device, moving-image decoding method, and moving-image encoding method - Google Patents

Abstract

To solve a problem that prediction to increase image quality utilizing gradient images does not lead to reduction in an amount of throughput.SOLUTION: There is provided a moving-image decoding method that generates a third prediction image by BDOF prediction, using a first prediction image, a second prediction image and a motion compensating correction value, the method including a step for deriving a difference absolute value sum of the first and second prediction images. The third prediction image is derived by the first and second prediction images, the motion compensating correction value, a first offset value, and a first shift value; the motion compensating correction value is derived by a gradient image in the horizontal and vertical directions of the first prediction image and a gradient image in the horizontal and vertical directions of the second prediction image; the gradient image in the horizontal and vertical directions of the first prediction image is derived by the first prediction image and the second shift value; and the gradient image in the horizontal and vertical directions of the second prediction image is derived by the second prediction image and the second shift value.SELECTED DRAWING: Figure 12

Description

Embodiments of the present invention relate to a video decoding device, a video encoding device, a video decoding method, and a video encoding method.

In order to efficiently transmit or record moving images, a moving image encoding device generates encoded data by encoding a moving image, and a moving image generates a decoded image by decoding the encoded data. An image decoding device is used.

Specific video encoding methods include, for example, H.264/AVC and HEVC (High-Efficiency Video Coding).

In such video encoding methods, the images (pictures) that make up a video are divided into slices obtained by dividing the image and coding tree units (CTU) obtained by dividing the slices. ), a coding unit (sometimes called a Coding Unit (CU)) obtained by splitting a coding tree unit, and a transform unit (TU:) obtained by splitting a coding unit. It is managed by a hierarchical structure consisting of CUs (Transform Units), and is encoded/decoded for each CU.

In addition, in such a video encoding method, a predicted image is usually generated based on a locally decoded image obtained by encoding/decoding the input image, and the predicted image is generated from the input image (original image). The prediction error obtained by subtraction (sometimes referred to as a "difference image" or "residual image") is encoded. Methods for generating predicted images include inter-screen prediction (inter prediction) and intra-screen prediction (intra prediction).

In addition, non-patent literature has been cited regarding recent video encoding and decoding techniques, and in particular, gradient images are used to improve image quality when deriving predicted images from bi-predictive motion compensation (interpolated images). BDOF technology is disclosed.

"Versatile Video Coding (Draft 5)", JVET-N1001, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 2019

In Non-Patent Document 1, the BDOF technique requires calculation of the sum of absolute differences between two interpolated images in order to determine whether or not to perform processing for each processing unit. Since processing can be aborted midway, the amount of processing can be reduced on average, but since there is an additional amount of processing for the sum of absolute differences, the amount of processing has not been reduced in the worst case. There was a problem.

A video decoding device according to one aspect of the present invention is a video decoding device that generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value. and a BDOF unit that derives the sum of absolute differences between the first predicted image and the second predicted image in subblock units, and the third predicted image is: (i) the first predicted image; and (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the sum of the first offset value, which is derived by shifting the sum to the right by a first shift value. , the motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, and a horizontal gradient image of the second predicted image, a vertical gradient image of the second predicted image, and the horizontal gradient image of the first predicted image is obtained by right-shifting the first predicted image by a second shift value. The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value. , the horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value, and the vertical gradient image of the second predicted image The directional gradient image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value.

A video encoding device according to one aspect of the present invention generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value. The third predicted image includes: (i) a BDOF unit that derives the sum of absolute differences between the first predicted image and the second predicted image in sub-block units; By right-shifting the sum of the predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value by a first shift value. The motion compensation correction value is derived from a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, and a horizontal gradient image of the second predicted image. and a vertical gradient image of the second predicted image, and the horizontal gradient image of the first predicted image shifts the first predicted image to the right by a second shift value. The vertical gradient image of the first predicted image is derived by taking the difference between the shifted values, and the vertical gradient image of the first predicted image is derived by taking the difference between the values of the first predicted image shifted to the right by the second shift value. The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value, and the horizontal gradient image of the second predicted image The vertical gradient image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value.

A video decoding method according to one aspect of the present invention is a video decoding method that generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value. (i) the third predicted image includes at least the step of deriving the sum of absolute differences between the first predicted image and the second predicted image in sub-block units; and (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the sum of the first offset value, which is derived by shifting the sum to the right by a first shift value. , the motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, and a horizontal gradient image of the second predicted image, a vertical gradient image of the second predicted image, and the horizontal gradient image of the first predicted image is obtained by right-shifting the first predicted image by a second shift value. The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value. , the horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value, and the vertical gradient image of the second predicted image The directional gradient image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value.

A video encoding method according to one aspect of the present invention is a video encoding method that generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value. The third predicted image includes at least the step of deriving the sum of absolute differences between the first predicted image and the second predicted image in sub-block units, and the third predicted image is: (i) the first predicted image; By right-shifting the sum of the predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value by a first shift value. The motion compensation correction value is derived from a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, and a horizontal gradient image of the second predicted image. and a vertical gradient image of the second predicted image, and the horizontal gradient image of the first predicted image shifts the first predicted image to the right by a second shift value. The vertical gradient image of the first predicted image is derived by taking the difference between the shifted values, and the vertical gradient image of the first predicted image is derived by taking the difference between the values of the first predicted image shifted to the right by the second shift value. The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value, and the horizontal gradient image of the second predicted image The vertical gradient image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value.

According to the above configuration, any of the above problems can be solved.

1 is a schematic diagram showing the configuration of an image transmission system according to the present embodiment. 1 is a diagram showing the configurations of a transmitting device equipped with a video encoding device and a receiving device equipped with a video decoding device according to the present embodiment. FIG. (a) shows a transmitting device equipped with a video encoding device, and (b) shows a receiving device equipped with a video decoding device. 1 is a diagram showing the configuration of a recording device equipped with a video encoding device and a playback device equipped with a video decoding device according to the present embodiment. (a) shows a recording device equipped with a video encoding device, and (b) shows a playback device equipped with a video decoding device. FIG. 3 is a diagram showing a hierarchical structure of data of an encoded stream. FIG. 3 is a diagram showing an example of CTU division. FIG. 2 is a conceptual diagram showing an example of a reference picture and a reference picture list. 1 is a schematic diagram showing the configuration of a moving image decoding device. FIG. 2 is a schematic diagram showing the configuration of an inter prediction parameter decoding section. FIG. 2 is a schematic diagram showing the configurations of a merge prediction parameter derivation unit and an AMVP prediction parameter derivation unit. FIG. 2 is a schematic diagram showing the configuration of an inter predicted image generation section. FIG. 3 is a diagram illustrating an example of a flowchart illustrating a process flow in which a motion compensation unit having a motion compensation function using BDOF prediction according to the present embodiment derives a predicted image. FIG. 2 is a schematic diagram showing the configuration of a BDOF unit according to the present embodiment. FIG. 3 is a diagram illustrating an example of a flowchart illustrating the flow of processing in which the BDOF unit according to the present embodiment generates a predicted image. FIG. 3 is a diagram illustrating an example of an area where the BDOF unit according to the present embodiment performs BDOF padding. FIG. 3 is a diagram illustrating an example of a BDOF processing unit and a reading area of a BDOF unit according to the present embodiment. FIG. 1 is a block diagram showing the configuration of a video encoding device. FIG. 2 is a schematic diagram showing the configuration of an inter prediction parameter encoding unit.

(First embodiment)
Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to the present embodiment.

The image transmission system 1 is a system that transmits an encoded stream obtained by encoding an image to be encoded, decodes the transmitted encoded stream, and displays the image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41. .

An image T is input to the moving image encoding device 11.

The network 21 transmits the encoded stream Te generated by the video encoding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily a bidirectional communication network, but may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Further, the network 21 may be replaced by a storage medium on which the encoded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

The video decoding device 31 decodes each encoded stream Te transmitted by the network 21, and generates one or more decoded images Td.

The moving image display device 41 displays all or part of one or more decoded images Td generated by the moving image decoding device 31. The moving image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Display formats include stationary, mobile, HMD, etc. Further, when the video decoding device 31 has high processing capacity, it displays a high quality image, and when it has only a lower processing capacity, it displays an image that does not require high processing capacity or display capacity. .

<Operator>
The operators used in this specification are described below.

>> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || indicates a logical OR.

x?y:z is a ternary operator that takes y if x is true (other than 0) and z if x is false (0).

Clip3(a,b,c) is a function that clips c to a value greater than or equal to a and less than or equal to b, and returns a if c<a, returns b if c>b, and otherwise is a function that returns c (where a<=b).

abs(a) is a function that returns the absolute value of a.

Int(a) is a function that returns the integer value of a.

floor(a) is a function that returns the largest integer less than or equal to a.

ceil(a) is a function that returns the smallest integer greater than or equal to a.

a/d represents the division of a by d (rounding down to the nearest whole number).

a^b represents a raised to the b power.

sign(a) is a function that returns the sign of a. sign(a) = a>0? 1 : a==0? 0 : -1
log2(a) is a function that returns the logarithm of a with base 2.

Max(a, b) is a function that returns a when a>=b and b when a<b.

Min(a, b) is a function that returns a when a<=b and b when a>b.

Round(a) is a function that returns the rounded value of a. Round(a) = sign(a)*floor(abs(a)+0.5).

<Structure of encoded stream Te>
Prior to a detailed explanation of the video encoding device 11 and the video decoding device 31 according to the present embodiment, data of the encoded stream Te generated by the video encoding device 11 and decoded by the video decoding device 31 will be described. Explain the structure.

FIG. 4 is a diagram showing the hierarchical structure of data in the encoded stream Te. The encoded stream Te exemplarily includes a sequence and a plurality of pictures that constitute the sequence. (a) to (f) in FIG. 4 respectively represent an encoded video sequence that defines the sequence SEQ, an encoded picture that defines the picture PICT, an encoded slice that defines the slice S, and an encoded slice that defines the slice data. FIG. 3 is a diagram showing data, a coding tree unit included in coded slice data, and a coding unit included in the coding tree unit.

(encoded video sequence)
In the encoded video sequence, a set of data that the moving image decoding device 31 refers to in order to decode the sequence SEQ to be processed is defined. As shown in Figure 4(a), the sequence SEQ includes a video parameter set (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), a picture PICT, and an additional extension. Contains information SEI (Supplemental Enhancement Information).

Video parameter set VPS is a set of encoding parameters common to multiple video images and encoding parameters related to multiple layers and individual layers included in the video image. A set is defined.

The sequence parameter set SPS defines a set of encoding parameters that the video decoding device 31 refers to in order to decode the target sequence. For example, the width and height of the picture are defined. Note that a plurality of SPSs may exist. In that case, select one of the multiple SPSs from the PPSs.

The picture parameter set PPS defines a set of encoding parameters that the video decoding device 31 refers to in order to decode each picture in the target sequence. For example, it includes a reference value for the quantization width used in picture decoding (pic_init_qp_minus26) and a flag indicating application of weighted prediction (weighted_pred_flag). Note that multiple PPSs may exist. In that case, one of the plurality of PPSs is selected from each picture in the target sequence.

(encoded picture)
In the encoded picture, a set of data that the video decoding device 31 refers to in order to decode the picture PICT to be processed is defined. As shown in FIG. 4(b), the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

Note that hereinafter, if there is no need to distinguish each of slices 0 to NS-1, the subscripts of the symbols may be omitted in the description. Further, the same applies to other data included in the encoded stream Te described below and having subscripts attached thereto.

(encoded slice)
In the encoded slice, a set of data that the video decoding device 31 refers to in order to decode the slice S to be processed is defined. A slice includes a slice header and slice data, as shown in FIG. 4(c).

The slice header includes a group of encoding parameters that the video decoding device 31 refers to in order to determine the decoding method for the target slice. Slice type designation information (slice_type) that designates the slice type is an example of an encoding parameter included in the slice header.

Slice types that can be specified by the slice type designation information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses unidirectional prediction or intra prediction during encoding, (3) Examples include B slices that use unidirectional prediction, bidirectional prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P and B slices, they refer to slices that include blocks for which inter prediction can be used.

Note that the slice header may include a reference (pic_parameter_set_id) to the picture parameter set PPS.

(encoded slice data)
The encoded slice data defines a set of data that the video decoding device 31 refers to in order to decode the slice data to be processed. The slice data includes CTUs, as shown in FIG. 4(d). A CTU is a block of fixed size (for example, 64x64) that constitutes a slice, and is also called a largest coding unit (LCU).

(encoding tree unit)
In FIG. 4(e), a set of data that the video decoding device 31 refers to in order to decode the CTU to be processed is defined. CTU is the basic encoding process using recursive quad tree partitioning (QT (Quad Tree) partitioning), binary tree partitioning (BT (Binary Tree) partitioning), or ternary tree partitioning (TT (Ternary Tree) partitioning). It is divided into encoding units CU, which are standard units. The combination of BT partitioning and TT partitioning is called multi-tree partitioning (MT (Multi Tree) partitioning). A tree-structured node obtained by recursive quadtree partitioning is called a coding node. An intermediate node of a quadtree, a binary tree, and a tertiary tree is a coding node, and the CTU itself is defined as the topmost coding node.

CT includes, as CT information, a QT splitting flag (qt_split_cu_flag) indicating whether to perform QT splitting, an MT splitting flag (mtt_split_cu_flag) indicating whether or not to perform MT splitting, an MT splitting direction (mtt_split_cu_vertical_flag) indicating the splitting direction of MT splitting, Contains the MT split type (mtt_split_cu_binary_flag) that indicates the split type of MT split. qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

FIG. 5 is a diagram showing an example of CTU division. When qt_split_cu_flag is 1, the encoding node is divided into four encoding nodes (FIG. 5(b)).

When qt_split_cu_flag is 0 and mtt_split_cu_flag is 0, the encoding node is not divided and has one CU as a node (FIG. 5(a)). CU is the terminal node of the encoding node and is not further divided. A CU is a basic unit of encoding processing.

When mtt_split_cu_flag is 1, the encoding node is divided into MTs as follows. When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 1, the encoding node is horizontally divided into two encoding nodes (Figure 5(d)), and when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 1, the encoding node is horizontally divided into two encoding nodes. vertically divided into nodes (Figure 5(c)). Furthermore, when mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 0, the encoding node is horizontally divided into 3 encoding nodes (Figure 5(f)), and when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 0, the encoding node is divided into 3 encoding nodes. It is vertically divided into two encoding nodes (Fig. 5(e)). These are shown in Figure 5(g).

Also, if the CTU size is 64x64 pixels, the CU size is 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels. , 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels. .

(encoding unit)
As shown in FIG. 4(f), a set of data that the video decoding device 31 refers to in order to decode the encoding unit to be processed is defined. Specifically, the CU includes a CU header CUH, prediction parameters, transformation parameters, quantized transformation coefficients, and the like. The prediction mode etc. are defined in the CU header.

The prediction process may be performed on a CU basis or on a sub-CU basis, which is obtained by further dividing a CU. If the sizes of the CU and sub-CU are equal, there is one sub-CU in the CU. If the CU is larger than the sub-CU size, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into four sub-CUs, two horizontally and two vertically.

There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is prediction within the same picture, and inter prediction refers to prediction processing performed between mutually different pictures (for example, between display times).

Although the transform/quantization process is performed in units of CUs, the quantized transform coefficients may be entropy encoded in units of subblocks such as 4x4.

(prediction parameter)
A predicted image is derived by prediction parameters associated with a block. The prediction parameters include intra prediction and inter prediction parameters.

The prediction parameters for inter prediction will be explained below. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indexes refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. The prediction list usage flags predFlagL0 and predFlagL1 are flags indicating whether reference picture lists called L0 list and L1 list are used, respectively, and when the value is 1, the corresponding reference picture list is used. In this specification, when the term "flag indicating whether or not XX" is used, a flag other than 0 (for example, 1) is XX, and 0 is not XX, and logical negation, logical product, etc. Treat 1 as true and 0 as false (the same applies hereafter). However, in actual devices and methods, other values can be used as true values and false values.

Syntax elements for deriving inter prediction parameters include, for example, affine flag affine_flag, merge flag merge_flag, merge index merge_idx, inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX, motion vector precision. There is a mode amvr_mode.

(Reference picture list)
The reference picture list is a list of reference pictures stored in reference picture memory 306. FIG. 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In Figure 6(a), the rectangles are pictures, the arrows are reference relationships between pictures, the horizontal axis is time, I, P, and B in the rectangles are intra pictures, uni-predicted pictures, and bi-predicted pictures, and the numbers in the rectangles are decoding Indicates the order. As shown in the figure, the decoding order of pictures is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. FIG. 6(b) shows an example of a reference picture list for picture B3 (target picture). The reference picture list is a list representing reference picture candidates, and one picture (slice) may have one or more reference picture lists. In the illustrated example, target picture B3 has two reference picture lists: L0 list RefPicList0 and L1 list RefPicList1. For each CU, the reference picture index refIdxLX specifies which picture in the reference picture list RefPicListX (X=0 or 1) is actually referred to. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a description method used when not distinguishing between L0 prediction and L1 prediction, and hereinafter, by replacing LX with L0 and L1, parameters for the L0 list and parameters for the L1 list are distinguished.

(Merge prediction and AMVP prediction)
Prediction parameter decoding (encoding) methods include merge prediction (merge) mode and AMVP (Advanced Motion Vector Prediction, adaptive motion vector prediction) mode, and the merge flag merge_flag is a flag for identifying these modes. Merge prediction mode is a mode used in which the prediction list usage flag predFlagLX (or inter prediction identifier inter_pred_idc), reference picture index refIdxLX, and motion vector mvLX are derived from the prediction parameters of already processed neighboring blocks without including them in the encoded data. . AMVP mode is a mode in which the inter prediction identifier inter_pred_idc, reference picture index refIdxLX, and motion vector mvLX are included in encoded data. Note that the motion vector mvLX is encoded as a predictive vector index mvp_LX_idx that identifies the predictive vector mvpLX, a difference vector mvdLX, and a motion vector accuracy mode amvr_mode. In addition to the merge prediction mode, there may be an affine prediction mode identified by the affine flag affine_flag. As one form of merge prediction mode, there may be a skip mode identified by a skip flag skip_flag. Note that the skip mode is a mode in which prediction parameters are derived and used in the same manner as the merge mode, and a prediction error (residual image) is not included in encoded data. In other words, when the skip flag skip_flag is 1, for the target CU, only the skip flag skip_flag and syntax related to the merge mode such as the merge index merge_idx are included, and motion vectors and the like are not included in the encoded data. Therefore, when the skip flag skip_flag indicates that the skip mode is applied to the target CU, decoding of prediction parameters other than the skip flag skip_flag is omitted.

(motion vector)
The motion vector mvLX indicates the amount of shift between blocks on two different pictures. A predicted vector and a difference vector regarding the motion vector mvLX are called a predicted vector mvpLX and a difference vector mvdLX, respectively.

(Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX)
The inter prediction identifier inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction BiPred using two reference pictures managed by L0 list and L1 list.

The merge index merge_idx is an index indicating which prediction parameter is used as the prediction parameter of the target block among the prediction parameter candidates (merging candidates) derived from the block for which processing has been completed.

The relationship between the inter prediction identifier inter_pred_idc and the prediction list usage flags predFlagL0 and predFlagL1 is as follows, and they are mutually convertible.

inter_pred_idc = (predFlagL1<<1)+predFlagL0
predFlagL0 = inter_pred_idc & 1
predFlagL1 = inter_pred_idc >> 1
(Judgment of bi-predictive biPred)
The flag biPred indicating whether it is bi-predictive BiPred can be derived depending on whether the two prediction list usage flags are both 1 or not. For example, it can be derived using the following formula.

biPred = (predFlagL0==1 && predFlagL1==1)
Alternatively, the flag biPred can also be derived depending on whether the inter prediction identifier has a value indicating that two prediction lists (reference pictures) are used. For example, it can be derived using the following formula.

biPred = (inter_pred_idc==PRED_BI) ? 1 : 0
(Configuration of moving image decoding device)
The configuration of the moving image decoding device 31 (FIG. 7) according to this embodiment will be explained.

The video decoding device 31 includes an entropy decoding section 301, a parameter decoding section (predicted image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a predicted image generation section (predicted image generation device) 308, and an inverse It is configured to include a quantization/inverse transformation section 311 and an addition section 312. Note that there is also a configuration in which the loop filter 305 is not included in the video decoding device 31 in accordance with the video encoding device 11 described later.

The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), which are not shown, and the CU decoding unit 3022 further includes a TU decoding unit 3024. ing. These may be collectively called a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS, and a slice header (slice information) from encoded data. CT information decoding section 3021 decodes CT from encoded data. CU decoding section 3022 decodes CU from encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from encoded data when a prediction error is included in the TU.

Further, the parameter decoding unit 302 includes an inter prediction parameter decoding unit (predicted image generation device) 303 and an intra prediction parameter decoding unit 304 (not shown). The predicted image generation unit 308 includes an inter predicted image generation unit (predicted image generation device) 309 and an intra predicted image generation unit 310.

Furthermore, although an example will be described below in which CTUs and CUs are used as processing units, the processing is not limited to this example, and processing may be performed in sub-CU units. Alternatively, CTU and CU may be read as blocks and sub-CUs as sub-blocks, and processing may be performed in units of blocks or sub-blocks.

The entropy decoding unit 301 performs entropy decoding on the encoded stream Te input from the outside, and decodes each code (syntax element).

Entropy decoding section 301 outputs the decoded code to parameter decoding section 302. The decoded codes include, for example, prediction mode predMode, merge flag merge_flag, merge index merge_idx, inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX, motion vector accuracy mode amvr_mode, and the like. Control of which code to decode is performed based on instructions from parameter decoding section 302.

(Configuration of inter prediction parameter decoding unit)
Inter prediction parameter decoding section 303 decodes inter prediction parameters based on the code input from entropy decoding section 301 with reference to the prediction parameters stored in prediction parameter memory 307. Further, the inter prediction parameter decoding unit 303 outputs the decoded inter prediction parameter to the predicted image generation unit 308 and stores it in the prediction parameter memory 307.

FIG. 8 is a schematic diagram showing the configuration of inter prediction parameter decoding section 303 according to this embodiment. The inter prediction parameter decoding unit 303 includes a merge prediction unit 30374, a DMVR unit 30375, a subblock prediction unit (affine prediction unit) 30372, an MMVD prediction unit (motion vector derivation unit) 30376, a Triangle prediction unit 30377, and an AMVP prediction parameter derivation unit 3032. , an addition section 3038. The merge prediction unit 30374 is configured to include a merge prediction parameter derivation unit 3036. The AMVP prediction parameter derivation unit 3032, merge prediction parameter derivation unit 3036, and affine prediction unit 30372 are means common to the video encoding device and the video decoding device, so they are collectively referred to as the motion vector derivation unit (motion vector It may also be called a derivation device).

The inter prediction parameter decoding unit 303 instructs the entropy decoding unit 301 to decode syntax elements related to inter prediction, and decodes syntax elements included in the encoded data, such as an affine flag affine_flag, a merge flag merge_flag, and a merge index merge_idx. , inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX, and motion vector accuracy mode amvr_mode are extracted.

When the affine flag affine_flag is 1, ie, indicates an affine prediction mode, the affine prediction unit 30372 derives the inter prediction parameter of the subblock.

When the merge flag merge_flag is 1, that is, indicates the merge prediction mode, the merge index merge_idx is decoded and output to the merge prediction parameter deriving unit 3036.

When the merge flag merge_flag is 0, that is, indicates AMVP prediction mode, the inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, and difference vector mvdLX are decoded as AMVP prediction parameters, for example. AMVP prediction parameter derivation unit 3032 derives prediction vector mvpLX from prediction vector index mvp_LX_idx. Addition unit 3038 adds the derived predicted vector mvpLX and difference vector mvdLX to derive a motion vector mvLX.

(Affine prediction department)
The affine prediction unit 30372 derives affine prediction parameters for the target block. In this embodiment, motion vectors (mv0_x, mv0_y) (mv1_x, mv1_y) of two control points (V0, V1) of the target block are derived as affine prediction parameters. Specifically, the motion vector of each control point may be derived by predicting it from the motion vector of a block adjacent to the target block, or it may be derived from the predicted vector derived as the motion vector of the control point and encoded data. The motion vector of each control point may be derived from the sum of the difference vectors.

Note that the affine prediction unit 30372 may appropriately derive parameters used for 4-parameter MVD affine prediction or parameters used for 6-parameter MVD affine prediction.

(merge prediction)
FIG. 9(a) is a schematic diagram showing the configuration of the merge prediction parameter deriving unit 3036 included in the merge prediction unit 30374. The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. Note that the merging candidate includes a prediction list usage flag predFlagLX, a motion vector mvLX, and a reference picture index refIdxLX, and is stored in the merging candidate list. Merge candidates stored in the merge candidate list are assigned indexes according to predetermined rules.

The merging candidate deriving unit 30361 derives a merging candidate using the decoded motion vector of the adjacent block and the reference picture index refIdxLX as they are. In addition, the merging candidate deriving unit 30361 may apply a spatial merging candidate deriving process, a temporal merging candidate deriving process, a pairwise merging candidate deriving process, and a zero merging candidate deriving process, which will be described later.

As a spatial merging candidate derivation process, the merging candidate deriving unit 30361 reads out the prediction parameters stored in the prediction parameter memory 307 according to a predetermined rule, and sets them as merging candidates. The reference picture can be specified by, for example, adjacent blocks within a predetermined range from the target block (for example, all or part of the blocks that are in contact with the left A1, right B1, upper right B0, lower left A0, and upper left B2 of the target block, respectively). ) are the prediction parameters for each of them. Each merge candidate is called A1, B1, B0, A0, and B2.

Here, A1, B1, B0, A0, and B2 are motion information derived from blocks including the following coordinates.

A1: (xCb - 1, yCb + cbHeight - 1)
B1: (xCb + cbWidth - 1, yCb - 1)
B0: (xCb + cbWidth, yCb - 1)
A0: (xCb - 1, yCb + cbHeight)
B2: (xCb - 1, yCb - 1)
As the temporal merge derivation process, the merging candidate deriving unit 30361 reads out the prediction parameters of the block C in the reference image including the lower right CBR or center coordinates of the target block from the prediction parameter memory 307 and sets it as a merging candidate Col. Store in the merge candidate list mergeCandList[].

The pairwise derivation unit derives the pairwise candidate avgK and stores it in the merge candidate list mergeCandList[].

The merging candidate deriving unit 30361 derives zero merging candidates Z0,...,ZM in which the reference picture index refIdxLX is 0...M and the X component and Y component of the motion vector mvLX are both 0, and stores them in the merging candidate list.

The order in which the merge candidate derivation unit 30361 or the pairwise derivation unit stores each merge candidate in the merge candidate list mergeCandList[] is, for example, spatial merge candidates (A1, B1, B0, A0, B2), temporal merge candidates Col, and pairwise candidates. AvgK, zero merge candidate ZeroCandK. Note that reference blocks that are not available (blocks are intra-predicted, etc.) are not stored in the merging candidate list.
i = 0
if( availableFlagA1 )
mergeCandList[ i++ ] = A1
if( availableFlagB1 )
mergeCandList[ i++ ] = B1
if( availableFlagB0 )
mergeCandList[ i++ ] = B0
if( availableFlagA0 )
mergeCandList[ i++ ] = A0
if( availableFlagB2 )
mergeCandList[ i++ ] = B2
if( availableFlagCol )
mergeCandList[ i++ ] = Col
if( availableFlagAvgK )
mergeCandList[ i++ ] = avgK
if( i < MaxNumMergeCand )
mergeCandList[ i++ ] = ZK
Note that the upper left coordinates of the target block are (xCb, yCb), the width of the target block is cbWidth, and the height of the target block is cbHeight.

The merge candidate selection unit 30362 selects the merge candidate N indicated by the merge index merge_idx from among the merge candidates included in the merge candidate list using the following formula.

N = mergeCandList[merge_idx]
Here, N is a label indicating a merging candidate, such as A1, B1, B0, A0, B2, Col, AvgK, ZeroCandK. The motion information of the merging candidate indicated by label N is indicated by (mvLXN[0], mvLXN[1]), predFlagLXN, refIdxLXN.

The merging candidate selection unit 30362 selects the motion information (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN of the selected merging candidate as inter prediction parameters of the target block. The merging candidate selection unit 30362 stores the selected inter prediction parameter in the prediction parameter memory 307 and outputs it to the predicted image generation unit 308.

(MMVD Prediction Department 30373)
The MMVD prediction unit 30373 adds the difference vector mvdLX to the center vector mvdLX (motion vector of the merging candidate) derived by the merging candidate deriving unit 30361 to derive a motion vector.

The MMVD prediction unit 30376 derives a motion vector mvLX[] using the merge candidate mergeCandList[] and the syntax base_candidate_idx, direction_idx, and distance_idx for decoding or encoding from encoded data into encoded data. Furthermore, the syntax distance_list_idx for selecting a distance table may be encoded or decoded.

The MMVD prediction unit 30376 selects the center vector mvLN[] using base_candidate_idx.

N = mergeCandList[base_candidate_idx]
The MMVD prediction unit 30376 derives the base distance (mvdUnit[0], mvdUnit[1]) and the distance DistFromBaseMV.

dir_table_x[] = { 8, -8, 0, 0, 6, -6, -6, 6 }
dir_table_y[] = { 0, 0, 8, -8, 6, -6, 6, -6 }
mvdUnit[0] = dir_table_x[direction_idx]
mvdUnit[1] = dir_table_y[direction_idx]
DistFromBaseMV = DistanceTable[distance_idx]
The MMVD prediction unit 30376 derives a difference vector refineMv[].

firstMv[0] = (DistFromBaseMV<<shiftMMVD) * mvdUnit[0]
firstMv[1] = (DistFromBaseMV<<shiftMMVD) * mvdUnit[1]
Here, shiftMMVD is a value that adjusts the magnitude of the difference vector to match the motion vector accuracy MVPREC in the motion compensation unit 3091 (interpolation unit).

refineMvL0[0] = firstMv[0]
refineMvL0[1] = firstMv[1]
refineMvL1[0] = -firstMv[0]
refineMvL1[1] = -firstMv[1]
Finally, the MMVD prediction unit 30376 derives the motion vector of the MMVD merging candidate from the difference vector refineMvLX and the center vector mvLXN as follows.

mvL0[ 0 ] = mvL0N[ 0 ] + refineMvL0[0]
mvL0[ 1 ] = mvL0N[ 1 ] + refineMvL0[1]
mvL1[ 0 ] = mvL1N[ 0 ] + refineMvL1[0]
mvL1[ 1 ] = mvL1N[ 1 ] + refineMvL1[1]
(AMVP prediction)
FIG. 9(b) is a schematic diagram showing the configuration of the AMVP prediction parameter deriving unit 3032 according to this embodiment. The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives a predicted vector candidate from the motion vector mvLX of the decoded adjacent block stored in the prediction parameter memory 307 based on the reference picture index refIdxLX, and stores it in the predicted vector candidate list mvpListLX[].

The vector candidate selection unit 3034 selects the motion vector mvpListLX[mvp_LX_idx] indicated by the predictive vector index mvp_LX_idx from among the predictive vector candidates in the predictive vector candidate list mvpListLX[] as the predictive vector mvpLX. Vector candidate selection section 3034 outputs the selected predicted vector mvpLX to addition section 3038.

Adding unit 3038 calculates motion vector mvLX by adding predicted vector mvpLX input from AMVP prediction parameter deriving unit 3032 and decoded difference vector mvdLX. Addition unit 3038 outputs the calculated motion vector mvLX to predicted image generation unit 308 and prediction parameter memory 307.

mvLX[0] = mvpLX[0]+mvdLX[0]
mvLX[1] = mvpLX[1]+mvdLX[1]
The motion vector precision mode amvr_mode is a syntax for switching the precision of motion vectors derived in AMVP mode. For example, when amvr_mode=0, 1, 2, the motion vector precision mode amvr_mode switches between 1/4 pixel, 1 pixel, and 4 pixel precision.

If the motion vector precision is 1/16 precision, in order to change the motion vector difference with 1/4, 1, and 4 pixel precision to the motion vector difference with 1/16 pixel precision, it is derived from amvr_mode as shown below. Inverse quantization may be performed using MvShift (=1<<amvr_mode).

mvdLX[0] = mvdLX[0] << (MvShift + 2)
mvdLX[1] = mvdLX[1] << (MvShift + 2)
Note that the parameter decoding unit 302 may further derive mvdLX[] by decoding the following syntax.
・abs_mvd_greater0_flag
・abs_mvd_minus2
・mvd_sign_flag
Decrypt and do. Then, the parameter decoding unit 302 decodes the difference vector lMvd[] from the syntax by using the following equation.

lMvd[ compIdx ] = abs_mvd_greater0_flag[ compIdx ] * ( abs_mvd_minus2[ compIdx] + 2 ) * ( 1 - 2 * mvd_sign_flag[ compIdx ] )
Furthermore, the decoded difference vector lMvd[] is set to mvdLX in the case of translational MVD (MotionModelIdc[ x ][ y ] ==0), and in the case of control point MVD (MotionModelIdc[ x ][ y ] != 0) Set to mvdCpLX.

if (MotionModelIdc[ x ][ y ] == 0)
mvdLX[ x0 ][ y0 ][ compIdx ] = lMvd[ compIdx ]
else
mvdCpLX[ x0 ][ y0 ][ compIdx ] = lMvd[ compIdx ]<<2
(DMVR)
Next, DMVR (Decoder side Motion Vector Refinement) processing performed by the DMVR unit 30375 will be described. If the merge flag merge_flag indicates that the merge prediction mode is applied to the target CU, or if the skip flag skip_flag indicates that the skip mode is applied to the target CU, the merge prediction unit 30374 The motion vector mvLX of the target CU derived by is corrected using the reference image.

Specifically, when the prediction parameters derived by the merge prediction unit 30374 are bi-prediction, the motion vector is modified using predicted images derived from motion vectors corresponding to two reference pictures. The corrected motion vector mvLX is supplied to the inter predicted image generation unit 309.

(Triangle prediction)
Next, we will explain Triangle prediction. In Triangle prediction, a target CU is divided into two triangular prediction units using a diagonal line or an opposite angle line as a boundary. The predicted image in each triangular prediction unit is derived by subjecting each pixel of the predicted image of the target CU (rectangular block including the triangular prediction unit) to weighted mask processing according to the position of the pixel. For example, a triangular image can be derived from a rectangular image by multiplying by a mask in which pixels in a triangular area within a rectangular area are set to 1 and pixels in non-triangular areas are set to 0. In addition, after generating the inter-predicted image, the adaptive weighting process is applied to both areas on both sides of the diagonal line, and the adaptive weighting process using the two predicted images allows one of the target CUs (rectangular blocks) to Two predicted images are derived. This process is called Triangle synthesis process. Then, transformation (inverse transformation) and quantization (inverse quantization) processing is applied to the entire target CU. Note that Triangle prediction is applied only in merge prediction mode or skip mode.

The Triangle prediction unit 30377 derives prediction parameters corresponding to two triangular areas used for Triangle prediction and supplies them to the inter prediction image generation unit 309. Triangle prediction may be configured without using bi-prediction to simplify processing. In this case, inter prediction parameters for unidirectional prediction are derived in one triangular region. Note that derivation of two predicted images and synthesis using the predicted images are performed by a motion compensation unit 3091 and a triangle synthesis unit 30952.

The loop filter 305 is a filter provided in the encoding loop, and is a filter that removes block distortion and ringing distortion and improves image quality. The loop filter 305 applies filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the addition unit 312.

The reference picture memory 306 stores the decoded image of the CU generated by the addition unit 312 at a predetermined position for each target picture and target CU.

The prediction parameter memory 307 stores prediction parameters in a predetermined position for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302, prediction mode predMode decoded by the entropy decoding unit 301, and the like.

A prediction mode predMode, prediction parameters, etc. are input to the predicted image generation unit 308. The predicted image generation unit 308 also reads a reference picture from the reference picture memory 306. The predicted image generation unit 308 generates a predicted image of a block or subblock using the prediction parameter and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, the reference picture block is a set of pixels on a reference picture (usually called a block because it is rectangular), and is an area to be referenced to generate a predicted image.

(Inter predicted image generation unit 309)
When the prediction mode predMode indicates inter prediction mode, the inter prediction image generation unit 309 generates a predicted image of the block or subblock by inter prediction using the inter prediction parameters input from the inter prediction parameter decoding unit 303 and the read reference picture. generate.

FIG. 10 is a schematic diagram showing the configuration of an inter predicted image generation unit 309 included in the predicted image generation unit 308 according to the present embodiment. The inter predicted image generation unit 309 includes a motion compensation unit (predicted image generation device) 3091 and a synthesis unit 3095.

(motion compensation)
The motion compensation unit 3091 (interpolation image generation unit 3091) generates a reference picture memory based on the inter prediction parameters (prediction list usage flag predFlagLX, reference picture index refIdxLX, motion vector mvLX) input from the inter prediction parameter decoding unit 303. From 306, an interpolated image (motion compensated image) is generated by reading a block located at a position shifted by the motion vector mvLX from the position of the target block in the reference picture RefPicLX specified by the reference picture index refIdxLX. Here, if the precision of the motion vector mvLX is not integer precision, a filter for generating pixels at decimal positions called a motion compensation filter is applied to generate a motion compensated image.

The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the intra-prediction block coordinates (x, y) using the following equations.

xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x
xFrac = mvLX[0]&(MVPREC -1)
yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y
yFrac = mvLX[1]&(MVPREC -1)
Here, (xPb,yPb) are the upper left coordinates of the bW*bH size block, x=0...bW-1, y=0...bH-1, and MVPREC is the precision of the motion vector mvLX (1/MVPREC pixel accuracy). For example, MVPREC=16.

The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation processing on the reference picture refImg using an interpolation filter. Σ below is the sum of k=0..NTAP-1 with respect to k, shift1 is a normalization parameter that adjusts the range of values, and offset1=1<<(shift1-1).

temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP/2+1][yInt]+offset1)>>shift1
Next, the motion compensation unit 3091 performs vertical interpolation processing on the temporary image temp[][] to derive an interpolated image Pred[][]. Σ below is the sum of k=0..NTAP-1 with respect to k, shift2 is a normalization parameter that adjusts the range of values, and offset2=1<<(shift2-1).

Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP/2+1]+offset2)>>shift2
The above interpolation image generation process may be expressed as Interpolation(refImg,xPb,yPb,bW,bH,mvLX).

(Synthesis department)
The synthesis unit 3095 performs prediction by referring to the interpolation image supplied from the motion compensation unit 3091, the inter prediction parameters supplied from the inter prediction parameter decoding unit 303, and the intra image supplied from the intra prediction image generation unit 310. An image is generated, and the generated predicted image is supplied to the addition unit 312.

The synthesis section 3095 includes a combined intra/inter synthesis section 30951, a triangle synthesis section 30952, and a BDOF section 30954.

(Combined intra/inter synthesis processing)
The combined intra/inter synthesis unit 30951 generates a predicted image by compositely using a unidirectional predicted image in AMVP, a predicted image in skip mode or merge prediction mode, and an intra predicted image.

(Triangle synthesis process)
The Triangle synthesis unit 30952 generates a predicted image using the above-described Triangle prediction.

(BDOF processing)
The BDOF unit 30954 generates a predicted image by performing BDOF (Bi-directional optical flow; bi-predictive gradient change) processing. Details of the BDOF section 30954 will be described later.

(weight prediction)
In weight prediction, a predicted image of a block is generated by multiplying the motion compensated image PredLX by a weighting coefficient. If one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (uni-prediction) and weight prediction is not used, process the following formula to match the motion compensated image PredLX (LX is L0 or L1) to the pixel bit number bitDepth. I do.

Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)>>shift1)
Here, shift1=14-bitDepth, offset1=1<<(shift1-1).
In addition, when both reference list usage flags (predFlagL0 and predFlagL1) are 1 (bi-prediction BiPred) and weight prediction is not used, the following formula is used to average motion compensated images PredL0 and PredL1 to match the number of pixel bits. conduct.

Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)>>shift2)
Here, shift2=15-bitDepth, offset2=1<<(shift2-1).

Furthermore, when performing uni-prediction and weight prediction, the synthesis unit 3095 derives a weight prediction coefficient w0 and an offset o0 from the encoded data, and performs processing according to the following equation.

Pred[x][y] = Clip3(0,(1<<bitDepth)-1,((PredLX[x][y]*w0+2^(log2WD-1))>>log2WD)+o0)
Here, log2WD is a variable indicating a predetermined shift amount.

Furthermore, when performing bi-prediction BiPred and weight prediction, the synthesis unit 3095 derives weight prediction coefficients w0, w1, o0, and o1 from the encoded data, and performs processing according to the following equation.

Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))>>(log2WD+1))
Then, the predicted image of the generated block is output to the adding unit 312.

The inverse quantization/inverse transform unit 311 inversely quantizes the quantized transform coefficients input from the entropy decoding unit 301 to obtain transform coefficients. This quantized transform coefficient is obtained by performing frequency transform such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) on the prediction error and quantizing it in the encoding process. It is a coefficient. The inverse quantization/inverse transform unit 311 performs inverse frequency transform such as inverse DCT and inverse DST on the obtained transform coefficients to calculate a prediction error. The inverse quantization/inverse transformation section 311 outputs the prediction error to the addition section 312.

The addition unit 312 adds the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization/inverse transformation unit 311 for each pixel to generate a decoded image of the block. The adding unit 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305.

(BDOF prediction)
Next, details of prediction using BDOF processing (BDOF prediction) performed by the BDOF unit 30954 will be described. The BDOF unit 30954 generates a predicted image in bi-prediction mode by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

FIG. 11 is a flowchart illustrating the flow of processing for deriving a predicted image.

When the inter prediction parameter decoding unit 303 determines that L0 is unidirectionally predicted (inter_pred_idc is 0 in S101), the motion compensation unit 3091 generates an L0 predicted image PredL0[x][y] (S102). When the inter prediction parameter decoding unit 303 determines L1 unidirectional prediction (inter_pred_idc is 1 in S101), the motion compensation unit 3091 generates an L1 predicted image PredL1[x][y] (S103). On the other hand, if the inter prediction parameter decoding unit 303 determines that the bi-prediction mode is selected (inter_pred_idc is 2 in S101), the process continues to S104 below. In S104, the synthesis unit 3095 refers to bdofFlag, which indicates whether or not to perform BDOF processing, and determines whether BDOF processing is necessary. When bdofFlag indicates TRUE, the BDOF unit 30954 executes BDOF processing to generate a bidirectional predicted image (S106). When bdofFlag indicates FALSE, the synthesis unit 3095 generates a predicted image by normal bidirectional predicted image generation (S105).

bdofFlag, which indicates whether or not to perform BDOF processing, is set to TRUE under the following conditions.

The inter prediction parameter decoding unit 303 may derive TRUE to bdofFlag when the L0 reference image refImgL0 and the L1 reference image refImgL1 are different reference images, and the two pictures are in opposite directions with respect to the target picture. . Specifically, when the target image is currPic, if the condition of DiffPicOrderCnt(currPic,refImgL0)*DiffPicOrderCnt(currPic,refImgL1)<0 is satisfied, bdofFlag indicates TRUE. Here, DiffPicOrderCnt( ) is a function that derives the difference in POC (Picture Order Count: picture display order) between two images as follows.

DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB)
As a condition that bdofFlag is TRUE, a condition that the target block is not subblock prediction (affine prediction) may be added.

Further, as a condition for bdofFlag to be TRUE, a condition may be added that neither L0 prediction nor L1 prediction performs weighted prediction in weighted prediction. Specifically, luma_weight_l0_flag[refIdxL0] indicates whether the luminance weighting coefficient w0 and offset o0 exist in the L0 predicted picture, and luma_weight_l0_flag[refIdxL0] indicates whether the luminance weighting coefficient w1 and offset o1 exist in the L1 predicted picture. If both of the indicated luma_weight_l1_flag[refIdxL1] are FALSE, the condition is that bdofFlag is TRUE.

The specific contents of the process performed by the BDOF unit 30954 when bdofFlag is TRUE will be described using FIG. 12. The BDOF unit 30954 includes an L0, L1 predicted image generation unit 309541, a flag setting unit 309542, a gradient image generation unit 309543, a correlation parameter calculation unit 309544, a motion compensation correction value derivation unit 309545, and a bidirectional predicted image generation unit 309546. The BDOF unit 30954 generates a predicted image from the interpolated image received from the motion compensation unit 3091 and the inter prediction parameters received from the inter prediction parameter decoding unit 303, and outputs the generated predicted image to the addition unit 312. Note that the process of deriving a motion compensation correction value bdofOffset (motion compensation correction image) from the gradient image and correcting and deriving the predicted images of PredL0 and PredL1 is called bidirectional gradient change processing.

First, the L0, L1 predicted image generation unit 309541 generates L0, L1 predicted images used for BDOF processing. Here, the CU is divided into subblocks. This enables parallel processing in sub-block units and reduces the amount of memory required for one-time processing.

Specifically, the basic size of the sub-block may be 16x16. For example, when the size of a CU is cbWidth and cbHeight, the number of subblocks in the horizontal direction numSbX, the number of subblocks in the vertical direction numSbY, and the subblock sizes sbWidth and sbHeight in the CU are derived as follows.

numSbX = ( cbWidth > 16 ) ? ( cbWidth >> 4 ) : 1
numSbY = ( cbHeight > 16 ) ? ( cbHeight >> 4 ) : 1
sbWidth = cbWidth / numSbX
sbHeight = cbHeight / numSbY
Next, BDOF processing is performed based on the L0 and L1 predicted images for each subblock shown in Figure 14, but in order to obtain the gradient, interpolated image information for one pixel around the subblock is additionally required. shall be. For this interpolation image information, neighboring pixels are used instead of a normal interpolation filter in gradient image generation, which will be described later. Otherwise, it is a padding area that copies surrounding pixels, similar to the outside of the picture. Furthermore, the unit of BDOF processing is 4x4 pixels, which is a sub-block unit or less, and the processing itself is performed using 6x6 pixels, including one surrounding pixel.

The flag setting unit 309542 sets the values of the flag sbBdofFlag, which indicates whether to perform BDOF processing in sub-block units, and the flag bdofUtilizationFlag[xIdx][yIdx], which indicates whether to perform BDOF processing in 4x4 pixel units. conduct. Here, let the number of pixels of the width and height of the sub-block be sbWidth and sbHeight. xIdx is the horizontal address of a 4x4 pixel block, taking values from 0 to (sbWidth>>2)-1, and yIdx is the 4x4 pixel block taking values from 0 to (sbHeight>>2)-1. This is the vertical address of the pixel block. When sbBdofFlag and bdofUtilizationFlag[xIdx][yIdx] are TRUE, BDOF processing is performed, and when FALSE, bidirectional prediction processing is performed.

First, the threshold value sbDiffThres in sub-block units, the threshold value bdofBlkDiffThres in 4x4 pixel block units, and the sum of absolute differences sbSumDiff in sub-block units are derived as follows.

sbDiffThres = 4 * sbWidth*sbHeight
bdofBlkDiffThres = 128
In this embodiment, sbDiffThres is 4 per pixel and bdofBlkDiffThres is 128=8*4*4, so a value of 8 per pixel is used as the threshold, but values other than these may be used. For example, double the value, sbDiffThres = 8 * sbWidth*sbHeight, bdofBlkDiffThres = 256. Further, by setting sbDiffThres=0, switching in units of sub-blocks may be turned off, and by setting bdofBlkDiffThres=0, switching in units of 4x4 pixel blocks may be turned off.

First, set the value of sbSumDiff as shown below.

sbSumDiff = 0
Then, run a loop of xIdx=0..(sbWidth>>2)-1 and yIdx=0..(sbHeight>>2)-1 for each 4x4 pixel block, and calculate the absolute difference value in 4x4 pixel units. The sum bdofBlkSumDiff and bdofUtilizationFlag[xIdx][yIdx] are derived as follows.

bdofBlkSumDiff
= sum4(Abs((PredL0[(xIdx<<2)+1+i][(yIdx<<2)+1+j]>>shift4)
?(PredL1[(xIdx<<2)+1+i][(yIdx<<2)+1+j]>>shift4))) (Formula BDOF-1)
shift4=Max(InternalBitDepth-4, bitDepth+InternalBitDepth-16)
bdofUtilizationFlag[xIdx][yIdx] = (bdofBlkSumDiff >= bdofBlkDiffThres)
sbSumDiff += bdofBlkSumDiff
In other words, the sum of the absolute values of the differences between the value obtained by shifting PredL0 by shift4 and the value obtained by shifting PredL1 by shift4 is derived. In addition, in (Formula BDOF-1), instead of (x, y)=((xIdx<<2)+1+i, yIdx<<2)+1+j), (x,y) is further converted to CU You may also use values (hx, vy) clipped by the size (nCbW, nCbH). In this case, pixels that cross the border are derived with padding.

hx = Clip3( 1, nCbW, x )
vy = Clip3( 1, nCbH, y )
Here, sum4() is a function that calculates the sum of 4x4 pixel blocks with arguments i=0..3 and j=0..3, and shift4 is the same as shift4 defined in the correlation parameter calculation unit 309544 described later. shall belong to. Note that InternalBitDepth is a parameter for calculation accuracy in the BDOF section, and is a constant value independent of the input pixel bit length bitDepth, and is a value of 7 or more and 12 or less. For example, when InternalBitDepth=8, shift4 becomes shift4=Max(4, bitDepth-8).

When using an interpolation filter similar to HEVC, the values of PredL0 and PredL1 have a calculation precision of 14 bits when bitDepth is in the range of 8 to 12 bits, and when InternalBitDepth=8, shift4 = Max(4, bitDepth-8 ) = 4, and the sum of absolute values of the differences after right-shifting PredL0 and PredL1 shown in (Formula BDOF-1) with shift4 is calculated by calculating the sum of absolute differences with 10-bit precision regardless of bitDepth. There will be.

In the prior art document, bdofBlkDiff directly calculates the sum of absolute differences between PredL0 and PredL1 without right-shifting. However, in this embodiment, the calculation of the difference of theta[x][y] (formula BDOF-2) calculated by the correlation parameter calculation unit 309544 of the BDOF unit 30954, which will be described later, is the same as the calculation by the flag setting unit 309542. Processing can be shared between threshold value determination and predicted image derivation by the BDOF unit 30954. As a result, in the prior art document, the calculation of the flag setting unit 309542 simply required an additional amount of calculation, but in this embodiment, since the calculation of the difference can be shared, the additional part calculates the sum of absolute values. Since only the required part is included, the amount of calculation can be significantly reduced.

bdofUtilizationFlag[xIdx][yIdx] is set to TRUE if bdofBlkSumDiff is greater than or equal to bdofBlkDiffThres, otherwise set to FALSE.

Furthermore, the value of sbSumDiff is the total value of bdofBlkSumDiff.

Finally, sbBdofFlag is set to FALSE if sbSumDiff is less than sbDiffThres, otherwise set to TRUE.

Since the calculation of the difference is shared, such a configuration allows the flag setting unit 309542 to set the values of sbBdofFlag and bdofUtilizationFlag[xIdx][yIdx] with a smaller amount of processing.

In another embodiment of the flag setting unit 309542, bdofBlkSumDiff is calculated using the following formula.

bdofBlkSumDiff
= sum42(Abs((PredL0[(xIdx<<2)+1+i][(yIdx<<2)+1+2*j]>>shift4)
?(PredL1[(xIdx<<2)+1+i][(yIdx<<2)+1+2*j]>>shift4)))
Here, sum42() is a function that calculates the sum of 4x2 pixel blocks with arguments i=0..3 and j=0..1. bdofBlkSumDiff may be the sum of absolute differences of 4x2 pixels skipping one line. In this case, sbDiffThres and bdofBlkDiffThres are set to half the values of the previous example, and are set as follows.

sbDiffThres = 2 * sbWidth*sbHeight
bdofBlkDiffThres = 64
With this configuration, the amount of calculation for bdofBlkSumDiff can be halved. Since this calculation does not directly affect predicted image generation, there is almost no performance deterioration due to processing reduction.

FIG. 13 is a flowchart illustrating the process by which the BDOF unit 30954 generates a predicted image. First, in flag setting (S201), the values of the flag sbBdofFlag, which indicates whether or not to perform BDOF processing in sub-block units, and the flag, bdofUtilizationFlag[xIdx][yIdx], which indicates whether or not to perform BDOF processing in 4x4 pixel units. Configure settings. Next, sbBdofFlag is determined (S202), and if FALSE, bidirectional predicted image generation is performed in sub-block units (S203), and if TRUE, BDOF processing (S204)-(S210) is performed in sub-block units. Loop1 ((S204) to (S210)) is a vertical address loop, and Loop2 ((S205) to (S209)) is a horizontal address loop. bdofUtilizationFlag[xIdx][yIdx] is determined (S206), and if FALSE, bidirectional predicted image generation in 4x4 pixel units is performed (S207), and if TRUE, 4x4 pixel unit BDOF processing is performed (S208).

In this flowchart, it is determined whether or not to perform BDOF processing in two stages: in units of subblocks and in units of 4x4 pixel blocks, but it may also be implemented only in units of subblocks or only in units of 4x4 pixel blocks.

The gradient image generation unit 309543 generates a gradient image. In gradient change (optical flow), it is assumed that the pixel value of each point does not change, but only its position changes. This is a change in pixel value I in the horizontal direction (horizontal gradient value lx) and a change in its position Vx, a change in pixel value I in the vertical direction (vertical gradient value ly) and a change in its position Vy, and a change in pixel value I in the vertical direction (vertical gradient value ly). It can be expressed as below using the time change lt.

lx * Vx + ly * Vy + lt = 0
Hereinafter, the change in position (Vx, Vy) will be referred to as a correction weight vector (u, v).

Specifically, the gradient image generation unit 309543 derives gradient images lx0, ly0, lx1, ly1 from the following equations. lx0 and lx1 indicate gradients along the horizontal direction, and ly0 and ly1 indicate gradients along the vertical direction.

Specifically, the gradient image generation unit 309543 derives the gradient images lx0, ly0, lx1, and ly1 as follows.

lx0[x][y] = (PredL0[x+1][y]-PredL0[x-1][y])>>shift0 (formula BDOF-3)
ly0[x][y] = (PredL0[x][y+1]-PredL0[x][y-1])>>shift0
lx1[x][y] = (PredL1[x+1][y]-PredL1[x-1][y])>>shift0
ly1[x][y] = (PredL1[x][y+1]-PredL1[x][y-1])>>shift0
Here, shift0=14-InternalBitDepth.

Another method for deriving the gradient images lx0, ly0, lx1, ly1 may be set as shown in the following formula.

lx0[x][y] = (PredL0[x+1][y]>>shift0)-(PredL0[x-1][y]>>shift0) (formula BDOF-4)
ly0[x][y] = (PredL0[x][y+1]>>shift0)-(PredL0[x][y-1]>>shift0)
lx1[x][y] = (PredL1[x+1][y]>>shift0)-(PredL1[x-1][y]>>shift0)
ly1[x][y] = (PredL1[x][y+1]>>shift0)-(PredL1[x][y-1]>>shift0)
In the case of (Formula BDOF-3), if the range of the minimum and maximum values of PredL0 and PredL1 is 16 bits, the difference will be 17 bits, so 32-bit operation is required, but (Formula BDOF-4) In this case, since the right shift is performed first, all operations can be performed using 16-bit operations, which has the advantage of making it easier to use parallel operation instructions.

In any case, when using an interpolation filter similar to HEVC, the values of PredL0 and PredL1 have a calculation precision of 14 bits if the bitDepth is in the range of 8 to 12 bits, and the range of the minimum and maximum values is It is 16 bit. If bitDepth is greater than 12, the calculation precision is (bitDepth+2) bits, and the range between the minimum value and maximum value is (bitDepth+4) bits. In this embodiment, it is shifted to the right by shift0, which is a value according to InternalBitDepth. Therefore, the calculation precision of gradient images lx0, ly0, lx1, ly1 is (InternalBitDepth+1) bits when BitDedepth is 8 or more and 12 or less, and (bitDepth+InternalBitDepth-11) when bitDepth is 13 or more. .

Next, the correlation parameter calculation unit 309544 derives gradient product sums s1, s2, s3, s5, s6 for each block of 4x4 pixels in the sub-block. Here, s1, s2, s3, s5, and s6 are calculated from the pixel sum of a 6x6 pixel block using one pixel around the block.

s1 = sum6(phiX[x][y]* phiX[x][y])
s2 = sum6(phiX[x][y]* phiY[x][y])
s3 = sum6(-theta[x][y]* phiX[x][y])
s5 = sum6(phiY[x][y]* phiY[x][y])
s6 = sum6(-theta[x][y]* phiY[x][y])
Here, sum(a) represents the sum of a for coordinates (x, y) within a 6x6 pixel block.

theta[x][y]= -(PredL1[x][y]>>shift4)+(PredL0[x][y]>>shift4) (formula BDOF-2)
phiX[x][y] = (lx1[x][y] + lx0[x][y])>>shift5
phiY[x][y] = (ly1[x][y] + ly0[x][y])>>shift5
That is, theta[x][y] is derived from the difference between the value of PredL0[x][y] shifted by shift4 and the value of PredL1[x][y] shifted by shift4. This is the same as the difference value calculated in (Formula BDOF-1).
Here, shift4 and shift5 are derived below.

shift4=Max(InternalBitDepth-4, bitDept+InternalBitDepth-16)
shift5=Max(InternalBitDepth-7, bitDepth+InternalBitDepth-19)
Note that in (Formula BDOF-2), values (hx, vy) clipped by the CU size (nCbW, nCbH) may be used instead of (x, y). In this case, pixels that cross the border are derived with padding. Further, x and y may be in the range of x=xSb-1..xSb+4 and y=ySb-1..ySb+4, respectively.

xSb = (xIdx<<2)+1, ySb = (yIdx<<2)+1
hx = Clip3( 1, nCbW, x )
vy = Clip3( 1, nCbH, y )
At this time, the calculation precision of the value of theta is always (19-InternalBitDepth) bits if bitDepth is 8 or more. Also, if the bitDepth of the image is 8 or more, the calculation precision of phiX and phiY is always 9 bits. Therefore, the calculation precision of s1, s2, and s5, which is the total of 6x6 pixel blocks, is about 23 bits regardless of bitDepth. Similarly, the calculation accuracy of s3 and s6, which is the total of 6x6 pixel blocks, is about (33-InternalBitDepth) bits. The minimum and maximum values of PredL0 and PredL1 are in the range of 16 bits if bitDepth is 8 or more and 12 or less, and if bitDepth is 13 or more bits, they are in the range of (bitDepth+4) bits, so 32-bit integer arithmetic is required. realizable.

More specifically, when InternalBitDepth=8, this is achieved using the following formula.

shift0=6
shift4=Max(4, bitDepth-8)
shift5=Max(1, bitDepth-11)
When InternalBitDepth=7, this is achieved using the following formula.

shift0=7
shift4=Max(3, bitDepth-9)
shift5=Max(0, bitDepth-12)
As another configuration of the correlation parameter calculation unit 309544, the gradient product sums s1, s2, s3, s5, s6 may be calculated using a 4x4 pixel block instead of a 6x6 pixel block. Since it is a block of 4x4=16 pixels, the calculation bits required to calculate sum are (Ceil(log2(36))-Ceil(log2(16)))=2 bits less than the total of 6x6=36 pixels. Mosumu.

shift4=Max(InternalBitDepth-5, bitDept+InternalBitDepth-15)
shift5=Max(InternalBitDepth-8, bitDepth+InternalBitDepth-18)
Therefore, even if each value is 1 bit smaller as shown in the above equation, it can be realized using 32-bit integer operations. However, in this case, the value of InternalBitDepth should be 8 or more and 12 or less. Furthermore, the amount of computation for gradient product sum can be reduced. As shown in FIG. 15, since the unit of BDOF processing and the read area match, unlike the case of FIG. 14, a padding area of one pixel around the target sub-block is not required.

Next, the motion compensation correction value deriving unit 309545 derives a correction weight vector (u, v) in 4x4 pixel units using the derived gradient product sums s1, s2, s3, s5, and s6.

u = (s3<<3) >> floor(log2(s1))
v = ((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1)) >> floor(log2(s5))
Here, s2m=s2>>12, s2s=s2&((1<<12)-1).

Note that the range of u and v may be further limited by using clips as described below.

u=s1>0?Clip3(-th,th,-(s3<<3)>>floor(log2(s1))):0
v=s5>0?Clip3(-th,th,((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>floor(log2(s5 ))):0
Here, th is expressed by the following formula.

th = Max(2, 1<<(13-InternalBitDepth))
Since th is a value independent of bitDepth, the pixel-by-pixel correction weight vector (u, v) is clipped by a value related to InternalBitDepth. Regardless of the pixel bit length bitDepth, for example, if InternalBitDepth=8, th=1<<(13-8)=32. Also, if InternalBitDepth=7, th=1<<(13-7)=64.

At this time, since the pixel-based correction weight vector (u, v) can be considered to be a value related to the accuracy of the motion vector and the quantization width, a threshold is used to limit the correction weight vector (u, v). The value th may be expressed as a function of the quantization width Qp as shown in the following equation.

th0 = Max(1, 1<<(12-InternalBitDepth))
th = th0 +floor((Qp-32)/6)
Note that the gradient product sums s3 and s6 may be derived using the following formula without shifting 3 bits to the left.

u = s3 >> floor(log2(s1))
v = (s6-((((u*s2m)<<12)+u*s2s)>>1))>> floor(log2(s5))
However, shift4 is derived using the formula below.

shift4=Max(InternalBitDepth-7, bitDepth+InternalBitDepth-19)
In this case, there is no need to change shift0 and shift5. Specifically, when InternalBitDepth=8, shift4 is derived using the following formula.

shift4=Max(1, bitDepth-11)
The motion compensation correction value deriving unit 309545 derives a motion compensation correction value bdofOffset using the correction weight vector (u, v) in 4x4 pixel units and the gradient images lx0, ly0, lx1, ly1.

bdofOffset[x][y]=((lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v+1) >>1
Alternatively, bdofOffset may be derived as follows using a round function.

bdofOffset[x][y] = Round(((lx1[x][y]-lx0[x][y])*u)>>1)+Round(((ly1[x][y]-ly0[ x][y])*v)>>1)
Alternatively, by modifying the value of shift0 of the right shift values of gradient images lx0, ly0, lx1, and ly1, it is also possible to derive without using right shift or round function as shown below.

bdofOffset[x][y]=(lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v
shift0=15-InternalBitDepth
shift5=Max(InternalBitDepth-8, bitDepth+InternalBitDepth-20)
In this case, the selection range of InternalBitDepth is between 8 and 12, and there is no need to change shift4. Specifically, when InternalBitDepth=8, the following formula is obtained.

shift0=7
shift5=Max(0, bitDepth-12)
The bidirectional predicted image generation unit 309546 derives the pixel value Pred of the 4x4 pixel predicted image using the above-mentioned parameters according to the following formula.

Pred[x][y] = Clip3(0, (1<<bitDepth)-1,( PredL0[x][y]+PredL1[x][y]+bdofOffset[x][y]+offset2)>> shift2)
Here, shift2=Max(3,15-bitDepth), offset2=1<<(shift2-1).

With the above configuration, it is possible to reduce the amount of calculation required to set the flag indicating whether to perform BDOF processing in sub-block units or the flag indicating whether to perform BDOF processing in 4x4 pixel units. can.

(Configuration of video encoding device)
Next, the configuration of the video encoding device 11 according to this embodiment will be explained. FIG. 16 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a predicted image generation section 101, a subtraction section 102, a transformation/quantization section 103, an inverse quantization/inverse transformation section 105, an addition section 106, a loop filter 107, a prediction parameter memory (prediction parameter storage section , frame memory) 108, reference picture memory (reference image storage unit, frame memory) 109, encoding parameter determination unit 110, parameter encoding unit 111, and entropy encoding unit 104.

The predicted image generation unit 101 generates a predicted image for each CU, which is a region obtained by dividing each picture of the image T. The predicted image generation unit 101 operates in the same manner as the predicted image generation unit 308 already described, and the explanation will be omitted.

The subtraction unit 102 subtracts the pixel value of the predicted image of the block input from the predicted image generation unit 101 from the pixel value of the image T to generate a prediction error. Subtraction section 102 outputs the prediction error to conversion/quantization section 103.

The conversion/quantization unit 103 calculates a conversion coefficient by frequency conversion for the prediction error input from the subtraction unit 102, and derives a quantized conversion coefficient by quantization. Transformation/quantization section 103 outputs the quantized transformation coefficients to entropy encoding section 104 and inverse quantization/inverse transformation section 105.

The inverse quantization/inverse transformation unit 105 is the same as the inverse quantization/inverse transformation unit 311 (FIG. 7) in the moving image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to addition section 106.

Entropy encoding section 104 receives quantized transform coefficients from transform/quantization section 103 and inputs encoding parameters from parameter encoding section 111. The encoding parameters include, for example, codes such as a reference picture index refIdxLX, a predicted vector index mvp_LX_idx, a difference vector mvdLX, a motion vector accuracy mode amvr_mode, a prediction mode predMode, and a merge index merge_idx.

Entropy encoding section 104 performs entropy encoding on division information, prediction parameters, quantization transform coefficients, etc., generates encoded stream Te, and outputs the encoded stream Te.

The parameter encoding unit 111 includes a header encoding unit 1110 (not shown), a CT information encoding unit 1111, a CU encoding unit 1112 (prediction mode encoding unit), and an inter prediction parameter encoding unit 112 and an intra prediction parameter encoding unit. It is equipped with 113. CU encoding section 1112 further includes a TU encoding section 1114.

The general operation of each module will be explained below. The parameter encoding unit 111 performs encoding processing of parameters such as header information, division information, prediction information, and quantization transform coefficients.

The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information, etc. from encoded data.

The CU encoding unit 1112 encodes CU information, prediction information, TU division flag split_transform_flag, CU residual flags cbf_cb, cbf_cr, cbf_luma, etc.

The TU encoding unit 1114 encodes the QP update information (quantization correction value) and the quantization prediction error (residual_coding) when the TU includes a prediction error.

The CT information encoding unit 1111 and the CU encoding unit 1112 encode inter prediction parameters (prediction mode predMode, merge flag merge_flag, merge index merge_idx, inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX), Syntax elements such as intra prediction parameters and quantized transform coefficients are supplied to entropy encoding section 104.

(Configuration of inter prediction parameter encoding unit)
Parameter encoding section 112 derives inter prediction parameters based on the prediction parameters input from encoding parameter determining section 110. Parameter encoding section 112 includes a configuration that is partially the same as the configuration in which inter prediction parameter decoding section 303 derives inter prediction parameters.

The configuration of prediction parameter encoding section 112 will be explained. As shown in FIG. 17, a parameter encoding control unit 1121, a merge prediction unit 30374, a subblock prediction unit (affine prediction unit) 30372, a DMVR unit 30375, an MMVD prediction unit 30376, a Triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, It is configured to include a subtraction section 1123. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The parameter encoding control unit 1121 includes a merge index deriving unit 11211 and a vector candidate index deriving unit 11212. Further, in the parameter encoding control unit 1121, a merge index deriving unit 11211 derives merge_idx, affine_flag, base_candidate_idx, distance_idx, direction_idx, etc., and a vector candidate index deriving unit 11212 derives mvpLX, etc. The merge prediction parameter derivation unit 3036, AMVP prediction parameter derivation unit 3032, affine prediction unit 30372, MMVD prediction unit 30376, and Triangle prediction unit 30377 may be collectively referred to as a motion vector derivation unit (motion vector derivation device). The parameter encoding unit 112 outputs the motion vector (mvLX, subMvLX), reference picture index refIdxLX, inter prediction identifier inter_pred_idc, or information indicating these to the predicted image generation unit 101. Parameter encoding section 112 also outputs merge_flag, skip_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_lX_idx, mvdLX, amvr_mode, and affine_flag to entropy encoding section 104.

The merge index derivation unit 11211 derives the merge index merge_idx and outputs it to the merge prediction parameter derivation unit 3036 (merge prediction unit). Vector candidate index derivation unit 11212 derives predicted vector index mvp_lX_idx.

The merge prediction parameter derivation unit 3036 derives inter prediction parameters based on the merge index merge_idx.

The AMVP prediction parameter derivation unit 3032 derives a prediction vector mvpLX based on the motion vector mvLX. AMVP prediction parameter derivation section 3032 outputs prediction vector mvpLX to subtraction section 1123. Note that the reference picture index refIdxLX and the predictive vector index mvp_lX_idx are output to the entropy encoding unit 104.

The affine prediction unit 30372 derives inter prediction parameters (affine prediction parameters) of subblocks.

The subtraction unit 1123 subtracts the prediction vector mvpLX that is the output of the AMVP prediction parameter derivation unit 3032 from the motion vector mvLX input from the encoding parameter determination unit 110 to generate a difference vector mvdLX. The difference vector mvdLX is output to entropy encoding section 104.

Adding section 106 adds the pixel value of the predicted image of the block inputted from predicted image generation section 101 and the prediction error inputted from inverse quantization/inverse transformation section 105 for each pixel to generate a decoded image. Adding unit 106 stores the generated decoded image in reference picture memory 109.

The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the addition unit 106. Note that the loop filter 107 does not necessarily need to include the above three types of filters, and may have a configuration including only a deblocking filter, for example.

The prediction parameter memory 108 stores the prediction parameters generated by the encoding parameter determination unit 110 at predetermined positions for each target picture and CU.

Reference picture memory 109 stores the decoded image generated by loop filter 107 at a predetermined position for each target picture and CU.

Encoding parameter determining section 110 selects one set from among multiple sets of encoding parameters. The encoding parameter is the above-mentioned QT, BT or TT division information, a prediction parameter, or a parameter to be encoded that is generated in relation to these. Predicted image generation section 101 generates a predicted image using these encoding parameters.

Encoding parameter determination section 110 calculates an RD cost value indicating the amount of information and encoding error for each of the plurality of sets. Encoding parameter determining section 110 selects a set of encoding parameters that minimizes the calculated cost value. Thereby, entropy encoding section 104 outputs the selected set of encoding parameters as encoded stream Te. Encoding parameter determining section 110 stores the determined encoding parameters in prediction parameter memory 108.

Note that some of the video encoding device 11 and video decoding device 31 in the embodiment described above, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, and the dequantization/inverse Transform unit 311, addition unit 312, predicted image generation unit 101, subtraction unit 102, transformation/quantization unit 103, entropy encoding unit 104, inverse quantization/inverse transformation unit 105, loop filter 107, encoding parameter determination unit 110 , the parameter encoding unit 111 may be realized by a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In that case, it may also include something that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Further, part or all of the video encoding device 11 and the video decoding device 31 in the embodiments described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the moving image encoding device 11 and the moving image decoding device 31 may be made into a processor individually, or some or all of them may be integrated into a processor. Moreover, the method of circuit integration is not limited to LSI, but may be implemented using a dedicated circuit or a general-purpose processor. Further, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although one embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to that described above, and various design changes etc. may be made without departing from the gist of the present invention. It is possible to

[Application example]
The above-described video encoding device 11 and video decoding device 31 can be used by being installed in various devices that transmit, receive, record, and reproduce video images. Note that the moving image may be a natural moving image captured by a camera or the like, or an artificial moving image (including CG and GUI) generated by a computer or the like.

First, it will be explained with reference to FIG. 2 that the above-described video encoding device 11 and video decoding device 31 can be used for transmitting and receiving video images.

FIG. 2(a) is a block diagram showing the configuration of the transmitting device PROD_A equipped with the video encoding device 11. As shown in the figure, the transmitter PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding a moving image, and a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1. and a transmitter PROD_A3 that transmits the modulated signal obtained by the modulator PROD_A2. The video encoding device 11 described above is used as this encoding unit PROD_A1.

The transmitting device PROD_A serves as a source of moving images input to the encoding unit PROD_A1, and includes a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for inputting moving images from the outside, and , may further include an image processing section A7 that generates or processes images. In the figure, a configuration in which the transmitter PROD_A includes all of these is illustrated, but some may be omitted.

Note that the recording medium PROD_A5 may be a recording of an unencoded moving image, or a recording of a moving image encoded using a recording encoding method different from the encoding method for transmission. It may be something. In the latter case, a decoding unit (not shown) may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1, which decodes the encoded data read from the recording medium PROD_A5 according to the recording encoding method.

FIG. 2(b) is a block diagram showing the configuration of the receiving device PROD_B equipped with the moving image decoding device 31. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulating unit PROD_B2 that obtains encoded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a demodulating unit PROD_B2 that obtains encoded data. It includes a decoding unit PROD_B3 that obtains a moving image by decoding encoded data. The above-mentioned moving image decoding device 31 is used as this decoding unit PROD_B3.

The receiving device PROD_B supplies the moving images output by the decoding unit PROD_B3 to a display PROD_B4 for displaying the moving images, a recording medium PROD_B5 for recording the moving images, and an output terminal for outputting the moving images to the outside. It may further include PROD_B6. Although the figure illustrates a configuration in which the receiving device PROD_B includes all of these, some may be omitted.

Note that the recording medium PROD_B5 may be for recording unencoded moving images, or it may be one that is encoded using a recording encoding method that is different from the transmission encoding method. It's okay. In the latter case, an encoding unit (not shown) may be interposed between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the moving image obtained from the decoding unit PROD_B3 according to a recording encoding method.

Note that the transmission medium for transmitting the modulated signal may be wireless or wired. Furthermore, the transmission mode for transmitting the modulated signal may be broadcasting (here, refers to a transmission mode in which the destination is not specified in advance), or communication (here, transmission mode in which the destination is specified in advance). ) may also be used. That is, transmission of the modulated signal may be realized by any of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

For example, a broadcasting station (broadcasting equipment, etc.)/receiving station (television receiver, etc.) of digital terrestrial broadcasting is an example of a transmitting device PROD_A/receiving device PROD_B that transmits and receives a modulated signal by wireless broadcasting. Further, a cable television broadcasting station (broadcasting equipment, etc.)/receiving station (television receiver, etc.) is an example of a transmitting device PROD_A/receiving device PROD_B that transmits and receives a modulated signal by wire broadcasting.

In addition, servers (workstations, etc.)/clients (television receivers, personal computers, smartphones, etc.) for VOD (Video On Demand) services and video sharing services using the Internet are transmitting devices that transmit and receive modulated signals via communication. This is an example of PROD_A/receiving device PROD_B (generally, either wireless or wired is used as the transmission medium in LAN, and wired is used as the transmission medium in WAN). Here, the personal computer includes a desktop PC, a laptop PC, and a tablet PC. Smartphones also include multifunctional mobile phone terminals.

Note that the client of the video sharing service has a function of decoding encoded data downloaded from the server and displaying the decoded data on a display, as well as a function of encoding a moving image captured by a camera and uploading it to the server. That is, the client of the video sharing service functions as both the transmitting device PROD_A and the receiving device PROD_B.

Next, it will be explained with reference to FIG. 3 that the above-described moving image encoding device 11 and moving image decoding device 31 can be used for recording and reproducing moving images.

FIG. 3(a) is a block diagram showing the configuration of a recording device PROD_C equipped with the video encoding device 11 described above. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding a moving image, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 to a recording medium PROD_M. It is equipped with. The video encoding device 11 described above is used as this encoding unit PROD_C1.

Note that the recording medium PROD_M may be of the type built into the recording device PROD_C, such as (1) HDD (Hard Disk Drive) or SSD (Solid State Drive), or (2) SD memory. It may be of the type connected to the recording device PROD_C, such as a card or USB (Universal Serial Bus) flash memory, or (3) DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray). Disc (registered trademark) may be loaded into a drive device (not shown) built into the recording device PROD_C.

The recording device PROD_C also serves as a source of moving images input to the encoding unit PROD_C1, including a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for inputting moving images from the outside, and a reception terminal for receiving moving images. The image processing apparatus may further include a section PROD_C5 and an image processing section PROD_C6 that generates or processes images. Although the figure illustrates a configuration in which the recording device PROD_C includes all of these, some may be omitted.

Note that the receiving unit PROD_C5 may receive unencoded moving images, or may receive encoded data encoded using a transmission encoding method different from the recording encoding method. It may be something that does. In the latter case, a transmission decoding unit (not shown) may be interposed between the receiving unit PROD_C5 and the encoding unit PROD_C1 to decode encoded data encoded using the transmission encoding method.

Examples of such a recording device PROD_C include a DVD recorder, BD recorder, HDD (Hard Disk Drive) recorder, etc. (In this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of moving images) . In addition, camcorders (in this case, camera PROD_C3 is the main source of moving images), personal computers (in this case, receiver PROD_C5 or image processing section C6 are the main sources of moving images), smartphones (in this case, camera PROD_C3 is the main source of moving images), In this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of moving images) is an example of such a recording device PROD_C.

FIG. 3(b) is a block diagram showing the configuration of a playback device PROD_D equipped with the video decoding device 31 described above. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads encoded data written in a recording medium PROD_M, and a decoding unit PROD_D2 that obtains a moving image by decoding the encoded data read by the reading unit PROD_D1. , is equipped with. The above-mentioned moving image decoding device 31 is used as this decoding unit PROD_D2.

Note that the recording medium PROD_M may be (1) a type built into the playback device PROD_D, such as an HDD or an SSD, or (2) a type such as an SD memory card or a USB flash memory. It may be of the type that is connected to the playback device PROD_D, or (3) it may be loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD. good.

The playback device PROD_D also has a display PROD_D3 for displaying the moving image, an output terminal PROD_D4 for outputting the moving image to the outside, and a transmitter for transmitting the moving image, as a supply destination of the moving image output by the decoding unit PROD_D2. It may further include PROD_D5. Although the figure illustrates a configuration in which the playback device PROD_D includes all of these, some of them may be omitted.

Note that the transmitter PROD_D5 may transmit unencoded moving images, or may transmit encoded data encoded using a transmission encoding method different from the recording encoding method. It may be something that does. In the latter case, it is preferable to interpose an encoding unit (not shown) that encodes the moving image using a transmission encoding method between the decoding unit PROD_D2 and the transmitting unit PROD_D5.

Examples of such a playback device PROD_D include a DVD player, BD player, HDD player, etc. (In this case, the output terminal PROD_D4 to which a television receiver etc. is connected is the main source of video images.) . In addition, television receivers (in this case, display PROD_D3 is the main source of moving images), digital signage (also called electronic billboards, electronic bulletin boards, etc.), and display PROD_D3 or transmitter PROD_D5 are the main source of moving images. ), a desktop PC (in this case, the output terminal PROD_D4 or the transmitter PROD_D5 is the main source of the video), a laptop or tablet PC (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main source of the video) Examples of such a playback device PROD_D include a smartphone (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main source of moving images).

(Hardware realization and software realization)
Furthermore, each block of the video decoding device 31 and the video encoding device 11 described above may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be implemented using a CPU (Central Processing It may also be realized in software using

In the latter case, each of the above devices includes a CPU that executes instructions of programs that implement each function, a ROM (Read Only Memory) that stores the above programs, a RAM (Random Access Memory) that expands the above programs, and the above programs and various data. It is equipped with a storage device (recording medium) such as a memory for storing the information. The purpose of the embodiment of the present invention is to provide a computer-readable record of the program code (executable program, intermediate code program, source program) of the control program for each of the above devices, which is software that realizes the above-described functions. This can also be achieved by supplying a medium to each of the above devices, and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

Examples of the recording medium include tapes such as magnetic tape and cassette tape, magnetic disks such as floppy (registered trademark) disks/hard disks, and CD-ROM (Compact Disc Read-Only Memory)/MO disks (Magneto-Optical discs). ) / MD (Mini Disc) / DVD (Digital Versatile Disc: registered trademark) / CD-R (CD Recordable) / Blu-ray Disc (registered trademark), etc. ) / cards such as optical cards, mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark) / semiconductor memory such as flash ROM, or PLD (Programmable Logic circuits such as FPGA (Field Programmable Gate Array) and FPGA (Field Programmable Gate Array) can be used.

Further, each of the above devices may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. This communication network is not particularly limited as long as it can transmit program codes. Examples include the Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television/Cable Television) communications network, and Virtual Private Network (LAN). network), telephone line network, mobile communication network, satellite communication network, etc. Further, the transmission medium constituting this communication network may be any medium that can transmit program codes, and is not limited to a specific configuration or type. For example, IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line), etc., wired lines, IrDA (Infrared Data Association), remote control, etc. , BlueTooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone networks, satellite lines, terrestrial digital broadcasting networks, etc. It is also available wirelessly. Note that embodiments of the present invention may also be implemented in the form of a computer data signal embedded in a carrier wave, in which the program code is embodied in electronic transmission.

The embodiments of the present invention are not limited to the embodiments described above, and various changes can be made within the scope of the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

[Additional notes]
The above embodiment can also be expressed as follows.

A video decoding device according to one aspect of the present invention includes a predicted image generation unit using bidirectional gradient change processing that generates a predicted image from two interpolated images by changing a gradient, and the predicted image generation unit includes: An L0 and L1 prediction generation unit generates an L0 predicted image and an L1 predicted image for each coding unit from two interpolated images, and bidirectional gradient change processing is performed for each coding unit from the L0 predicted image and L1 predicted image. a flag setting unit that determines whether or not to perform the above; a gradient image generation unit that generates four gradient images in a horizontal direction and a vertical direction from the L0 predicted image and the L1 predicted image; and the L0 predicted image and the L1 predicted image. and a correlation parameter calculation unit that calculates a correlation parameter for each processing unit from a product-sum operation of the four gradient images; and a motion compensation unit that derives a value for correcting a bidirectional predicted image from the gradient image and the correlation parameter. a correction value derivation unit; a bidirectional predicted image generation unit that generates a predicted image from the L0 predicted image, the L1 predicted image, and the motion compensation correction value; It is characterized by making the calculation accuracy of the difference between the predicted image and the L1 predicted image the same.

Further, in the video decoding device according to one aspect of the present invention, in the flag setting unit and the correlation parameter calculation unit, the calculation accuracy of the difference between the L0 predicted image and the L1 predicted image is made constant regardless of the pixel bit length. It is characterized by

[Configuration 1]
It has a predicted image generation unit that uses bidirectional gradient change processing to generate a predicted image from two interpolated images by changing the gradient, and the predicted image generation unit generates L0 prediction for each coding unit from the two interpolated images. an L0 and L1 prediction generation unit that generates an image and an L1 prediction image; a flag setting unit that determines whether or not bidirectional gradient change processing is to be performed for each coding unit from the L0 prediction image and the L1 prediction image; A gradient image generation unit that generates four gradient images in the horizontal and vertical directions from the L0 predicted image and the L1 predicted image, and processing from the product-sum calculation of the L0 predicted image, the L1 predicted image, and the four gradient images. a correlation parameter calculation unit that calculates a correlation parameter for each unit; a motion compensation correction value derivation unit that derives a value for correcting a bidirectional predicted image from the gradient image and the correlation parameter; and the L0 prediction image and the L1 prediction. It has a bidirectional predicted image generation unit that generates a predicted image from the image and the motion compensation correction value, and the flag setting unit and the correlation parameter calculation unit have the same calculation precision for the difference between the L0 predicted image and the L1 predicted image. A moving image decoding device characterized by:

[Configuration 2]
Video decoding characterized in that in the flag setting section and the correlation parameter calculation section of the video decoding device according to Configuration 1, the calculation accuracy of the difference between the L0 predicted image and the L1 predicted image is made constant regardless of the pixel bit length. Device.

[Configuration 3]
It has a predicted image generation unit that uses bidirectional gradient change processing to generate a predicted image from two interpolated images by changing the gradient, and the predicted image generation unit generates L0 prediction for each coding unit from the two interpolated images. an L0 and L1 prediction generation unit that generates an image and an L1 prediction image; a flag setting unit that determines whether or not bidirectional gradient change processing is to be performed for each coding unit from the L0 prediction image and the L1 prediction image; A gradient image generation unit that generates four gradient images in the horizontal and vertical directions from the L0 predicted image and the L1 predicted image, and processing from the product-sum calculation of the L0 predicted image, the L1 predicted image, and the four gradient images. a correlation parameter calculation unit that calculates a correlation parameter for each unit; a motion compensation correction value derivation unit that derives a value for correcting a bidirectional predicted image from the gradient image and the correlation parameter; and the L0 prediction image and the L1 prediction. It has a bidirectional predicted image generation unit that generates a predicted image from the image and the motion compensation correction value, and the flag setting unit and the correlation parameter calculation unit have the same calculation precision for the difference between the L0 predicted image and the L1 predicted image. A video encoding device characterized by:

[Configuration 4]
A moving image characterized in that, in the flag setting section and the correlation parameter calculation section of the moving image encoding device according to configuration 3, the calculation accuracy of the difference between the L0 predicted image and the L1 predicted image is made constant regardless of the pixel bit length. Encoding device.

Embodiments of the present invention are suitably applied to a video decoding device that decodes encoded data obtained by encoding image data, and a video encoding device that generates encoded data obtained by encoding image data. be able to. Further, the present invention can be suitably applied to the data structure of encoded data generated by a video encoding device and referenced by a video decoding device.

31 Image decoding device
301 Entropy decoding section
302 Parameter decoding section
3020 Header decoding section
303 Inter prediction parameter decoding unit
304 Intra prediction parameter decoding unit
308 Predicted image generation unit
309 Inter predicted image generation unit
310 Intra predicted image generation unit
311 Inverse quantization/inverse transformation section
312 Addition section
11 Image encoding device
101 Predicted image generation unit
102 Subtraction section
103 Conversion/quantization section
104 Entropy encoder
105 Inverse quantization/inverse transformation section
107 Loop filter
110 Encoding parameter determination unit
111 Parameter encoding section
112 Inter prediction parameter encoder
113 Intra prediction parameter encoding unit
1110 Header encoder
1111 CT information encoder
1112 CU encoding unit (prediction mode encoding unit)
1114 TU encoding section
30954 BDOF section
309541 L0,L1 predicted image generation unit
309542 Flag setting section
309543 Gradient image generation unit
309544 Correlation parameter calculation section
309545 Motion compensation correction value derivation unit
309546 Bidirectional predictive image generation unit

Claims (5)

A moving image decoding device that generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value,
comprising a BDOF unit that derives a sum of absolute differences between the first predicted image and the second predicted image in sub-block units;
The third predicted image is a combination of (i) the first predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value. Derived by shifting the summation value to the right by the first shift value,
The motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, a horizontal gradient image of the second predicted image, and a horizontal gradient image of the first predicted image. a vertical gradient image of the second predicted image,
The horizontal gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right with a second shift value,
The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value,
The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value,
A video decoding device characterized in that the vertical gradient image of the second predicted image is derived by taking a difference between values obtained by shifting the second predicted image to the right by the second shift value. . The moving image decoding device according to claim 1, wherein the second shift value is six. A video encoding device that generates a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value,
comprising a BDOF unit that derives a sum of absolute differences between the first predicted image and the second predicted image in sub-block units;
The third predicted image is a combination of (i) the first predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value. Derived by right-shifting the summation value by the first shift value,
The motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, a horizontal gradient image of the second predicted image, and a horizontal gradient image of the first predicted image. a vertical gradient image of the second predicted image,
The horizontal gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right with a second shift value,
The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value,
The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value,
Video encoding characterized in that the vertical gradient image of the second predicted image is derived by taking a difference between values obtained by shifting the second predicted image to the right by the second shift value. Device. A video decoding method for generating a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value, the method comprising:
At least the step of deriving the sum of absolute differences between the first predicted image and the second predicted image in sub-block units,
The third predicted image is a combination of (i) the first predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value. Derived by shifting the summation value to the right by the first shift value,
The motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, a horizontal gradient image of the second predicted image, and a horizontal gradient image of the first predicted image. a vertical gradient image of the second predicted image,
The horizontal gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right with a second shift value,
The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value,
The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value,
A video decoding method characterized in that the vertical gradient image of the second predicted image is derived by taking a difference between values obtained by shifting the second predicted image to the right by the second shift value. . A video encoding method for generating a third predicted image by BDOF prediction using a first predicted image, a second predicted image, and a motion compensation correction value, the method comprising:
At least the step of deriving the sum of absolute differences between the first predicted image and the second predicted image in sub-block units,
The third predicted image is a combination of (i) the first predicted image, (ii) the second predicted image, (iii) the motion compensation correction value, and (iv) the first offset value. Derived by shifting the summation value to the right by the first shift value,
The motion compensation correction value includes a horizontal gradient image of the first predicted image, a vertical gradient image of the first predicted image, a horizontal gradient image of the second predicted image, and a horizontal gradient image of the first predicted image. a vertical gradient image of the second predicted image,
The horizontal gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right with a second shift value,
The vertical gradient image of the first predicted image is derived by taking the difference between the values obtained by shifting the first predicted image to the right by the second shift value,
The horizontal gradient image of the second predicted image is derived by taking the difference between the values obtained by shifting the second predicted image to the right by the second shift value,
Video encoding characterized in that the vertical gradient image of the second predicted image is derived by taking a difference between values obtained by shifting the second predicted image to the right by the second shift value. Method.

Images