CN114270867A

CN114270867A - Image decoding device, image decoding method, and program

Info

Publication number: CN114270867A
Application number: CN202080058192.5A
Authority: CN
Inventors: 海野恭平; 河村圭; 内藤整
Original assignee: KDDI Corp
Current assignee: KDDI Corp
Priority date: 2019-09-20
Filing date: 2020-08-21
Publication date: 2022-04-01
Also published as: WO2021054040A1; JP2021052228A; JP7200074B2

Abstract

The image decoding device 200 of the present invention includes: an affine prediction signal generating unit 241F configured to determine whether or not the PROF processing is appropriate for each block, and when the PROF processing is applied, perform the PROF processing for each sub-block obtained by dividing the block; and a loop filter processing unit 250 configured to determine whether or not deblocking filtering is appropriate for each block boundary, and to determine not to apply deblocking filtering to the sub-block boundary of the block to which the PROF processing has been applied.

Description

Image decoding device, image decoding method, and program

[ technical field ]

The present invention relates to an image decoding device, an image decoding method, and a program.

[ background art ]

Conventionally, a PROF (Prediction reference with Optical Flow Refinement) technique and a deblocking filter technique are known (for example, refer to non-patent document 1).

The PROF is a technique for generating a prediction signal with high accuracy by using a motion model corresponding to parallel translation and rotation in units of 4 × 4 pixel blocks, and can be expected to improve both subjective image quality and objective image quality compared to a case where the PROF processing is not applied.

The deblocking filter is a low-pass filter applied to a block boundary, and is expected to suppress block noise and improve subjective image quality.

[ Prior art documents ]

Non-patent document

Non-patent document 1: versatile Video Coding (Draft 6), JFET-N1001

[ summary of the invention ]

Problems to be solved by the invention

In the prior art, at block boundaries where the PROF processing is applied, it is also possible to apply deblocking filtering. However, there are the following problems: for a block whose subjective image quality has been sufficiently improved by the PROF process, there is a possibility that the subjective image quality is deteriorated by applying deblocking filtering.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image decoding device, an image decoding method, and a program that can prevent subjective image quality from deteriorating by not applying deblocking filtering to sub-block boundaries to which PROF processing is applied.

Means for solving the problems

A first aspect of the present invention is summarized as an image decoding apparatus including: an affine prediction signal generating unit configured to determine whether or not a PROF process is appropriate for each block, and when the PROF process is applied, perform the PROF process for each sub-block into which the block is divided; and a loop filter processing unit configured to determine whether or not deblocking filtering is appropriate for each block boundary, and to determine not to apply deblocking filtering to a sub-block boundary of the block to which the PROF processing has been applied.

A second feature of the present invention is summarized as an image decoding method including the steps of: determining whether or not PROF processing is appropriate for each block, and when the PROF processing is applied, performing the PROF processing for each sub-block into which the block is divided; and judging whether the deblocking filtering is suitable for each block boundary, and determining not to apply the deblocking filtering to the sub-block boundary of the block subjected to the PROF processing.

A third aspect of the present invention is summarized as a program used in an image decoding apparatus, the program causing a computer to execute: determining whether or not PROF processing is appropriate for each block, and when the PROF processing is applied, performing the PROF processing for each sub-block into which the block is divided; and judging whether the deblocking filtering is suitable for each block boundary, and determining not to apply the deblocking filtering to the sub-block boundary of the block subjected to the PROF processing.

ADVANTAGEOUS EFFECTS OF INVENTION

According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program that can prevent subjective image quality from deteriorating by not applying deblocking filtering to sub-block boundaries to which PROF processing is applied.

[ brief description of the drawings ]

Fig. 1 is a diagram showing an example of the configuration of an image processing system 10 according to an embodiment.

Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to the embodiment.

Fig. 3 is a diagram showing an example of the configuration of encoded data (bit stream) output from the encoding unit 140 of the image encoding device 100 according to the embodiment.

Fig. 4 is a diagram showing an example of control data included in SPS 141 shown in fig. 3.

Fig. 5 is a diagram showing an example of control data included in the slice header 143A shown in fig. 3.

Fig. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

Fig. 7 is a flowchart showing an example of the processing procedure of the thinning unit 111C of the inter prediction unit 111 of the image coding device 100 according to the embodiment.

Fig. 8 is a flowchart showing an example of the processing procedure of the prediction signal generation unit 111D of the inter prediction unit 111 of the image encoding device 100 according to the embodiment.

Fig. 9 is a flowchart showing an example of the processing procedure of the affine motion vector calculating unit 111E of the inter prediction unit 111 of the image encoding device 100 according to the embodiment.

Fig. 10 is a diagram for explaining an example of the processing procedure of the affine motion vector calculating unit 111E of the inter prediction unit 111 of the image encoding device 100 according to the embodiment.

Fig. 11 is a flowchart showing an example of the processing procedure of the affine prediction signal generating unit 111F of the inter prediction unit 111 of the image encoding device 100 according to the embodiment.

Fig. 12 is a diagram showing an example of functional blocks of the loop filter processing unit 150 of the image encoding device 100 according to one embodiment.

Fig. 13 is a diagram for explaining a processing procedure of the boundary strength determination unit 153(153A/153B) of the loop filter processing unit 150 of the image encoding device 100 according to the embodiment.

Fig. 14 is a diagram showing an example of functional blocks of the image decoding apparatus 200 according to the embodiment.

Fig. 15 is a diagram showing an example of functional blocks of the inter prediction unit 241 of the image decoding apparatus 200 according to the embodiment.

Fig. 16 is a diagram showing an example of functional blocks of the loop filter processing unit 250 of the image decoding device 200 according to one embodiment.

Fig. 17 is a diagram for explaining modification 1.

Fig. 18 is a diagram for explaining modification 1.

Fig. 19 is a diagram for explaining modification 2.

Fig. 20 is a diagram for explaining modification 2.

[ detailed description of the invention ]

Embodiments of the present invention will be described below with reference to the drawings. Further, the components in the following embodiments may be replaced with existing components as appropriate, and various modifications including combinations with other existing components may be made. Therefore, the contents of the invention described in the claims should not be limited to the description of the embodiments described below.

(first embodiment)

Next, an image processing system 10 according to a first embodiment of the present invention will be described with reference to fig. 1 to 16. Fig. 1 is a diagram showing an image processing system 10 of the present embodiment.

As shown in fig. 1, the image processing system 10 includes an image encoding device 100 and an image decoding device 200.

The image encoding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding apparatus 200 is configured to generate an output image signal by decoding the encoded data.

Here, the encoded data may be transmitted from the image encoding apparatus 100 to the image decoding apparatus 200 via a transmission path. The encoded data may be stored in a storage medium and then supplied from the image encoding apparatus 100 to the image decoding apparatus 200.

(image encoding apparatus 100)

Next, the image coding device 100 according to the present embodiment will be described with reference to fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to the present embodiment.

As shown in fig. 2, the image encoding device 100 includes an inter-frame prediction unit 111, an intra-frame prediction unit 112, a subtractor 121, an adder 122, a transformation and quantization unit 131, an inverse transformation and inverse quantization unit 132, an encoding unit 140, a loop filter processing unit 150, and a frame buffer 160.

The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter prediction).

Specifically, the inter-frame prediction unit 111 is configured to specify a reference block included in a reference frame by comparing a frame to be encoded (hereinafter, referred to as a target frame) with the reference frame stored in the frame buffer 160, and to specify a motion vector of the specified reference block.

The inter prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the object frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra prediction).

Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. The intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block to which a block to be predicted (hereinafter referred to as a target block) refers. For example, the reference block is a block adjacent to the object block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is a difference between a prediction signal generated by intra prediction or inter prediction and an input image signal.

The adder 122 is configured to generate a pre-filter decoded signal by adding the prediction signal to the prediction residual signal output from the inverse transform and inverse quantization unit 132, and output the pre-filter decoded signal to the intra prediction unit 112 and the loop filter processing unit 150.

Here, the decoded signal before the filtering process constitutes a reference block used in the intra prediction unit 112.

The transform and quantization unit 131 is configured to perform a transform process on the prediction residual signal and acquire a coefficient level value. The transform/quantization unit 131 may be configured to quantize the coefficient level value.

Here, the transform processing is processing for transforming the prediction residual signal into a frequency component signal. In this transformation process, a base model (transformation matrix) corresponding to Discrete Cosine Transform (DCT) may be used, or a base model (transformation matrix) corresponding to Discrete Sine Transform (DST) may be used.

The inverse transform and inverse quantization unit 132 is configured to perform an inverse transform process on the coefficient level values output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values before the inverse transform process.

Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131.

The encoding unit 140 is configured to encode the coefficient level value output from the transform and quantization unit 131 and output encoded data.

Here, for example, the encoding is entropy encoding in which codes of different lengths are assigned based on the generation probability of the coefficient level value.

The encoding unit 140 is configured to encode not only the coefficient level value but also control data used for decoding.

Here, the control data may include size data such as a Coding Unit (CU) size, a Prediction Unit (PU) size, a Transform Unit (TU) size, and the like.

The loop filter processing unit 150 is configured to perform filter processing on the pre-filter-processed decoded signal output from the adder 122, and to output the post-filter-processed decoded signal to the frame buffer 160.

Here, the filtering process is, for example, a deblocking filtering process that reduces distortion generated at a boundary portion of a block (coding block, prediction block, or transform block).

The frame buffer 160 is configured to store a reference frame used by the inter prediction unit 111.

Here, the decoded signal after the filtering process constitutes a reference frame used in the inter prediction unit 111.

(encoding section 140)

Next, the control data encoded by the encoding unit 140 will be described with reference to fig. 3 to 5. Fig. 3 shows an example of the configuration of encoded data (bit stream) output from the encoding unit 140.

First, the bitstream may include an SPS (Sequence Parameter Set) 141 at the beginning. The SPS 141 is a set of control data in units of a sequence (a set of pictures). Specific examples will be described below. Each SPS 141 includes at least id information for identifying each SPS 141 when there are a plurality of SPS 141.

The bitstream may also include a PPS (Picture Parameter Set) 142 following the SPS 141. The PPS 142 is a set of control data in units of pictures (a set of slices). Each PPS 142 includes at least id information for identifying each SPS 141 when there are a plurality of SPS 141. Each PPS 142 includes at least SPS id information for specifying the SPS 141 corresponding to each PPS 142.

The bitstream may also contain a slice header 143A after the PPS 142. The slice header 143A is a set of control data in units of slices. Specific examples will be described below. Each slice header 143A includes at least PPS id information for specifying the PPS 142 corresponding to each slice header 143A.

The bitstream may also contain slice data 144A after the slice header 143A. Slice data 144A may include coefficient level values and size data, etc., as described above.

As described above, the present invention is configured as follows: one slice header 143A/143B, PPS 142, SPS 141 each corresponds to each slice data 144A/144B. As described above, in the slice header 143A/143B, which PPS 142 is referred to is specified by PPS id information, and in the PPS 142, which SPS 141 is referred to is further specified by SPS id information, and therefore, the SPS 141 and the PPS 142 that are common to the plurality of slice data 144A/144B can be used.

In other words, the SPS 141 and PPS 142 need not be transmitted for each slice data 144A/144B. For example, as shown in fig. 3, the following may be configured: SPS 141 and PPS 142 are not encoded prior to slice header 143B and slice data 144B.

The configuration of fig. 3 is merely an example. In the configuration in which the control data specified by the slice header 144A/144B, PPS 142 and the SPS 141 corresponds to each slice data 144A/144B, elements other than the above may be added as components of the stream. Similarly, the shaping may be performed at the time of transmission so as to have a configuration different from that of fig. 3.

Fig. 4 is a diagram showing an example of control data included in SPS 141.

As described above, the SPS 141 includes at least id information (SPS _ seq _ parameter _ set _ id) for identifying each SPS 141.

SPS 141 may include a flag "SPS _ bdofjenabled _ flag" for controlling whether or not the following BDOF (Bi-Directional Optical Flow) process can be used (enabled or disabled). When the flag has a value of "0", it indicates that the BDOF process is not used in the slice corresponding to the SPS 141. On the other hand, when the flag has a value of "1", it indicates that the BDOF process is available in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether or not the BDOF processing is actually used in each block within the slice.

That is, "SPS _ BDOF _ enabled _ flag" is a flag (fourth flag) included in SPS 141 and indicating whether or not the BDOF process can be used in units of a sequence.

SPS 141 may also include a flag "SPS _ DMVR _ enabled _ flag" for controlling whether or not DMVR (Decoder-side Motion Vector reference) processing described below is available (valid, invalid). When the flag has a value of "0", it indicates that DMVR processing is not used in the slice corresponding to the SPS 141. On the other hand, when the value of the flag is "1", it indicates that DMVR processing is available in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether or not DMVR processing is actually used in each block within a slice.

That is, "SPS _ DMVR _ enabled _ flag" is a flag (third flag) included in SPS 141 indicating whether or not DMVR processing can be used in sequence units.

SPS 141 may also contain a flag "SPS _ affine _ enabled _ flag" for controlling the availability (active, inactive) of affine motion compensation described below. When the flag has a value of "0", it indicates that affine motion compensation is not used in the slice corresponding to the SPS 141. On the other hand, when the flag has a value of "1", it indicates that affine motion compensation is available in the slice corresponding to the SPS 141. In the following process, it is determined for each block whether affine motion compensation is actually used in the blocks within the slice.

That is, "SPS _ affine _ PROF _ enabled _ flag" is a flag (fifth flag) included in the SPS 141 indicating whether or not the PROF processing can be used in sequence units.

SPS 141 may additionally include a flag of "SPS _ affine _ type _ flag" when "SPS _ affine _ enabled _ flag" is "1". When the flag has a value of "0", it indicates that the number of parameters for affine motion compensation described below is always "4" in the slice corresponding to the SPS 141. On the other hand, if the flag has a value of "1", this indicates that affine motion compensation is performed in the slice corresponding to the SPS 141, and either "4" or "6" may be selected as the number of parameters for each block.

SPS 141 may additionally include a flag of "SPS _ affine _ prof _ enabled _ flag" when the value of "SPS _ affine _ enabled _ flag" is "1". When the flag has a value of "0", it indicates that the following PROF (prediction reference with Optical flow) processing is not used in the slice corresponding to the SPS 141. On the other hand, when the value of the flag is "1", it indicates that the PROF processing is available in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether or not the PROF processing is actually used in each block within the slice.

SPS 141 may additionally include a flag "SPS _ bdofdmvr _ prof _ slice _ present _ flag" when the value of at least any of the flags "SPS _ bdofjenabled _ flag", "SPS _ dmvr _ enabled _ flag" and "SPS _ affine _ prof _ enabled _ flag" is "1". When the flag is "1", it indicates that the following flag "slice _ disable _ bdef _ dmvr _ prof _ flag" is included in the slice header 143A/143B corresponding to the SPS 141. When the value of this flag is "0", it indicates that "slice _ disable _ bdef _ dmvr _ prof _ flag" is not included in the slice header 143A/143B corresponding to this SPS 141.

That is, "sps _ bdofj dmvr _ prof _ slice _ present _ flag" is a flag (second flag) indicating whether or not "slice _ disable _ bdofj dmvr _ prof _ flag" is included in the slice header 143A/143B. In addition, when the flag is not present, the value of the flag may be implicitly regarded as "0".

Fig. 5 is a diagram showing an example of control data included in the slice header 143A/143B.

As described above, the slice header 143A/143B includes at least the PPS id information "slice _ pic _ parameter _ set _ id" for specifying the PPS 142 corresponding to the slice. As described above, the SPS 141 referred to within the PPS specified by the PPS id information is the SPS 141 corresponding to the slice header 143A/143B.

When "SPS _ bdofdmvr _ prof _ slice _ present _ flag" is included in the SPS 141 corresponding to the slice header 143A/143B and the flag has a value of "1", the slice header 143A/143B may also include "slice _ disable _ bdofdmvr _ prof _ flag". When the flag has a value of "1", the BDOF process, DMVR process, or PROF process may be controlled not to be used in each block included in the slice, as described below. When the flag has a value of "0", the BDOF process, DMVR process, or PROF process can be controlled to be used for each block included in the slice.

That is, "slice _ disable _ bdofjdmvr _ PROF _ flag" is a flag (first flag) included in the slice header 143A/143B that collectively controls the availability of DMVR processing, BDOF processing, and PROF processing.

In other words, "slice _ disable _ bdofdmvr _ PROF _ flag" is a flag contained in the slice header 143A/143B for controlling the availability of the PROF process, a flag contained in the slice header 143A/143B for controlling the availability of the DMVR process, and a flag contained in the slice header 143A/143B for controlling the availability of the BDOF process.

The value of the flag described above is just an example. When the values ("0" and "1") given to the flags are made to have opposite meanings, equivalent processing can be realized by making the corresponding processing correspondingly opposite.

Further, as described above, since the image encoding device 100 side can clearly control not to use the function by setting the flag for controlling not to use the BDOF process, the DMVR process, or the PROF process in units of slices, the processing load and the power consumption of the image encoding device 100 and the corresponding image decoding device 200 can be reduced by not using the function.

(inter prediction unit 111)

Next, the inter prediction unit 111 of the image encoding device 100 according to the present embodiment will be described with reference to fig. 6. Fig. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to the present embodiment.

As shown in fig. 6, the inter-frame prediction unit 111 includes a motion vector search unit 111A, a motion vector encoding unit 111B, a thinning unit 111C, a prediction signal generation unit 111D, an affine motion vector calculation unit 111E, and an affine prediction signal generation unit 111F.

The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector search unit 111A is configured to specify a reference block included in a reference frame by comparing a target frame with the reference frame, and search for a motion vector of the specified reference block.

Then, the above search is performed on the plurality of candidate reference frames to determine the reference frame and the motion vector used for prediction in the prediction block. A maximum of two reference frames and motion vectors may be used for a block each. The case of using only one set of reference frames and motion vectors for a block is called unidirectional prediction, and the case of using two sets of reference frames and motion vectors is called bidirectional prediction. Hereinafter, the first group is referred to as L0, and the second group is referred to as L1.

The motion vector search unit 111A is configured to determine a reference frame and a method of encoding a motion vector. In addition to the conventional method of transmitting information of a reference frame and a motion vector separately, the encoding method includes the following merge mode and affine motion compensation.

In addition, since a known method can be adopted as a method for searching for a motion vector, a method for determining a reference frame, and a method for determining a reference frame and a method for encoding a motion vector, detailed description thereof will be omitted.

The motion vector encoding unit 111B is configured to encode the information of the reference frame and the motion vector, which are also determined by the motion vector search unit 111A, by using the encoding method determined by the motion vector search unit 111A. A specific reference frame and a coding method of motion vector information may be known, and thus, a detailed description thereof will be omitted.

When the encoding method of the block is the merge mode, the merge list for the block is first created on the image encoding apparatus 100 side. Here, the merge list is a list listing combinations of a plurality of reference frames and motion vectors.

An index is assigned to each combination, and image encoding apparatus 100 encodes only the index and transmits it to image decoding apparatus 200, instead of encoding information of the reference frame and the motion vector separately. By sharing the merge list creation method between the image encoding apparatus 100 and the image decoding apparatus 200 in advance, the image decoding apparatus 200 can decode the information of the reference frame and the motion vector only based on the information of the index.

In addition, as for the method of creating the merge list, a known method can be adopted, and therefore, detailed description thereof is omitted.

Affine motion compensation is the following method: a small number of parameters are transmitted for each block and based on the parameters and a predetermined model, a motion vector is derived for each sub-block into which the block is divided. The details of the method of deriving the motion vector will be described later.

In the case of affine motion compensation, the motion vector encoding unit 111B is configured to encode information of a control point motion vector and information of a reference frame, which will be described below. As described below, the control point motion vector transmits two or three motion vector information per block.

When the block is a bi-directional predicted block, two or three pieces of information of the control point motion vector are transmitted to each of L0 and L1. As for a specific encoding method, a known method can be employed, and thus, detailed description thereof is omitted.

When affine motion compensation is used in the block, the process moves to the affine motion vector calculation part 111E shown in fig. 6, and otherwise, the process moves to the thinning part 111C.

The thinning unit 111C is configured to perform thinning processing (for example, DMVR) for correcting the motion vector encoded by the motion vector encoding unit 111B.

Specifically, the thinning unit 111C is configured to perform thinning processing as follows: a search range is set with reference to the reference position specified by the motion vector encoded by the motion vector encoding unit 111B, a correction reference position with the minimum specific cost is specified from the search range, and the motion vector is corrected based on the correction reference position.

Fig. 7 is a flowchart showing an example of the processing procedure of the thinning unit 111C.

As shown in fig. 7, in step S71, the thinning section 111C determines whether a specific condition for applying the thinning process is satisfied. When all of the specific conditions are satisfied, the present processing sequence proceeds to step S72. On the other hand, when any of the specific conditions is not satisfied, the present processing sequence proceeds to step S75, and the thinning processing is ended.

Here, the specific condition includes a condition that the block is a bi-directionally predicted block.

Further, the specific conditions may include the following conditions: either one of the reference frame on the L0 side and the reference frame on the L1 side is a temporally past frame with respect to the frame, the other reference frame is a temporally subsequent frame with respect to the frame, and the temporal distances (for example, the absolute value of the difference of POC (Picture Order Count) from the frame) on the L0 side and the L1 side are equal.

Also, the specific condition may also include a condition that the motion vector is encoded in the merge mode.

Further, the specific conditions may include the following conditions: in the SPS 141 of the sequence to which the block belongs, the value of "SPS _ DMVR _ enabled _ flag" indicates that DMVR can be used in the sequence (for example, the value of the flag is "1").

Further, the specific conditions may also include the following conditions: in the slice header 143A/143B of the slice to which the block belongs, a value of "slice _ disable _ bdef _ DMVR _ prof _ flag" indicates that DMVR is usable in the slice (for example, the value of the flag is "0").

In step S72, the refinement unit 111C generates a search image based on the motion vector encoded by the motion vector encoding unit 111B and the information of the reference frame.

Here, when the motion vector points to a non-integer pixel position, the refinement section 111C applies a filter to interpolate a pixel of the non-integer pixel position to a pixel value of the reference frame. In this case, the thinning unit 111C can reduce the amount of calculation by using an interpolation filter having fewer taps than the interpolation filter used in the prediction signal generating unit 111D described below. For example, the thinning section 111C may interpolate the pixel value of the non-integer pixel position by bilinear interpolation.

In step S73, the thinning unit 111C performs a search with integer pixel accuracy using the search image generated in step S72. Here, the integer pixel precision refers to searching for only points that are spaced apart by integer pixels based on the motion vector encoded by the motion vector encoding unit 111B.

The refinement section 111C determines the modified motion vector at the integer-pixel interval position by the search of step S72. As the search method, a known method may be used.

For example, the thinning unit 111C may perform the search by the following method: only the points corresponding to the combinations obtained by inverting the signs of the differential motion vectors on the L0 side and the L1 side are searched for. In the search, the thinning-out unit 111C may calculate a search cost at each search point, for example, and correct the motion vector at the search point where the search cost is the smallest.

Here, the thinning-out unit 111C may calculate the search cost so that the search cost is reduced only at the search point corresponding to the motion vector encoded by the motion vector encoding unit 111B. For example, the refining unit 111C may set a value obtained by reducing the search cost calculated by the same method as for the other search points to 3/4 as the final search cost for the search point.

The search result in step S73 may also be the same value as the motion vector before the search.

In step S74, the refinement unit 111C searches for a motion vector with non-integer-pixel accuracy using the corrected motion vector with integer-pixel accuracy determined in step S73 as an initial value. As a search method of the motion vector, a known method can be used.

When a search is not actually performed, the thinning unit 111C may determine a vector with non-integer pixel accuracy by using a parametric model such as parabolic fitting, with the result of step S73 as input.

After determining the corrected motion vector with non-integer pixel accuracy in step S74, the thinning unit 111C proceeds to step S45 and terminates the thinning process. Here, for convenience, the expression of the motion vector after correction of non-integer-pixel accuracy is used, but the result may be the same value as the motion vector of integer-pixel accuracy obtained in step S73, depending on the search result in step S74.

The thinning unit 111C may divide a block larger than a predetermined threshold into smaller sub-blocks and perform thinning processing on each sub-block. For example, the thinning unit 111C may set the unit of performing thinning processing to 16 × 16 pixels in advance, and when the size of the block in the horizontal direction or the vertical direction is larger than 16 pixels, the thinning unit may be divided into 16 pixels or less. At this time, the motion vector of the block encoded by the motion vector encoding unit 111B is used as the reference motion vector for the thinning-out processing for all sub-blocks within the same block.

When processing each sub-block, the thinning section 111C may perform all the steps of fig. 7 for each sub-block. The thinning unit 111C may perform the partial processing of fig. 7 only for each sub-block. Specifically, the thinning unit 111C may perform the processes of steps S71 and S72 of fig. 7 for each block, and perform the processes of steps S73 and S74 for each sub-block.

The prediction signal generation unit 111D is configured to generate a prediction signal based on the corrected motion vector output from the refinement unit 111C.

Here, the prediction signal generation unit 111D is configured to determine whether or not to perform the BDOF process for each block based on information (for example, search cost) calculated in the refinement process described above, as described below.

Specifically, when the motion vector is not corrected, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector encoding unit 111B. On the other hand, when the motion vector is corrected, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the thinning-out unit 111C.

Fig. 8 is a flowchart showing an example of the processing procedure of the prediction signal generation unit 111D. Here, when the thinning processing is performed in units of sub-blocks by the thinning unit 111C, the processing of the prediction signal generating unit 111D is also performed in units of sub-blocks. At this time, the term block in the following description may be replaced with a sub-block as appropriate.

As shown in fig. 8, in step S81, the prediction signal generation unit 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D receives the motion vector encoded by the motion vector encoding unit 111B or the motion vector encoded by the thinning unit 111C as an input, and when the position indicated by the motion vector is a non-integer pixel position, applies a filter to interpolate a pixel at the non-integer pixel position to the pixel value of the reference frame. Here, as for a specific filter, the maximum 8-tap horizontal-vertical separable type filter disclosed in non-patent document 1 can be applied.

When the block is a bi-directionally predicted block, the prediction signal generator 111D generates both a prediction signal based on the first group (hereinafter referred to as L0) of reference frames and motion vectors and a prediction signal based on the second group (hereinafter referred to as L1) of reference frames and motion vectors.

In step S82, the prediction signal generation unit 111D checks for each block whether or not the following application condition of the BDOF process is satisfied.

The application condition at least includes a condition that the block is a bi-directionally predicted block.

The application conditions may include the following conditions: in the integer pixel position search in step S73, the refinement unit 111C determines that the search cost of the search point with the smallest search cost is greater than or equal to a predetermined threshold value.

Further, the specific conditions may include the following conditions: in the SPS 141 of the sequence to which the block belongs, the value of "SPS _ BDOF _ enabled _ flag" indicates that BDOF can be used in the sequence (for example, the value of this flag is "1").

The application conditions may include the following conditions: in the slice header 143A/143B of the slice to which the block belongs, a value of "slice _ disable _ bdofj dmvr _ prof _ flag" indicates that the BDOF process is usable in the slice (for example, the value of the flag is "0").

The application conditions may include the following conditions: either one of the reference frame on the L0 side and the reference frame on the L1 side is a temporally past frame with respect to the frame, the other reference frame is a temporally subsequent frame with respect to the frame, and the ratio of the temporal distance (for example, the absolute value of the difference of POC) from the frame on the L0 side and the L1 side is a predetermined ratio.

Here, as a method of determining the ratio, for example, the ratio can be determined as follows: the weighting coefficient is proportional to the reciprocal of the weighting coefficient when the gradient or the luminance value on the L0 side and the L1 side in the below-described BDOF process is used for calculation.

For example, when the weight coefficient on the L0 side is "2/3" and the weight coefficient on the L1 side is "1/3", the distance between L1 and the inter frame is "2" when the distance between L0 and the inter frame is "1" as the ratio of the distance of the reference frame, and only in this case, the BDOF process is applied.

Similarly, for example, when the weight coefficient on the L0 side is "1/2" and the weight coefficient on the L1 side is "1/2", if the distance between L0 and the inter frame is "1" as the ratio of the distance of the reference frame, the distance between L1 and the inter frame is "1", that is, the distance between the frame and L0 is equal to the distance between the frame and L1, and only in this case, the BDOF process is applied.

That is, the prediction signal generation unit 111D may include, as the application condition: the temporal distance between the two reference frames (L0/L1) and the frame is correlated in proportion to the reciprocal of the weight coefficient when the calculation is performed using the pixel values of the two reference frames (L0/L1) or the respective values calculated from the pixel values of the two reference frames (L0/L1).

The prediction signal generation unit 111D may include, as the application condition, the following condition: when the weight coefficients are equal, as a condition related to the temporal distance, the temporal distance of each of the two reference frames (L0/L1) from the frame is equal.

Note that, although the case where the weight coefficient is "1/2" is described as an example, when the same weight (for example, "1") is used on the L0 side and the L1 side, the BDOF process is applied only when the distance between the frame and L0 is equal to the distance between the frame and L1.

When it is determined that the application condition is satisfied, the processing sequence proceeds to step S83, and when the application condition is not satisfied, the processing sequence proceeds to step S84.

In step S83, the prediction signal generation unit 111D executes the BDOF process and generates a prediction signal. Since the BDOF process itself can be performed by a known method, only an outline thereof will be described, and a detailed description thereof will be omitted.

First, the prediction signal generation unit 111D calculates the gradient of the luminance value in the vertical direction and the gradient of the luminance value in the horizontal direction for each pixel in the reference block of the block. As a specific calculation method, for example, the method described in non-patent document 1 can be used. Here, the horizontal gradient of L0 is referred to as "gradientHL 0", the vertical gradient of L0 is referred to as "gradientVL 0", the horizontal gradient of L1 is referred to as "gradientHL 1", and the vertical gradient of L1 is referred to as "gradientVL 1".

Second, the prediction signal generation unit 111D calculates the corrected MV (vx, vy) of the block. As a specific calculation method, for example, the method described in non-patent document 1 can be used. Here, the sum of the gradients (tempH, tempV) and the difference in luminance value (diff) between the reference blocks of L0 and L1 are calculated as follows, and may be used to calculate the above (vx, vy) as described in non-patent document 1.

diff＝predSampleL0×weightL0－predSampleL1×weightL1

tempH＝gradientHL0×weightL0+gradientHL1×weightL1

tempV＝gradientVL0×weightL0+gradientVL1×weightL1

Here, "predSampleL 0" and "predSampleL 1" are luminance values of the reference blocks of L0 and L1, respectively, and "weight L0" and "weight L1" are weight coefficients for calculating the gradient and the luminance value difference, respectively.

Here, an example in which three parameters "diff", "tempH", and "tempV" are used as an example and a common weight is used has been described, and the same weight can be set when information on the L0 side and information on the L1 side are subjected to addition, subtraction, multiplication, and division.

Further, the application condition of the BDOF process relating to the distance from the reference frame may be set as described above in proportion to the reciprocal of the weight. In other words, the weight for addition, subtraction, multiplication, and division may be set as described above in proportion to the reciprocal of the ratio of the distance between the frame and each of the reference frames L0 and L1.

By setting in this way, when it is assumed that the gradient, the luminance value, or the like linearly changes between L0 and L1, an appropriate weight coefficient can be set for the time position of the frame. Furthermore, by setting the application conditions of the BDOF process corresponding to the weight coefficient, it is possible to ensure that an appropriate weight coefficient is always used.

Third, the prediction signal generation unit 111D calculates a luminance value correction amount "bdofOffset" for each pixel, for example, as follows, using the gradient and the correction MV obtained as described above.

bdofOffset＝vx×(weightL0×gradientHL0―weightL1×gradientHL1)+vy×(weightL0×gradientVL0―weightL1×gradientVL1)

Fourth, the prediction signal generation unit 111D can calculate a prediction value of the luminance value after correction for each pixel using "bdofOffset", for example, as described in non-patent document 1.

That is, the prediction signal generator 111D is configured to include a condition relating to the temporal distance between two reference frames (L0/L1) and the frame in the above application condition, based on a weight coefficient when performing computation using the pixel values of the two reference frames or the respective values calculated from the pixel values of the two reference frames (L0/L1) in the BDOF process.

After the BDOF process is performed, the process proceeds to step S85, and the process ends.

In step S84, when the block is a bi-directional prediction block, the prediction signal generator 111D synthesizes the prediction signals of L0 and L1 generated in step S81 to generate a final prediction signal. As for a specific synthesis method, a known method can be used, and thus, detailed description thereof is omitted. If the block is a block for performing unidirectional prediction, the prediction signal generation unit 111D directly uses the prediction signal generated in step S81 as the final prediction signal.

After generating the final prediction signal in the above-described order, the prediction signal generation unit 111D proceeds to step S85 and ends the process.

The affine motion vector calculating unit 111E is configured to calculate a motion vector for each sub-block into which the block is divided, based on the control point motion vector encoded by the motion vector encoding unit 111B, when the motion vector searching unit 111A determines that affine motion compensation is performed in the block.

Fig. 9 is a flowchart showing an example of the processing procedure of the affine motion vector calculating unit 111E.

As shown in fig. 9, in step S91, the affine motion vector calculation unit 111E calculates the values of four parameters, "dHorX", "dhorY", "dvrx", and "dvey".

Here, the affine motion vector calculating unit 111E uses the value of the control point motion vector encoded by the motion vector encoding unit 111B when calculating the parameter.

As described above, there are two or three control point motion vectors per block. In fig. 10, an example is shown when the control point motion vectors are three ("CPMV 0", "CPMV 1", "CPMV 2").

In fig. 10, the solid line indicates the boundary of the block, the broken line indicates the boundary of the sub-block into which the block is divided, the solid line arrow indicates the control point motion vector, and the broken line arrow indicates the motion vector of each sub-block.

When the control point motion vectors are two, the states of only CPMV0 and CPMV1 of fig. 10 are shown. A specific method of calculating the values of four parameters of "dHorX", "dhorY", "dvirx", and "dviry" from the values of the control point motion vectors may use a known method, and thus, detailed description thereof is omitted.

In step S92, the affine motion vector calculation unit 111E determines whether or not to apply "fallback mode" to the block.

Here, "fallback mode" is a mode in which the same motion vector is used for all sub-blocks into which the block is divided. Since a known method can be used for the judgment method of "fallback mode", detailed description thereof is omitted. When the "fallback mode" is applied to the block, the processing sequence proceeds to step S93, and when the "fallback mode" is not applied to the block, the processing sequence proceeds to step S94.

In step S93, the affine motion vector calculation unit 111E calculates the motion vector of each sub-block when "fallback mode" is used.

As described above, when it is "fallback mode", the values of the motion vectors of all sub-blocks in the same block are the same. Since a known method can be used for a specific calculation method, detailed description thereof will be omitted.

For example, in non-patent document 1, the motion vector is calculated by substituting the values of "dHorX", "dHorY", "dvirx", and "dviry" and the coordinate value of the block center (the values obtained by making the height and width of the block "1/2", respectively) when the coordinate on the upper left of the block is the origin into a predetermined model. After completion of step S93, the process proceeds to step S97, where the process ends.

In step S94, the affine motion vector calculation unit 111E calculates the motion vector of each sub-block to which "fallback mode" is not applied. Since a known method can be used for a specific calculation method, detailed description thereof will be omitted.

For example, in non-patent document 1, the values of "dHorX", "dHorY", "dvirx", and "dviry" and the coordinate value of the center of each sub-block when the upper-left coordinate of the block is the origin are substituted into a predetermined model to calculate a motion vector. After the end of step S94, the process proceeds to step S95.

In step S95, the affine motion vector calculation unit 111E determines whether or not the condition for applying the PROF processing to the block is satisfied.

The application condition may include a condition that the value of "sps _ affine _ prof _ enabled _ flag" is "1".

The application condition may include a condition that a value of "slice _ disable _ bdef _ dmvr _ PROF _ flag" indicates that PROF processing is available in the slice (for example, the value of the flag is "0").

Furthermore, the applying condition may also include a condition that "fallback mode" is not applied to the block.

When determining that all the application conditions are satisfied, the affine motion vector calculation unit 111E determines to apply the PROF processing to the block, and the process proceeds to step S96. Otherwise, it is determined not to apply the PROF processing to the block, and the process proceeds to step S97 to end the process.

The above has described an example in which the affine motion vector calculation unit 111E determines whether or not all the application conditions are satisfied, but the affine motion vector calculation unit 111E may also determine by other logically equivalent methods.

Specifically, all the application conditions described above may be reversed and defined as conditions for not applying the PROF process, and the affine motion vector calculation unit 111E may determine not to apply the PROF process to the block and advance to step S97 when any of the conditions for not applying the PROF process is satisfied, and determine to apply the PROF process to the block and advance to step S96 when any of the conditions for not applying the PROF process is not satisfied.

The condition for not applying the PROF process corresponding to the above example may be defined as follows.

The condition not applied may include a condition that a value of "sps _ affine _ prof _ enabled _ flag" is "0".

The condition not to be applied may include a condition that the value of "slice _ disable _ bdef _ dmvr _ PROF _ flag" indicates that the use of the PROF process in the slice is prohibited (for example, the value of the flag is "1").

In addition, the condition not applied may also include applying a "fallback mode" condition to the block.

In step S96, the affine motion vector calculation unit 111E calculates the value of "diffMV" used in the PROF processing. For example, the value of "diffMV" is calculated for each pixel position when the upper left coordinate of each sub-block is the origin. When the sub-blocks in the block have the same size (for example, 4 × 4 pixels), the "diffMV" of each pixel position of one sub-block is calculated, and the value can be used for the other sub-blocks.

For example, "diffMV" can be calculated as follows.

diffMV[x][y][0]＝x×(dHorX＜＜2)+y×(dVerX＜＜2)－posOffsetX

diffMV [ x ] [ y ] [1], (xhory < 2) + yx (dVerY < 2) -posoffset where x and y denote the pixel positions (x, y) with the upper-left coordinates of the respective sub-blocks as the origin, [0] denotes the x-direction component of "diffMV", and [1] denotes the y-direction component of "diffMV".

In addition, the following relationship holds.

posOffsetX＝6×dHorX+6×dVerX

posOffsetY＝6×dHorY+6×dVerY

Then, as described in non-patent document 1, the affine motion vector calculation unit 111E performs right shift and clipping processing.

After "diffMV" is calculated, the processing procedure proceeds to step S97, and the processing ends.

The above description is about the processing of one reference frame in the block. For example, when bi-prediction with two reference frames is performed, the processes of steps S91 to S97 described above are performed on L0 and L1, respectively. Therefore, the determination result of whether or not the PROF process is applied and the calculation result of "diffMV" may differ for L0 and L1, respectively.

Fig. 11 is a flowchart illustrating an example of the processing of the affine prediction signal generating unit 111F.

The affine prediction signal generating unit 111F is configured to perform the PROF process when "slice _ disable _ bdof _ dmvr _ PROF _ flag" indicates that the PROF process is usable in the slice corresponding to the slice header (when the value of the flag is "0"), and to generate the prediction signal without performing the PROF process when it indicates that the PROF process is not usable (when the value of the flag is "0").

In addition, when the slice header does not include the "slice _ disable _ bdef _ dmvr _ prof _ flag", the affine prediction signal generating unit 111F may be configured to implicitly interpret that the flag indicates that the processing can be performed using (the value of the flag is "0").

In step S111, the affine prediction signal generating unit 111F generates a prediction signal.

Specifically, the affine prediction signal generating unit 111F receives as input the motion vector for each sub-block calculated by the affine motion vector calculating unit 111E, and when the position indicated by the motion vector is a non-integer pixel position, applies a filter to interpolate a pixel at the non-integer pixel position to the pixel value of the reference frame. Here, as for a specific filter, the maximum 6-tap horizontal-vertical separable type filter disclosed in non-patent document 1 can be applied.

When the block is a bi-directional prediction block, the affine prediction signal generator 111F generates both prediction signals of L0 and L1.

In step S112, the affine prediction signal generating unit 111F determines whether or not the PROF processing is applied in the prediction direction (L0 or L1) of the block. In this determination, the determination result in step S95 described above may be used as it is.

When it is determined that the PROF processing is applied in the prediction direction of the block, the present processing sequence proceeds to step S113. On the other hand, when it is determined that the PROF processing is not applied in the prediction direction of the block, the present processing sequence proceeds to step S114.

In step S113, the affine prediction signal generating section 111F performs the PROF process. A specific process of the PROF process may use a known method, and thus, a detailed description thereof is omitted. The affine prediction signal generating unit 111F corrects the prediction signal by the PROF process.

In step S114, if the block is a bi-directional prediction block, the affine prediction signal generator 111F synthesizes the prediction signals of L0 and L1 generated in step S111 or the prediction signal obtained by correcting the prediction signal in step S113, and generates a final prediction signal. A specific synthesis method may use a known method, and thus, detailed description thereof is omitted. When the block is a block for performing unidirectional prediction, the affine prediction signal generating unit 111F directly uses the prediction signal generated in step S111 or the prediction signal obtained by correcting the prediction signal in step S113 as the final prediction signal.

After the generation of the final prediction signal is completed, the present processing sequence proceeds to step S115, and the process ends.

In the above example, for convenience, the prediction signal generation unit 111D and the affine prediction signal generation unit 111F have been described as separate processing blocks, but some of the processes may be shared or may be a single processing block.

For example, steps S81 and S111, and steps S84 and S114 may be regarded as the same processing.

In this case, if it is determined as no in step S82 of fig. 8, steps S112, S113, and S114 of fig. 10 are sequentially executed, and the flowcharts of fig. 8 and 10 can be integrated into a single processing block. At this time, the BDOF process and the PROF process are exclusively applied in block units.

(Loop Filter processing section 150)

Next, the loop filter processing unit 150 of the present embodiment will be explained. Fig. 12 is a diagram showing the loop filter processing unit 150 according to the present embodiment.

As shown in fig. 12, the loop filter processing unit 150 includes a block boundary detecting unit 151, a boundary strength determining unit 153, a filter determining unit 154, and a filter processing unit 155.

Here, the configuration denoted by "a" at the end is related to deblocking filter processing at a block boundary in the vertical direction, and the configuration denoted by "B" at the end is related to deblocking filter processing at a block boundary in the horizontal direction.

Next, a case will be described in which deblocking filtering processing is performed on a block boundary in the vertical direction, and then deblocking filtering processing is performed on a block boundary in the horizontal direction.

As described above, the deblocking filtering process may be applied to the encoding block, may be applied to the prediction block, and may also be applied to the transform block. The deblocking filter process may be applied to sub blocks obtained by dividing each block. That is, the target block and the adjacent block may be coding blocks, prediction blocks, transform blocks, or subblocks obtained by dividing them.

The sub-block definition includes the sub-blocks described as the processing units of the thinning unit 111C and the prediction signal generating unit 111D. The sub-block definition includes sub-blocks described as processing units of the affine motion vector calculation unit 111E and the affine prediction signal generation unit 111E. When applying deblocking filtering to sub-blocks, the blocks in the following description may be replaced with sub-blocks as appropriate.

Since the vertical block boundary deblocking filter process is the same as the horizontal block boundary deblocking filter process, the vertical block boundary deblocking filter process will be described below.

The block boundary detecting unit 151A is configured to detect a block boundary based on control data indicating a block size. Here, the block is a coding block (CU), a prediction block (PU), a transform block (TU). A specific detection method can be applied to a known method, and thus, a detailed description thereof is omitted.

The boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block.

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the target block and the adjacent block are intra prediction blocks.

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "2" when at least one of the target block and the adjacent block is an intra-frame prediction block (that is, when at least one of the blocks on both sides of the block boundary is an intra-frame prediction block).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not a non-zero (zero) orthogonal transform coefficient is included in the target block and the adjacent block, and whether or not the block boundary is a boundary of the transform block.

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when at least one of the target block and the adjacent block includes a non-zero orthogonal transform coefficient and the block boundary is a boundary of a transform block (that is, when at least one of the blocks on both sides of the block boundary has a non-zero transform coefficient and is a boundary of a TU).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold value (for example, 1/2 pixels).

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold value (e.g., 1/2 pixels) (i.e., when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is equal to or greater than a threshold value (e.g., 1/2 pixels)).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the reference block to which prediction of the motion vector of the target block and the adjacent block refers differs.

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the reference blocks to which the motion vectors of the target block and the adjacent blocks are predicted are different (that is, when the reference images are different in the blocks on both sides of the block boundary).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the number of motion vectors of the target block and the adjacent block is different.

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the number of motion vectors of the target block and the adjacent block is different (that is, the number of motion vectors is different in the blocks on both sides of the block boundary).

For example, as shown in fig. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "0" when none of the above conditions is satisfied.

In addition, it is likely that the larger the value of the boundary strength, the larger the block distortion generated at the block boundary.

The boundary strength determination method may be performed by a common method for the luminance signal and the color difference signal, or may be performed by using partially different conditions.

The filter determination unit 154A is configured to determine the type of filter processing (for example, deblocking filter processing) applied to the block boundary.

For example, the filter specification unit 154A may be configured to specify whether or not to apply the filtering process to the block boundary, and whether or not to apply any of the weak filtering process and the strong filtering process, based on the boundary strength of the block boundary, or the quantization parameters included in the target block and the adjacent blocks.

The filter determination unit 154A may be configured to determine not to apply the filtering process when the boundary strength of the block boundary is "0".

The filter processing unit 155A is configured to process the image before deblocking based on the determination result of the filter determination unit 154A. The processing performed on the image before deblocking is filter-less processing, weak filter processing, strong filter processing, or the like.

(image decoding apparatus 200)

Next, an image decoding device 200 according to the present embodiment will be described with reference to fig. 14. Fig. 14 is a diagram showing an example of functional blocks of the image decoding apparatus 200 according to the present embodiment.

As shown in fig. 14, the image decoding apparatus 200 includes a decoding unit 210, an inverse transform and inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, a loop filter processing unit 250, and a frame buffer 260.

The decoding unit 210 is configured to decode encoded data generated by the image encoding device 100 and decode the coefficient level value.

Here, for example, the decoding is entropy decoding in the order reverse to the entropy encoding performed by the encoding unit 140.

The decoding unit 210 may be configured to acquire the control data by decoding the encoded data.

In addition, as described above, the control data may include size data such as a coding block size, a prediction block size, a transform block size, and the like.

As described above, the control data may include header information such as a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), and a slice header.

For example, the decoding unit 21 may be configured to decode a flag for controlling the availability of the PROF process, a flag for controlling the availability of the DMVR process, the BDOF process, and the PROF process, and the like from the slice header.

The inverse transform and inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level values output from the decoding unit 210. Here, the inverse transform and inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level values before the inverse transform process.

The adder 230 is configured to generate a pre-filter decoded signal by adding the prediction signal to the prediction residual signal output from the inverse transform and inverse quantization unit 220, and to output the pre-filter decoded signal to the intra prediction unit 242 and the loop filter processing unit 250.

Here, the decoded signal before the filtering process constitutes a reference block used in the intra prediction unit 242.

The inter prediction unit 241 is configured to generate a prediction signal by inter prediction (inter prediction) as in the inter prediction unit 111.

Specifically, the inter-prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.

The intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intra prediction), as in the intra prediction unit 112.

Specifically, the intra prediction unit 242 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.

Similarly to the loop filter processing unit 150, the loop filter processing unit 250 is configured to perform filter processing on the pre-filter-processed decoded signal output from the adder 230, and to output the post-filter-processed decoded signal to the frame buffer 260.

Here, the filtering process is, for example, a deblocking filtering process that reduces distortion generated at a boundary portion of a block (an encoding block, a prediction block, a transform block, or a sub-block into which they are divided).

Like the frame buffer 160, the frame buffer 260 is configured to store a reference frame used by the inter prediction unit 241.

Here, the decoded signal after the filtering process constitutes a reference frame used in the inter prediction unit 241.

(inter prediction unit 241)

Next, the inter prediction unit 241 according to the present embodiment will be described with reference to fig. 15. Fig. 15 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to the present embodiment.

As shown in fig. 15, the inter-frame prediction unit 241 includes a motion vector decoding unit 241B, a thinning unit 241C, a prediction signal generating unit 241D, an affine motion vector calculating unit 241E, and an affine prediction signal generating unit 241F.

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector decoding unit 241B is configured to acquire a motion vector by decoding the control data received from the image encoding device 100.

The motion vector decoding unit 241B is configured to decode information on whether or not to use affine motion compensation for the block.

Here, when affine motion compensation is not used for the block, the processing by the thinning processing unit 241C and the prediction signal generating unit 241D is performed, and when affine motion compensation is used for the block, the processing by the affine motion vector calculating unit 241E and the affine prediction signal generating unit 241F is performed.

The thinning unit 241C is configured to perform thinning processing for correcting a motion vector, as in the thinning unit 111C.

The prediction signal generator 241D is configured to generate a prediction signal based on the motion vector, as in the prediction signal generator 111D.

Similarly to the affine motion vector calculating unit 111E, the affine motion vector calculating unit 241E is configured to calculate a motion vector of each sub-block using the control point motion vector decoded by the motion vector decoding unit 241B.

The affine prediction signal generator 241F may generate the prediction signal using the motion vector of each sub-block calculated by the affine motion vector calculator 241E, as with the affine prediction signal generator 111F, and the prediction signal generator 241D and the affine prediction signal generator 241F may be integrated into the same processing block, as with the prediction signal generator 111D and the affine prediction signal generator 111F.

(Loop Filter processing section 250)

Next, the loop filter processing unit 250 of the present embodiment will be explained. Fig. 16 is a diagram showing the loop filter processing unit 250 according to the present embodiment.

As shown in fig. 16, the loop filter processing unit 250 includes a block boundary detecting unit 251, a boundary strength determining unit 253, a filter determining unit 254, and a filter processing unit 255.

Here, a case is exemplified in which deblocking filter processing is performed on a block boundary in the vertical direction and then deblocking filter processing is performed on a block boundary in the horizontal direction.

The block boundary detecting unit 251A is configured to detect the boundary of a block based on control data indicating the block size, as in the block boundary detecting unit 151A.

Similarly to the boundary strength determination unit 153A, the boundary strength determination unit 253A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block. The method of determining the boundary strength of the block boundary is as described above.

The filter specification unit 254A is configured to specify the type of deblocking filter processing to be applied to the block boundary, as in the filter specification unit 154A. The method of determining the type of deblocking filter processing is as described above.

The filter processing unit 255A is configured to process the image before deblocking based on the determination result of the filter determination unit 254A, similarly to the filter processing unit 155A. The processing performed on the image before deblocking is filter-less processing, weak filter processing, strong filter processing, or the like.

(modification 1)

Next, a modified example 1 of the above embodiment will be described with reference to fig. 17 and 18. The following mainly explains differences from the above embodiments. In the above embodiment, the BDOF process and the PROF process are not described in detail, but in the present modification, a specific example of the process is described.

First, a specific example of the BDOF process of step S83 will be described with reference to fig. 17.

As shown in fig. 17, in step S171, the prediction

signal generation units

111D and 241D calculate the luminance value gradients in the horizontal direction and the vertical direction for each pixel in the reference blocks of L0 and L1 corresponding to the block or sub-block to which the BDOF process is applied.

Specifically, the prediction signal generation unit 111D/241D calculates the gradient for each pixel position (x, y) in the block using the following equation.

gradientHL0[x][y]＝(predSamplesL0[hx+1][vy]＞＞shift1)－(predSamplesL0[hx－1][vy]＞＞shift1)

gradientV0[x][y]＝(predSamplesL0[hx][vy+1]＞shift1)－(predSamplesL0[hx][vy－1]＞＞shift1)

gradientHL1[x][y]＝(predSamplesL1[hx+1][vy]＞＞shift1)－(predSamplesL1[hx－1][vy]＞＞shift1)

gradientV1[x][y]＝(predSamplesL1[hx][vy+1]＞shift1)－(predSamplesL0[hx][vy－1]＞＞shift1)

Here, "predSamplesL 0" and "predSamplesL 1" are prediction signals of L0 and L1 calculated in step S81. Further, "hx" and "vy" are defined by the following formulae.

In the following equation, "nCbW" is the width of the block, "nCbH" is the height of the block, "Clip 3(min, max, input)" is a function that clips the value of "input" at "min ≦ input ≦ max".

hx＝Clip3(1，nCbW，x)

hy＝Clip3(1，nCbH，y)

In addition, "shift 1" is defined by the following formula.

shift1＝Max(6，bitDepth－6)

Here, "bitDepth" is a bit depth at the time of processing a pixel value.

In step S172, the prediction signal generation unit 111D/241D calculates the corrected MV.

First, the prediction signal generation unit 111D/241D calculates the following three parameter values for each pixel of the block or sub-block.

diff[x][y]＝(predSamplesL0[hx][vy]＞＞shift2)－(predSamplesL1[hx][vy]＞＞shift2)

tempH[x][y]＝(gradientHL0[x][y]+gradientHL1[x][y])＞＞shift3

tempV[x][y]＝(gradientVL0[x][y]+gradientVL1[x][y])＞＞shift3

Here, "shift 2" and "shift 3" are defined by the following formulas.

shift2Max(4，bitDepth-8)

shift3＝Max(3，15-bitDepth)

Second, the prediction signal generation unit 111D/241D divides the block or sub-block into 4 × 4 pixel blocks, and calculates parameters for each of the 4 × 4 pixel blocks as described below.

[ mathematical formula 1]

sGx2＝∑_i∑_j Abs(tempH[xSb+i][ySb+j])with i，j＝-1..4

sGy2＝∑_i∑_j Abs(tempV[xSb+i][ySb+j])with i，j＝-1..4

sGxGy＝∑_i∑_j(Sign(tempV[xSb+i][ySb+j])*tempH[xSb+i][ySb+j])with i，j＝-1..4

sGxGym＝sGxGy＞＞12

sGxGy_s＝sGxGy&((1＜＜12)-1)

sGxdI＝∑_i∑_j(-Sign(tempH[xSb+i][ySb+j])*diff[xSb+i][ySb+j])with i.j＝-1..4

sGydI＝∑_i∑_j(-Sign(tempV[xSb+i][ySb+j])*diff[xSb+i][ySb+j])with i.j＝-1..4

Here, "xSb" and "yXb" are coordinates of the upper left pixel of each 4 × 4 pixel block when the upper left pixel of the block or sub-block is the origin. And, "sign (input)" is a function as follows: "1" is returned when the value of "input" is positive, "1" is returned when the value of "input" is negative, and "0" is returned when the value of "input" is 0.

As described below, the prediction signal generation unit 111D/241D calculates the corrected MV (vx, vy) using the values calculated as described above.

vx＝sGx2＞0Clip3(-mvRefineThres，mvRefineThres-1，-(sGxdI＜＜3)＞＞Floor(Log2(sGx2)))：0

vy＝sGx2＞0Clip3(－mvRefineThres，mvRefineThres－1，((sGydI＜＜3)－((vx×sGxGy_m)＜＜12+vx×sGxGy_s)＞＞1)＞＞Floor(Log2(sGx2)))：0

Here, "mvRefineThres" is defined by the following formula.

mvRefineThres＝1＜＜Max(5，bitDepth－7)

With the above configuration, the bit width of each component of the corrected MV (vx, vy) can be suppressed to within (including the sign of) bits "1 < Max (5, bitDepth-7) + 1".

In step S173, the prediction

signal generation units

111D and 241D correct the prediction signals of the respective pixels as described below.

bdofOffset＝(vx×(gradientHL0[x+1][y+1]－gradientHL1[x+1][y+1]))＞＞1+(vy×(gradientVL0[x+1][y+1]－gradientVL1[x+1][y+1]))＞＞1

pbSamples[x][y]＝Clip3(0，(2^bitDepth)－1，(predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset)＞＞shift4)

Here, "offset 4" and "shift 4" are defined by the following formulas.

offset4＝1＜＜(shift4－1)

shift4＝Max(3，15－bitDepth)

After performing all the above processes, the prediction signal generation unit 111D/241D proceeds to step S174 and ends the process.

Next, a specific example of the PROF process of step S113 will be described with reference to fig. 18.

As shown in fig. 18, in step S181, the affine prediction

signal generating units

111F and 241F calculate the horizontal and vertical luminance value gradients for each pixel in the reference blocks of L0 and L1 to which the PROF process is applied.

Specifically, the affine prediction signal generating unit 111F/241F calculates a gradient for each pixel position (x, y) in the block using the following expression.

gradientH[x][y]＝(predSamples[x+2][y]＞＞shift1)－(predSamples[x][y]＞＞shift1)

gradientV[x][y]＝(predSamples[x][y+2]＞＞shift1)－(predSamples[x][y]＞＞shift1)

Here, "predSamples" is the prediction signal of L0 or L1 generated in step S111. Here, shift1 is Max (6, bitDepth-6).

When the PROF processing is applied to both L0 and L1, the affine prediction signal generating units 111F/241F calculate the "gradientH" and the "gradientV" described above for L0 and L1, respectively. At this time, the calculation content in step S181 is completely the same as that in step S171 in terms of pixel units.

In step S182, the affine prediction signal generating unit 111F/241F calculates the corrected MV. This processing is actually performed as step S96 inside the affine motion vector calculation unit 111E/241E, but for convenience, the processing performed in step S182 will be described as follows. Actually, the affine prediction signal generating unit 111F/241F may use the value calculated in step S96 as it is.

First, as described in step S96, the affine prediction signal generating unit 111F/241F calculates the value of "diffMv".

diffMv[x][y][0]＝x×(dHorX＜＜2)+y×(dVerX＜＜2)－posOffsetX

diffMv[x][y][1]＝x×(dHorY＜＜2)+y×(dVerY＜＜2)－posOffsetY

Second, the affine prediction signal generating unit 111F/241F right-shifts the value of "diffMv" by 7 bits, respectively.

Third, as described below, the affine prediction signal generating unit 111F/241F performs a clipping process.

diffMv[x][y][i]＝Clip3(－dmvLimit，dmvLimit－1，diffMv[x][y][i])

Here, the number of the first and second electrodes,

dmvLimit＝1＜＜Max(5，bitDepth－7)。

with the above configuration, the bit width of each component of "diffMv" can be suppressed to within (including a symbol) bits of "1 < Max (5, bitDepth-7) + 1". The bit width is the same as the corrected MV calculated in step S172.

In step S183, the affine prediction signal generating unit 111F/241F corrects the prediction signal.

As described below, the affine prediction signal generation unit 111F/241F corrects the value of the prediction signal "pbSamples" for the block and the prediction direction to which the PROF process is applied.

dI＝gradientH[x][y]×diffMv[x][y][0]+gradientV[x][y]×diffMv[x][y][1]

pbSamples[x][y]＝predSamples[x+1][y+1]+(dI＞＞1)

When the PROF processing is applied in both L0 and L1, the affine prediction signal generating part 111F/241F performs the above-described processing for L0 and L1, respectively.

After the correction is completed, the affine prediction signal generating unit 111F/241F proceeds to step S184, and ends the PROF process.

After the PROF processing is completed, the affine prediction signal generating unit 111F/241F executes the prediction value synthesizing process described in step S114. The synthesis of the predicted values may be performed by the following equation, for example.

predSamples[x][y]＝Clip3(0，(1＜＜bitDepth)－1，(predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)＞＞shift2)

Here, the number of the first and second electrodes,

offset2＝1＜＜(shift2－1)

shift2＝Max(3，15－bitDepth)。

when both of the prediction signal "predSamplesL 0" on the L0 side and the prediction signal "predSamplesL 1" on the L1 side are corrected by the PROF process, the following modifications can be made by substituting the calculation of step S183 into "predSamplesL 0" and "predSamplesL 1".

dI_Bi＝(dI_L0＞＞1)+(dI_L1＞＞1)

pbSamples[x][y]＝Clips(0，(1＜＜bitDepth)－1，(predSamplesL0_orig[x][y]+offset2+predSamplesL1_orig[x][y]+dI_Bi)＞＞shift2)

Here, "dI _ L0" and "dI _ L1" are values of dI of L0 and L1, respectively, "predSamplesL 0_ orig" and "predSamplesL 1_ orig" are prediction signals before correction by the PROF process.

Since the gradient and the corrected MV values calculated in the BDOF process and the PROF process described above are clipped so as to have the same bit width, the bit widths of "bdofOffset" and "dI _ Bi" calculated therefrom are also the same.

That is, the calculation may be performed as follows: the offset amounts calculated in the BDOF processing by the prediction

signal generation units

111D and 241D are the same as the bit widths of the offset amount synthesized values calculated for the two reference frames (L0 and L1) in the PROF processing by the affine prediction

signal generation units

111F and 241F.

The values of "offset 2" and "shift 2" when synthesizing the prediction signal after the PROF processing are also the same as the values of "offset 4" and "shift 4" in the BDOF processing. Therefore, the predicted value generation processing when applying the BDOF processing and the PROF processing to both L0 and L1 is the same equation.

By configuring the BDOF process and the PROF process as described above, the following effects are obtained.

First, the gradient calculation process in step S171 and step S181 can be completely the same.

That is, when the PROF process is applied to two reference frames (L0/L1), the gradient calculation process in the BDOF process performed by the prediction signal generation units 111D/241D and the gradient calculation process in the PROF process performed by the affine prediction signal generation units 111F/241F can be made the same.

Second, the bit width of the corrected MV, which is the output of step S172 and step S182, can be made exactly the same.

That is, it is possible to perform the slicing processing so that the corrected MV amount in the BDOF processing by the prediction

signal generation units

111D and 241D and the corrected MV amount used in the PROF processing by the affine prediction

signal generation units

111F and 241F have the same bit width.

Third, the predicted value correction processing in step S173 can be made completely the same as the predicted value correction processing in step S183 and the predicted value synthesis processing in step S114.

That is, the final prediction signal generation processing by the prediction

signal generation units

111D and 241D using the offset amount can be the same as the final prediction signal generation processing by the affine prediction

signal generation units

111F and 241F using the value synthesized by the offset amount.

Since the BDOF processing and the PROF processing are exclusively applied to each block as described above, when the same processing section is implemented in hardware, a processing circuit can be commonly used in the BDOF processing and the PROF processing. This can reduce the circuit scale.

(modification 2)

Next, a modified example 2 of the above embodiment will be described with reference to fig. 19 and 20. The following mainly explains differences from the above embodiments. In the present modification, a description will be given of a block boundary detection method in the block boundary detector 151/251 and a boundary strength determination method in the boundary strength determination unit 153/253, using a different example from the above-described embodiment.

The block boundary detector 151A will be described below, but the block boundary detector 151B/251A/251B may detect a block boundary by similar processing.

Fig. 19 is a flowchart illustrating an example of the processing of the block boundary detecting unit 151A. Next, a case where processing is performed on a CU basis will be described as an example.

In step S191, the block boundary detector 151A detects a CU boundary. Since the block boundary detection unit 151A is a processing block that detects a vertical boundary, the vertical boundary on the left side of the CU is detected as a block boundary to be subjected to filter processing.

The "detection as a block boundary" corresponds to, for example, preparing a two-dimensional flag sequence (for example, "edgeFlags [ x ] [ y ]") indicating whether or not the pixel position in the screen is a boundary in advance, setting a flag value of a pixel position detected as a boundary to "1", and setting a flag value of a pixel position other than a boundary to "0".

Here, when the vertical boundary on the left side of the CU is located at a boundary of a coding unit such as a picture boundary, a slice boundary, or a tile boundary, the block boundary detecting unit 151A may not detect the boundary as a block boundary.

After detecting the CU boundaries in the above-described order, the block boundary detector 151A proceeds to step S192.

In step S192, the block boundary detection unit 151A detects a TU boundary (transform block boundary).

The block boundary detection unit 151A checks whether or not each pixel position is a TU boundary, while increasing the horizontal coordinate by 4 pixels and the vertical pixel position by 1 pixel, with the vertical boundary on the left side of the CU as a starting point. When the pixel position is a TU boundary, the block boundary detecting section 151A detects the pixel position as a block boundary.

After detecting the TU boundary in the above-described order, the block boundary detecting unit 151A proceeds to step S193.

In step S193, the block boundary detector 151A detects sub-block boundaries.

In step S193, the sub-block boundary to be detected may be, for example, a sub-block boundary of a CU subjected to inter prediction.

In this case, the sub-block boundary may include both a boundary of a sub-block (for example, 16 × 16 pixel size) in the DMVR process performed by the thinning unit 111C/241C and a boundary of a sub-block (for example, 4 × 4 pixel size) in the affine motion compensation performed by the affine motion vector calculating unit 111E/241E and the affine prediction signal generating unit 111F/241F.

In step S193, the sub-block boundary to be detected may be, for example, a sub-block boundary of the CU to which affine motion compensation or "merge _ sub" is applied.

In addition, when the PROF processing is applied in the CU, the sub-block of affine motion compensation coincides with the sub-block of the PROF processing.

Here, when the sub-block boundary of the affine motion compensation is to be detected in step S193, the block boundary detecting unit 151A may not set the sub-block boundary of the affine motion compensation and the sub-block boundary of the PROF process as the detection targets when the PROF process is applied to the CU.

That is, the block boundary detecting unit 151A may set the sub-block boundary of the affine motion compensation as the detection target only when the affine motion compensation is applied to the CU and the PROF processing is not applied to the CU. Of course, the block boundary detection unit 151A may detect the sub-block boundary of affine motion compensation and the sub-block boundary of the PROF process for the CU to which the PROF process is applied.

When the PROF processing is applied to at least one of L0 and L1, the block boundary detector 151A may regard the block as a "CU to which the PROF processing is applied" with respect to the sub-block boundary of the block subjected to bidirectional prediction. Note that, only when the PROF processing is applied to both of L0 and L1, the block boundary detecting unit 151A may regard the block as "CU to which the PROF processing is applied". The same explanation is also possible in the following description.

The block boundary detection unit 151A determines whether or not each pixel position is a sub-block boundary by increasing the horizontal coordinate by the sub-block size calculated by the following method every time, and by increasing the vertical pixel position by 1 pixel every time, using the vertical boundary on the left side of the CU as a starting point.

When the pixel position is a sub-block boundary, the block boundary detecting section 151A detects the pixel position as a block boundary. Here, the following processing may be performed: even if the position is a sub-block boundary, the block boundary detecting unit 151A does not detect it as a sub-block boundary when the PROF processing is applied to the CU.

For example, the sub-block size can be calculated as follows.

Sub-block size Max (8, (nCbW/NumSbX))

Here, "nCbW" is the width of the CU (the number of pixels in the horizontal direction), "NumSbX" is the number of subblocks included in the CU in the horizontal direction.

When the above expression is used, it is determined whether or not the sub-block boundary is at a position of every 8 pixels minimum in the horizontal direction starting from the vertical boundary on the left side of the CU.

For example, the sub-block size may be calculated as follows.

Sub-block size Max (4, (nCbW/NumSbX))

In this case, it is determined whether or not the sub-block boundary is a sub-block boundary at a position of every 4 pixels minimum in the horizontal direction, starting from the vertical boundary on the left side of the CU.

That is, the block boundary detecting unit 151A is configured to determine whether or not the target pixel position is a sub-block boundary at an interval of at least 4 pixels based on the coordinates of the left end or the upper end of the target block.

After detecting the sub-block boundaries in the above-described order, the block boundary detector 151A proceeds to step S194, and ends the process.

In the above description, the CU boundary, the TU boundary, and the sub-block boundary are not separately distinguished, but the block boundary is simply detected, but they may be separately stored.

For example, the CU boundary, the TU boundary, and the sub-block boundary may be distinguished and stored, or the TU boundary and the motion vector boundary may be distinguished. At this time, CU boundaries are detected as both TU boundaries and motion vector boundaries. And, TU boundaries are detected as TU boundaries, and sub-block boundaries are detected as motion vector boundaries. This division can be used, for example, in the boundary strength determination unit 153A described below.

The boundary strength determination unit 153A will be described below, but the same determination may be performed by the boundary

strength determination units

153B, 253A, and 253B.

The boundary strength determination unit 153A may be configured to determine the boundary strength as "0" when the target pixel position is not the block boundary.

That is, the block boundary detecting unit 151 may be configured not to detect a sub-block within the block to which the PROF process is applied as a block boundary, the boundary strength judging unit 153A may be configured to judge the boundary strength of a pixel position not detected as a block boundary to be "0", and the filter determining unit 154 may be configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is "0".

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "2" when at least one of the target block and the adjacent block is an intra-frame prediction block (that is, when at least one of the blocks on both sides of the block boundary is an intra-frame prediction block).

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when at least one of the target block and the adjacent block includes a non-zero orthogonal transform coefficient and the block boundary is a boundary of a transform block (that is, when at least one of the blocks on both sides of the block boundary has a non-zero transform coefficient and is a boundary of a TU).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold value (for example, 1/2 pixels). In this case, the boundary strength of the boundary may be determined based on whether or not the block boundary is a sub-block boundary in the block to which the PROF process is applied.

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold (for example, 1/2 pixels) (that is, when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is equal to or greater than a threshold (for example, 1/2 pixels)), and the block boundary is not a sub-block (4 × 4 pixel block) boundary of the block to which the PROF process is determined in step S95.

That is, the boundary strength determination unit 153A may be configured to determine the boundary strength for each block boundary, and determine the boundary strength to be "1" when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is not less than 1/2 pixels and the block boundary is not a sub-block boundary to which the PROF process is applied.

Here, the filter specification unit 154 may be configured to specify that the deblocking filter is not to be applied to the block boundary when the boundary strength is "0".

For example, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold (e.g., 1/2 pixels) (i.e., when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is equal to or greater than a threshold (e.g., 1/2 pixels)), and when the blocks on both sides of the block boundary are not the blocks to which the PROF process is determined to be applied in step S95.

For example, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the absolute value of the difference between the motion vectors of the target block and the adjacent blocks is equal to or greater than a threshold (e.g., 1/2 pixels) (i.e., when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is equal to or greater than a threshold (e.g., 1/2 pixels)), and when at least one of the blocks on the block boundary is not the block to which the PROF process is determined to be applied in step S95.

When the PROF processing is applied to at least one of L0 and L1 with respect to the sub-block boundary of the block subjected to bidirectional prediction, the block may be regarded as "a block to which the PROF processing is determined to be applied". That is, in this case, if the PROF processing is not applied to both of L0 and L1, the boundary strength is determined to be "1" when the difference between the motion vectors is equal to or greater than the threshold value.

That is, the boundary strength determination unit 153A may be configured to consider that the PROF process is applied to the target block when the target block is bi-directionally predicted and when the PROF process is applied to at least one of the two reference frames.

It is also possible to regard "a block to which the PROF processing is determined to be applied" only when the PROF processing is applied in both of L0 and L1. That is, in this case, when the PROF processing is applied only to L0 or L1 and the difference in motion vector is equal to or greater than the threshold, the boundary strength is determined to be "1".

As another example, the following determination may be made: for a block boundary to which the PROF process is applied, when determining the boundary strength, the difference of motion vectors is not determined or is regarded as "0". As a specific example, the determination may be made as follows.

When the block is a block subjected to unidirectional prediction and the PROF is applied to the block, the difference of the motion vectors is not determined or is regarded as "0" when the boundary strength of the sub-block boundary is determined.

When the block is a bi-directionally predicted block and the reference frame on the L0 side is different from the reference frame on the L1 side, the difference in motion vector is not determined or is regarded as "0" in the prediction direction (L0 only, L1 only, or both) to which the PROF is applied on the L0 side and the L1 side.

When the block is a bi-directionally predicted block and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, when the PROF is applied on the L0 side, the difference between the motion vectors of L0 on both sides of the sub-block boundary is not determined or the difference between the motion vectors of L0 on both sides of the sub-block boundary is regarded as "0".

Likewise, when the PROF is applied on the L1 side, the difference between the motion vectors of L1 on both sides of the sub-block boundary is not judged, or the difference between the motion vectors of L1 on both sides of the sub-block boundary is regarded as "0".

In this case, as in the related art, the difference between the motion vectors of L0 and L1 is used to determine the boundary strength. Specifically, for example, the difference between the motion vector on the L0 side of the sub-block on the left side of the sub-block boundary and the motion vector on the L1 side of the sub-block on the right side of the sub-block boundary is used to determine the boundary strength.

That is, when determining the difference between the motion vectors of the blocks on both sides of the target boundary in the same prediction direction, the boundary strength determination unit 153A may be configured not to determine the difference between the motion vectors in the prediction direction or to consider the difference between the motion vectors in the prediction direction as "0" when applying the PROF process to the prediction direction.

Alternatively, when the block is a bi-directionally predicted block and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, if the motion vectors on both sides of the sub-block boundary are each applied with the PROF, the difference between the motion vectors may not be determined or may be regarded as "0".

In this case, for example, when applying the PROF to both the L0 side and the L1 side, it is different from the above example in that the difference between the motion vectors between L0 and L1 or the difference between the motion vectors between L0 and L1 is not determined to be "0".

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the reference blocks to which the motion vectors of the target block and the adjacent blocks are predicted are different (that is, when the reference images are different in the blocks on both sides of the block boundary).

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the number of motion vectors of the target block and the adjacent block is different (that is, the number of motion vectors is different in the blocks on both sides of the block boundary).

Although the above description has been made of an example of a method of determining the boundary strength using a motion vector or a reference block, the determination using a motion vector or a reference block may be performed only for a block boundary detected as the CU boundary, the sub-block boundary, or the motion vector boundary.

In other words, the determination using the motion vector or the reference block may be skipped at a block boundary which is a TU boundary but not a sub-block boundary or a motion vector boundary. At this time, at the TU boundary, it is confirmed whether or not there is a non-zero orthogonal transform coefficient only in at least any one of the blocks on both sides of the boundary.

For example, as shown in fig. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "0" when none of the above conditions is satisfied.

That is, the affine prediction signal generating unit 111F/241F is configured to determine whether or not the PROF is appropriate for each block, and when applying the PROF processing, perform the PROF processing for each sub-block obtained by dividing the block.

The loop filter processing unit 150/250 is configured to determine whether or not deblocking filtering is appropriate for each block boundary, and to determine that deblocking filtering is not to be applied to the sub-block boundary of the block to which the PROF processing has been applied.

With the above configuration, the boundary strength of the block boundary or sub-block boundary after applying the PROF processing is "0", and the deblocking filter is not applied. Since the PROF process has an effect of improving subjective image quality, the following can be prevented by adopting such a configuration: the pixel values at the block boundaries are corrected by deblocking filtering, which adversely impairs subjective image quality.

The image encoding device 100 and the image decoding device 200 may be realized by a program that causes a computer to execute each function (each step).

In the above embodiments, the present invention has been described as being applied to the image encoding device 100 and the image decoding device 200, but the present invention is not limited to this example, and can be similarly applied to an image encoding/decoding system having each function of the image encoding device 100 and the image decoding device 200.

According to the present invention, it is possible to prevent subjective image quality from deteriorating by not applying deblocking filtering to sub-block boundaries to which PROF processing is applied.

[ description of symbols ]

10: an image processing system; 100: an image encoding device; 111. 241: an inter-frame prediction unit; 111A: a motion vector search unit; 111B: a motion vector encoding unit; 111C, 241C: a thinning section; 111D, 241D: a prediction signal generation unit; 111E, 241E: an affine motion vector calculation unit; 111F, 241F: an affine prediction signal generating unit; 112. 242: an intra prediction unit; 121: a subtractor; 122. 230: an adder; 131: a transformation and quantization unit; 132. 220, and (2) a step of: an inverse transformation and inverse quantization unit; 140: an encoding unit; 150. 250: a loop filter processing unit; 160. 260: a frame buffer; 200: an image decoding device; 210: a decoding unit; 241B: and a motion vector decoding unit.

Claims

1. An image decoding apparatus, comprising:

an affine prediction signal generating unit configured to determine whether or not a PROF (prediction reference with Optical flow) process is appropriate for each block, and when the PROF process is applied, perform the PROF process for each sub-block into which the block is divided; and

and a loop filter processing unit configured to determine whether or not deblocking filtering is appropriate for each block boundary, and to determine not to apply deblocking filtering to a sub-block boundary of the block to which the PROF processing has been applied.

2. The image decoding device according to claim 1, wherein the loop filter processing unit further includes a block boundary detecting unit configured to determine whether or not a target pixel position is the sub-block boundary at an interval of at least 4 pixels with reference to a coordinate of a left end or an upper end of a target block.

3. The image decoding apparatus according to claim 1 or 2, wherein the loop filter processing section includes:

a boundary strength determination unit configured to determine a boundary strength for each of the block boundaries, and configured to determine the boundary strength as "1" when an absolute value of a difference between motion vectors of blocks on both sides of the block boundary is 1/2 pixels or more and the boundary is not a sub-block boundary to which the PROF process is applied; and

and a filter determination unit configured to determine not to apply deblocking filtering to the block boundary when the boundary strength is "0".

4. The image decoding apparatus according to claim 1 or 2, wherein the loop filter processing section includes:

a block boundary detection unit configured not to detect a sub-block in the block to which the PROF processing is applied as the block boundary;

a boundary strength determination unit configured to determine a boundary strength of a pixel position not detected as the block boundary to be "0"; and

5. The image decoding device according to claim 3 or 4, wherein the boundary strength determination unit is configured to, when a target block is bidirectionally predicted and the PROF processing is applied to at least one of two reference frames, regard that the PROF processing is applied to the target block.

6. The image decoding device according to claim 3 or 4, wherein the boundary strength determination unit is configured to, when a target block is bidirectionally predicted and the PROF processing is applied to both reference frames, regard that the PROF processing is applied to the target block.

7. The image decoding device according to claim 1 or 2, wherein the boundary strength determination unit is configured not to determine the difference between the motion vectors in the prediction direction or to consider the difference between the motion vectors in the prediction direction as "0" when the PROF processing is applied to the prediction direction when determining the difference between the motion vectors in the same prediction direction for the blocks on both sides of the target boundary.

8. An image decoding method, characterized by comprising the steps of:

determining whether or not a PROF (prediction reference with optical flow) process is appropriate for each block, and when the PROF process is applied, performing the PROF process for each sub-block obtained by dividing the block; and

and judging whether the deblocking filtering is suitable for each block boundary, and determining not to apply the deblocking filtering to the sub-block boundary of the block subjected to the PROF processing.

9. A program used in an image decoding apparatus, the program causing a computer to execute: