JP7034363B2 - Image decoder, image decoding method and program - Google Patents

Description

The present invention relates to an image decoding device, an image decoding method and a program.

Conventionally, with respect to a technique called BODF (Bi-Directional Optical Flow), in order to shorten the execution time of software, the absolute value difference sum of pixel values between two reference images used for BDOF processing is calculated, and the absolute value difference sum is calculated. Discloses a technique for skipping BDOF processing in the block when is less than a predetermined threshold (see, for example, Non-Patent Document 1).

On the other hand, from the viewpoint of reducing the processing delay at the time of hardware mounting, a technique for deleting the BDOF skip processing by calculating the above-mentioned absolute value difference sum is also disclosed (see, for example, Non-Patent Document 2).

Versatile Video Coding (Draft 4), JVET-M1001 CE9-related: BDOF buffer reduction and enabling VPDU based application, JVET-M0890

However, for example, in the technology disclosed in Non-Patent Document 1, the execution time when the technology is implemented by software can be shortened, but the execution time when the technology is implemented by hardware increases. There was a problem.

On the other hand, the technique disclosed in Non-Patent Document 2 can shorten the execution time in hardware, but has a problem that the execution time in software increases.

Therefore, the above-mentioned conventional technique has a problem that the processing time cannot be shortened both at the time of software mounting and at the time of hardware mounting.

Therefore, the present invention has been made in view of the above-mentioned problems, and is hardware by controlling the suitability of the BDOF processing by using the information calculated in the process of the refinement processing performed prior to the BDOF processing. From the viewpoint of hardware implementation, the processing amount can be reduced by diverting the already calculated values, and from the viewpoint of software implementation, the processing time can be shortened by reducing the number of blocks to which BDOF processing is applied. It is an object of the present invention to provide a decryption method and a program.

The first feature of the present invention is an image decoding device, which comprises a motion vector decoding unit configured to decode a motion vector from encoded data, and a refinement process for correcting the decoded motion vector. It has a refinement unit configured to perform, and a prediction signal generation unit configured to generate a prediction signal based on the modified motion vector output from the refinement unit. The prediction signal generation unit determines whether or not the BDOF application condition is satisfied for each subblock that divides the block, and the BDOF for the subblock that satisfies the BDOF application condition based on the search cost in the refinement process. It is configured to determine whether or not to apply the process, and the gist is that the horizontal and vertical sizes of the subblocks are 16 pixels or less, respectively.

The second feature of the present invention is an image decoding method, which is a step A of decoding a motion vector from encoded data, a step B of performing a refinement process for correcting the decoded motion vector, and the step B. The process C includes a step C of generating a prediction signal based on the motion vector modified in the above step C. It has a step of determining whether or not to apply the BDOF process to the sub-block that satisfies the application conditions based on the search cost in the refinement process, and the horizontal and vertical sizes of the sub-block. The gist is that each has 16 pixels or less.

A third feature of the present invention is a program used in an image decoding device, in which a computer is subjected to a step A of decoding a motion vector from encoded data and a step of performing a refinement process of correcting the decoded motion vector. B and step C for generating a prediction signal based on the motion vector modified in step B are executed, and step C determines whether the BDOF application condition is satisfied for each subblock in which the block is divided. A step of determining whether or not to apply the BDOF process to the sub-block satisfying the BDOF application condition based on the search cost in the refinement process is performed, and the sub-block is horizontal. The gist is that the size in the direction and the size in the vertical direction are 16 pixels or less, respectively.

According to the present invention, it is possible to provide an image decoding device, an image decoding method and a program capable of reducing the amount of processing related to BDOF processing at both the time of hardware mounting and the time of software mounting.

It is a figure which shows an example of the structure of the image processing system 10 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image coding apparatus 100 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the interprediction unit 111 of the image coding apparatus 100 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the refinement part 111C of the inter prediction part 111 of the moving image decoding apparatus 30 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the prediction signal generation part 111D of the inter prediction part 111 of the moving image decoding apparatus 30 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the in-loop filter processing unit 150 of the image coding apparatus 100 which concerns on one Embodiment. It is a figure for demonstrating an example of the determination by the boundary strength determination unit 153 of the in-loop filter processing unit 150 of the image coding apparatus 100 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image decoding apparatus 200 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the inter prediction part 241 of the image decoding apparatus 200 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the in-loop filter processing unit 250 of the image decoding apparatus 200 which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The components in the following embodiments can be replaced with existing components as appropriate, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

(First Embodiment)
Hereinafter, the image processing system 10 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 10. FIG. 1 is a diagram showing an image processing system 10 according to an embodiment according to the present embodiment.

As shown in FIG. 1, the image processing system 10 includes an image coding device 100 and an image decoding device 200.

The image coding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding the coded data.

Here, such coded data may be transmitted from the image coding device 100 to the image decoding device 200 via a transmission path. Further, the coded data may be stored in the storage medium and then provided from the image coding device 100 to the image decoding device 200.

(Image Coding Device 100)
Hereinafter, the image coding apparatus 100 according to the present embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a functional block of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 2, the image coding apparatus 100 includes an inter-prediction unit 111, an intra-prediction unit 112, a subtractor 121, an adder 122, a conversion / quantization unit 131, and an inverse conversion / dequantization. It has a unit 132, a coding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

The inter-prediction unit 111 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 111 identifies and identifies the reference block included in the reference frame by comparing the frame to be encoded (hereinafter referred to as the target frame) with the reference frame stored in the frame buffer 160. It is configured to determine the motion vector for the reference block.

Further, the inter-prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter-prediction unit 111 is configured to output a prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. Further, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block referred to with respect to the block to be predicted (hereinafter referred to as the target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the conversion / quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal which is a difference between the prediction signal generated by the intra prediction or the inter prediction and the input image signal.

The adder 122 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 132 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is used by the intra prediction unit 112 and the input. It is configured to output to the loop filter processing unit 150.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 112.

The conversion / quantization unit 131 is configured to perform conversion processing of the predicted residual signal and acquire a coefficient level value. Further, the conversion / quantization unit 131 may be configured to quantize the coefficient level value.

Here, the conversion process is a process of converting the predicted residual signal into a frequency component signal. In such a conversion process, a base pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a base pattern (transformation matrix) corresponding to a discrete sine transform (DST). May be used.

The inverse conversion / inverse quantization unit 132 is configured to perform an inverse conversion process of the coefficient level value output from the conversion / quantization unit 131. Here, the inverse transformation / inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation process.

Here, the inverse conversion process and the inverse quantization are performed in the reverse procedure of the conversion process and the quantization performed by the conversion / quantization unit 131.

The coding unit 140 is configured to encode the coefficient level value output from the conversion / quantization unit 131 and output the coded data.

Here, for example, coding is entropy coding in which codes of different lengths are assigned based on the probability of occurrence of a coefficient level value.

Further, the coding unit 140 is configured to encode the control data used in the decoding process in addition to the coefficient level value.

Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Precision Unit) size, and a conversion block (TU: Transfer Unit) size.

The in-loop filter processing unit 150 is configured to perform filter processing on the pre-filter processing decoding signal output from the adder 122 and output the post-filter processing decoding signal to the frame buffer 160.

Here, for example, the filtering process is a deblocking filtering process that reduces the distortion generated at the boundary portion of the block (encoded block, prediction block, or conversion block).

The frame buffer 160 is configured to store reference frames used by the inter-prediction unit 111.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 111.

(Inter prediction unit 111)
Hereinafter, the inter-prediction unit 111 of the image coding apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram showing an example of a functional block of the inter-prediction unit 111 of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 3, the inter-prediction unit 111 includes a motion vector search unit 111A, a motion vector coding unit 111B, a refinement unit 111C, and a prediction signal generation unit 111D.

The inter-prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector search unit 111A is configured to identify a reference block included in the reference frame by comparing the target frame with the reference frame, and search for a motion vector for the specified reference block.

Further, the above-mentioned search process is performed on a plurality of reference frame candidates, and the reference frame and the motion vector used for prediction in the prediction block are determined. Up to two reference frames and motion vectors can be used for one block. The case where only one set of reference frame and motion vector is used for one block is called one-sided prediction, and the case where two sets of reference frame and motion vector are used is called bi-prediction. Hereinafter, the first set is referred to as L0, and the second set is referred to as L1.

Further, the motion vector search unit 111A is configured to determine the reference frame and the motion vector coding method. The coding method includes a merge mode described later, a Symmetric MVD mode described in Non-Patent Document 2, and the like, in addition to a normal method for transmitting information on a reference frame and a motion vector, respectively.

As for the method of searching the motion vector, the method of determining the reference frame, and the method of determining the method of encoding the reference frame and the motion vector, known methods can be adopted, and the details thereof will be omitted.

The motion vector coding unit 111B is configured to encode the information of the reference frame and the motion vector determined by the motion vector search unit 111A by using the coding method also determined by the motion vector search unit 111A.

When the encoding method of the block is the merge mode, first, a merge list for the block is created. A merge list is a list in which a plurality of combinations of reference frames and motion vectors are listed. An index is assigned to each combination, and instead of individually encoding the reference frame and motion vector information, only the index is encoded and transmitted to the decoding side. By sharing the method of creating the merge list between the coding side and the decoding side, the decoding side can decode the reference frame and motion vector information only from the index information. As for the method of creating the merge list, a known method can be adopted, so the details thereof will be omitted.

The Symmetric MVD mode is a coding method that can only be used when performing bi-prediction in the block. In the Symmetric MVD mode, of the two (L0, L1) reference frames and the two (L0, L1) motion vectors, which are the information to be transmitted to the decoding side, only the L0 motion vector (difference motion vector) is coded. To become. The remaining motion vector of L1 and the information of the two reference frames are uniquely determined on the coding side and the decoding side by a predetermined method.

Regarding the coding of motion vector information, first, the predicted motion vector, which is the predicted value of the motion vector to be coded, is generated, and then the differential motion vector, which is the difference value between the predicted motion vector and the motion vector to be actually encoded, is coded. To become.

In the Symmetric MVD mode, the code of the encoded difference motion vector of L0 is inverted as the difference motion vector of L1. As a specific method, for example, the method described in Non-Patent Document 2 can be used.

The refinement unit 111C is configured to perform a refinement process (for example, DMVR) for modifying the motion vector encoded by the motion vector coding unit 111B.

Specifically, the refinement unit 111C sets the search range with reference to the reference position specified by the motion vector encoded by the motion vector coding unit 111B, and corrects the search range with the smallest predetermined cost. It is configured to perform refinement processing that identifies the reference position and corrects the motion vector based on the correction reference position.

FIG. 4 is a flowchart showing an example of the processing procedure of the refinement unit 111C.

As shown in FIG. 4, in step S41, the refinement unit 111C determines whether or not the predetermined conditions for applying the refinement process are satisfied. When all the predetermined conditions are satisfied, this processing procedure proceeds to step S42. If any one of the predetermined conditions is not satisfied, the processing procedure proceeds to step S45 and ends the refining process.

Here, the predetermined condition includes the condition that the block is a block for performing bi-prediction.

Further, the predetermined condition may include the condition that the motion vector is encoded in the merge mode.

Further, the predetermined condition may include the condition that the motion vector is encoded in the Symmetric MVD mode.

Further, the predetermined condition includes a condition that the motion vector is encoded in the Symmetric MVD mode and the magnitude of the differential motion vector (MVD of L0) transmitted in the Symmetric MVD mode is within a preset threshold value. But it may be.

Here, the magnitude of the differential motion vector can be defined, for example, by the absolute value of each of the horizontal and vertical components of the differential motion vector.

As such a threshold value, different threshold values may be used for the horizontal and vertical components of the motion vector, or a common threshold value may be used for the horizontal and vertical components. It is also possible to set the threshold value to 0. In this case, it means that the predicted motion vector and the encoded motion vector have the same value. Further, the threshold value may be defined by a minimum value and a maximum value. In this case, the predetermined condition includes the condition that the value or the absolute value of the differential motion vector is equal to or more than a predetermined minimum value and not more than the maximum value.

Further, the predetermined condition may include a condition that the motion vector is encoded in the merge mode or the Symmetric MVD mode. Similarly, the predetermined condition is that when the motion vector is encoded in merge mode or Symmetric MVD mode and encoded in Symmetric MVD mode, the magnitude of the transmitted differential motion vector is a preset threshold. It may include the condition that it is within.

In step S42, the refinement unit 111C generates a search image based on the information of the motion vector and the reference frame encoded by the motion vector coding unit 111B.

Here, when the motion vector points to a non-integer pixel position, the refinement unit 111C applies a filter to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. At this time, the refinement unit 111C can reduce the amount of calculation by using the interpolation filter having a smaller number of taps than the interpolation filter used in the prediction signal generation unit 111D described later. For example, the refinement unit 111C can interpolate pixel values at non-integer pixel positions by bilinear interpolation.

In step S43, the refinement unit 111C performs a search with integer pixel accuracy using the search image generated in step S42. Here, the integer pixel accuracy means to search only the points having the integer pixel spacing with reference to the motion vector encoded by the motion vector coding unit 111B.

The refinement unit 111C determines the corrected motion vector at the integer pixel spacing position by the search in step S42. As a search method, a known method can be used. For example, the refinement unit 111C can also search by a method of searching only a point where the difference motion vector on the L0 side and the L1 side is a combination in which only the sign is inverted. Here, as a result of the search in step S43, the value may be the same as the motion vector before the search.

In step S44, the refinement unit 111C performs a motion vector search with non-integer pixel accuracy with the motion vector after correction with integer pixel accuracy determined in step S43 as an initial value. As a motion vector search method, a known method can be used.

Further, the refinement unit 111C can also determine a vector with non-integer pixel accuracy by using a parabolic fitting or the like parabolic model as an input of the result of step S43 without actually performing a search.

In step S44, the refinement unit 111C determines the motion vector after the correction with non-integer pixel accuracy, and then proceeds to step S45 to end the refinement process. Here, for convenience, the expression of the modified motion vector with non-integer pixel accuracy is used, but the search result in step S44 may result in the same value as the motion vector with integer pixel accuracy obtained in step S43. There is also.

The refinement unit 111C may divide a block larger than a predetermined threshold into smaller subblocks and execute the refinement process for each subblock. For example, the refinement unit 111C sets the execution unit of the refinement process to 16 × 16 pixels, and when the horizontal or vertical size of the block is larger than 16 pixels, it is divided into 16 pixels or less. can do. At this time, as the motion vector used as the reference for the refinement process, the motion vector of the block encoded by the motion vector coding unit 111B is used for all the sub-blocks in the same block.

When processing is performed for each sub-block, the refinement unit 111C may execute all the procedures of FIG. 4 for each sub-block. Further, the refinement unit 111C may process only a part of the process of FIG. 4 for each subblock. Specifically, the refinement unit 111C may process the steps S41 and S42 of FIG. 4 for each block, and may process only the steps S43 and S44 for each sub-block.

The prediction signal generation unit 111D is configured to generate a prediction signal based on the modified motion vector output from the refinement unit 111C.

Here, as will be described later, the prediction signal generation unit 111D determines whether or not to perform BDOF processing for each block based on the information calculated in the process of the above-mentioned refinement processing (for example, search cost). It is configured as follows.

Specifically, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector coding unit 111B when the motion vector is not modified. On the other hand, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C when the motion vector is modified.

FIG. 5 is a flowchart showing an example of the processing procedure of the prediction signal generation unit 111D. Here, when the refinement process is performed in the sub-block unit by the refinement unit 111C, the process of the prediction signal generation unit 111D is also executed in the sub-block unit. In that case, the word block in the following description can be appropriately read as a subblock.

As shown in FIG. 5, in step S51, the prediction signal generation unit 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D takes a motion vector encoded by the motion vector coding unit 111B or a motion vector encoded by the refinement unit 111C as an input, and the position pointed to by the motion vector is a non-integer. In the case of a pixel position, a filter is applied to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. Here, as a specific filter, a horizontal / vertical separable filter with a maximum of 8 taps disclosed in Non-Patent Document 3 can be applied.

When the block is a block that performs bi-prediction, it is based on the first (hereinafter referred to as L0) reference frame and motion vector prediction signal and the second (hereinafter referred to as L1) reference frame and motion vector. Generate both predictive signals.

In step S52, the prediction signal generation unit 111D confirms whether or not the application conditions of BDOF (Bi-Directional Optical Flow) described later are satisfied.

As such application conditions, the conditions described in Non-Patent Document 3 can be applied. The applicable condition includes at least the condition that the block is a block that makes a bi-prediction. Further, the applicable condition may include a condition that the motion vector of the block is not encoded in the Symmetric MVD mode as described in Non-Patent Document 1.

Further, the application condition is that the motion vector of the block is not encoded in the Symmetric MVD mode, or the magnitude of the differential motion vector to be transmitted is set in advance when it is encoded in the Symbolic MVD mode. It may include the condition that it is within the threshold value. Here, the magnitude of the differential motion vector can be determined by the same method as in step S41 described above. The threshold value can also be set to 0 in the same manner as in step S41 described above.

If the applicable conditions are not satisfied, the processing procedure proceeds to step S55 to end the processing. At this time, the prediction signal generation unit 111D outputs the prediction signal generated in step S51 as the final prediction signal.

On the other hand, when all the applicable conditions are satisfied, this processing procedure proceeds to step S53. In step S53, this processing procedure determines whether or not to actually execute the BDOF processing of step S54 for the block satisfying the applicable conditions.

For example, the prediction signal generation unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and if the value is equal to or less than a predetermined threshold value, the BDOF process is not performed. Make a judgment.

Here, the prediction signal generation unit 111D can also use the result of the refinement processing for whether or not the BDOF is applied to the block for which the refinement process is executed by the refinement unit 111C.

For example, as a result of performing the refinement process, when the difference between the motion vectors before and after the correction is equal to or less than a predetermined threshold value, the prediction signal generation unit 111D can determine that the BDOF is not applied. When such a threshold value is set to "0" for both the horizontal and vertical components, it is determined that the BDOF is not applied when the motion vector does not change as a result of the refinement process. It becomes equivalent.

The prediction signal generation unit 111D uses the search cost calculated in the process of the above-mentioned refinement process (for example, the absolute value difference sum of the pixel value of the reference block on the L0 side and the pixel value of the reference block on the L1 side). , It is also possible to determine whether or not to apply BDOF.

In the following, the case where the absolute value difference sum is used as the search cost will be described as an example, but other indexes can also be used for the search cost. For example, it may be an index value for determining the similarity between image signals, such as the absolute value difference sum of signals after removing the local mean value and the square error sum.

For example, in the integer pixel position search in step S43, the prediction signal generation unit 111D determines that the absolute value difference sum of the search points at which the search cost (absolute value difference sum) is minimized is smaller than a predetermined threshold value. , BDOF can be determined not to be applied.

Further, the prediction signal generation unit 111D can also determine the suitability of the BFOF process by combining the method using the change of the motion vector before and after the refinement process described above and the method using the search cost of the refinement process described above. ..

For example, the prediction signal generation unit 111D determines that the BDOF process is not applied when the difference between the motion vectors before and after the refinement process is equal to or less than a predetermined threshold value and the search cost of the refinement process is equal to or less than a predetermined threshold value. be able to.

Here, when the threshold value of the difference between the motion vectors before and after the refinement process is set to 0, the motion vector before the refinement process (= motion vector after the refinement process) is determined as the above-mentioned search cost. It is the sum of the absolute value differences between the reference blocks.

Further, the prediction signal generation unit 111D may determine by the method based on the result of the refinement process in the block in which the refinement process is executed, and by the method by the absolute value difference sum in the other blocks.

Further, the prediction signal generation unit 111D is obtained from the result of the refinement processing without newly performing the processing of calculating the absolute value difference sum between the prediction signal on the L0 side and the prediction signal on the L1 side, as described above. It is also possible to use only the information to determine the suitability of the BDOF. In this case, in step S53, the prediction signal generation unit 111D determines that the BDOF is always applied to the block for which the refinement process has not been executed.

According to such a configuration, in this case, it is not necessary to perform the calculation processing of the absolute value difference sum in the prediction signal generation unit 111D, so that the processing amount and the processing delay can be reduced from the viewpoint of hardware implementation.

Further, according to such a configuration, from the viewpoint of software implementation, the coding efficiency is improved by using the result of the refinement processing and preventing the BDOF processing from being executed in the block where the effect of the BDOF processing is presumed to be low. While maintaining it, the processing time for the entire image can be shortened.

Further, the determination process itself using the result of the above-mentioned refinement process is executed inside the refinement unit 111C, and the information indicating the result is transmitted to the prediction signal generation unit 111D, so that the prediction signal generation unit 111D , It is also possible to determine the suitability of BDOF processing.

For example, as described above, the motion vector and the value of the search cost before and after the refinement process are determined, and if the condition that the BDOF is not applied is satisfied, the value is "1". Is prepared, a flag that becomes "0" when is not applied, and the prediction signal generation unit 111D can determine the suitability of BDOF by referring to the value of the flag.

Further, although the steps S52 and S53 have been described as different steps here for convenience, it is also possible to simultaneously perform the determinations in the steps S52 and S53.

In the determination as described above, the present processing procedure proceeds to step S55 for the block for which it is determined that the BDOF is not applied. For the other blocks, this processing procedure proceeds to step S54.

In step S54, the prediction signal generation unit 111D executes BDOF processing. Since a known method can be used for the BDOF processing itself, a detailed description thereof will be omitted. After the BDOF process is performed, the process moves to step S55 and ends the process.

(In-loop filter processing unit 150)
Hereinafter, the in-loop filter processing unit 150 according to the present embodiment will be described. FIG. 6 is a diagram showing an in-loop filter processing unit 150 according to the present embodiment.

As shown in FIG. 6, the in-loop filter processing unit 150 includes a target block boundary detection unit 151, an adjacent block boundary detection unit 152, a boundary strength determination unit 153, a filter determination unit 154, and a filter processing unit 155. Have.

Here, the configuration with "A" at the end is the configuration related to the deblocking filter processing for the vertical block boundary, and the configuration with "B" at the end is the configuration for the horizontal block boundary. This is a configuration related to deblocking filter processing.

Hereinafter, a case where the deblocking filter processing for the horizontal block boundary is performed after the deblocking filter processing for the vertical block boundary is performed will be illustrated.

As described above, the deblocking filtering process may be applied to the coded block, the predicted block, or the transformed block. Further, it may be applied to a sub-block obtained by dividing each of the above blocks. That is, the target block and the adjacent block may be a coded block, a prediction block, a conversion block, or a subblock obtained by dividing them.

The definition of the sub-block includes the sub-block described as the processing unit of the refinement unit 111C and the prediction signal generation unit 111D. When applying a deblocking filter to a subblock, the block described below can be read as a subblock as appropriate.

Since the deblocking filter processing for the vertical block boundary and the deblocking filter processing for the horizontal block boundary are the same processing, the deblocking filter processing for the vertical block boundary will be described below.

The target block boundary detection unit 151A is configured to detect the boundary of the target block based on the control data indicating the block size of the target block.

The adjacent block boundary detection unit 152A is configured to detect the boundary of the adjacent block based on the control data indicating the block size of the adjacent block.

The boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block.

Further, the boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on the control data indicating whether or not the target block and the adjacent block are intra prediction blocks.

For example, as shown in FIG. 7, in the boundary strength determination unit 153A, when at least one block of the target block and the adjacent block is an intra prediction block (that is, at least one of the blocks on both sides of the block boundary is intra. In the case of a predicted block), it may be configured to determine that the boundary strength of the block boundary is "2".

Further, the boundary strength determination unit 153A is based on control data indicating whether or not the target block and the adjacent block include a non-zero orthogonal transformation coefficient and whether or not the block boundary is the boundary of the transformation block. It may be configured to determine the boundary strength of the block boundary.

For example, as shown in FIG. 7, in the boundary strength determination unit 153A, when at least one of the target block and the adjacent block contains a non-zero orthogonal transformation coefficient and the block boundary is the boundary of the transformation block. (Ie, if there is a non-zero conversion factor in at least one of the blocks on either side of the block boundary and it is the boundary of the TU), it is configured to determine that the boundary strength of the block boundary is "1". May be.

Further, the boundary strength determination unit 153A determines the boundary strength of the block boundary based on the control data indicating whether or not the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than the threshold value (for example, 1 pixel). It may be configured to do so.

For example, as shown in FIG. 7, the boundary strength determination unit 153A has a case where the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or more than a threshold value (for example, one pixel) (that is, blocks on both sides of the block boundary). It may be configured to determine that the boundary strength of the block boundary is "1" when the absolute value of the difference between the motion vectors of is equal to or larger than the threshold value (for example, one pixel).

Further, the boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary based on the control data indicating whether or not the reference blocks referred to in the prediction of the motion vectors of the target block and the adjacent block are different. You may.

For example, as shown in FIG. 7, when the boundary strength determination unit 153A has different reference blocks referred to in the prediction of the motion vector of the target block and the adjacent block (that is, the reference images are different in the blocks on both sides of the block boundary). ) May be configured to determine that the boundary strength of the block boundary is "1".

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on the control data indicating whether or not the numbers of motion vectors of the target block and the adjacent block are different.

For example, as shown in FIG. 7, the boundary strength determination unit 153A blocks when the number of motion vectors of the target block and the adjacent block is different (that is, when the number of motion vectors is different between the blocks on both sides of the block boundary). It may be configured to determine that the boundary strength of the boundary is "1".

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary depending on whether or not the refinement process by the refinement unit 111C is applied to the target block and the adjacent block.

For example, as shown in FIG. 7, in the boundary strength determination unit 153A, when the target block and the adjacent block are both blocks to which the refinement process is applied by the refinement unit 111C, the boundary strength of the block boundary is "1". It may be configured to determine that.

Here, the boundary strength determination unit 153A may determine that "the refinement process has been applied" when all the predetermined conditions in step S41 of FIG. 4 are satisfied in the block. Further, a flag indicating the determination result of step S41 in FIG. 4 may be prepared, and the boundary strength determination unit 153A may be configured to determine whether or not the refinement process is applied based on the value of the flag.

Further, the boundary strength determination unit 153A determines that the boundary strength of the block boundary is "1" when at least one of the target block or the adjacent block is a block to which the refinement process by the refinement unit 111C is applied. It may be configured to do so.

Alternatively, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "1" when the boundary is a sub-block boundary in the refinement process by the refinement unit 111C. good.

Further, even if the boundary strength determination unit 153A is configured to determine that the boundary strength of the block boundary is "1" when DMVR is applied to at least one of the blocks on both sides of the block boundary. good.

For example, as shown in FIG. 7, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is “0” when none of the above conditions are satisfied.

It should be noted that the larger the value of the boundary strength, the higher the possibility that the block distortion generated at the block boundary is large.

The above-mentioned boundary strength determination method may be determined by a method common to the luminance signal and the color difference signal, or may be determined by using some different conditions. For example, the above-mentioned conditions relating to the refinement process may be applied to both the luminance signal and the color difference signal, or may be applied only to the luminance signal or only the color difference signal.

Further, a flag for controlling whether or not the result of the refinement process is taken into consideration when determining the boundary strength may be provided in a header called SPS (Sequence Parameter Set) or PPS (Picture Parameter Set).

The filter determination unit 154A is configured to determine the type of filter processing (for example, deblocking filter processing) applied to the block boundary.

For example, the filter determination unit 154A determines whether or not to apply the filter processing to the block boundary based on the boundary strength of the block boundary, the quantization parameter included in the target block and the adjacent block, and the weak filter processing and the strong. It may be configured to determine which of the filtering filters to apply.

The filter determination unit 154A may be configured to determine that the filter processing is not applied when the boundary strength of the block boundary is “0”.

The filter processing unit 155A is configured to perform processing on the pre-deblocking image based on the determination of the filter determination unit 154A. The processing for the image before deblocking is no filter processing, weak filter processing, strong filter processing, or the like.

(Image Decoding Device 200)
Hereinafter, the image decoding apparatus 200 according to the present embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an example of a functional block of the image decoding apparatus 200 according to the present embodiment.

As shown in FIG. 8, the image decoding device 200 includes a decoding unit 210, an inverse conversion / inverse quantization unit 220, an adder 230, an inter-prediction unit 241 and an intra-prediction unit 242, and an in-loop filter processing unit. It has 250 and a frame buffer 260.

The decoding unit 210 is configured to decode the coded data generated by the image coding device 100 and decode the coefficient level value.

Here, for example, the decoding is the entropy decoding in the reverse procedure of the entropy coding performed by the coding unit 140.

Further, the decoding unit 210 may be configured to acquire control data by decoding processing of encoded data.

As described above, the control data may include size data such as a coded block size, a predicted block size, and a conversion block size.

The inverse conversion / inverse quantization unit 220 is configured to perform an inverse conversion process of the coefficient level value output from the decoding unit 210. Here, the inverse transformation / inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation process.

Here, the inverse conversion process and the inverse quantization are performed in the reverse procedure of the conversion process and the quantization performed by the conversion / quantization unit 131.

The adder 230 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 220 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is used by the intra prediction unit 242 and the in-loop. It is configured to output to the filter processing unit 250.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 242.

Similar to the inter-prediction unit 111, the inter-prediction unit 241 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 241 is configured to generate a prediction signal for each prediction block based on the motion vector decoded from the coded data and the reference signal included in the reference frame. The inter-prediction unit 241 is configured to output a prediction signal to the adder 230.

Like the intra prediction unit 112, the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 242 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. The intra prediction unit 242 is configured to output a prediction signal to the adder 230.

Similar to the in-loop filter processing unit 150, the in-loop filter processing unit 250 performs filter processing on the pre-filter processing decoded signal output from the adder 230, and outputs the post-filter processing decoded signal to the frame buffer 260. It is configured to do.

Here, for example, the filtering process is a deblocking filter process that reduces the distortion that occurs at the boundary portion of a block (encoded block, prediction block, conversion block, or sub-block that divides them).

Like the frame buffer 160, the frame buffer 260 is configured to store reference frames used by the inter-prediction unit 241.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 241.

(Inter prediction unit 241)
Hereinafter, the inter-prediction unit 241 according to the present embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of a functional block of the inter-prediction unit 241 according to the present embodiment.

As shown in FIG. 9, the inter-prediction unit 241 has a motion vector decoding unit 241B, a refinement unit 241C, and a prediction signal generation unit 241D.

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in the prediction block based on a motion vector.

The motion vector decoding unit 241B is configured to acquire a motion vector by decoding control data received from the image coding device 100.

The refinement unit 241C is configured to execute the refinement process for modifying the motion vector, similarly to the refinement unit 111C.

Like the prediction signal generation unit 111D, the prediction signal generation unit 241D is configured to generate a prediction signal based on the motion vector.

(In-loop filter processing unit 250)
Hereinafter, the in-loop filter processing unit 250 according to the present embodiment will be described. FIG. 10 is a diagram showing an in-loop filter processing unit 250 according to the present embodiment.

As shown in FIG. 10, the in-loop filter processing unit 250 includes a target block boundary detection unit 251, an adjacent block boundary detection unit 252, a boundary strength determination unit 253, a filter determination unit 254, and a filter processing unit 255. Have.

Here, the configuration with "A" at the end is the configuration related to the deblocking filter processing for the vertical block boundary, and the configuration with "B" at the end is the configuration for the horizontal block boundary. This is a configuration related to deblocking filter processing.

Here, an example is described in which a deblocking filter process is performed on a vertical block boundary and then a deblocking filter process is performed on a horizontal block boundary.

As described above, the deblocking filtering process may be applied to the coded block, the predicted block, or the transformed block. Further, it may be applied to the subblocks obtained by dividing each of the above blocks. That is, the target block and the adjacent block may be a coded block, a prediction block, a conversion block, or a subblock obtained by dividing them.

Since the deblocking filter processing for the vertical block boundary and the deblocking filter processing for the horizontal block boundary are the same processing, the deblocking filter processing for the vertical block boundary will be described below.

Similar to the target block boundary detection unit 151A, the target block boundary detection unit 251A is configured to detect the boundary of the target block based on the control data indicating the block size of the target block.

The adjacent block boundary detection unit 252A is configured to detect the boundary of the adjacent block based on the control data indicating the block size of the adjacent block, similarly to the adjacent block boundary detection unit 152A.

The boundary strength determination unit 253A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block, similarly to the boundary strength determination unit 153A. The method for determining the boundary strength of the block boundary is as described above.

Like the filter determination unit 154A, the filter determination unit 254A is configured to determine the type of deblocking filter processing applied to the block boundary. The method for determining the type of deblocking filter processing is as described above.

Similar to the filter processing unit 155A, the filter processing unit 255A is configured to perform processing on the pre-deblocking image based on the determination of the filter determination unit 254A. The processing for the image before deblocking is no filter processing, weak filter processing, strong filter processing, or the like.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, when the boundary strength is determined by the boundary strength determination units 153 and 253, the refinement units 111C and 241C are used in the blocks adjacent to the boundary. Consider whether the refinement process has been applied.

For example, as described above, when the refinement process is applied to at least one of the two blocks adjacent to the boundary, the boundary strength of the boundary is set to 1.

For boundaries with a boundary strength of "1" or more, the filter determination units 154 and 254 consider whether or not a deblocking filter is applied to the block boundary in consideration of parameters such as quantization parameters, and the deblocking filter. Determine the type.

With such a configuration, even if the value of the motion vector after refinement cannot be used for determining the boundary strength due to the limitation of hardware implementation, it is appropriate at the boundary of the block subjected to the refinement process. A deblocking filter can be applied, block noise can be suppressed, and subjective image quality can be improved.

Various methods for determining the suitability of the deblocking filter and determining the boundary strength have been proposed. For example, Patent Document 1 discloses a technique for omitting the application of a deblocking filter based on syntax information such as whether the block is in skip mode. Further, for example, Patent Document 2 discloses a technique of omitting the application of a deblocking filter by using a quantization parameter.

However, neither of them considers the suitability of the refinement process. One of the problems solved by the present invention is that the corrected motion vector value cannot be used for the application determination of the deblocking filter in the refinement process in which the motion vector value decoded from the syntax is corrected on the decoding side. In this case, it is a problem peculiar to the refinement process that the application judgment cannot be made appropriately.

Therefore, this problem cannot be solved by the methods of Patent Document 1 and Patent Document 2 that do not consider the suitability of the refinement process. On the other hand, it is possible to combine the methods of Patent Document 1 and Patent Document 2 with the application determination of the deblocking filter of the present invention.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the motion vector that is the reference of the processing is encoded in the Symmetric MVD mode as the execution condition of the refinement processing in the refinement units 111C and 241C. Consider whether or not it has been done.

In such a refinement process, when searching only the points where the absolute values of the difference motion vectors on the L0 side and the L1 side are the same and the signs are inverted, a motion vector similar to the case where the motion vector is transmitted in the Symmetric MVD mode is obtained. Can be done.

Therefore, when the motion vector originally transmitted is obtained by the above-mentioned refinement process, the code amount related to the differential motion vector can be reduced by making the differential motion vector transmitted in the Symmetric MVD mode as small as possible.

In particular, if the difference motion vector is a coding method that transmits only the flag when it is a specific value (0, etc.) and directly encodes the difference value when it is another value, such refinement processing is performed. Even if the difference in the motion vector corrected by is small, the effect of reducing the code amount may be large.

Further, as the execution condition of the above-mentioned refinement process, the motion vector that is the reference of the refinement process is encoded in the Symmetric MVD mode, and the value of the difference motion vector is equal to or less than a predetermined threshold value. By doing so, it is also possible to implicitly switch whether or not to execute the refinement process depending on the value of the differential motion vector.

Similarly, the motion vector of the block is encoded in the Symmetric MVD mode under the application conditions of the BDOF processing in the prediction signal generation units 111D and 241D, and the value of the differential motion vector is equal to or less than a predetermined threshold value. By using the condition, it is possible to implicitly switch whether to apply the BDOF processing depending on the value of the differential motion vector.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the result of the refinement processing is taken into consideration in determining whether or not the BDOF processing is executed in the prediction signal generation units 111D and 241D. In Non-Patent Document 3, the determination is performed using the absolute value difference value of the predicted signals of L0 and L1, but instead, the process of calculating the absolute value difference is performed by performing the determination based on the result of the refinement process. Can be reduced.

Further, the above-mentioned image coding device 100 and image decoding device 200 may be realized by a program that causes a computer to execute each function (each step).

In each of the above embodiments, the present invention has been described by taking application to the image coding device 100 and the image decoding device 200 as an example, but the present invention is not limited to this, and the coding device and the coding device and the present invention are described. It can be similarly applied to a coding / decoding system having each function of the decoding device.
According to the present invention, even when the refined motion vector cannot be used for determining the application of the deblocking filter, the deblocking filter is appropriately applied to the boundary of the refined block to suppress the block noise. It is possible to improve the subjective image quality.

10 ... Image processing system 100 ... Image coding device 111, 241 ... Inter prediction unit 111A ... Motion vector search unit 111B ... Motion vector coding unit 111C, 241C ... Refinement unit 111D, 241D ... Prediction signal generation unit 112, 242 ... Intra prediction unit 121 ... subtractor 122, 230 ... adder 131 ... conversion / quantization unit 132, 220 ... inverse conversion / inverse quantization unit 140 ... coding unit 150, 250 ... in-loop filter processing unit 151, 251 ... target Block boundary detection unit 152, 252 ... Adjacent block boundary detection unit 153, 253 ... Boundary strength determination unit 154, 254 ... Filter determination unit 155, 255 ... Filter processing unit 160, 260 ... Frame buffer 200 ... Image decoding device 210 ... Decoding unit 241B ... Motion vector decoding unit

Claims (3)

It is an image decoder
A motion vector decoder configured to decode motion vectors from coded data,
A refinement unit configured to perform a refinement process for correcting the decoded motion vector, and a refinement unit.
It has a prediction signal generation unit configured to generate a prediction signal based on the modified motion vector output from the refinement unit.
The prediction signal generation unit is
It is determined whether the BDOF application condition is satisfied for each subblock in which the block is divided, and it is determined.
It is configured to determine whether or not to apply the BDOF process to the sub-block that satisfies the BDOF application condition based on the search cost in the refinement process.
An image decoding device characterized in that the horizontal and vertical sizes of the subblocks are 16 pixels or less, respectively. Step A to decode the motion vector from the coded data,
Step B, which performs a refinement process for correcting the decoded motion vector, and
It has a step C of generating a prediction signal based on the motion vector modified in the step B.
The step C is
The process of determining whether the BDOF application condition is satisfied for each subblock obtained by dividing the block, and
It has a step of determining whether or not to apply the BDOF process to the sub-block satisfying the BDOF application condition based on the search cost in the refinement process.
An image decoding method characterized in that the horizontal and vertical sizes of the subblocks are 16 pixels or less, respectively. A program used in an image decoder, which can be used on a computer.
Step A to decode the motion vector from the coded data,
Step B, which performs a refinement process for correcting the decoded motion vector, and
A step C of generating a prediction signal based on the motion vector corrected in the step B is executed.
The step C is
The process of determining whether the BDOF application condition is satisfied for each subblock obtained by dividing the block, and
It has a step of determining whether or not to apply the BDOF process to the sub-block satisfying the BDOF application condition based on the search cost in the refinement process.
A program characterized in that the horizontal and vertical sizes of the subblocks are 16 pixels or less, respectively.

Images