JP2021078136A - Image decoding device, image decoding method, and program - Google Patents

Abstract

To prevent reduction in encoding efficiency by not performing refinement processing for a block with low correlation in a time direction.SOLUTION: An image decoding device 200 according to the present invention includes: a motion vector decoding unit 241B configured to decode a motion vector from encoded data; and a refining unit 241C configured to perform search for a motion vector by employing, as an initial value, a value of the motion vector decoded by the motion vector decoding unit 241B, and to determine the decoded motion vector as a final motion vector when a search cost at an initial search point is greater than a predetermined threshold or when the search cost at the initial search point is equal to or greater than the threshold.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image decoding device, an image decoding method and a program.

Conventionally, a technique of applying refinement processing to a block that satisfies an applicable condition consisting of only information that can be acquired on the decoding device side, that is, DMVR (Decoder-side)
A technique called motion vector refinement (decoding side motion vector refinement) is known (see, for example, Non-Patent Document 1).

Versatile Video Coding (Draft 5), JVET-N1001

However, for example, in the technique disclosed in Non-Patent Document 1, since the refinement process is always executed in the block satisfying the above-mentioned application condition, the refinement process is executed even in the block having a low correlation in the time direction. On the contrary, there is a problem that the coding efficiency may decrease.

Therefore, the present invention has been made in view of the above-mentioned problems, and image decoding can prevent a decrease in coding efficiency by not performing refinement processing on a block having a low correlation in the time direction. It is an object of the present invention to provide an apparatus, an image decoding method and a program.

The first feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. The motion vector is searched with the value as the initial value, and the decoded data is obtained when the search cost at the initial search point is larger than a predetermined threshold value or when the search cost at the initial search point is equal to or higher than the threshold value. The gist is to have a refinement unit configured to determine the motion vector as the final motion vector.

The second feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. A motion vector is searched using a value as an initial value, and the decoded motion vector is obtained when the minimum search cost in the search point is larger than a predetermined threshold or when the minimum search cost is equal to or higher than the threshold. The gist is to have a refinement unit that is configured to determine as the final motion vector.

A third feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. The motion vector is searched with the value as the initial value, and when the difference value between the search cost at the initial search point and the minimum search cost in the search point is smaller than a predetermined threshold value or the difference value is equal to or less than the threshold value. In the case of, it is a gist to include a refinement unit configured to determine the decoded motion vector as the final motion vector.

The fourth feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. A refinement unit configured to search for a motion vector with a value as an initial value, and a prediction signal generation unit configured to generate a prediction signal based on the motion vector output from the refinement unit. When the index value representing the similarity between the block on the first reference frame side and the block on the second reference frame side is larger than a predetermined threshold value, or the index value is the threshold value. If the above is the case, BDOF (Bi-)
The gist is that it is configured to determine not to execute the Directional Optical Flow) process.

A fifth feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. It includes a refinement unit configured to search for a motion vector with a value as the initial value, and the refinement unit includes the norm of the difference vector between the initial search point and the search point in the search cost. The gist is that it is composed.

The sixth feature of the present invention is a step of decoding a motion vector from encoded data, a step of searching for a motion vector using the decoded value of the motion vector as an initial value, and a search cost at an initial search point. The gist is to have a step of determining the decoded motion vector as the final motion vector when it is larger than a predetermined threshold or when the search cost at the initial search point is equal to or greater than the threshold. ..

The seventh feature of the present invention is a program used in an image decoding device, which is used in a computer.
A step of decoding a motion vector from coded data, a step of searching for a motion vector using the decoded value of the motion vector as an initial value, and a case where the search cost at the initial search point is larger than a predetermined threshold value or When the search cost at the initial search point is equal to or greater than the threshold value,
The gist is to execute the step of determining the decoded motion vector as the final motion vector.

According to the present invention, it is possible to provide an image decoding device, an image decoding method and a program capable of preventing a decrease in coding efficiency by not performing refinement processing on a block having a low correlation in the time direction. it can.

It is a figure which shows an example of the structure of the image processing system 10 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image coding apparatus 100 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the inter-prediction part 111 of the image coding apparatus 100 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the refinement part 111C of the inter-prediction part 111 of the image coding apparatus 100 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the prediction signal generation unit 111D of the inter-prediction unit 111 of the moving image decoding apparatus 30 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image decoding apparatus 200 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the inter-prediction part 241 of the image decoding apparatus 200 which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The components in the following embodiments can be replaced with existing components as appropriate, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

(First Embodiment)
Hereinafter, the image processing system 10 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 7. FIG. 1 is a diagram showing an image processing system 10 according to an embodiment according to the present embodiment.

As shown in FIG. 1, the image processing system 10 includes an image coding device 100 and an image decoding device 200.

The image coding device 100 is configured to generate coded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

Here, such coded data may be transmitted from the image coding device 100 to the image decoding device 200 via a transmission line. Further, the coded data may be stored in the storage medium and then provided from the image coding device 100 to the image decoding device 200.

(Image Encoding Device 100)
Hereinafter, the image coding apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an example of a functional block of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 2, the image coding device 100 includes an inter-prediction unit 111, an intra-prediction unit 112, a subtractor 121, an adder 122, a conversion / quantization unit 131, and an inverse transformation / dequantization. It has a unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

The inter-prediction unit 111 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 111 is a frame to be encoded (hereinafter referred to as a target frame).
Is configured to identify the reference block contained in the reference frame and determine the motion vector for the identified reference block by comparing with the reference frame stored in the frame buffer 160.

Further, the inter-prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter-prediction unit 111 is configured to output a prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. Further, the intra prediction unit 112 is configured to output a prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block that is referred to for the block to be predicted (hereinafter, the target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the conversion / quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal, which is the difference between the prediction signal generated by the intra prediction or the inter prediction and the input image signal.

The adder 122 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 132 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is combined with the intra prediction unit 112 and the input. It is configured to output to the loop filter processing unit 150.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 112.

The conversion / quantization unit 131 is configured to perform conversion processing of the predicted residual signal and acquire a coefficient level value. Further, the conversion / quantization unit 131 may be configured to quantize the coefficient level value.

Here, the conversion process is a process of converting the predicted residual signal into a frequency component signal. In such a conversion process, the Discrete Cosine Transform (DCT)
The basis pattern (transformation matrix) corresponding to the form) may be used, and the discrete sine transformation (transformation matrix) may be used.
DST; Discrete Sine Transfer) corresponding to the base pattern (
A transformation matrix) may be used.

The inverse transformation / inverse quantization unit 132 is configured to perform an inverse transformation process of the coefficient level value output from the conversion / quantization unit 131. Here, the inverse transformation / inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation processing.

Here, the inverse transformation processing and the inverse quantization are performed in the reverse procedure of the conversion processing and the quantization performed by the conversion / quantization unit 131.

The coding unit 140 is configured to encode the coefficient level value output from the conversion / quantization unit 131 and output the coded data.

Here, for example, coding is entropy coding in which codes of different lengths are assigned based on the probability of occurrence of coefficient level values.

Further, the coding unit 140 is configured to encode the control data used in the decoding process in addition to the coefficient level value.

Here, the control data includes a coding block (CU: Coding Unit) size, a prediction block (PU: Precision Unit) size, and a conversion block (TU: T).
It may include size data such as landform Unit) size.

The in-loop filter processing unit 150 is configured to perform filter processing on the pre-filter processing decoding signal output from the adder 122 and output the post-filter processing decoding signal to the frame buffer 160.

Here, for example, the filtering process is a deblocking filtering process that reduces the distortion that occurs at the boundary portion of the block (encoded block, prediction block, or conversion block).

The frame buffer 160 is configured to store reference frames used by the inter-prediction unit 111.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 111.

(Inter Prediction Unit 111)
Hereinafter, with reference to FIG. 3, the inter-prediction unit 11 of the image coding apparatus 100 according to the present embodiment
1 will be described. FIG. 3 is a diagram showing an example of a functional block of the inter-prediction unit 111 of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 3, the inter-prediction unit 111 includes a motion vector search unit 111A, a motion vector coding unit 111B, a refinement unit 111C, and a prediction signal generation unit 111D.

The inter-prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector search unit 111A is configured to identify the reference block included in the reference frame by comparing the target frame with the reference frame, and search for the motion vector for the specified reference block.

In addition, the above-mentioned search is performed on a plurality of reference frame candidates, and the reference frame and motion vector used for prediction in the prediction block are determined. A maximum of two reference frames and motion vectors can be used for one block. The case where only one set of reference frames and motion vectors is used for one block is called one-sided prediction, and the case where two sets of reference frames and motion vectors are used is called bi-prediction. Hereinafter, the first set will be referred to as L0, and the second set will be referred to as L1.

Further, the motion vector search unit 111A is configured to determine the reference frame and the motion vector coding method. The coding method includes a merge mode and the like, which will be described later, in addition to the usual method of transmitting information on the reference frame and the motion vector, respectively.

As for the motion vector search method, the reference frame determination method, and the reference frame and motion vector coding method determination method, known methods can be adopted, and details thereof will be omitted.

The motion vector coding unit 111B is configured to encode the information of the reference frame and the motion vector determined by the motion vector search unit 111A by using the coding method also determined by the motion vector search unit 111A.

When the coding method of the block is the merge mode, the image coding apparatus 100 side
First, a merge list for the block is created. Here, the merge list is a list in which a plurality of combinations of reference frames and motion vectors are listed.

An index is assigned to each combination, and the image coding apparatus 100 encodes only such an index and transmits it to the image decoding apparatus 200, instead of encoding the information of the reference frame and the motion vector individually. Image encoding device 100 side and image decoding device 200
By sharing the method of creating the merge list with the side, the image decoding apparatus 200 side can decode the information of the reference frame and the motion vector only from the information related to the index.

As for the method of creating the merge list, a known method can be adopted, and the details thereof will be omitted.

Regarding the coding of motion vector information, first, a predicted motion vector, which is a predicted value of the motion vector to be encoded, is generated, and then a differential motion, which is a difference value between the predicted motion vector and the motion vector to be actually encoded. Encode the vector.

The refinement unit 111C is configured to perform a refinement process (for example, DMVR) for modifying the motion vector encoded by the motion vector coding unit 111B.

Specifically, the refinement unit 111C sets the search range with reference to the reference position specified by the motion vector encoded by the motion vector coding unit 111B, and corrects the search range with the smallest predetermined cost. It is configured to perform refinement processing that identifies the reference position and corrects the motion vector based on the correction reference position.

FIG. 4 is a flowchart showing an example of the processing procedure of the refinement unit 111C.

As shown in FIG. 4, in step S41, the refinement unit 111C determines whether or not the predetermined conditions for applying the refinement process are satisfied. When all the predetermined conditions are satisfied, the processing procedure proceeds to step S42. On the other hand, if any one of the predetermined conditions is not satisfied, the present processing procedure proceeds to step S48 and ends the refining process.

Here, the predetermined condition includes a condition that the block is a block that performs bi-prediction.
Further, the predetermined condition may include the condition that the motion vector is encoded in the merge mode.

In step S42, the refinement unit 111C generates a search image based on the information of the motion vector and the reference frame encoded by the motion vector coding unit 111B.

Here, when the motion vector points to the non-integer pixel position, the refinement unit 111C applies a filter to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. At this time, the refinement unit 111C can reduce the amount of calculation by using the interpolation filter having a smaller number of taps than the interpolation filter used in the prediction signal generation unit 111D described later. For example, the refinement unit 111C can interpolate pixel values at non-integer pixel positions by bilinear interpolation.

In step S43, the refinement unit 111C calculates the search cost at the initial position using the search image generated in step S42.

Here, the initial position is the position pointed to by the motion vector encoded by the motion vector coding unit 111B. The search cost is an index value of the degree of similarity between the reference block on the L0 side (first reference frame side) and the reference block on the L1 side (second reference frame side) pointed to by the motion vector, for example, a pixel. Absolute value error sums and square error sums between values can be used.

In step S44, the refinement unit 111C uses the search cost at the initial position calculated in step S43 as an input to determine whether or not the end condition for terminating the refinement process of the block is satisfied. When all the termination conditions are satisfied, the processing procedure proceeds to step S45. On the other hand, if any one of the termination conditions is not satisfied, the processing procedure proceeds to step S48 to end the refining process.

Here, in the end condition (discontinuation condition) in step S44, for example, the search cost at the above-mentioned initial position is smaller than the predetermined first threshold value (or the above-mentioned search cost at the initial position is predetermined. The condition (being equal to or less than the first threshold value) can be included.
Further, the end condition (discontinuation condition) in step S44 includes, for example, that the search cost at the above-mentioned initial position is larger than the predetermined second threshold value (or the above-mentioned search cost at the initial position is a predetermined first value. The condition that it is 1 threshold value or more) can be included.

In step S45, the refinement unit 111C performs a search with integer pixel accuracy using the search image generated in step S42.

Here, the integer pixel accuracy means that only the points that are the integer pixel intervals are searched with reference to the motion vector encoded by the motion vector coding unit 111B.

The refinement unit 111C determines the corrected motion vector at the integer pixel interval position by the search in step S45. Here, a known method can be used as the search method.

For example, the refinement unit 111C can also search by a method of searching only a point where the difference motion vector on the L0 side and the L1 side is a combination in which only the sign is inverted.

Here, the search point with the lowest search cost at each search position is used as the corrected motion vector at the integer pixel interval position. As the search cost, as described above, an index such as an absolute value difference sum or a square error sum can be used. At this time, the refinement unit 111C adds the Lp norm of the difference vector between the corrected motion vector and the motion vector at the initial position to the index for evaluating the similarity between blocks such as the above-mentioned absolute value difference sum. May be used as the search cost.

Specifically, the refinement unit 111C may use, for example, the sum of the absolute value difference sum and the L1 norm of the difference vector as the search cost. Further, the refinement unit 111C may add the L1 norm to the absolute value difference sum after weighting it at a predetermined ratio. For example, the refinement unit 111C may add a value obtained by doubling the L1 norm to the absolute value difference sum. At this time, if the magnification is a power of 2, equivalent processing can be realized by bit shifting.
As a result of the search in step S45, the value may be the same as the motion vector before the search.

In step S46, the refinement unit 111C satisfies the end condition for discontinuing the refinement process in the block by using the search cost corresponding to the corrected motion vector at the integer pixel interval position determined in step S45. Judge whether or not it is. When all the termination conditions are satisfied, the processing procedure proceeds to step S47. On the other hand, if any one of the termination conditions is not satisfied, the processing procedure proceeds to step S48 to end the refining process.

Here, the end condition (discontinuation condition) in step S46 includes, for example, that the above-mentioned search cost is larger than a predetermined third threshold value (or that the above-mentioned search cost is equal to or more than a predetermined third threshold value). Can be included. At this time, the third threshold value may be set to the same value as the above-mentioned second threshold value.

Further, when the refinement unit 111C determines that the end condition is satisfied, the search result in step S45 is discarded, and the same motion vector (that is, the motion vector) as in the case where the refinement process is not performed is performed. The motion vector encoded by the coding unit 111B) may be used as the final motion vector of the block.

Further, the end condition (discontinuation condition) in step S46 includes, for example, step S4.
The difference value between the search cost at the initial position calculated in 3 and the search cost corresponding to the corrected motion vector calculated in step S45 is smaller than the predetermined fourth threshold value (or the above-mentioned difference value is in advance. It may include the condition that it is equal to or less than the specified fourth threshold value). At this time, the fourth threshold value may be set to the same value as the first threshold value.

Further, when the refinement unit 111C determines that the end condition is satisfied, the search result in step S45 is discarded, and the same motion vector as in the case where the refinement process is not performed is used in the block. It may be the final motion vector.

In step S47, the refinement unit 111C searches for a motion vector with non-integer pixel accuracy by using the motion vector after correction with integer pixel accuracy determined in step S43 as an initial value. Here, as a motion vector search method, a known method can be used.

Further, the refinement unit 111C can determine the motion vector with non-integer pixel accuracy by using the parametric model such as parabolic fitting by inputting the result of step S43 without actually performing the search.

In step S47, the refinement unit 111C determines the motion vector after correction with non-integer pixel accuracy, and then proceeds to step S48 to end the refinement process. Here, for convenience, the expression of the motion vector after the correction of the non-integer pixel accuracy is used, but the search result in step S47 may result in the same value as the motion vector with the integer pixel accuracy obtained in step S45. There is also.

In the above, for convenience, step S43 and step S45 have been described as separate steps, but both processes may be executed in the same step. For example, this processing procedure moves to step S45 immediately after step S42, and the refinement unit 111C has both the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel spacing position in step S45. Can be calculated. After that, the present processing procedure moves to step S46, and the refinement unit 111C satisfies at least one of the conditions described as step S44 and the condition described as step S46, and the end condition is satisfied. It is possible to make a judgment as to whether or not it is present.

Further, for example, in step S48, the refinement unit 111C may determine whether to discard the search result by using the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel spacing position. it can.

For example, when the search cost at the initial position is larger than the second threshold value (or when it is equal to or higher than the second threshold value), the refinement unit 111C discards the search result and does not perform the refinement process. The same motion vector as the above (that is, the motion vector encoded by the motion vector coding unit 111B) may be used as the final motion vector of the block.

Further, for example, when the search cost corresponding to the corrected motion vector at the integer pixel interval position is larger than the third threshold value (or is equal to or higher than the third threshold value), the refinement unit 111C discards the search result. Then, the same motion vector as in the case where the refinement process is not performed may be used as the final motion vector of the block.

Further, for example, when the difference value between the search cost at the initial position and the search cost corresponding to the corrected motion vector at the integer pixel interval position is smaller than the fourth threshold value (or when it is equal to or less than the fourth threshold value). The refinement unit 111C may discard the search result and use the same motion vector as in the case where the refinement process is not performed as the final motion vector of the block.

In the above, the configuration including all steps S41 to S48 has been described, but step S4
4 and step S46 do not necessarily have to be included in the configuration.

The refinement unit 111C may divide a block larger than a predetermined threshold value into smaller subblocks and execute the refinement process for each subblock. For example, the refinement unit 111C
The execution unit of the refinement process is set to 16 × 16 pixels, and when the size of the block in the horizontal direction or the vertical direction is larger than 16 pixels, it can be divided into 16 pixels or less. At this time, as the motion vector that serves as the reference for the refinement process, the motion vector of the block encoded by the motion vector coding unit 111B is used for all the sub-blocks in the same block.

When processing is performed for each sub-block, the refinement unit 111C may execute all the procedures of FIG. 4 for each sub-block. Further, the refinement unit 111C may process only a part of the process of FIG. 4 for each subblock. Specifically, the refinement unit 111C is in step S4 of FIG.
1 and S42 may be processed for each block, and steps S43 to S48 may be processed for each subblock.

The prediction signal generation unit 111D is configured to generate a prediction signal based on the modified motion vector output from the refinement unit 111C.

Here, as will be described later, the prediction signal generation unit 111D has a BDOF (Bi-Dire) for each block based on the information (for example, search cost) calculated in the process of the above-mentioned refinement processing.
It is configured to determine whether or not to perform a partial optical flow) process.

Specifically, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector coding unit 111B when the motion vector is not modified. On the other hand, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C when the motion vector is corrected.

FIG. 5 is a flowchart showing an example of the processing procedure of the prediction signal generation unit 111D. Here, when the refinement process is performed in the sub-block unit by the refinement unit 111C, the process of the prediction signal generation unit 111D is also executed in the sub-block unit. In that case, the word block in the following description can be appropriately read as a subblock.

Further, even for a block that has not been refined by the refinement unit 111C, if the block size is larger than a predetermined threshold value, the block may be divided into smaller subblocks and the refinement process may be executed for each subblock. Good. For example, as in the refinement unit 111C, when the execution unit of the prediction signal generation process is set to 16 × 16 pixels and the horizontal or vertical size of the block is larger than 16 pixels, the number of pixels is 16 or less, respectively. Can be divided as follows. In this case as well, the word block in the following description can be appropriately read as a subblock.

As shown in FIG. 5, in step S51, the prediction signal generation unit 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D takes a motion vector encoded by the motion vector coding unit 111B or a motion vector encoded by the refinement unit 111C as an input, and the position pointed to by the motion vector is a non-integer. In the case of the pixel position, a filter is applied to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. Here, as a specific filter, a horizontal / vertical separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a block that performs bi-prediction, the prediction signal generation unit 111D includes a prediction signal based on the first (hereinafter referred to as L0) reference frame and motion vector, and the second (hereinafter referred to as L1). Generates both the reference frame and the motion vector prediction signal.

In step S52, the prediction signal generation unit 111D confirms whether or not the application conditions of the BDOF processing described later are satisfied.

As such application conditions, the conditions described in Non-Patent Document 1 can be applied. The applicable condition includes at least the condition that the block is a block that performs bi-prediction. Further, as an applicable condition, as described in Non-Patent Document 1, the motion vector of the block is Symmetric.
It may include the condition that it is not encoded in MVD mode.

If the applicable conditions are not satisfied, the processing procedure proceeds to step S55 to end the processing. At this time, the prediction signal generation unit 111D outputs the prediction signal generated in step S51 as the final prediction signal.

On the other hand, when all the applicable conditions are satisfied, the present processing procedure proceeds to step S53. In step S53, this processing procedure actually applies to blocks that satisfy the applicable conditions.
It is determined whether or not to execute the BDOF process in step S54.

In the following, the case where the absolute value difference sum is used as the search cost will be described as an example, but other indexes can also be used for the search cost. For example, any index value for determining the similarity between image signals, such as the sum of absolute value differences and the sum of squared errors of signals after removing the local average value, can be used as the search cost. ..

For example, the prediction signal generation unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and when the calculated value is smaller than a predetermined threshold value (or the calculated value is in advance). If it is less than or equal to the predetermined threshold value), it is determined that the BDOF process is not performed.

Further, for example, the prediction signal generation unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and when the calculated value is larger than a predetermined threshold value (or the calculated value is larger than the predetermined threshold value). If it is equal to or higher than a predetermined threshold value), it is possible to make a determination that the BDOF process is not performed.

Here, the prediction signal generation unit 111D can also use the result of the refinement process for determining whether or not the BDOF process is applied to the block for which the refinement process is executed by the refinement unit 111C.

The prediction signal generation unit 111D uses the search cost calculated in the process of the above-mentioned refinement processing (for example, the absolute value difference sum of the pixel value of the reference block on the L0 side and the pixel value of the reference block on the L1 side). , It is also possible to determine whether or not to apply the BDOF process.

For example, in the prediction signal generation unit 111D, in the search with integer pixel accuracy in step S45, the absolute value difference sum of the search points at which the above-mentioned search cost (absolute value difference sum) is minimized is greater than a predetermined fifth threshold value. If is also small (or if it is less than or equal to the predetermined fifth threshold value), B
It is possible to determine that the DOF process is not applied. At this time, the fifth threshold value may be set to the same value as the first threshold value.

For example, in the prediction signal generation unit 111D, in the search with integer pixel accuracy in step S45, the absolute value difference sum of the search points at which the above-mentioned search cost (absolute value difference sum) is minimized is greater than a predetermined sixth threshold value. If it is also large (or if it is equal to or higher than the predetermined sixth threshold value), B
It is possible to determine that the DOF process is not applied. At this time, the sixth threshold value may be set to the same value as the second threshold value or the third threshold value.

For example, the prediction signal generation unit 111D performs BDOF processing when the search cost at the initial position calculated in step S43 is smaller than the predetermined fifth threshold value (or is equal to or less than the predetermined fifth threshold value). Can be determined not to apply. At this time, the fifth threshold value may be set to the same value as the first threshold value.

For example, the prediction signal generation unit 111D performs BDOF processing when the search cost at the initial position calculated in step S43 is larger than the predetermined sixth threshold value (or is equal to or higher than the predetermined sixth threshold value). Can be determined not to apply. At this time, the sixth threshold value may be set to the same value as the second threshold value or the third threshold value.

For example, in the prediction signal generation unit 111D, the difference value between the search cost at the initial position calculated in step S43 and the minimum search cost in the search with integer pixel accuracy in step S45 is larger than a predetermined seventh threshold value. If it is small (or if it is equal to or less than a predetermined seventh threshold value), it can be determined that the BDOF process is not applied. At this time, the seventh threshold value may be set to the same value as the first threshold value or the fourth threshold value.

Further, the prediction signal generation unit 111D may make a determination by a method based on the result of the refinement process in the block in which the refinement process is executed, and a method by the absolute value difference sum in the other blocks.

Further, the prediction signal generation unit 111D is obtained from the result of the refinement processing without newly performing the processing of calculating the absolute value difference sum of the prediction signal on the L0 side and the prediction signal on the L1 side as described above. It is also possible to use only the information to determine the suitability of the BDOF process.
In this case, the prediction signal generation unit 111D, in step S53, the prediction signal generation unit 11
1D will always determine that the BDOF process is applied to the block for which the refinement process has not been executed.

According to such a configuration, in this case, it is not necessary to perform the calculation process of the absolute value difference sum in the prediction signal generation unit 111D, so that the processing amount and the processing delay can be reduced from the viewpoint of hardware implementation.

Further, according to such a configuration, from the viewpoint of software implementation, the coding efficiency is improved by using the result of the refinement processing and preventing the BDOF processing from being executed in the block where the effect of the BDOF processing is presumed to be low. It is possible to reduce the processing time of the entire image while maintaining it.

Further, the determination process itself using the result of the above-mentioned refinement process is executed inside the refinement unit 111C, and the information indicating the result is transmitted to the prediction signal generation unit 111D, so that the prediction signal generation unit 111D , It is also possible to determine the suitability of BDOF processing.

For example, as described above, the motion vector and the value of the search cost before and after the refinement process are determined, and B
Prepare a flag that will be "1" if the condition to which the DOF processing is not applied is met, and will be "0" if the condition to which the BDOF processing is not applied is not applied or the refinement processing is not applied. , The prediction signal generation unit 111D refers to the value of such a flag and B
The suitability of DOF processing can be determined.

Further, here, for convenience, steps S52 and S53 have been described as different steps, but the determinations in steps S52 and S53 can be performed at the same time.

In the determination as described above, the present processing procedure proceeds to step S55 for the block determined not to apply the BDOF processing. For the other blocks, this processing procedure proceeds to step S54.

In step S54, the prediction signal generation unit 111D executes BDOF processing. BD
Since a known method can be used for the OF processing itself, detailed description thereof will be omitted. After the BDOF process is performed, the process moves to step S55 to end the process.

The above-mentioned first threshold value to seventh threshold value are defined so as to change their values according to the block size (product of the height and width of the block) or the number of pixels used for calculating the search cost such as SAD in the block. You may.

The above-mentioned first threshold value to the seventh threshold value may be defined so as to change their values according to the quantization parameter (QP) of the block. For example, if the cost value tends to increase when the quantization parameter is large, the threshold value can be defined to increase as the quantization parameter increases.

(Image Decoding Device 200)
Hereinafter, the image decoding apparatus 200 according to the present embodiment will be described with reference to FIG. Figure 6
Is a diagram showing an example of a functional block of the image decoding apparatus 200 according to the present embodiment.

As shown in FIG. 6, the image decoding device 200 includes a decoding unit 210 and an inverse transformation / inverse quantization unit 22.
It has 0, an adder 230, an inter-prediction unit 241, an intra-prediction unit 242, an in-loop filter processing unit 250, and a frame buffer 260.

The decoding unit 210 is configured to decode the coded data generated by the image coding device 100 and decode the coefficient level value.

Here, for example, the decoding is the entropy decoding in the reverse procedure of the entropy coding performed by the coding unit 140.

Further, the decoding unit 210 may be configured to acquire control data by decoding the encoded data.

As described above, the control data includes the coded block size, the predicted block size, and the like.
It may include size data such as conversion block size.

The inverse transformation / inverse quantization unit 220 is configured to perform an inverse transformation process of the coefficient level value output from the decoding unit 210. Here, the inverse transformation / inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation processing.

Here, the inverse transformation processing and the inverse quantization are performed in the reverse procedure of the conversion processing and the quantization performed by the conversion / quantization unit 131.

The adder 230 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 220 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is used as an intra prediction unit 2.
It is configured to output to 42 and the in-loop filter processing unit 250.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 242.

Like the inter-prediction unit 111, the inter-prediction unit 241 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 241 is configured to generate a prediction signal for each prediction block based on the motion vector decoded from the coded data and the reference signal included in the reference frame. The inter-prediction unit 241 is configured to output a prediction signal to the adder 230.

Like the intra prediction unit 112, the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 242 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. The intra prediction unit 242 is configured to output a prediction signal to the adder 230.

Similar to the in-loop filter processing unit 150, the in-loop filter processing unit 250 performs filter processing on the pre-filter processing decoding signal output from the adder 230, and also performs filter processing.
It is configured to output the filtered decoding signal to the frame buffer 260.

Here, for example, the filtering process is a deblocking filtering process that reduces the distortion that occurs at the boundary portion of a block (encoded block, prediction block, conversion block, or sub-block that divides them).

The frame buffer 260 is the inter-prediction unit 241 like the frame buffer 160.
It is configured to accumulate the reference frames used in.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 241.

(Inter prediction unit 241)
Hereinafter, the inter-prediction unit 241 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a functional block of the inter-prediction unit 241 according to the present embodiment.

As shown in FIG. 7, the inter-prediction unit 241 includes a motion vector decoding unit 241B, a refinement unit 241C, and a prediction signal generation unit 241D.

The inter-prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector decoding unit 241B is configured to acquire the motion vector by decoding the control data received from the image coding device 100.

The refinement unit 241C is configured to execute a refinement process for modifying the motion vector, similarly to the refinement unit 111C.

Like the prediction signal generation unit 111D, the prediction signal generation unit 241D is configured to generate a prediction signal based on the motion vector.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the refinement unit 1
In 11C and 241C, when the search cost is larger than the predetermined threshold value (or when the search cost is equal to or higher than the predetermined threshold value), the search result can be discarded. Here, the refinement process can improve the coding efficiency when the correlation between the pixel values of the block, the reference block on the L0 side, and the reference block on the L1 side is high.
With the above configuration, the refinement process is performed in the block having a small correlation in the time direction without transmitting additional information (information about whether or not the refinement process is applied) from the image coding apparatus 100 side. It can be prevented, and a decrease in coding efficiency can be prevented.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the refinement unit 1
In 11C and 241C, when the difference value between the search cost at the initial position and the search cost corresponding to the point after the search is smaller than the predetermined threshold value (or less than or equal to the predetermined threshold value), It is possible to take a configuration in which the search result is discarded. As a result, it is possible to avoid unnecessarily correcting the motion vector at the point where a search cost slightly smaller than the initial position is accidentally obtained due to the influence of noise or the like, and it is possible to prevent a decrease in coding efficiency. Can be done.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the refinement unit 1
In 11C and 241C, the norm of the difference vector between the initial position and the search point can be included in the search cost. In this case, if the index values for evaluating the similarity such as the sum of absolute values and errors are almost the same at each search point, the motion vector correction amount is unnecessarily increased by regularizing with the norm of the difference vector. It can be prevented from becoming large, and a decrease in coding efficiency can be prevented.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the reference block on the L0 side and the reference block on the L1 side are used to determine whether or not to execute the BDOF processing in the prediction signal generation units 111D and 241D. The condition that the degree of similarity (for example, the sum of absolute value errors) is larger than a predetermined threshold value (or the degree of similarity is equal to or more than a predetermined threshold value) can be included. Similar to the refinement process, the BDOF process is also an effective process when the correlation between the block, the reference block on the L0 side, and the reference block on the L1 side is high. Therefore, with the above configuration, BDOF processing is performed in a block having a small correlation in the time direction without transmitting additional information (information about whether or not BDOF processing is applied) from the image coding apparatus 100 side. It can be prevented, and a decrease in coding efficiency can be prevented.
In addition, the result of the above-mentioned refinement processing can be used for the above-mentioned determination. As a result, the process of calculating the absolute value difference described above can be reduced.

Further, the above-mentioned image coding device 100 and image decoding device 200 may be realized by a program that causes a computer to execute each function (each step).

In each of the above embodiments, the present invention is applied to the image coding device 100 and the image decoding device 200.
Although the application to the above has been described as an example, the present invention is not limited to such an example, and the present invention is also applied to an image coding / decoding system having each function of the image coding device 100 and the image decoding device 200. It can be applied in the same way.

According to the present invention, it is possible to prevent a decrease in coding efficiency by not performing the refining process on a block having a low correlation in the time direction.

10 ... Image processing system 100 ... Image coding device 111, 241 ... Inter prediction unit 111A ... Motion vector search unit 111B ... Motion vector coding unit 111C, 241C ... Refinement unit 111D, 241D ... Prediction signal generation unit 112, 242 ... Intra prediction unit 121 ... subtractor 122, 230 ... adder 131 ... conversion / quantization unit 132, 220 ... inverse conversion / inverse quantization unit 140 ... coding unit 150, 250 ... in-loop filter processing unit 160, 260 ... frame Buffer 200 ... Image decoding device 210 ... Decoding unit 241B ... Motion vector decoding unit

Claims (8)

It is an image decoding device
A motion vector decoding unit configured to decode a motion vector from encoded data,
A motion vector search is performed using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and the search cost at the initial search point is larger than a predetermined threshold value or the search cost at the initial search point. An image decoding device comprising a refinement unit configured to determine the decoded motion vector as a final motion vector when is greater than or equal to the threshold. It is an image decoding device
A motion vector decoding unit configured to decode a motion vector from encoded data,
A motion vector search is performed using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and the minimum search cost in the search point is larger than a predetermined threshold value or the minimum search cost is the said. An image decoding device comprising a refinement unit configured to determine the decoded motion vector as a final motion vector when the value is equal to or greater than a threshold value. It is an image decoding device
A motion vector decoding unit configured to decode a motion vector from encoded data,
The motion vector is searched using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and the difference value between the search cost at the initial search point and the minimum search cost among the search points is predetermined. It is characterized by including a refinement unit configured to determine the decoded motion vector as the final motion vector when it is smaller than the threshold value or when the difference value is equal to or less than the threshold value. Image decoding device. It is an image decoding device
A motion vector decoding unit configured to decode a motion vector from encoded data,
A refinement unit configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and a refinement unit.
It includes a prediction signal generation unit configured to generate a prediction signal based on a motion vector output from the refinement unit.
In the prediction signal generation unit, when the index value representing the similarity between the block on the first reference frame side and the block on the second reference frame side is larger than a predetermined threshold value, or when the index value is equal to or higher than the threshold value. In addition, BDOF (Bi-Signal Optical Fl)
ow) An image decoding device characterized in that it is configured to determine not to perform processing. The image decoding apparatus according to claim 4, wherein the prediction signal generation unit is configured to use the search cost calculated by the refinement unit as the index value. It is an image decoding device
A motion vector decoding unit configured to decode a motion vector from encoded data,
It is provided with a refinement unit configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value.
The refinement unit is an image decoding device characterized in that the search cost includes the norm of the difference vector between the initial search point and the search point. The process of decoding the motion vector from the coded data and
A step of searching for a motion vector using the decoded motion vector value as an initial value, and
A step of determining the decoded motion vector as the final motion vector when the search cost at the initial search point is larger than a predetermined threshold or when the search cost at the initial search point is equal to or greater than the threshold. An image decoding method characterized by having and. A program used in an image decoding device that can be used on a computer.
The process of decoding the motion vector from the coded data and
A step of searching for a motion vector using the decoded motion vector value as an initial value, and
A step of determining the decoded motion vector as the final motion vector when the search cost at the initial search point is larger than a predetermined threshold or when the search cost at the initial search point is equal to or greater than the threshold. A program characterized by executing and.

Images