JP7026276B2 - Image decoder, image decoding method and program - Google Patents

Description

The present invention relates to an image decoding device, an image decoding method and a program.

Conventionally, a technique of applying a refinement process to a block that satisfies an applicable condition consisting of only information that can be acquired on the decoder side, that is, a technique called DMVR (Decoder-side motion vector refinement). Is known (see, for example, Non-Patent Document 1).

Versatile Video Coding (Draft 5), JVET-N1001

However, for example, in the technique disclosed in Non-Patent Document 1, since the refinement process is always executed in the block satisfying the above-mentioned application condition, the refinement process is executed even in the block having a low correlation in the time direction. On the contrary, there is a problem that the coding efficiency may decrease.

Therefore, the present invention has been made in view of the above-mentioned problems, and image decoding can prevent a decrease in coding efficiency by not performing a refinement process on a block having a low correlation in the time direction. It is an object of the present invention to provide an apparatus, an image decoding method and a program.

The first feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. The motion vector is searched with the value as the initial value, and the decoded data is obtained when the search cost at the initial search point is larger than a predetermined threshold value or when the search cost at the initial search point is equal to or higher than the threshold value. The gist is to have a refinement unit configured to determine the motion vector as the final motion vector.

The second feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. A motion vector is searched with a value as an initial value, and the decoded motion vector is obtained when the minimum search cost in the search point is larger than a predetermined threshold or when the minimum search cost is equal to or higher than the threshold. The gist is to have a refinement unit configured to determine as the final motion vector.

The third feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. The motion vector is searched with the value as the initial value, and when the difference value between the search cost at the initial search point and the minimum search cost in the search point is smaller than a predetermined threshold value or the difference value is equal to or less than the threshold value. In this case, the gist is to include a refinement unit configured to determine the decoded motion vector as the final motion vector.

The fourth feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from encoded data, and a motion vector decoded by the motion vector decoding unit. A refinement unit configured to search for a motion vector with a value as an initial value, and a prediction signal generation unit configured to generate a prediction signal based on the motion vector output from the refinement unit. When the index value representing the similarity between the block on the first reference frame side and the block on the second reference frame side is larger than a predetermined threshold value, or the index value is the threshold value. In the above cases, the gist is that the configuration is such that it is determined not to execute the BDOF (Bio-Signal Optical Flow) process.

The fifth feature of the present invention is an image decoding device, which is a motion vector decoding unit configured to decode a motion vector from coded data, and a motion vector decoded by the motion vector decoding unit. It includes a refinement unit configured to search for a motion vector with a value as the initial value, and the refinement unit includes the norm of the difference vector between the initial search point and the search point in the search cost. The gist is that it is composed.

The sixth feature of the present invention is a step of decoding a motion vector from coded data, a step of searching for a motion vector using the decoded motion vector value as an initial value, and a search cost at an initial search point. The gist is to have a step of determining the decoded motion vector as the final motion vector when it is larger than a predetermined threshold value or when the search cost at the initial search point is equal to or higher than the threshold value. ..

The seventh feature of the present invention is a program used in an image decoding device, in which a computer decodes a motion vector from encoded data and searches for a motion vector using the decoded motion vector value as an initial value. And when the search cost at the initial search point is larger than a predetermined threshold value or when the search cost at the initial search point is equal to or higher than the threshold value, the decoded motion vector is finally moved. The gist is to execute the process of determining as a vector.

According to the present invention, it is possible to provide an image decoding device, an image decoding method and a program capable of preventing a decrease in coding efficiency by not performing a refinement process on a block having a low correlation in the time direction. can.

It is a figure which shows an example of the structure of the image processing system 10 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image coding apparatus 100 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the inter-prediction part 111 of the image coding apparatus 100 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the refinement part 111C of the inter-prediction part 111 of the image coding apparatus 100 which concerns on one Embodiment. It is a flowchart which shows an example of the processing procedure of the prediction signal generation unit 111D of the inter-prediction unit 111 of the moving image decoding apparatus 30 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the image decoding apparatus 200 which concerns on one Embodiment. It is a figure which shows an example of the functional block of the inter-prediction part 241 of the image decoding apparatus 200 which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The components in the following embodiments can be replaced with existing components as appropriate, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

(First Embodiment)
Hereinafter, the image processing system 10 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 7. FIG. 1 is a diagram showing an image processing system 10 according to an embodiment according to the present embodiment.

As shown in FIG. 1, the image processing system 10 includes an image coding device 100 and an image decoding device 200.

The image coding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding the coded data.

Here, such coded data may be transmitted from the image coding device 100 to the image decoding device 200 via a transmission path. Further, the coded data may be stored in the storage medium and then provided from the image coding device 100 to the image decoding device 200.

(Image Coding Device 100)
Hereinafter, the image coding apparatus 100 according to the present embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a functional block of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 2, the image coding apparatus 100 includes an inter-prediction unit 111, an intra-prediction unit 112, a subtractor 121, an adder 122, a conversion / quantization unit 131, and an inverse conversion / dequantization. It has a unit 132, a coding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

The inter-prediction unit 111 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 111 identifies and identifies the reference block included in the reference frame by comparing the frame to be encoded (hereinafter referred to as the target frame) with the reference frame stored in the frame buffer 160. It is configured to determine the motion vector for the reference block.

Further, the inter-prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter-prediction unit 111 is configured to output a prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. Further, the intra prediction unit 112 is configured to output a prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block referred to for the block to be predicted (hereinafter referred to as the target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the conversion / quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal, which is the difference between the prediction signal generated by the intra prediction or the inter prediction and the input image signal.

The adder 122 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 132 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is used by the intra prediction unit 112 and the input. It is configured to output to the loop filter processing unit 150.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 112.

The conversion / quantization unit 131 is configured to perform conversion processing of the predicted residual signal and acquire a coefficient level value. Further, the conversion / quantization unit 131 may be configured to quantize the coefficient level value.

Here, the conversion process is a process of converting the predicted residual signal into a frequency component signal. In such a conversion process, a base pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a base pattern (transformation matrix) corresponding to a discrete sine transform (DST). May be used.

The inverse conversion / inverse quantization unit 132 is configured to perform an inverse conversion process of the coefficient level value output from the conversion / quantization unit 131. Here, the inverse transformation / inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation process.

Here, the inverse conversion process and the inverse quantization are performed in the reverse procedure of the conversion process and the quantization performed by the conversion / quantization unit 131.

The coding unit 140 is configured to encode the coefficient level value output from the conversion / quantization unit 131 and output the coded data.

Here, for example, coding is entropy coding in which codes of different lengths are assigned based on the probability of occurrence of a coefficient level value.

Further, the coding unit 140 is configured to encode the control data used in the decoding process in addition to the coefficient level value.

Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Precision Unit) size, and a conversion block (TU: Transfer Unit) size.

The in-loop filter processing unit 150 is configured to perform filter processing on the pre-filter processing decoding signal output from the adder 122 and output the post-filter processing decoding signal to the frame buffer 160.

Here, for example, the filtering process is a deblocking filtering process that reduces the distortion generated at the boundary portion of the block (encoded block, prediction block, or conversion block).

The frame buffer 160 is configured to store reference frames used by the inter-prediction unit 111.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 111.

(Inter prediction unit 111)
Hereinafter, the inter-prediction unit 111 of the image coding apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram showing an example of a functional block of the inter-prediction unit 111 of the image coding apparatus 100 according to the present embodiment.

As shown in FIG. 3, the inter-prediction unit 111 includes a motion vector search unit 111A, a motion vector coding unit 111B, a refinement unit 111C, and a prediction signal generation unit 111D.

The inter-prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector search unit 111A is configured to identify a reference block included in the reference frame by comparing the target frame with the reference frame, and search for a motion vector for the specified reference block.

In addition, the above-mentioned search is performed on a plurality of reference frame candidates, and the reference frame and motion vector used for prediction in the prediction block are determined. Up to two reference frames and motion vectors can be used for one block. The case where only one set of reference frames and motion vectors is used for one block is called one-sided prediction, and the case where two sets of reference frames and motion vectors are used is called bi-prediction. Hereinafter, the first set is referred to as L0, and the second set is referred to as L1.

Further, the motion vector search unit 111A is configured to determine the reference frame and the motion vector coding method. The coding method includes a merge mode and the like, which will be described later, in addition to a normal method of transmitting information on a reference frame and a motion vector, respectively.

As for the method of searching the motion vector, the method of determining the reference frame, and the method of determining the method of encoding the reference frame and the motion vector, known methods can be adopted, and the details thereof will be omitted.

The motion vector coding unit 111B is configured to encode the information of the reference frame and the motion vector determined by the motion vector search unit 111A by using the coding method also determined by the motion vector search unit 111A.

When the coding method of the block is the merge mode, the image coding apparatus 100 first creates a merge list for the block. Here, the merge list is a list in which a plurality of combinations of reference frames and motion vectors are listed.

An index is assigned to each combination, and the image coding apparatus 100 encodes only such an index and transmits it to the image decoding apparatus 200 instead of individually encoding the information of the reference frame and the motion vector. By sharing the method of creating the merge list between the image coding device 100 side and the image decoding device 200 side, the image decoding device 200 side decodes the reference frame and motion vector information only from the information related to the index. can do.

As for the method of creating the merge list, a known method can be adopted, so the details thereof will be omitted.

Regarding the coding of motion vector information, first, a predicted motion vector, which is a predicted value of the motion vector to be encoded, is generated, and then a differential motion, which is a difference value between the predicted motion vector and the motion vector actually desired to be encoded. Encode the vector.

The refinement unit 111C is configured to perform a refinement process (for example, DMVR) for modifying the motion vector encoded by the motion vector coding unit 111B.

Specifically, the refinement unit 111C sets the search range with reference to the reference position specified by the motion vector encoded by the motion vector coding unit 111B, and corrects the search range with the smallest predetermined cost. It is configured to perform refinement processing that identifies the reference position and corrects the motion vector based on the correction reference position.

FIG. 4 is a flowchart showing an example of the processing procedure of the refinement unit 111C.

As shown in FIG. 4, in step S41, the refinement unit 111C determines whether or not the predetermined conditions for applying the refinement process are satisfied. When all the predetermined conditions are satisfied, the processing procedure proceeds to step S42. On the other hand, if any one of the predetermined conditions is not satisfied, the process proceeds to step S48 and the refinement process is terminated.

Here, the predetermined condition includes the condition that the block is a block for performing bi-prediction. Further, the predetermined condition may include the condition that the motion vector is encoded in the merge mode.

In step S42, the refinement unit 111C generates a search image based on the information of the motion vector and the reference frame encoded by the motion vector coding unit 111B.

Here, when the motion vector points to a non-integer pixel position, the refinement unit 111C applies a filter to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. At this time, the refinement unit 111C can reduce the amount of calculation by using the interpolation filter having a smaller number of taps than the interpolation filter used in the prediction signal generation unit 111D described later. For example, the refinement unit 111C can interpolate pixel values at non-integer pixel positions by bilinear interpolation.

In step S43, the refinement unit 111C calculates the search cost at the initial position using the search image generated in step S42.

Here, the initial position is the position pointed to by the motion vector encoded by the motion vector coding unit 111B. Further, the search cost is an index value of the degree of similarity between the reference block on the L0 side (first reference frame side) and the reference block on the L1 side (second reference frame side) pointed to by the motion vector, and is, for example, a pixel. Absolute value error sums and square error sums between values can be used.

In step S44, the refinement unit 111C uses the search cost at the initial position calculated in step S43 as an input to determine whether or not the end condition for terminating the refinement process of the block is satisfied. When all the termination conditions are satisfied, the processing procedure proceeds to step S45. On the other hand, if any one of the termination conditions is not satisfied, the processing procedure proceeds to step S48 and the refinement processing is terminated.

Here, in the end condition (discontinuation condition) in step S44, for example, the search cost at the above-mentioned initial position is smaller than the predetermined first threshold value (or the above-mentioned search cost at the initial position is predetermined. The condition that it is equal to or less than the first threshold value) can be included.
Further, in the end condition (discontinuation condition) in step S44, for example, the search cost at the above-mentioned initial position is larger than the predetermined second threshold value (or the above-mentioned search cost at the initial position is predetermined. It is possible to include the condition that it is 1 threshold value or more).

In step S45, the refinement unit 111C performs a search with integer pixel accuracy using the search image generated in step S42.

Here, the integer pixel accuracy means that only the points having the integer pixel spacing are searched with reference to the motion vector encoded by the motion vector coding unit 111B.

The refinement unit 111C determines the corrected motion vector at the integer pixel spacing position by the search in step S45. Here, a known method can be used as the search method.

For example, the refinement unit 111C can also search by a method of searching only a point where the difference motion vector on the L0 side and the L1 side is a combination in which only the sign is inverted.

Here, the search point with the smallest search cost at each search position is used as a motion vector after correction at the integer pixel spacing position. As the search cost, as described above, an index such as an absolute value difference sum or a square error sum can be used. At this time, the refinement unit 111C adds the Lp norm of the difference vector between the corrected motion vector and the motion vector at the initial position to the index for evaluating the similarity between blocks such as the above-mentioned absolute value difference sum. May be used as the search cost.

Specifically, the refinement unit 111C may use, for example, the sum of the absolute value difference sum and the L1 norm of the difference vector as the search cost. Further, the refinement unit 111C may add weights at a predetermined ratio when adding the L1 norm to the absolute value difference sum. For example, the refinement unit 111C may add a value obtained by doubling the L1 norm to the absolute value difference sum. At this time, if the magnification is a power of 2, equivalent processing can be realized by bit shift.
As a result of the search in step S45, the value may be the same as the motion vector before the search.

In step S46, the refinement unit 111C satisfies the end condition for terminating the refinement process in the block by using the search cost corresponding to the corrected motion vector at the integer pixel spacing position determined in step S45. Judge whether or not it is. When all the termination conditions are satisfied, the processing procedure proceeds to step S47. On the other hand, if any one of the termination conditions is not satisfied, the processing procedure proceeds to step S48 and the refinement processing is terminated.

Here, the end condition (discontinuation condition) in step S46 is, for example, that the above-mentioned search cost is larger than a predetermined third threshold value (or that the above-mentioned search cost is equal to or higher than a predetermined third threshold value). Can be included. At this time, the third threshold value may be set to the same value as the above-mentioned second threshold value.

Further, when the refinement unit 111C determines that the end condition is satisfied, the search result in step S45 is discarded, and the same motion vector (that is, the motion vector) as in the case where the refinement process is not performed is performed. The motion vector encoded by the coding unit 111B) may be used as the final motion vector of the block.

Further, in the end condition (discontinuation condition) in step S46, for example, a difference value between the search cost at the initial position calculated in step S43 and the search cost corresponding to the corrected motion vector calculated in step S45 is previously set. It may include the condition that it is smaller than the predetermined fourth threshold value (or the above-mentioned difference value is equal to or less than the predetermined fourth threshold value). At this time, the fourth threshold value may be set to the same value as the first threshold value.

Further, when the refinement unit 111C determines that the end condition is satisfied, the search result in step S45 is discarded, and the same motion vector as in the case where the refinement process is not performed is used in the block. It may be the final motion vector.

In step S47, the refinement unit 111C searches for a motion vector with non-integer pixel accuracy by using the motion vector after correction with integer pixel accuracy determined in step S43 as an initial value. Here, as a motion vector search method, a known method can be used.

Further, the refinement unit 111C can also determine a motion vector with non-integer pixel accuracy by using a parametric model such as parabolic fitting with the result of step S43 as an input without actually performing a search.

In step S47, the refinement unit 111C determines the motion vector after the correction with non-integer pixel accuracy, and then proceeds to step S48 to end the refinement process. Here, for convenience, the expression of the modified motion vector with non-integer pixel accuracy is used, but the search result in step S47 may result in the same value as the motion vector with integer pixel accuracy obtained in step S45. There is also.

In the above, for convenience, step S43 and step S45 have been described as separate steps, but both processes may be executed in the same step. For example, this processing procedure moves to step S45 immediately after step S42, and the refinement unit 111C has both a search cost at the initial position and a search cost corresponding to the corrected motion vector at the pixel spacing position in step S45. Can be calculated. After that, the present processing procedure moves to step S46, and the refinement unit 111C considers at least one of the conditions described as step S44 and the conditions described as step S46, and the termination condition is satisfied. It is possible to make a judgment as to whether or not it is present.

Further, for example, in step S48, the refinement unit 111C may determine whether to discard the search result by using the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel spacing position. can.

For example, when the search cost at the initial position is larger than the second threshold value (or when it is equal to or higher than the second threshold value), the refinement unit 111C discards the search result and does not perform the refinement process. The same motion vector as above (that is, the motion vector encoded by the motion vector coding unit 111B) may be used as the final motion vector of the block.

Further, for example, when the search cost corresponding to the corrected motion vector at the integer pixel spacing position is larger than the third threshold value (or is equal to or higher than the third threshold value), the refinement unit 111C discards the search result. Then, the same motion vector as in the case where the refinement process is not performed may be used as the final motion vector of the block.

Further, for example, when the difference value between the search cost at the initial position and the search cost corresponding to the corrected motion vector at the integer pixel spacing position is smaller than the fourth threshold value (or when it is equal to or less than the fourth threshold value). The refinement unit 111C may discard the search result and use the same motion vector as in the case where the refinement process is not performed as the final motion vector of the block.

In the above, the configuration including all of steps S41 to S48 has been described, but steps S44 and S46 do not necessarily have to be included in the configuration.

The refinement unit 111C may divide a block larger than a predetermined threshold into smaller subblocks and execute the refinement process for each subblock. For example, the refinement unit 111C sets the execution unit of the refinement process to 16 × 16 pixels, and when the horizontal or vertical size of the block is larger than 16 pixels, it is divided into 16 pixels or less. can do. At this time, as the motion vector used as the reference for the refinement process, the motion vector of the block encoded by the motion vector coding unit 111B is used for all the sub-blocks in the same block.

When processing is performed for each sub-block, the refinement unit 111C may execute all the procedures of FIG. 4 for each sub-block. Further, the refinement unit 111C may process only a part of the process of FIG. 4 for each subblock. Specifically, the refinement unit 111C may process steps S41 and S42 in FIG. 4 for each block, and steps S43 to S48 for each sub-block.

The prediction signal generation unit 111D is configured to generate a prediction signal based on the modified motion vector output from the refinement unit 111C.

Here, as will be described later, the prediction signal generation unit 111D performs BDOF (Bi-Directional Optical Flow) processing for each block based on the information calculated in the process of the above-mentioned refinement processing (for example, search cost). It is configured to determine whether or not to do so.

Specifically, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector coding unit 111B when the motion vector is not modified. On the other hand, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C when the motion vector is modified.

FIG. 5 is a flowchart showing an example of the processing procedure of the prediction signal generation unit 111D. Here, when the refinement process is performed in the sub-block unit by the refinement unit 111C, the process of the prediction signal generation unit 111D is also executed in the sub-block unit. In that case, the word block in the following description can be appropriately read as a subblock.

Further, even if the block that has not been refined by the refinement unit 111C is divided into smaller subblocks if the block size is larger than the predetermined threshold value, the refinement process may be executed for each subblock. good. For example, as in the refinement unit 111C, when the execution unit of the prediction signal generation process is set to 16 × 16 pixels and the horizontal or vertical size of the block is larger than 16 pixels, the size is 16 pixels or less, respectively. Can be divided as follows. In this case as well, the word block in the following description can be appropriately read as a subblock.

As shown in FIG. 5, in step S51, the prediction signal generation unit 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D takes a motion vector encoded by the motion vector coding unit 111B or a motion vector encoded by the refinement unit 111C as an input, and the position pointed to by the motion vector is a non-integer. In the case of a pixel position, a filter is applied to the pixel value of the reference frame to interpolate the pixel at the non-integer pixel position. Here, as a specific filter, a horizontal / vertical separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a block that performs bi-prediction, the prediction signal generation unit 111D includes a prediction signal based on the first (hereinafter referred to as L0) reference frame and motion vector, and the second (hereinafter referred to as L1). Generates both the reference frame and the motion vector prediction signal.

In step S52, the prediction signal generation unit 111D confirms whether or not the application conditions of the BDOF processing described later are satisfied.

As such application conditions, the conditions described in Non-Patent Document 1 can be applied. The applicable condition includes at least the condition that the block is a block that makes a bi-prediction. Further, the applicable condition may include a condition that the motion vector of the block is not encoded in the Symmetric MVD mode as described in Non-Patent Document 1.

If the applicable conditions are not satisfied, the processing procedure proceeds to step S55 to end the processing. At this time, the prediction signal generation unit 111D outputs the prediction signal generated in step S51 as the final prediction signal.

On the other hand, when all the applicable conditions are satisfied, this processing procedure proceeds to step S53. In step S53, this processing procedure determines whether or not to actually execute the BDOF processing of step S54 for the block satisfying the applicable conditions.

In the following, the case where the absolute value difference sum is used as the search cost will be described as an example, but other indexes can also be used for the search cost. For example, any index value for determining the similarity between image signals, such as the absolute value difference sum and the square error sum of the signals after removing the local mean value, can be used as the search cost. ..

For example, the prediction signal generation unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and when the calculated value is smaller than the predetermined threshold value (or the calculated value is predetermined). If it is equal to or less than a predetermined threshold value), it is determined that the BDOF process is not performed.

Further, for example, the prediction signal generation unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and when the calculated value is larger than the predetermined threshold value (or the calculated value is larger than the predetermined threshold value). If it is equal to or higher than a predetermined threshold value), it is possible to make a determination that the BDOF process is not performed.

Here, the prediction signal generation unit 111D can also use the result of the refinement processing for determining whether or not the BDOF processing is applied to the block for which the refinement processing is executed by the refinement unit 111C.

The prediction signal generation unit 111D uses the search cost calculated in the process of the above-mentioned refinement process (for example, the absolute value difference sum of the pixel value of the reference block on the L0 side and the pixel value of the reference block on the L1 side). , It is also possible to determine whether or not to apply the BDOF process.

For example, in the prediction signal generation unit 111D, in the search with integer pixel accuracy in step S45, the absolute value difference sum of the search points at which the search cost (absolute value difference sum) is minimized is higher than the predetermined fifth threshold value. If it is small (or if it is equal to or less than a predetermined fifth threshold value), it can be determined that the BDOF process is not applied. At this time, the fifth threshold value may be set to the same value as the first threshold value.

For example, in the prediction signal generation unit 111D, in the search with integer pixel accuracy in step S45, the absolute value difference sum of the search points at which the above-mentioned search cost (absolute value difference sum) is minimized is greater than a predetermined sixth threshold value. If it is also large (or if it is equal to or higher than a predetermined sixth threshold value), it can be determined that the BDOF process is not applied. At this time, the sixth threshold value may be set to the same value as the second threshold value or the third threshold value.

For example, the prediction signal generation unit 111D performs BDOF processing when the search cost at the initial position calculated in step S43 is smaller than the predetermined fifth threshold value (or when it is equal to or less than the predetermined fifth threshold value). Can be determined not to apply. At this time, the fifth threshold value may be set to the same value as the first threshold value.

For example, the prediction signal generation unit 111D performs BDOF processing when the search cost at the initial position calculated in step S43 is larger than the predetermined sixth threshold value (or when it is equal to or higher than the predetermined sixth threshold value). Can be determined not to apply. At this time, the sixth threshold value may be set to the same value as the second threshold value or the third threshold value.

For example, in the prediction signal generation unit 111D, the difference value between the search cost at the initial position calculated in step S43 and the minimum search cost in the search with integer pixel accuracy in step S45 is larger than the predetermined seventh threshold value. If it is small (or if it is equal to or less than a predetermined seventh threshold value), it can be determined that the BDOF process is not applied. At this time, the seventh threshold value may be set to the same value as the first threshold value or the fourth threshold value.

Further, the prediction signal generation unit 111D may be determined by a method based on the result of the refinement process in the block in which the refinement process is executed, and by a method based on the absolute value difference sum in the other blocks.

Further, the prediction signal generation unit 111D is obtained from the result of the refinement processing without newly performing the processing of calculating the absolute value difference sum between the prediction signal on the L0 side and the prediction signal on the L1 side, as described above. It is also possible to use only the information to determine the suitability of the BDOF process. In this case, the prediction signal generation unit 111D determines in step S53 that the prediction signal generation unit 111D always applies the BDOF processing to the block for which the refinement processing has not been executed.

According to such a configuration, in this case, it is not necessary to perform the calculation processing of the absolute value difference sum in the prediction signal generation unit 111D, so that the processing amount and the processing delay can be reduced from the viewpoint of hardware implementation.

Further, according to such a configuration, from the viewpoint of software implementation, the coding efficiency is improved by using the result of the refinement processing and preventing the BDOF processing from being executed in the block where the effect of the BDOF processing is presumed to be low. While maintaining it, the processing time for the entire image can be shortened.

Further, the determination process itself using the result of the above-mentioned refinement process is executed inside the refinement unit 111C, and the information indicating the result is transmitted to the prediction signal generation unit 111D, so that the prediction signal generation unit 111D , It is also possible to determine the suitability of BDOF processing.

For example, as described above, the motion vector and the value of the search cost before and after the refinement process are determined, and if the condition that the BDOF process is not applied is applied, the value is "1". A flag that becomes "0" when the conversion process is not applied is prepared, and the prediction signal generation unit 111D can determine the suitability of the BDOF process by referring to the value of the flag.

Further, although the steps S52 and S53 have been described as different steps here for convenience, it is also possible to simultaneously perform the determinations in the steps S52 and S53.

In the determination as described above, the present processing procedure proceeds to step S55 for the block determined not to apply the BDOF processing. For the other blocks, this processing procedure proceeds to step S54.

In step S54, the prediction signal generation unit 111D executes BDOF processing. Since the BDOF process itself can use a known method, detailed description thereof will be omitted. After the BDOF process is performed, the process moves to step S55 and ends the process.

The above-mentioned first threshold value to the seventh threshold value are defined so as to change their values according to the block size (product of the height and width of the block) or the number of pixels used for calculating the search cost such as SAD in the block. You may.

The above-mentioned first threshold value to the seventh threshold value may be defined so as to change their values according to the quantization parameter (QP) of the block. For example, if the cost value tends to increase when the quantization parameter is large, the threshold value can be defined to increase as the quantization parameter increases.

(Image Decoding Device 200)
Hereinafter, the image decoding apparatus 200 according to the present embodiment will be described with reference to FIG. FIG. 6 is a diagram showing an example of a functional block of the image decoding apparatus 200 according to the present embodiment.

As shown in FIG. 6, the image decoding device 200 includes a decoding unit 210, an inverse conversion / inverse quantization unit 220, an adder 230, an inter-prediction unit 241 and an intra-prediction unit 242, and an in-loop filter processing unit. It has 250 and a frame buffer 260.

The decoding unit 210 is configured to decode the coded data generated by the image coding device 100 and decode the coefficient level value.

Here, for example, the decoding is the entropy decoding in the reverse procedure of the entropy coding performed by the coding unit 140.

Further, the decoding unit 210 may be configured to acquire the control data by the decoding process of the coded data.

As described above, the control data may include size data such as a coded block size, a predicted block size, and a conversion block size.

The inverse conversion / inverse quantization unit 220 is configured to perform an inverse conversion process of the coefficient level value output from the decoding unit 210. Here, the inverse transformation / inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level value prior to the inverse transformation process.

Here, the inverse conversion process and the inverse quantization are performed in the reverse procedure of the conversion process and the quantization performed by the conversion / quantization unit 131.

The adder 230 adds a prediction signal to the prediction residual signal output from the inverse conversion / inverse quantization unit 220 to generate a pre-filter processing decoding signal, and the pre-filter processing decoding signal is used by the intra prediction unit 242 and the in-loop. It is configured to output to the filter processing unit 250.

Here, the pre-filtered decoding signal constitutes a reference block used by the intra prediction unit 242.

Similar to the inter-prediction unit 111, the inter-prediction unit 241 is configured to generate a prediction signal by inter-prediction (inter-frame prediction).

Specifically, the inter-prediction unit 241 is configured to generate a prediction signal for each prediction block based on the motion vector decoded from the coded data and the reference signal included in the reference frame. The inter-prediction unit 241 is configured to output a prediction signal to the adder 230.

Like the intra prediction unit 112, the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (in-frame prediction).

Specifically, the intra prediction unit 242 is configured to specify a reference block included in the target frame and generate a prediction signal for each prediction block based on the specified reference block. The intra prediction unit 242 is configured to output a prediction signal to the adder 230.

Similar to the in-loop filter processing unit 150, the in-loop filter processing unit 250 performs filter processing on the pre-filter processing decoded signal output from the adder 230, and outputs the post-filter processing decoded signal to the frame buffer 260. It is configured to do.

Here, for example, the filtering process is a deblocking filtering process that reduces the distortion generated at the boundary portion of a block (encoded block, prediction block, conversion block, or sub-block obtained by dividing them).

Like the frame buffer 160, the frame buffer 260 is configured to store reference frames used by the inter-prediction unit 241.

Here, the filtered decoded signal constitutes a reference frame used by the inter-prediction unit 241.

(Inter prediction unit 241)
Hereinafter, the inter-prediction unit 241 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a functional block of the inter-prediction unit 241 according to the present embodiment.

As shown in FIG. 7, the inter-prediction unit 241 includes a motion vector decoding unit 241B, a refinement unit 241C, and a prediction signal generation unit 241D.

The inter-prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector decoding unit 241B is configured to acquire a motion vector by decoding control data received from the image coding device 100.

The refinement unit 241C is configured to execute the refinement process for modifying the motion vector, similarly to the refinement unit 111C.

Like the prediction signal generation unit 111D, the prediction signal generation unit 241D is configured to generate a prediction signal based on the motion vector.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, in the refinement units 111C and 241C, when the search cost is larger than the predetermined threshold value (or when the search cost is equal to or higher than the predetermined threshold value). In some cases), the search result can be discarded. Here, the refinement process can improve the coding efficiency when the correlation between the pixel values of the block, the reference block on the L0 side, and the reference block on the L1 side is high. With the above configuration, the refinement process is performed in the block having a small correlation in the time direction without transmitting additional information (information about whether or not the refinement process is applied) from the image coding apparatus 100 side. It is possible to prevent the coding efficiency from being lowered, and it is possible to prevent a decrease in coding efficiency.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, in the refinement units 111C and 241C, the difference value between the search cost at the initial position and the search cost corresponding to the point after the search is set. When the value is smaller than the predetermined threshold value (or the value is equal to or lower than the predetermined threshold value), the search result can be discarded. As a result, it is possible to avoid unnecessarily correcting the motion vector at the point where a search cost slightly smaller than the initial position is accidentally obtained due to the influence of noise or the like, and it is possible to prevent a decrease in coding efficiency. Can be done.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the refinement units 111C and 241C can be configured to include the norm of the difference vector between the initial position and the search point in the search cost. In this case, when the index value for evaluating the similarity such as the absolute value error sum becomes almost the same value at each search point, the motion vector is unnecessarily corrected by regularizing with the norm of the difference vector. It can be prevented from becoming large, and a decrease in coding efficiency can be prevented.

According to the image coding device 100 and the image decoding device 200 according to the present embodiment, the reference block on the L0 side and the reference block on the L1 side are used to determine whether or not the BDOF processing is executed in the prediction signal generation units 111D and 241D. It is possible to include the condition that the degree of similarity (for example, the sum of absolute value errors) is larger than a predetermined threshold value (or the degree of similarity is equal to or more than a predetermined threshold value). Similar to the refinement process, the BDOF process is also an effective process when the correlation between the block, the reference block on the L0 side, and the reference block on the L1 side is high. Therefore, with the above configuration, the BDOF processing is performed in the block having a small correlation in the time direction without transmitting additional information (information about whether or not the BDOF processing is applied) from the image coding apparatus 100 side. It is possible to prevent the coding efficiency from being lowered, and it is possible to prevent a decrease in coding efficiency. Further, the result of the above-mentioned refinement processing can be used for the above-mentioned determination. This makes it possible to reduce the process of calculating the absolute value difference described above.

Further, the above-mentioned image coding device 100 and image decoding device 200 may be realized by a program that causes a computer to execute each function (each step).

In each of the above embodiments, the present invention has been described by using the application to the image coding device 100 and the image decoding device 200 as an example, but the present invention is not limited to such an example, and the image code is not limited to the above. It can be similarly applied to an image coding / decoding system having each function of the computer 100 and the image decoding device 200.

According to the present invention, it is possible to prevent a decrease in coding efficiency by not performing a refinement process on a block having a low correlation in the time direction.

10 ... Image processing system 100 ... Image coding device 111, 241 ... Inter prediction unit 111A ... Motion vector search unit 111B ... Motion vector coding unit 111C, 241C ... Refinement unit 111D, 241D ... Prediction signal generation unit 112, 242 ... Intra prediction unit 121 ... subtractor 122, 230 ... adder 131 ... conversion / quantization unit 132, 220 ... inverse conversion / inverse quantization unit 140 ... coding unit 150, 250 ... in-loop filter processing unit 160, 260 ... frame Buffer 200 ... Image decoding device 210 ... Decoding unit 241B ... Motion vector decoding unit

Claims (3)

It is an image decoder
A motion vector decoder configured to decode motion vectors from coded data,
Equipped with a refinement department
The refinement unit determines whether or not the predetermined conditions for applying the refinement process are satisfied, and when it is determined that all the predetermined conditions are satisfied, the refinement unit determines.
A motion vector search is performed using the value of the motion vector decoded by the motion vector decoding unit as an initial value.
When the search cost at the initial search point is smaller than the predetermined first threshold value, the search for the motion vector is terminated.
When the search cost at the initial search point is equal to or higher than the first threshold value, the search is performed at the integer pixel spacing position.
The integer pixel when the difference value between the search cost at the initial search point and the minimum search cost in the search at the integer pixel spacing position is smaller than a predetermined threshold value or when the difference value is equal to or less than the threshold value. An image decoding device characterized in that it is configured to discard search results at spaced positions. The process of decoding the motion vector from the coded data and
It is determined whether or not the predetermined conditions for applying the refinement process are satisfied, and when it is determined that all the predetermined conditions are satisfied, it is determined.
A step of searching for a motion vector using the decoded motion vector value as an initial value, and
When the search cost at the initial search point is smaller than the predetermined first threshold value, the step of ending the search of the motion vector and the step of ending the search.
When the search cost at the initial search point is equal to or higher than the first threshold value, the step of performing the search at the integer pixel interval position and the step of performing the search.
The integer pixel when the difference value between the search cost at the initial search point and the minimum search cost in the search at the integer pixel spacing position is smaller than a predetermined threshold value or when the difference value is equal to or less than the threshold value. An image decoding method comprising a step of discarding a search result at an interval position. A program used in an image decoder, which can be used on a computer.
The process of decoding the motion vector from the coded data and
It is determined whether or not the predetermined conditions for applying the refinement process are satisfied, and when it is determined that all the predetermined conditions are satisfied, it is determined.
A step of searching for a motion vector using the decoded motion vector value as an initial value, and
When the search cost at the initial search point is smaller than the predetermined first threshold value, the step of ending the search of the motion vector and the step of ending the search.
When the search cost at the initial search point is equal to or higher than the first threshold value, the step of performing the search at the integer pixel interval position and the step of performing the search.
The integer pixel when the difference value between the search cost at the initial search point and the minimum search cost in the search at the integer pixel spacing position is smaller than a predetermined threshold value or when the difference value is equal to or less than the threshold value. A program characterized by executing a process of discarding search results at interval positions.

Images