JP4563982B2 - Motion estimation method, apparatus, program thereof, and recording medium thereof - Google Patents

Description

  The present invention relates to a high-efficiency image signal encoding method, and more particularly, to speed up the amount of shift estimation with integer pixel accuracy in moving image encoding with inter-frame prediction using motion compensation using a plurality of reference frames. The present invention relates to a motion estimation method, apparatus, program thereof, and recording medium thereof.

  In recent years, there are growing expectations for super-high-quality images such as large-screen sports images and digital cinema that are full of realism. In response to this, research on high-quality video has been vigorously conducted. The following four elements are necessary to realize ultra-high-quality video. That is, spatial resolution, pixel value depth, color reproducibility, and temporal resolution. In response, the former three elements are being studied in applications such as digital cinema and natural vision projects.

  However, sufficient studies have not been made on improving time resolution, that is, increasing the frame rate of an image, which is indispensable for expressing the natural movement of a subject. According to Spillmann et al., Physiological knowledge that the upper limit of the number of pulses output by ganglion cells as output cells of the retina is about 300 to 400 per second is shown. For this reason, it is presumed that the human visual system is the shortest and can perceive a difference in light emission from 1/150 to 1/200 second. This means that the perceivable frame rate detection limit is 150 to 200 [frames / second]. The current video frame rate of 30, 60 [frames / second] is determined from the flicker detection limit and is not a sufficient value to express natural motion.

  On the other hand, the super high image quality of the video leads to an explosive increase in the amount of data, so an efficient encoding method is required. Inter-frame prediction by motion compensation is effective for reducing redundancy of moving image data in the time axis direction. In order to reduce prediction errors using motion compensation, it is necessary to accurately estimate the amount of shift between frames by motion estimation. When an accurate shift amount between adjacent frames is estimated for a high frame rate video, there is a possibility that a shift amount less than one pixel may occur. Therefore, motion estimation with decimal pixel accuracy is required. For example, in the case of a high frame rate video of 1000 [frames / second], there is a possibility that a shift amount of about 1/1000 pixel accuracy may occur.

  In motion estimation with decimal pixel accuracy, an interpolation filter is used to obtain the pixel value at the decimal pixel position in the reference frame. However, it is generally reported that the prediction performance is saturated by interpolation with 1/4 to 1/8 pixel accuracy due to the restriction due to the low-pass characteristic of the interpolation filter. For this reason, there is a limit in dealing with a minute amount of shift included in a high frame rate video by fractional pixel precision motion estimation using an interpolation filter.

  On the other hand, there is a method (multiple reference frames) that uses a plurality of frames as reference frames during motion compensation. In the case of a high frame rate video, since sampling in the time axis direction is densely performed, there is a high possibility that there is a frame that can be associated with the frame at integer pixel positions. That is, there is a high possibility that there is a frame in which the shift amount between the frame and the reference frame can be expressed with integer pixel accuracy.

  If the amount of shift can be expressed with integer pixel accuracy, interpolation filter processing is not necessary, and the possibility of keeping prediction errors low is increased. For this reason, it can be said that motion compensation with integer pixel accuracy using multiple reference frames is effective in motion compensation for high frame rate video. However, when the amount of shift is determined by brute force, the number of frames subject to reference frames increases, so both the calculation amount and the memory consumption increase. For example, in the case of a pixel having a shift amount of 1 [pixel / sec], when estimating a shift amount by motion compensation for a frame having a frame rate of 1000 [frames / sec], 1000 [frames] is used as a reference frame. It is necessary to estimate the amount of displacement. For this reason, a means for narrowing down the reference frame is required.

The following investigations have been made on speeding up motion search for multiple reference frames. In the following non-patent document 1, a motion search is performed on adjacent frames, and the motion search range in other reference frames is narrowed down on the basis of the search result assuming constant motion. The conventional method as shown in Non-Patent Document 1 is a method aimed at reducing the amount of calculation of motion search with decimal pixel accuracy. For this reason, narrowing down of reference frames for performing motion search with integer pixel precision is not considered. Furthermore, it is a premise that extrapolation is performed based on the motion search result obtained in the adjacent frame. For this reason, when the reference frame for performing the motion search with integer pixel accuracy is set without being limited to the adjacent frames as in the above-described case, the amount of calculation cannot be expected.
Shohei Matsuo, Isao Nagayoshi, Go Hanamura, Hideyoshi Tominaga, “Examination of Efficient Motion Search Using Multiple Reference Frames”, IEICE Technical Report, 2005-AVM-50, Oct. 2005

  The present invention has been made in view of such circumstances, and establishes a high-speed motion estimation method in the case of performing integer pixel precision shift amount estimation using a plurality of reference frames in encoding processing for a high frame rate video signal. The purpose is to do.

  In order to solve the above problems, the present invention uses the following method. In addition, although the example which uses the value of a brightness | luminance as a pixel value is demonstrated, you may use pixel values other than a brightness | luminance.

  In the first invention, when motion compensation using one or a plurality of reference frames (multiple reference frame motion compensation) is used in moving picture coding with inter-frame prediction, the amount of time variation required for a certain spatial transition A time shift amount that minimizes a cost function composed of a weighted sum of a spatial luminance gradient between pixels at a distance of the spatial shift in the processing target frame and a temporal luminance gradient between adjacent frames in the processing target frame, Motion estimation is performed using a frame corresponding to the obtained time shift amount as a reference frame. Alternatively, motion estimation is performed using a frame corresponding to the obtained time shift amount and a plurality of frames before and after the frame as reference frames.

  In the second invention, when motion compensation using one or a plurality of reference frames (multiple reference frame motion compensation) is used in moving picture coding with inter-frame prediction, for each pixel in the local region Thus, the amount of time shift required for a certain spatial shift is estimated by the ratio of the luminance gradient between the pixels at the distance of the spatial shift in the frame and the luminance gradient between adjacent frames, and the estimated time shift amount Based on the above, a histogram of the time shift amount in the local region is generated, the time shift amount of the maximum frequency in the histogram is obtained, and motion estimation is performed using a frame corresponding to the time shift amount as a reference frame. Alternatively, motion estimation is performed using a frame corresponding to the time shift amount and a plurality of frames before and after the frame as reference frames. Details are as follows.

[First invention]
In the following, for simplicity, a one-dimensional signal will be described as an example. The pixel value at the position x of the t-th frame is assumed to be f (x, t). Let the frame interval be _Δt . Assuming a translation model and assuming that the number of frames required for shifting one pixel (time shift amount) is b, the following relationship holds.

f (x, t) = f (x-1, t-bΔ t) (1)
The right-hand side of the above equation can be approximated by the first-order approximation of Taylor expansion as

Further, the second term on the right side of the above equation is approximated by difference.

Using the above relationship, the following equation is obtained.

Solving for b, assuming that both sides of the above equation are equal, the following equation is obtained.

The above equation gives the amount of time shift required for shifting one pixel for each pixel.

Furthermore, when the amount of transition is generalized and the number of frames necessary for the transition of a pixel (time variation amount) is b _a , the following equation is obtained in the same manner as described above. Note that a is an integer.

The above equation gives the amount of time shift required for shifting a pixel for each pixel. The b shown in the equation (5) is a case where a = 1 in the equation (6) and corresponds to b ₁ .

On the other hand, from the viewpoint of encoding efficiency, it is desirable to give a common time shift amount to a plurality of pixels. Therefore, assuming equal time displacement amount of each point in the local region, the above relation in the local region to estimate the best fit b _a sense of the least square error. That is, a time shift amount that minimizes the following expression in the local region R is obtained.

Here, argmin _b returns b that minimizes the next function. The cost function E (b _a ) is as follows:

In equation (8),
e _s (x, t, a) = f (x−a, t) −f (x, t), and
e _t (x, t, a) = f (x−a, t− _Δt ) −f (x−a, t)
Then, E (b _a ) can be expanded as follows. Note that e _s (x, t, a) is called a spatial luminance gradient, and e _t (x, t, a) is called a temporal luminance gradient.

Therefore, the value of b that minimizes the above equation is as follows.

The motion estimation for f (x, t) is based on the frame interval b _opt shown in Equation (7), and the motion estimation with integer pixel accuracy is performed using the frame f (x, t−b _{opt Δt} ) as a reference frame. . As will be described in detail in the second invention, a plurality of frames before and after the frame f (x, t−b _{opt Δt} ) may be used as reference frames.

[Second invention]
Using equation (6), the time shift amount is obtained for all the pixels in the local region R. The time shift amount at f (x, t) (xεR) obtained as a result is set as b (x, t, a). Let b _opt be the value with the highest frequency of b (x, t, a) (xεR), and let the (t−b _opt ) frame be the reference frame.

Further, a plurality of frames (2M-1 frames) before and after the (t-b _opt ) frame may be used as reference frames. In this case, the τ-th frame (t−b _opt −M ≦ τ <t−b _opt + M) is a reference frame to be searched for motion estimation. Here, M is an integer, and the value of M is given from the outside. If there is no means for giving the value of M from the outside, the same value is generated internally. For example, assuming that the probability of the maximum frequency of b (x, t, a) (xεR) is r, M is given by the following equation.

Here, F is the frame rate of the sequence, and N is the number of pixels in the region R.

  According to the present invention, when motion estimation in which a reference frame is not limited to adjacent frames is applied to a high frame rate video, it is possible to quickly determine a reference frame to be searched for motion estimation with integer pixel accuracy. This makes it possible to estimate the amount of displacement at high speed. By estimating the shift amount by motion estimation with integer pixel accuracy, it is possible to avoid missing high-frequency components in the reference signal due to the application of the interpolation filter, thereby reducing the prediction residual in motion compensation interframe prediction.

Hereinafter, the pixel value f (x, t) of the motion estimation target frame is referred to as the pixel value of the frame, f (x−a, t) is referred to as the transition position pixel value of the frame, and f (x−a , T−Δ _t ) is called the transition position pixel value of the adjacent frame.

[Flow of Motion Estimation Processing (Example 1)]
A flow of motion estimation processing according to the first embodiment will be described with reference to FIG.

  Step S101: The spatial shift amount a is read and stored. In this embodiment, the spatial shift amount a is assumed to be given from the outside as a parameter at the time of encoding. For example, in the case where a = 1 is set in advance, such as a = 1, this spatial shift amount reading process is not necessary.

  Step S102: The frame is read into the frame buffer. Note that the frame number of the frame is t.

  Step S103: Read an adjacent frame of the frame into the frame buffer.

  Step S104: The registers A1 and A2 are initialized to zero values.

  Step S105: Based on the read spatial shift amount a, the processing target pixel value f (x, t) of the frame and the shift position pixel value f (x−a, t) of the frame are read from the frame buffer. A process for obtaining a difference between these values is performed, and the result is written in the register Es.

Step S106: Based on the read spatial shift amount a, the shift position pixel value f (x−a, t) of the frame and the shift position pixel value f (x−a, t− _Δt ) of the adjacent frame are obtained. Reading from each frame buffer, obtaining a difference between these values, and writing the obtained result to the register Et.

  Step S107: The value of the register Et is read, the square value of the same value is obtained, the obtained result is added to the value of the register A1, and the addition result is written to the register A1.

  Step S108: The values of the registers Es and Et are read, the product of both values is obtained, the obtained result is added to the value of the register A2, and the addition result is written to the register A2.

  Steps S109 and S110: It is determined whether or not the processing has been completed for all the pixels in the local region R. If the processing has been completed, the process proceeds to step S111. Otherwise, the processing target pixel is set as the next pixel in the local region R, and the processing in steps S105 to S108 is repeated in the same manner.

  Step S111: Read the values of the registers A1 and A2, perform division processing on the value of A2 using the value of A1, and write the result to the register b.

  Step S112: The value of the register b and the value of M storing the number of reference frames are read, and the 2M + 1 frame from the (t−b−M) th frame to the (t−b + M) th frame is a reference for motion estimation search. Write as a frame to the frame buffer. t is the frame number of the frame.

  Step S113: The pixel value of the frame in the frame buffer written out in step S102 and the pixel value of the reference frame in the frame buffer written out in step S112 are read as inputs, motion estimation processing is performed, and the estimated amount of change is calculated. Write to register. Note that a specific motion estimation method is given from the outside. As a motion estimation method, for example, there is motion estimation based on block matching.

[Motion Estimation Device (Example 1)]
FIG. 2 is a configuration diagram of the motion estimation apparatus according to the first embodiment. With reference to FIG. 2, the motion estimation apparatus of Example 1 is demonstrated.

  Frame reading unit 101: Reads the frame and stores it in the frame storage unit 102.

  Spatial shift reading unit 103: Reads the spatial shift a and stores it in the spatial shift storage unit 104.

  Adjacent frame reading unit 105: Reads the frame and stores it in the adjacent frame storage unit 106.

  Reference frame number reading unit 107: Reads the reference frame number M and stores it in the reference frame number storage unit 108.

  Spatial luminance gradient calculation unit 109: Based on the spatial shift amount a in the spatial shift amount storage unit 104, the processing target pixel value f (x, t) of the frame from the frame storage unit 102 and the shift position of the frame The pixel value f (x−a, t) is read out, a process for obtaining a difference between the two pixel values is performed, and the obtained result is written in the spatial luminance gradient storage unit 110.

Temporal luminance gradient calculation unit 111: Based on the spatial shift amount a in the spatial shift amount storage unit 104, the transition position pixel value f (x−a, t) of the frame is read from the frame storage unit 102, and the adjacent frame The transition position pixel value f (x−a, t− _Δt ) of the adjacent frame is read from the storage unit 106, a process for calculating the difference between the two values is performed, and the calculated result is written in the temporal luminance gradient storage unit 112.

  Integration processing unit 113: Reads the value of the spatial luminance gradient storage unit 110 and the value of the temporal luminance gradient storage unit 112 as inputs, performs a product operation of both values, and writes the calculation result to the integrated value storage unit 114.

  Square calculation processing unit 115: Reads the value of the time luminance gradient storage unit 112 as an input, performs a square calculation on the same value, and writes the calculation result to the square value storage unit 116.

  Final pixel determination processing unit 117: Control is performed to repeat the above processing for all the pixels in the local region.

  Division processing unit 118: The value of the square value storage unit 116 and the value of the integrated value storage unit 114 are read as inputs, the latter is divided by the former, the number is rounded to an integer value, and the result is divided by the division value storage unit 119. Export to

  Reference frame reading unit 120: Reads the value of the division value storage unit 119 and the value of the reference frame number storage unit 108 as inputs, and sets both values as b and M, respectively, from the (t−b−M) frame to the ( 2M + 1 frames up to (t−b + M) frames are written in the reference frame storage unit 121 as reference frames to be searched for motion estimation. t is the frame number of the frame.

  Motion estimation processing unit 122: Reads the pixel value of the frame in the frame storage unit 102 and the pixel value of the reference frame in the reference frame storage unit 121 as input, performs a motion estimation process, and stores the estimated shift amount as a shift amount storage unit Write to 123. Note that a specific motion estimation method is given from the outside. For example, there is motion estimation based on block matching.

[Flow of Motion Estimation Processing (Example 2)]
A flow of motion estimation processing according to the second embodiment will be described with reference to FIG.

  Steps S201 to S203: The same as steps S101 to S103 of the first embodiment.

  Step S204: Each element of the histogram H is initialized to zero.

  Steps S205 and S206: The same as steps S105 and S106 of the first embodiment.

  Step S207: Read the values of the registers Es and Et, divide both values Es / Et, round the obtained result to an integer value, and write the result to the register b.

  Step S208: Read the value of the register b and the histogram H as input, add 1 to the frequency of the b-th element of the histogram H, and overwrite the frequency after the addition as the frequency of the b-th element of the histogram H.

  Steps S209 and S210: It is determined whether or not the processing has been completed for all the pixels in the local region R. If the processing is completed, the process proceeds to step S211. Otherwise, the process target pixel is set as the next pixel in the local region R, and the processes in steps S205 to S208 are repeated in the same manner.

  Step S211: The histogram H is read as an input, the value of the element having the maximum frequency in the histogram is obtained and written in the register b.

  Steps S212 and S213: Same as steps S112 and S113 of the first embodiment.

[Motion Estimation Device (Example 2)]
FIG. 4 is a configuration diagram of the motion estimation apparatus according to the second embodiment. With reference to FIG. 4, the motion estimation apparatus of Example 2 is demonstrated.

  Frame reading unit 201: Reads the frame and stores it in the frame storage unit 202.

  Spatial shift reading unit 203: Reads the spatial shift a and stores it in the spatial shift storage unit 204.

  Adjacent frame reading unit 205: Reads the frame and stores it in the adjacent frame storage unit 206.

  Reference frame number reading unit 207: Reads the reference frame number M and stores it in the reference frame number storage unit 208.

  Spatial luminance gradient calculation unit 209: based on the spatial shift amount a in the spatial shift amount storage unit 204, the processing target pixel value f (x, t) of the frame from the frame storage unit 202 and the shift position of the frame The pixel value f (x−a, t) is read out, a process for obtaining the difference between the two pixel values is performed, and the obtained result is written in the spatial luminance gradient storage unit 210.

Temporal luminance gradient calculating unit 211: Based on the spatial shift amount a in the spatial shift amount storage unit 204, the transition position pixel value f (x−a, t) of the frame is read from the frame storage unit 202, and the adjacent frame The transition position pixel value f (x−a, t− _Δt ) of the adjacent frame is read from the storage unit 206, a process for calculating the difference between the two values is performed, and the calculated result is written in the temporal luminance gradient storage unit 212.

  Division processing unit 213: Reads the value of the spatial luminance gradient storage unit 210 and the value of the temporal luminance gradient storage unit 212 as input, performs an operation of dividing the former by the latter, performs processing to round to an integer value, and divides the result into division values Write to the storage unit 214.

  Histogram update unit 215: Reads the value of the division value storage unit 214 and the histogram as input, and if the value of the division value storage unit 214 is b, 1 is added to the frequency of the b-th element of the histogram, and the frequency after the addition is The frequency is written in the histogram storage unit 216 as the frequency of the b-th element of the histogram.

  Final pixel determination processing unit 217: Control is performed to repeat the above processing for all pixels in the local region.

  Maximum frequency element detection unit 218: Reads the histogram of the histogram storage unit 216 as input, obtains the value of the element having the maximum frequency in the histogram, and writes this value to the maximum frequency element storage unit 219.

  Reference frame reading unit 220: Reads the value of the maximum frequency element storage unit 219 and the value of the reference frame number storage unit 208 as inputs, where both values are b and M, respectively, from the (t−b−M) frame to the second 2M + 1 frames up to (t−b + M) frames are written in the reference frame storage unit 221 as reference frames to be searched for motion estimation. t is the frame number of the frame.

  Motion estimation processing unit 222: Reads the pixel value of the frame of the frame storage unit 202 and the pixel value of the reference frame of the reference frame storage unit 221 as input, performs a motion estimation process, and stores the estimated shift amount as a shift amount storage unit Write to 223. Note that a specific motion estimation method is given from the outside. For example, there is motion estimation based on block matching.

  In order to make the explanation easy to understand, the description has been given by taking a one-dimensional signal as an example, but it goes without saying that the present technology can be easily applied to a two-dimensional image signal from the description of the present embodiment.

  The motion estimation process described above can be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or provided through a network.

3 is a flowchart illustrating a flow of motion estimation processing according to the first embodiment. It is a block diagram of the motion estimation apparatus of Example 1. It is a flowchart which shows the flow of the motion estimation process of Example 2. It is a block diagram of the motion estimation apparatus of Example 2.

Explanation of symbols

101, 201 Frame reading unit 102, 202 Frame storage unit 103, 203 Spatial displacement reading unit 104, 204 Spatial transition storage unit 105, 205 Adjacent frame reading unit 106, 206 Adjacent frame storage unit 107, 207 Number of reference frames Reading unit 108, 208 Reference frame number storage unit 109, 209 Spatial luminance gradient calculation unit 110, 210 Spatial luminance gradient storage unit 111, 211 Time luminance gradient calculation unit 112, 212 Time luminance gradient storage unit 113 Integration processing unit 114 Integration value storage Unit 115 Square arithmetic processing unit 116 Square value storage unit 117, 217 Final pixel determination processing unit 118, 213 Division processing unit 119, 214 Division value storage unit 120, 220 Reference frame reading unit 121, 221 Reference frame storage unit 122, 222 Motion Estimation process Units 123 and 223 Transition amount storage unit 215 Histogram update unit 216 Histogram storage unit 218 Maximum frequency element detection unit 219 Maximum frequency element storage unit

Claims (6)

In a motion estimation method in video coding with inter-frame prediction using motion compensation using one or more reference frames,
A spatial pixel value gradient calculating step for calculating a spatial pixel value gradient between the processing target pixel in the processing target frame and a pixel separated from the processing target pixel by a spatial shift amount corresponding to a predetermined integer pixel;
Time for calculating a time pixel value gradient between a transition position pixel separated from the processing target pixel in the processing target frame by the spatial shift amount and a transition position pixel in a position corresponding to the transition position pixel in an adjacent frame A pixel value gradient calculating step;
A time shift amount calculating step for calculating a time shift amount that minimizes a cost function including a weighted sum of the space pixel value gradient and the time pixel value gradient, the time shift amount necessary for the space shift amount as a weight value; When,
A motion estimation method comprising: a motion estimation step of estimating motion in motion compensation using a frame corresponding to the calculated time shift amount or a plurality of frames before and after the frame as reference frames. In a motion estimation method in video coding with inter-frame prediction using motion compensation using one or more reference frames,
The spatial pixel value gradient between the processing target pixel in the processing target frame and a pixel separated from the processing target pixel by a predetermined integer pixel spatial shift amount is calculated for each pixel in the local region in the processing target frame. A spatial pixel value gradient calculating step,
A temporal pixel value gradient between a transition position pixel separated from the processing target pixel in the processing target frame by the spatial shift amount and a transition position pixel in a position corresponding to the transition position pixel in an adjacent frame is represented by the processing. A temporal pixel value gradient calculating step for calculating each pixel in the local region in the target frame;
Calculating a ratio between the spatial pixel value gradient and the temporal pixel value gradient, and estimating a time shift amount for each pixel in the local region;
A time shift amount calculating step for generating a time shift amount histogram for all pixels in the local region based on the estimated time shift amount for each pixel, and obtaining a maximum frequency time shift amount in the histogram;
A motion estimation step of estimating a motion in motion compensation using a frame corresponding to the calculated time shift amount or a plurality of frames before and after the frame as a reference frame. In a motion estimation apparatus in video coding with inter-frame prediction using motion compensation using one or a plurality of reference frames,
A spatial pixel value gradient calculating means for calculating a spatial pixel value gradient between a processing target pixel in the processing target frame and a pixel separated from the processing target pixel by a spatial shift amount corresponding to a predetermined integer pixel;
Time for calculating a time pixel value gradient between a transition position pixel separated from the processing target pixel in the processing target frame by the spatial shift amount and a transition position pixel in a position corresponding to the transition position pixel in an adjacent frame Pixel value gradient calculating means;
Time shift amount calculating means for calculating a time shift amount that minimizes a cost function composed of a weighted sum of the space pixel value gradient and the time pixel value gradient using a time shift amount required for the space shift amount as a weight value. When,
A motion estimation apparatus comprising: a motion estimation unit configured to estimate motion in motion compensation using a frame corresponding to the calculated time shift amount or a plurality of frames before and after the frame as reference frames. In a motion estimation apparatus in video coding with inter-frame prediction using motion compensation using one or a plurality of reference frames,
The spatial pixel value gradient between the processing target pixel in the processing target frame and a pixel separated from the processing target pixel by a predetermined integer pixel spatial shift amount is calculated for each pixel in the local region in the processing target frame. Spatial pixel value gradient calculating means for
A temporal pixel value gradient between a transition position pixel separated from the processing target pixel in the processing target frame by the spatial shift amount and a transition position pixel in a position corresponding to the transition position pixel in an adjacent frame is represented by the processing. A time pixel value gradient calculating means for calculating each pixel in the local region in the target frame;
Calculating a ratio between the spatial pixel value gradient and the temporal pixel value gradient, and estimating a time shift amount for each pixel in the local region;
A time shift amount calculating means for generating a histogram of time shift amounts for all the pixels in the local region based on the estimated time shift amount for each pixel, and obtaining a time shift amount having a maximum frequency in the histogram;
A motion estimation apparatus comprising: a motion estimation unit configured to estimate motion in motion compensation using a frame corresponding to the calculated time shift amount or a plurality of frames before and after the frame as reference frames.   A motion estimation program for causing a computer to execute the motion estimation method according to claim 1.   A computer-readable recording medium recording a motion estimation program for causing a computer to execute the motion estimation method according to claim 1.

Images