JP5309097B2 - Motion estimation apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion estimation device estimating a motion of a moving image and to provide a program. <P>SOLUTION: A motion estimation device 1 includes: a time space band spectrum information detecting unit 11 performing a frequency band separation processing of time space on an image of a processing object and calculating time space frequency band power information; and a motion estimating unit 13 performing first motion estimation for estimating a hierarchy motion by using a spatial low frequency image group of each hierarchy on which the band separation processing is performed, second motion estimation for estimating the motion by using a horizontal low/vertical high frequency image group of the first hierarchy on which the band separation processing is performed, third motion estimation for estimating the motion by using a horizontal high/vertical low frequency image group of the first hierarchy on which the band separation processing is performed, and fourth motion estimation for estimating the motion by using a horizontal high/vertical high frequency image group of the first hierarchy on which the band separation processing is performed, and deciding a motion estimation value by a motion estimation rate based on information of frequency band power of the time space for each motion estimation information obtained by the first motion estimation to the fourth motion estimation. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present invention relates to a motion estimation apparatus and program for hierarchically analyzing the amount of motion of a moving image based on the temporal and spatial spectral power of the moving image and detecting the motion vector of the moving image with high accuracy.

  The conventional basic method of motion estimation is the full search block matching method. As a way to develop this full search block matching method, a hierarchical image with different resolutions is obtained by performing frequency band resolution processing on the moving image in the spatial direction, and hierarchical motions using the hierarchical images with different resolutions are obtained. Methods for performing estimation (see, for example, Patent Documents 1 and 2), diamond search methods (for example, see Patent Document 3), and the like are known.

Japanese Patent No. 3334271 Japanese Patent No. 4195978 JP 2007-300169 A

  Hierarchical motion estimation and the diamond search method as described above are effective techniques for detecting a large amount of motion in a moving image, and have an effect of reducing the calculation cost as compared with the all-search block matching method.

  That is, a motion estimation device using a small block size (for example, 2 × 2 pixels) is suitable for a pattern having a large number of spatial high-frequency components. On the other hand, since a pattern with a small number of spatial high-frequency components is likely to be an erroneous motion estimation device with a small block size, a motion estimation device using a large block size (for example, 16 × 16 pixels) is effective. Therefore, by performing hierarchical motion estimation, motion estimation using a plurality of types of block sizes is equivalently performed, which is advantageous in reducing calculation cost.

  On the other hand, a large amount of motion increases the amount of motion blur in the motion region and decreases the power ratio in the spatial high frequency region. Also, a large dynamic area increases the time variation area of the dynamic area and increases the power ratio of the time high frequency area. Therefore, it cannot be said that the amount of movement of various sizes can be sufficiently covered only by hierarchizing based on the power in the spatial frequency domain, and there is room for improvement.

  Accordingly, an object of the present invention is to hierarchically analyze the amount of motion of a moving image based on the spectral power of the moving image in the time direction and the spatial direction (that is, the spatio-temporal direction), and to obtain motion estimation information of the moving image with high accuracy and high accuracy. The object is to provide a motion estimation device and a program for obtaining accuracy.

  As described above, there is often a certain correlation between the amount of motion and the power in the high-frequency region in the space-time direction. It is effective to estimate a block size and detect a motion vector by spatially hierarchizing the estimated block size.

  Considering the spectrum in the spatio-temporal direction of the moving image, the power of the high-frequency region in the moving direction in the moving region decreases as the area of the motion amount increases. That is, a moving image in which a large-area moving object has a large amount of motion has a tendency that the power in the high-frequency region in the spatial direction decreases, but the power in the high-frequency region in the time direction tends to increase. This is because when a large area object moves greatly on the screen, the variation in the time direction becomes large.

  Therefore, the motion estimation apparatus of the present invention performs spatio-temporal frequency band decomposition processing on a moving image to be processed, calculates spatio-temporal frequency band power information, and performs each frequency band decomposition processing on each layer. Horizontal low / vertical high-frequency images of each layer (preferably the first layer when calculation cost is taken into account) subjected to the first hierarchical motion estimation using the spatial low-frequency image group and subjected to frequency band decomposition processing By using the horizontal high / vertical low frequency image group of each layer (preferably the first layer when calculation cost is considered), the second hierarchical motion estimation is performed using the group, and the frequency band decomposition processing is performed. A fourth hierarchical type using a horizontal high / vertical high-frequency image group of each layer (preferably the first layer in consideration of calculation cost) subjected to the third hierarchical motion estimation and subjected to the frequency band decomposition processing. Motion estimation, and each motion estimation From the motion estimation information obtained Te, determines a motion estimation value by the motion estimation percentage based on the information of the frequency band power of the space-time direction.

  That is, the motion estimation apparatus of the present invention is a motion estimation apparatus that obtains a motion estimation value of a moving image, performs a spatiotemporal frequency band decomposition process on the processing target image, and calculates a spatiotemporal frequency band power. Frequency band decomposition processing means for calculating information, first motion estimation means for performing hierarchical motion estimation using spatial low frequency image groups of each layer subjected to frequency band decomposition processing, and frequency band decomposition processing A second motion estimation unit that performs hierarchical motion estimation using a horizontal low / vertical high-frequency image group of each layer, and a hierarchical type using a horizontal high / vertical low-frequency image group of each layer subjected to frequency band decomposition processing Third motion estimation means for performing motion estimation, fourth motion estimation means for performing hierarchical motion estimation using a horizontal high / vertical high-frequency image group of each hierarchy subjected to frequency band decomposition processing, and the first Motion estimation means, the second Motion estimation value based on the motion estimation ratio based on the information on the frequency band power of the spatio-temporal for each motion estimation information obtained from the motion estimation means, the third motion estimation means, and the fourth motion estimation means And means for determining.

  The motion estimation apparatus of the present invention is a motion estimation apparatus for obtaining a motion estimation value of a moving image, performing a spatio-temporal frequency band decomposition process on a processing target image, and also calculating a spatio-temporal frequency band power. Frequency band decomposition processing means for calculating information, first motion estimation means for performing hierarchical motion estimation using spatial low frequency image groups of each layer subjected to frequency band decomposition processing, and frequency band decomposition processing Motion estimation using a second motion estimation unit that performs motion estimation using a first layer of horizontal low / vertical high-frequency images and a first layer of horizontal high / vertical low-frequency images subjected to frequency band decomposition processing 3rd motion estimation means for performing motion estimation, 4th motion estimation means for performing motion estimation using a horizontal high / vertical high frequency image group in the first layer subjected to frequency band decomposition processing, and the first motion estimation means , The second motion estimation A motion estimation value is determined by a motion estimation ratio based on the information on the frequency band power of the spatio-temporal for each motion estimation information obtained from the means, the third motion estimation means, and the fourth motion estimation means And means for performing.

  In the motion estimation apparatus of the present invention, the frequency band decomposition processing is octave decomposition processing by wavelet transform.

  In addition, the program of the present invention performs spatio-temporal frequency band decomposition processing on an image to be processed on a computer that is configured as a motion estimation device that obtains a motion estimation value of a moving image, and also calculates spatio-temporal frequency band power. A step of calculating information, a step of performing hierarchical motion estimation using a spatial low-frequency image group of each layer subjected to frequency band decomposition processing, and a horizontal low / vertical high-frequency image of each layer subjected to frequency band decomposition processing Performing hierarchical motion estimation using a group, performing hierarchical motion estimation using a horizontal high / vertical low frequency image group of each layer subjected to frequency band decomposition processing, and applying frequency band decomposition processing A step of performing a hierarchical motion estimation using a horizontal high / vertical high-frequency image group of each layer; the first motion estimation means; the second motion estimation means; and the third motion estimation means. And a step of determining a motion estimation value for each motion estimation information obtained from the fourth motion estimation means based on a motion estimation ratio based on information of the frequency band power of the spatio-temporal space. Configured as a program.

  In addition, the program of the present invention performs spatio-temporal frequency band decomposition processing on an image to be processed on a computer that is configured as a motion estimation device that obtains a motion estimation value of a moving image, and also calculates spatio-temporal frequency band power. A step of calculating information, a step of performing hierarchical motion estimation using a group of spatial low-frequency images subjected to frequency band decomposition processing, and a horizontal low / vertical high frequency of the first layer subjected to frequency band decomposition processing A step of performing motion estimation using an image group, a step of performing motion estimation using a horizontal high / vertical low frequency image group of a first layer subjected to frequency band decomposition processing, and a layer subjected to frequency band decomposition processing A step of performing motion estimation using a horizontal high / vertical high-frequency image group of eyes, the first motion estimation means, the second motion estimation means, the third motion estimation means, and the fourth And a step of determining a motion estimation value based on a motion estimation ratio based on the information on the frequency band power of the spatio-temporal for each motion estimation information obtained from the motion estimation means. .

  According to the present invention, not only hierarchical motion estimation is performed using a spatial low-frequency image group of each layer subjected to frequency band decomposition processing in the spatio-temporal direction, but also motion estimation processing using an image including spatial high-frequency components. Using the frequency band power information in the spatio-temporal direction obtained by the frequency band decomposition process, and motion estimation for motion estimation information obtained from spatial low-frequency images and motion estimation information obtained from images containing spatial high-frequency components It is possible to improve the accuracy of motion estimation by determining the motion estimation value by controlling the ratio.

It is the schematic of the motion estimation apparatus of one Example by this invention. It is the schematic of the block division part in the motion estimation apparatus of one Example by this invention. It is the schematic of the spatio-temporal band spectrum information analysis part in the motion estimation apparatus of one Example by this invention. It is the schematic of the motion estimation rank determination part in the motion estimation apparatus of one Example by this invention. It is the schematic of the motion estimation part in the motion estimation apparatus of one Example by this invention. It is an operation | movement flowchart which shows operation | movement of the motion estimation apparatus of one Example by this invention. It is a figure which shows the block division | segmentation row | line | column in the motion estimation apparatus of one Example by this invention. It is explanatory drawing of the area | region which performs space-time direction frequency decomposition in the motion estimation apparatus of one Example by this invention. It is explanatory drawing of the space-time direction frequency decomposition in the motion estimation apparatus of one Example by this invention. (A), (b) It is explanatory drawing of the two-dimensional 2nd-order discrete wavelet decomposition | disassembly which concerns on the motion estimation apparatus of one Example by this invention. (A), (b), (c), (d) It is explanatory drawing concerning the hierarchical motion detection in the motion estimation apparatus of one Example by this invention. It is explanatory drawing of the block matching method of the decimal pixel position by quadratic function approximation which concerns on the motion estimation apparatus of one Example by this invention.

  Hereinafter, a motion estimation apparatus according to an embodiment of the present invention will be described.

  As a motion estimation apparatus 1 according to an embodiment, a case where octave decomposition processing by wavelet transform is used for frequency analysis in a time direction and a spatial direction will be described. For the frequency analysis in the time direction and the spatial direction, other orthogonal transforms or FFT (Fast Fourier Transform) can be used besides the wavelet transform, but the same processing is repeated as the resolution of the image is reduced. In consideration of performing frequency analysis in the time direction and in the spatial direction, it is particularly advantageous to use octave decomposition processing by wavelet transform because the processing efficiency is improved.

[Device configuration]
FIG. 1 shows a motion estimation apparatus 1 according to an embodiment of the present invention. The motion estimation apparatus 1 according to the present embodiment includes a block division unit 10, a spatio-temporal band spectrum information detection unit 11, a motion estimation rank determination unit 12, and a motion estimation unit 13. Note that image data necessary for processing by each component can be appropriately stored in a storage unit (not shown) included in the motion estimation device 1 and read out.

The block division unit 10 is a functional unit that divides a frame image into blocks, and includes time information t _c of the motion estimation reference frame, information of the frame image F (t _c ) at time t _c, and block size information designated in advance. Bx and By are input, and the frame image F (t _c ) is divided into blocks each having a block size of horizontal Bx pixels × vertical By pixels, and the divided horizontal i-th and vertical j-th block positions B (i, j, t _c ), and a block division information generation unit 101 that outputs block division information (see FIG. 2).

Space-time band spectrum information detection unit 11, a time-position information t _c of the motion estimation reference frame, and time position information t _r motion estimation reference frame, and information of the frame image sequence F (t), block position B (i , J, t _c ) (also expressed as block division information B (i, j, t _c )), motion estimation search range information Sx, Sy, and predetermined maximum hierarchy number information D, frame image F _{(t c)} on the block position _{B (i, j, t c} ) in the block position _{B (i, j, t c} ) from the center ± Sx of the spatial region of ± Sy pixels SA (i, j, t), and the same spatial area SA (i, j, t) of 2 ^D-1 frames before and after the frame image F (t _c ); t _c −2 ^D−1 ≦ t ≦ t _c +2 ^D−1 It is a functional unit that uses space-time power detection.

  More specifically, as shown in FIG. 3, the spatiotemporal band spectrum information detection unit 11 includes a decomposition rank / space region determination unit 111, a one-dimensional D floor wavelet decomposition unit 112, and a spatiotemporal band spectrum information generation unit. 113.

Decomposition rank / space region determination unit 111 receives block position B (i, j, t _c ) information, motion estimation search range information Sx, Sy, and predetermined maximum hierarchy number information D, and decomposes them. An instruction is sent to the one-dimensional D-th floor wavelet decomposition unit 112 to perform octave decomposition at rank D, and ± Sx and ± Sy centered on the block position B (i, j, t _c ) are used as motion search space area information SA ( i, j, t _c ) to the one-dimensional D-th wavelet decomposition unit 112 and the motion estimation rank determination unit 12.

The one-dimensional D-th floor wavelet decomposition unit 112 has the same spatial area SA (i, j, t) of 2 ^D-1 frames before and after the frame image F (t _c ) in the decomposition hierarchy D; t _c −2 ^D−1 ≦ Using t ≦ t _c +2 ^D−1 , one-dimensional D-order wavelet decomposition is performed for each of the time direction, the horizontal direction, and the vertical direction in the input frame image sequence F (t). The frequency component information is sent to the spatio-temporal band spectrum information generation unit 113.

The spatio-temporal band spectrum information generation unit 113 has the same spatial area SA (i, j, t) of 2 ^D-1 frames before and after the frame image F (t _c ); t _c −2 ^D−1 ≦ t ≦ t _c +2 Generate time / horizontal / vertical band spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) in ^{D-1 and} estimate the motion rank The data is sent to the determination unit 12.

The motion estimation rank determining unit 12 includes space spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) in the spatio-temporal direction. Information on the area SA (i, j, t) and the block position B (i, j, t _c ) is input and determined for each spectrum power in the spatiotemporal direction and each spectrum power in the spatiotemporal direction. The spectrum power threshold value is compared to determine whether or not the moving amount of the moving object is large. When it is determined that the moving amount of the moving object is large, two predetermined number of hierarchy setting values (for example, In the case where it is determined that the larger one of the maximum hierarchies D> 2 and the number of hierarchies (one hierarchy) is the motion estimation hierarchy number, and it is determined that the amount of motion of the moving object is small, the two predetermined hierarchy numbers The smaller of the set values is the motion estimation hierarchy It is determined as.

  More specifically, as shown in FIG. 4, the motion estimation rank determination unit 12 includes a comparison control unit 121, comparison units 122, 123, and 124, and a motion estimation hierarchy number determination unit 125.

The comparison control unit 121 inputs information on the space area SA (i, j, t) for motion search and the block position B (i, j, t _c ), and is input to the comparison units 122, 123, and 124, respectively. Spectral power threshold Th _τ (i, j) defined as band spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) ), Th _μ (i, j) and Th _v (i, j) are controlled.

The motion estimation hierarchy number determination unit 125 determines whether or not the moving amount of the moving object is large based on the comparison results by the comparing units 122, 123, and 124, and when it is determined that the moving amount of the moving object is large. Determines the larger of the two predetermined layer number setting values as the motion estimation layer number, and if it is determined that the amount of motion of the moving object is small, the two predetermined layer number setting values The smaller one of them is determined as the number of motion estimation layers, and information on the determined number of motion estimation layers is output as the number of motion estimation layers R (i, j, t _c ).

For example, assuming that the moving area of a moving object existing on the frame image sequence F (t) is MA (i, j, t) and it is not an image that is blurred due to out-of-focus or the like when the moving object is stationary, To determine whether the amount of motion is large or the amount of motion of the moving object is small, band spectrum information P _τ (i, j, t), P _μ (i, j, t) in the spatio-temporal direction , P _v (i, j, t), the power ratios are divided into four cases.
[1. (When the amount of motion of the moving object is large)
(1.1) The motion area MA (i, j, t) of a moving object that is large relative to the spatial area of the spatial region SA (i, j, t) has a high power ratio in the high-frequency band in the time direction, and the space The power ratio of the high frequency band in the direction is high.
(1.2) The motion area MA (i, j, t) of the moving object that is small relative to the spatial area of the spatial region SA (i, j, t) has a low power ratio in the high-frequency band in the time direction, and the space The power ratio in the high frequency band in the direction is low.
[2. (When the amount of motion of the moving object is small)
(2.1) The movement area MA (i, j, t) of a moving object that is large relative to the spatial area of the spatial area SA (i, j, t) has a low power ratio in the high-frequency band in the time direction, and the space The power ratio of the high frequency band in the direction is high.
(2.2) The moving area MA (i, j, t) of the moving object that is small relative to the spatial area of the spatial area SA (i, j, t) has a low power ratio in the high-frequency band in the time direction, and the space The power ratio in the high frequency band in the direction is low.

Therefore, the motion estimation rank determination unit 12 is adapted to the motion patterns (1.1) to (2.2) of the above four types of moving objects, so that the spectrum power threshold Th _τ (i, j) in the time direction is used. By setting the horizontal spectrum power threshold Th _μ (i, j) and the vertical spectrum power threshold Th _v (i, j) as a reference, the movement amount of the moving object can be increased. It is determined whether the amount of movement of the object is small.

The motion estimation rank determining unit 12 determines that each of the spatial spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) in the spatio-temporal direction is a motion pattern. In the case of (1.1) and (1.2), it is determined that the moving amount of the moving object is large, and the larger one of the two predetermined layer number setting values is determined as the motion estimation layer number.

The motion estimation rank determining unit 12 determines that each of the spatial spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) in the spatio-temporal direction is a motion pattern. In the case of (2.1) and (2.2), it is determined that the amount of motion of the moving object is small, and the smaller of the two predetermined layer number setting values is determined as the motion estimation layer number.

In this way, the motion estimation floor determination unit 12 can output the determined motion estimation hierarchy number R (i, j, t _c ).

The motion estimation unit 13 includes horizontal and vertical band spectrum information P _μ (i, j, t), P _v (i, j, t), time position information t _c of the motion estimation reference frame, and motion estimation. and time position information _{t r} of the reference frame, a frame image sequence data F (t), the motion estimation hierarchy number information _{R (i, j, t c} ) and, the block division information _{B (i, j, t c} ) and, Spatial low frequency of each layer reconstructed in the motion search space area SA (i, j, t) for each block B (i, j, t _c ) by inputting information of the predetermined maximum number of layers D In addition to performing hierarchical motion estimation using an image group to obtain motion estimation information in the 0th layer, an image composed of horizontal low-frequency components and vertical high-frequency components in the first layer (first-order wavelet decomposition) (hereinafter, “ Reconstruct the horizontal low / vertical high frequency image) Estimate motion and reconstruct an image consisting of horizontal high-frequency components and vertical low-frequency components (hereinafter referred to as “horizontal high / vertical low-frequency images”) in the first layer (first-order wavelet decomposition) to perform motion estimation. By reconstructing an image composed of horizontal high-frequency components and vertical high-frequency components (hereinafter referred to as “horizontal high / vertical high-frequency images”) in the first layer (first floor wavelet decomposition), four types of motion estimation are performed. The motion estimation information is acquired, and the four types of motion estimation information are classified into four types using the horizontal and vertical band spectrum information P _μ (i, j, t) and P _v (i, j, t). The final motion estimation value is determined by calculating the frequency band power ratio corresponding to the motion estimation information and weighting and adding the four types of motion estimation information according to the frequency band power ratio.

  More specifically, as shown in FIG. 5, the motion estimation unit 13 includes a hierarchical motion estimation unit 131, a spatial hierarchical wavelet reconstruction unit 132, a two-dimensional R-order wavelet decomposition unit 133, and an octave decomposition power component. A calculation unit 134, a power estimation unit 135 for each band, a spatial first-order wavelet reconstruction unit 136, a first-order wavelet decomposition horizontal / vertical high-frequency component extraction unit 137, and a motion vector correction unit 138 are provided.

First, the motion estimation unit 13 first performs a two-dimensional D-order wavelet decomposition process with the maximum number of layers D on the frame image sequences F (t _c ) and F (t _r ) by the two-dimensional R-order wavelet decomposition unit 133. If each layer of the low-frequency image is d = 1,..., D by the octave decomposition power component calculation unit 134, the octave decomposition power components P ^(d) _LL (t _c ), P ^{(d )} _LH (t _c ), P ^(d) _HL (t _c ), P ^(d) _HH (t _c ) and P ^(d) _LL ( _tr ), P ^(d) _LH ( _tr ), P ^{( d)} _HL ( _tr ), P ^(d) _HH ( _tr ) is obtained. Here, LL, LH, HL, and HH are respectively a band component composed of a horizontal low frequency component and a vertical low frequency component, a band component composed of a horizontal low frequency component and a vertical high frequency component, and a horizontal high frequency component and a vertical low frequency component. A band component consisting of a horizontal component and a horizontal high-frequency component / vertical high-frequency component.

Next, the motion estimation unit 13 uses the hierarchical motion estimation unit 131 to perform motion search space area SA (i, j, _tc ) for all blocks B (i, j, t _c ) on the frame image F (t _c ). t) using the spatial image group of each layer reconstructed based on the motion estimation layer number information R (i, j, t _c ) within t), and performing motion vector v ^r (i, j, t _c ) (where r = 0,..., R). Here, for convenience of explanation, the two-dimensional R-order wavelet decomposition unit 133 is described as performing a two-dimensional D-order wavelet decomposition process with the maximum number of layers D. However, since R ≦ D, A two-dimensional R floor wavelet decomposition process may be configured for nesting. r = 0 is a layer corresponding to the frame images F (t _c ) and F (t _r ) that are the frame original images. The spatial hierarchy wavelet reconstruction unit 132 reconstructs the spatial image of each layer using the motion estimation layer number R (i, j, t _c ) and the octave decomposition power component from the octave decomposition power component calculation unit 134. .

For example, the hierarchical motion estimation unit 131 starts motion estimation from the hierarchy r = R. For an arbitrary block B (i, j, t _c ), a reference frame image F based on a block at the same spatial position of the horizontal low / vertical low frequency band components P ^(r) _LL (t _c ) of the R-order wavelet decomposition _{(t r)} R floor wavelet decomposition horizontal low and vertical low frequency band component ^P of the _(r) LL _{(t r)} of the ^{va r-1} (2i shown from the same spatial location in the formula (1), 2j, _{t c} mainly) shifted by location, ± Sx, searching in the spatial domain of ± Sy pixels, ^{va r-1} to the motion estimator obtained (2i, 2j, a plus _{t c)} va ^r (i , J, t _c ). By repeating this until r = 0 while r = r−1, motion estimation information by hierarchical motion estimation is acquired.

^{va r-1 (2i, 2j} , t c) = (v r-1 (2i, 2j, t c)
+ V ^r-1 (2i, 2j-1, t _c )
+ V ^r-1 (2i-1, 2j, t _c )
^{+ V r-1 (2i-} 1,2j-1, t c)) / 4 (1)

Thereby, the motion estimation part 13 can perform the motion estimation of a hierarchy using the spatial low-frequency image group of each hierarchy, and can acquire the motion estimation information in 0th hierarchy. Note that hierarchical motion estimation is performed with a fractional pixel accuracy by a parabolic fitting block matching method. That is, the hierarchical motion estimation unit 131 obtains the motion vector v ⁰ (i, j, t _c ) of the block B (i, j, t _c ) by hierarchical motion estimation using the spatial R-order decomposition low frequency component. .

The motion estimation unit 13 performs hierarchical motion estimation until r = 0 by the hierarchical motion estimation unit 131, and then corrects motion estimation information (motion vector v ⁰ ) obtained by this hierarchical motion estimation. Then, motion estimation by band is performed.

More specifically, the motion estimation unit 13 first performs motion estimation using the horizontal low / vertical high frequency band image information of the first-order wavelet decomposition by the band-specific motion estimation unit 135. The motion estimation unit 135 for each band is based on the block at the same spatial position of the horizontal low / vertical high frequency band component P ⁽¹⁾ _LH (t _c ) of the first-order wavelet decomposition for the block B (i, j, t _c ). From the same spatial position of the horizontal low / vertical high frequency band components P ⁽¹⁾ _LH (t _r ) of the R-order wavelet decomposition of the reference frame image F (t _r ), va ² (2i, 2j, t _c ) Searching in a spatial region of ± Sx, ± Sy pixels centered on a place shifted by a), and adding the obtained motion estimation amount to va ² (2i, 2j, t _c ) is a motion vector v ¹ _LH Obtained as (i, j, t _c ). This motion estimation is performed with a fractional pixel accuracy by a parabolic fitting block matching method. The spatial first-order wavelet reconstruction unit 136 reconstructs the spatial image of the first layer using the motion estimation layer number R (i, j, t _c ) and the octave decomposition power component from the octave decomposition power component calculation unit 134. Configure.

Further, the motion estimation unit for each band 135 similarly performs motion estimation using the horizontal high / vertical low frequency band image information of the first-order wavelet decomposition, and obtains a motion vector v ¹ _HL (i, j, t _c ). In addition, similarly, motion estimation is performed using the horizontal high / vertical high frequency band image information of the first-order wavelet decomposition to obtain a motion vector v ¹ _HH (i, j, t _c ). In any case, this motion estimation is performed with a fractional pixel accuracy by a parabolic fitting block matching method.

The first-order wavelet decomposition horizontal / vertical high-frequency component extraction unit 137 receives horizontal and vertical band spectrum information P _μ (i, j, t) and P _v (i, j, t) and inputs 1 The horizontal high-frequency component P ¹ _μ and the vertical high-frequency component P ¹ _v of the floor wavelet decomposition are extracted.

The motion vector correction unit 138 sets the previously calculated motion vector v ⁰ (i, j, t _c ) as the horizontal motion amount v _x and the vertical motion amount as v _y, and calculates the calculated motion vector v ¹ _LH (i , J, t _c ), v ¹ _HL (i, j, t _c ), v ¹ _HH (i, j, t _c ), horizontal high frequency component P ¹ _μ and vertical high frequency component P ¹ _v , According to (2), the motion vector v ⁰ (i, j, t _c ) is finely corrected to calculate motion estimation information v ⁰ _ADJ (i, j, t _c ).

In equation (2), the weighting factor for adjustment according to the rate of movement in the horizontal and vertical directions is multiplied by the amount of motion blur in the direction in which the moving object is moving. This is to adjust the weight ratio in consideration. Further, from the horizontal high-frequency component P ¹ _μ and the vertical high-frequency component P ¹ _v of the first-order wavelet decomposition in the band spectrum information P _μ (i, j, t) and P _v (i, j, t) in the horizontal and vertical directions. The reason for calculating the ratio of the high-frequency band power in the horizontal direction and the vertical direction is to adjust the weight ratio in consideration of the direction of the high-frequency band power because the amount of motion blur in the direction in which the high-frequency band power is high is small.

  As described above, the motion estimation apparatus 1 according to the present embodiment not only performs the hierarchical motion estimation using the spatial low-frequency image group of each layer subjected to the frequency band decomposition process in the spatio-temporal direction, but also the spatial high-frequency component. An image containing motion estimation information and spatial high-frequency components obtained from spatial low-frequency images using information on frequency-band power in the spatio-temporal direction obtained by frequency-band decomposition processing. The accuracy of motion estimation is improved by determining the motion estimation value by controlling the motion estimation ratio with respect to the motion estimation information obtained in (1).

  Hereinafter, the operation of the motion estimation apparatus 1 according to an embodiment of the present invention will be described in more detail.

[Device operation]
FIG. 6 is an operation flow showing the operation of the motion estimation apparatus of one embodiment according to the present invention. In step S1, the motion estimation device 1 includes the frame image sequence F (t ₀ ) at time t = t ₀ ... T _m including the base frame F (t _c ) and the reference frame F (t _r ) _,. .., F (t _c ),..., F (t _r ),..., F (t _m ) can be input and read as appropriate to a storage unit (not shown) included in the motion estimation device 1. To store.

In step S2, the block division unit 10 performs block division of the reference frame F (t _c ) (see FIG. 7).

In step S3, the spatio-temporal band spectrum information detection unit 11 obtains block position B (i, j, t _c ) information, motion estimation search range information Sx, Sy, and predetermined maximum hierarchy number information D. Then, ± Sx and ± Sy centered on the block position B (i, j, t _c ) are determined as motion search space area information SA (i, j, t _c ) (see FIG. 8). As shown in FIG. 8, the same spatial area SA (i, j, t) of 2 ^D-1 frames before and after the frame image F (t _c ); t _c −2 ^D−1 ≦ t ≦ t _c +2 ^D−1 Is determined.

In step S4, the spatio-temporal band spectrum information detection unit 11 performs one-dimensional D-order wavelet decomposition for each of the time direction, the horizontal direction, and the vertical direction in the input frame image sequence F (t) to obtain the frame image F ( The same spatial area SA (i, j, t) of 2 ^D-1 frames before and after t _c ); t _c −2 ^D−1 ≦ t ≦ t _c +2 Band spectrum information in time, horizontal and vertical directions at ^D−1 P _τ (i, j, t), P _μ (i, j, t), and P _v (i, j, t) are generated (see FIG. 9).

The spatio-temporal band spectrum information detection unit 11 is a space in the frame image sequence F (t ₀ ), ..., F (t _c ), ..., F (t _r ), ..., F (t _m ). After the area SA (i, j, t) is decomposed into a frequency area of the maximum rank (D floor) defined in advance in the spatio-temporal direction, the power for each spatio-temporal frequency band in all pixels can be calculated. For example, as shown in FIG. 9, if a frame image sequence F (t) of 2 ^D-1 frames before and after is decomposed into the D floor in the spatio-temporal direction, when D = 1, the low frequency region L ¹ and the high frequency region H ¹ (see FIG. 9A), and when D = 2, it can be divided into the low frequency region L ² and the high frequency regions H ¹ and H ² (see FIG. 9B). = 3 can be divided into a low frequency region L ³ and high frequency regions H ¹ , H ² , H ³ (see FIG. 9C), and when D = 4, the low frequency region L ⁴ and the high frequency region H ¹ can be divided. , H ² , H ³ , and H ⁴ (see FIG. 9D).

In step S5, the motion estimation rank determining unit 12 performs band spectrum information P _τ (i, j, t), P _μ (i, j, t), P _v (i, j, t) in the spatio-temporal direction. , Information on motion search space area SA (i, j, t) and block position B (i, j, t _c ) is input, and each spectrum power in the spatio-temporal direction and each spectrum power in the spatio-temporal direction are input. If the moving amount of the moving object is determined to be large by comparing the determined spectrum power thresholds with each other, and if it is determined that the moving amount of the moving object is large, two predetermined number of layers The larger one of the set values (for example, the maximum number of hierarchies D> 1 and the number of two hierarchies) is determined as the number of motion estimation hierarchies. Of the two hierarchy number setting values To determine the differences in how as motion estimation hierarchy number.

In step S6, the motion estimation unit 13, and the time-position information t _c of the motion estimation reference frame, and time position information t _r motion estimation reference frame, a frame image sequence data F (t), the motion estimation hierarchy number information Based on R (i, j, t _c ) and block division information B (i, j, t _c ), a reconstructed image with spatial R-order decomposition low-frequency components is used to generate the highest layer (ie, the original hierarchy). The motion level is sequentially decremented until it reaches the image level (step S7), block matching motion vector detection is performed at the decimal pixel position, and motion vector v ⁰ (i, j, t _c ) is obtained. Can be determined (step S8).

For example, as shown in FIG. 10A, a power value for each frequency band is calculated by performing a two-dimensional second-order discrete wavelet decomposition in the spatial direction on all the pixels of the reference frame F (t _c ). can be, as shown in FIG. 10 (b), the low-frequency region of the spatial direction of the reference frame F _{(t c)} (e.g., LL ²⁾ low-frequency region only extracted by the reference frame F _{(t c)} only Can be reconstructed.

That is, as shown in FIG. 11, hierarchical motion detection can be performed using a reconstructed image based on spatial R-order decomposition low frequency components. For example, when R = 4, the motion detection start rank n _s = 4 can be used to reconstruct an image of only the four-layer low-frequency region (see FIG. 11D). In addition, it is possible to reconstruct an image of only three layers of low-frequency regions (in this case, LL ³ ) with motion detection start rank n _s = 3 (see FIG. 11C). Similarly, it is possible to reconstruct an image of only two layers of low-frequency regions (in this case, LL ² ) with the motion detection start rank n _s = 2 (see FIG. 11B), and the motion detection start rank n _s. = 1, it is possible to reconstruct an image of only one layer of a low frequency region (in this case, LL ¹ ).

  It is preferable to use the block matching method of decimal pixel positions by quadratic function approximation only for the highest rank (that is, the first floor), which is given by equation (3) (FIG. 12). reference).

   When the pixel position at the search position is x, SSD (x) represents the SSD value (sum of squares of error) at the pixel position. More specifically, SSD (0) is the SSD value at the center position, SSD (−1) represents the SSD value at the position of −Sx (Sy) pixel from the center position, and SSD (1) represents the SSD value at the position of + Sx (Sy) pixel from the center position. From equation (1), pixel positions (decimal pixel positions) with decimal pixel precision in the horizontal or vertical direction can be calculated respectively. For example, in the example of FIG. 12, a quadratic function approximation can be performed from Equation (3) to obtain, for example, −0.33 as the decimal pixel position.

In step S9, a spatial low-frequency image group of each layer reconstructed in the motion search space area SA (i, j, t) by the motion estimation unit 13 for each block B (i, j, t _c ). It is used to perform hierarchical motion estimation to obtain motion estimation information in the 0th layer, and further, an image composed of horizontal low frequency components and vertical high frequency components in the first layer (first floor wavelet decomposition) ("horizontal low / vertical high frequency Reconstruct the image "), perform motion estimation, and reconstruct the image consisting of the horizontal high-frequency component and vertical low-frequency component (" horizontal high / vertical low-frequency image ") in the first layer (first floor wavelet decomposition) A motion vector v is obtained by performing motion estimation by reconstructing an image (“horizontal high / vertical high-frequency image”) composed of horizontal high-frequency components and vertical high-frequency components in the first layer (first-order wavelet decomposition). ¹ _LH (i, j, t _c ), v ¹ _HL (i, j, t _c ), v ¹ _HH (i, j, t _c ) are calculated.

In step S10, the motion estimator 13 uses the horizontal and vertical band spectrum information P _μ (i, j, t), P _v (i, j, t) to generate the horizontal high-frequency component P of the first-order wavelet decomposition. ¹ _μ and the vertical high-frequency component P ¹ _v are extracted.

In step S11, the motion estimation unit 13 causes the previously calculated motion vector v ⁰ (i, j, t _c ) and the already calculated motion vector v ¹ _LH (i, j, t _c ), v ¹ _HL (i , J, t _c ), v ¹ _HH (i, j, t _c ), horizontal high-frequency component P ¹ _μ, and vertical high-frequency component P ¹ _v , according to equation (2), motion vector v ⁰ (i, The motion vector obtained by finely correcting j, t _c ), that is, motion estimation information v ⁰ _ADJ (i, j, t _c ) is calculated.

  By applying the motion vector obtained with high accuracy and efficiency in this way to encoding processing and super-resolution processing of an existing encoding device, further improvement in quality can be expected.

  In the above-described embodiment, the frequency band decomposition processing is performed on the motion vector obtained by performing the hierarchical motion estimation using the spatial low-frequency image group of each layer subjected to the frequency band decomposition processing. Motion estimation is performed using the motion vector obtained by performing the motion estimation using the horizontal low / vertical high-frequency image of the hierarchy and the horizontal high / vertical low-frequency image of the first hierarchy subjected to the frequency band decomposition processing. Spatio-temporal frequency band power information using the obtained motion vector and the motion vector obtained by performing motion estimation using the first-level horizontal high / vertical high-frequency image subjected to frequency band decomposition processing. An example of correction based on the above has been described. The present invention is not limited to this, and the frequency band decomposition processing is performed on the motion vector obtained by performing the hierarchical motion estimation using the spatial low-frequency image group of each layer subjected to the frequency band decomposition processing. Hierarchy using the motion vector obtained by performing hierarchical motion estimation using the horizontal low / vertical high-frequency image group of each layer and the horizontal high / vertical low-frequency image group of each layer subjected to frequency band decomposition processing Using the motion vector obtained by performing the motion estimation and the motion vector obtained by performing the hierarchical motion estimation using the horizontal high / vertical high frequency image group of each layer subjected to the frequency band decomposition process, It is also possible to make corrections based on information on the spatio-temporal frequency band power.

  Furthermore, as one aspect of the present invention, the motion estimation apparatus of the present invention can be configured as a computer. A program for causing a computer to realize each component of the motion estimation device of the present invention described above is stored in a storage unit provided inside or outside the computer. Such a storage unit can be realized by an external storage device such as an external hard disk or an internal storage device such as ROM or RAM. The control unit provided in the computer can be realized by controlling a central processing unit (CPU) or the like. In other words, the CPU can appropriately read from the storage unit a program in which the processing content for realizing the function of each component is described, and realize the function of each component on the computer. Here, the function of each component may be realized by a part of hardware.

  In addition, the program describing the processing contents can be distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM, and such a program can be distributed on a server on a network, for example. Can be distributed by transferring the program from the server to another computer via the network.

  In addition, a computer that executes such a program can temporarily store, for example, a program recorded on a portable recording medium or a program transferred from a server in its own storage unit. As another embodiment of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and each time the program is transferred from the server to the computer. In addition, the processing according to the received program may be executed sequentially.

  While the embodiments of the present invention have been described in detail with specific examples, it will be apparent to those skilled in the art that various modifications and changes can be made without departing from the scope of the claims of the present invention.

  According to the present invention, the present invention can be applied to any technique that uses motion estimation of moving images, such as frame interpolation, image coding, and image super-resolution processing.

DESCRIPTION OF SYMBOLS 1 Motion estimation apparatus 10 Block division part 11 Spatio-temporal band spectrum information detection part 12 Motion estimation rank determination part 13 Motion estimation part 101 Block division | segmentation information generation part 111 Decomposition rank / space area determination part 112 1-dimensional D floor wavelet decomposition part 113 Spatial band spectrum information generation unit 121 Comparison control unit 122, 123, 124 Comparison unit 125 Motion estimation layer number determination unit 131 Hierarchical motion estimation unit 132 Spatial layer wavelet reconstruction unit 133 Two-dimensional R-order wavelet decomposition unit 134 Octave decomposition power component Calculation unit 135 Band-specific power estimation unit 136 Spatial first-order wavelet reconstruction unit 137 First-order wavelet decomposition horizontal / vertical high-frequency component extraction unit 138 Motion vector correction unit

Claims (5)

A motion estimation device for obtaining a motion estimation value of a moving image,
A frequency band decomposition processing unit that performs spatio-temporal frequency band decomposition processing on an image to be processed, and calculates information of spatio-temporal frequency band power;
First motion estimation means for performing hierarchical motion estimation using a spatial low-frequency image group of each layer subjected to frequency band decomposition processing;
Second motion estimation means for performing hierarchical motion estimation using a horizontal low / vertical high frequency image group of each layer subjected to frequency band decomposition processing;
Third motion estimation means for performing hierarchical motion estimation using a horizontal high / vertical low frequency image group of each layer subjected to frequency band decomposition processing;
Fourth motion estimation means for performing hierarchical motion estimation using horizontal and vertical high-frequency image groups of each layer subjected to frequency band decomposition processing;
For each motion estimation information obtained from the first motion estimation means, the second motion estimation means, the third motion estimation means, and the fourth motion estimation means, the spatio-temporal frequency band power Means for determining a motion estimation value by a motion estimation ratio based on the information of
A motion estimation device comprising: A motion estimation device for obtaining a motion estimation value of a moving image,
A frequency band decomposition processing unit that performs spatio-temporal frequency band decomposition processing on an image to be processed, and calculates information of spatio-temporal frequency band power;
First motion estimation means for performing hierarchical motion estimation using a spatial low-frequency image group of each layer subjected to frequency band decomposition processing;
Second motion estimation means for performing motion estimation using a horizontal low / vertical high frequency image group in the first layer subjected to frequency band decomposition processing;
A third motion estimator that performs motion estimation using a first-level horizontal high / vertical low-frequency image group subjected to frequency band decomposition processing;
A fourth motion estimation means for performing motion estimation using a first-level horizontal high / vertical high-frequency image group subjected to frequency band decomposition processing;
For each motion estimation information obtained from the first motion estimation means, the second motion estimation means, the third motion estimation means, and the fourth motion estimation means, the spatio-temporal frequency band power Means for determining a motion estimation value by a motion estimation ratio based on the information of
A motion estimation device comprising:   The motion estimation apparatus according to claim 1, wherein the frequency band decomposition process is an octave decomposition process using a wavelet transform. In a computer configured as a motion estimation device for obtaining a motion estimation value of a moving image,
Performing a spatio-temporal frequency band decomposition process on the image to be processed and calculating spatio-temporal frequency band power information;
Performing hierarchical motion estimation using spatial low-frequency image groups of each layer subjected to frequency band decomposition processing;
Performing hierarchical motion estimation using a horizontal low / vertical high frequency image group of each layer subjected to frequency band decomposition processing;
Performing hierarchical motion estimation using a horizontal high / vertical low frequency image group of each layer subjected to frequency band decomposition processing;
Performing hierarchical motion estimation using horizontal and vertical high-frequency image groups of each layer subjected to frequency band decomposition processing;
For each motion estimation information obtained from the first motion estimation means, the second motion estimation means, the third motion estimation means, and the fourth motion estimation means, the spatio-temporal frequency band power Determining a motion estimation value based on a motion estimation ratio based on the information of;
A program for running In a computer configured as a motion estimation device for obtaining a motion estimation value of a moving image,
Performing a spatio-temporal frequency band decomposition process on the image to be processed and calculating spatio-temporal frequency band power information;
Performing hierarchical motion estimation using spatial low-frequency image groups of each layer subjected to frequency band decomposition processing;
Performing motion estimation using a first layer horizontal low / vertical high frequency image group subjected to frequency band decomposition processing;
Performing motion estimation using a first-level horizontal high / vertical low-frequency image group subjected to frequency band decomposition processing;
Performing motion estimation using a horizontal high / vertical high frequency image group in the first layer subjected to frequency band decomposition processing;
For each motion estimation information obtained from the first motion estimation means, the second motion estimation means, the third motion estimation means, and the fourth motion estimation means, the spatio-temporal frequency band power Determining a motion estimation value based on a motion estimation ratio based on the information of;
A program for running

Images