JPH10111945A - Moving amount estimation device for dynamic image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate the moving amounts which are continuous in terms of time with reflection of continuity of the time direction. SOLUTION: This device consists of a rectangular parallelepiped block extraction circuit 1 which extracts a 1st rectangular parallelepiped block of a space position (x, y) where the n-th frame is defined as the center of a time position and plural 2nd rectangular parallelepiped blocks having the same size as the 1st rectangular parallelepiped block where every (n-1)-th frame is defined as the center of a time position and every space position (x, y) is displaced by a pixel (Δx, Δy) from the input moving image signals respectively, a matching evaluation function value arithmetic circuit 2 which calculates a difference of the corresponding pixel value between the 1st and 2nd rectangular parallelepiped blocks and outputs the matching evaluation function value for every 2nd rectangular parallelepiped block, and a motion estimation circuit 3 which estimates the motion vector of a two-dimensional block of the n-th frame based on the pixel displacement (Δs, Δy) of the 2nd rectangular parallelepiped block that secures the least evaluation function value and outputs the motion vector.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for estimating the amount of motion of a moving image, and more particularly to an apparatus for estimating the amount of motion of a moving image for estimating the amount of motion for each block or pixel on the moving image.

[0002]

2. Description of the Related Art Conventionally, this kind of motion image motion estimation apparatus is roughly classified into an apparatus using a gradient method and an apparatus using a block matching method. The method using the gradient method estimates the amount of motion from the spatiotemporal luminance gradient of each pixel in the image block, on the assumption that all pixels in the image block of a predetermined size have the same amount of motion. It is a technique to do. (For example, blowout “Measurement of moving amount and speed of moving object by image signal”, IEICE Technical Report, IE78-67, pp. 146-64).
35-41 (1978)).

[0003] This gradient method can estimate a good motion amount for a small motion amount, but it is often pointed out that the estimation accuracy of the motion amount estimation deteriorates as the motion amount increases. . The calculation of the gradient value in the gradient method is as follows.
The motion estimation at time t _0, a past frame adjacent thereto and frame at time t ₀ or one frame before and after, the use of the two frames together,
There is also a disadvantage that they are susceptible to random noise contained in these frame images.

On the other hand, the method using the block matching method is a method of detecting the position of the image block in which the sum of the pixel values is minimized in the image of the previous frame with respect to the image block of the current frame. (For example, Ninomiya “Motion Correction in Interframe Coding”, IEICE Technical Report, IE78-6, p.
p. 1-10 (1978)).

As an application of the block matching method, there is a method of improving the accuracy of motion detection by hierarchically reducing the block size. (For example, Tominaga et al., “Motion Amount Detection Method in Moving Image Using Hierarchical Image Information”, IEICE, Vol. J72-D-II, No. 3, pp. 3)
95-403 (1989)). In the motion estimation using the hierarchical image information according to this method, the estimation error can be reduced and the processing efficiency can be improved by using the estimation result of the motion amount of the upper hierarchical image for the estimation in the lower hierarchical image. . However, the above method uses a square block in one of two temporally consecutive frames as a matching block, similar to a general block matching method, and considers the spatial continuity of motion. However, temporal continuity is not used.

As an attempt to use the temporal continuity of the amount of motion, when estimating the amount of motion of the image of the n-th frame at time t ₀ , the estimated value of the previous (n−1) -th frame is set to the initial value. Then, there is a filtering method for sequentially correcting the amount of motion as time passes. (For example, James C. B
railean, et al. “A Recursive Nonstationary MAPDis
placement Vector Field Estimation Algorithm ”, IEE
E Transactions on Image Processing, Vol. 4, No.
4, April 1995). However, according to this method, the previous frame (for example, the n-th frame is
When an estimation error occurs in the estimation of the motion amount in (−1 frame), the estimation error propagates to subsequent frames one after another, and may degrade the estimation accuracy of the motion amount in spite of expectations.

[0007]

As described above, in the prior art, the motion amount is estimated using images of two temporally continuous frames. As described above, in the prior art, since the temporally continuous motion amount is estimated as completely independent, the estimation result does not reflect the continuity in the time direction, and a random estimation error may occur. There's a problem. In particular, in order to obtain a motion vector field having a high spatial resolution, it is necessary to reduce the block size. In this case, random estimation errors are more likely to occur.

[0008]

However, in a general moving image, the movement of an object or a background in an image has temporal continuity. Due to this continuity, the movement (direction, Size) can be assumed to be constant.

In view of this fact, in the present invention, instead of the conventional two-dimensional block in one frame, pixels included in a rectangular parallelepiped on a spatiotemporal space composed of two-dimensional blocks in the same spatial position in a plurality of frames are used. To estimate the amount of motion, that is, to estimate the amount of motion by a matching method using a rectangular parallelepiped (three-dimensional) block. This allows
The motion estimation is less susceptible to noise in a two-dimensional block of a specific frame, reduces temporal and spatial random estimation errors, and enables more accurate and smooth motion estimation. It becomes possible.

In other words, the moving picture motion amount estimating apparatus according to the present invention includes a first rectangular parallelepiped block at a spatial position (x, y) centered on the n-th frame and the same as the first rectangular parallelepiped block. And the n-th frame is the center of the time position, and the spatial position is shifted by (Δx, Δy) pixels with respect to the (x, y) using Δx and Δy as parameters. A rectangular parallelepiped block extraction circuit for extracting a plurality of second rectangular parallelepiped blocks positioned from the input video signal, respectively, between the first rectangular parallelepiped block extracted from the rectangular parallelepiped block extraction circuit and the plurality of second rectangular parallelepiped blocks Performs a difference operation between the corresponding pixel values and a calculation of the sum of the difference operation results, and outputs a matching evaluation function value for each of the second rectangular parallelepiped blocks. Quenching evaluation function value calculation circuit, and one of the matching evaluation function value outputted from the matching evaluation function value calculation circuit, the pixel deviation ([Delta] x of the second rectangular solid block providing the minimum evaluation function value, delta
y)

(Equation 3) Is estimated to be a motion vector of a two-dimensional block of the n-th frame located at the center of the first rectangular parallelepiped block of the moving image, and is output.

In the motion picture motion amount estimating apparatus according to the present invention, the first rectangular parallelepiped block at the spatial position (x, y) whose n-th frame is the center of the time position is the same as the first rectangular parallelepiped block. Is a rectangular parallelepiped block having a size of (x, y), where the (n−1) th and (n + 1) th frames are the centers of time positions, and the spatial position is (x, y) with respect to the (x, y) using Δx and Δy as parameters. ,
A rectangular parallelepiped block extraction circuit for extracting three types of rectangular parallelepiped blocks each composed of a second and a third rectangular parallelepiped block deviated by (−Δx, −Δy) pixels from an input video signal, and the rectangular parallelepiped block extraction circuit Calculating the difference between each corresponding pixel value between the first rectangular parallelepiped block and the plurality of second rectangular parallelepiped blocks, and between the first rectangular parallelepiped block and the plurality of third rectangular parallelepiped blocks, respectively. A first and a second matching evaluation function value calculation circuit for performing a calculation for summing up the difference calculation results and outputting a matching evaluation function value for each of the second and third rectangular parallelepiped blocks;
And the pixel deviations (Δx, Δy), (−x) of the second and third rectangular parallelepiped blocks that provide the minimum evaluation function value among the matching evaluation function values output from the second matching evaluation function value calculation circuit Δx, -Δy)

(Equation 4) Is estimated to be a motion vector of a two-dimensional block of the n-th frame located at the center of the first rectangular parallelepiped block of the moving image, and is output.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below based on embodiments of the present invention with reference to the accompanying drawings. FIG.
FIG. 1 is a block diagram showing a first embodiment of a motion estimation device for a moving image according to the present invention. In FIG. 1, reference numeral 1 denotes a rectangular parallelepiped block extraction circuit to which a moving image signal to be processed and position / size information of an image block described below are supplied, 2 a matching evaluation function value operation circuit, and 3
Is a motion estimation circuit.

The operation will be described. In the apparatus of the present invention, as a first step, a moving image signal whose motion amount is to be estimated is supplied to the rectangular parallelepiped block extracting circuit 1 and, from the moving image signal, supplied to the rectangular parallelepiped block extracting circuit 1 as well. A rectangular parallelepiped block is extracted at the two-dimensional position (x, y) specified as information and at the size M × N × (2K + 1) specified as block size information. This rectangular parallelepiped block of size M × N × (2K + 1) is composed of a total of 2K + 1 frames in the same space (two-dimensional) position (x, y) on the screen in the vertical direction on the screen, with the frame at a certain time as the center. M,
It is composed of pixels included in a rectangular parallelepiped composed of blocks having N pixels (hereinafter, this rectangular parallelepiped region is referred to as Φ).

FIG. 2 is a diagram showing a rectangular parallelepiped block when K = 2 as an example of a rectangular parallelepiped block of size M × N × (2K + 1). In the figure, a rectangular parallelepiped box surrounded by a broken line is a rectangular parallelepiped block. Further, in the present embodiment (the embodiment shown in FIG. 1), M × N that defines the range of the two-dimensional position in the rectangular parallelepiped block is set to M = N = 5 (for K, as shown in FIG. 2). K = 2) will be described. The values of M, N, and K are not limited to the above, and may be any positive integer of 1 or more.

In the apparatus of the present invention, the amount of motion is estimated by performing block matching (the method of block matching has been described in the section of the prior art) in units of this rectangular parallelepiped block. Also, in the present embodiment shown in FIGS. 1 and 2, 2
It is assumed that the motion amount is estimated based on two rectangular parallelepiped blocks. Here, the values of M, N, and K of the rectangular parallelepiped block are 5, 5, and 2, respectively, as described above.

That is, as a first stage in the present embodiment, referring to FIG. 2, in this case, the rectangular parallelepiped block extraction circuit 1 determines whether the supplied moving images have different n-2, n-1 and n-1 From the images of the n, (n + 1) th and (n + 2) th frames, a first cuboid block of 5 pixels × 5 pixels × 5 frames at a spatial position (x, y) centered on the nth frame is extracted, and The same as the first rectangular parallelepiped block located at the spatial position (x + Δx, y + Δy) from the images of the (n-2) -th, (n−1) -th, and (n + 1) -th frames centering on the (n−1) -th frame and using Δx and Δy as parameters. A second rectangular parallelepiped block having a size is extracted.

As a second stage of the present invention, in the matching evaluation function operation circuit 2, the rectangular parallelepiped block extraction circuit 1
, A first rectangular parallelepiped block Φ at a spatial position (x, y) centered on the n-th frame, and a spatial position centered on the (n-1) th frame, and the spatial position is calculated from the position (x, y) by ( [Delta] x, [Delta] y) The difference sum of each pixel value is calculated with the second rectangular parallelepiped block deviated by pixels. The sum of the differences between the pixel values is calculated, for example, between the upper left pixel (out of the 5 × 5 pixels in the frame) of the n−2 frame of the first rectangular parallelepiped block and the n−3 frame of the second rectangular parallelepiped block. Similarly, the difference between the upper left pixel (of the same 5 × 5 pixels) is calculated, and thereafter, the difference between the next pixel (for example, the upper right pixel on the right) is calculated. , All pixels (all possible pixels of a rectangular parallelepiped block are M × N × (2K +
1) At the point in time when the differences for (= 5 × 5 × 5) are found, the sum of these differences is found. Hereinafter, description will be made using mathematical expressions.

The calculation method in this embodiment will be described below. For a rectangular parallelepiped block Φ, an evaluation function defined by the following equation for a plurality of rectangular blocks having the same size in a predetermined search range and having the (n-1) th frame as a center frame

(Equation 5) Is calculated by the following equation (1), and the evaluation function value of the operation result is output.

(Equation 6) here,

(Equation 7) Represents a pixel value in a rectangular parallelepiped block Φ centered on the n-th frame. K is 2K + which constitutes a rectangular parallelepiped block.
Indicates the frame number of one frame. In the present embodiment, k = n−2, n−1, n, n + 1, and n + 2. Also, Δx, Δy∈−L, L, where −
L and L are search ranges for motion amount estimation. For a general moving image, it is sufficient to set L to about 14 pixels.

Further, as a third stage, the motion estimating circuit 3 calculates a plurality of

(Equation 8) Is used to estimate the motion vector. The estimation method in the present embodiment is shown in the following equation (2).

(Equation 9) That is, one set of values that gives the smallest matching evaluation function value in the search range

(Equation 10) Of the n-th frame located at the center of the rectangular parallelepiped block Φ
It is estimated that the motion vector is a motion vector of a dimensional block (a two-dimensional block indicated by a dotted surface in FIG. 2).

In the first embodiment, the motion amount (motion vector) is estimated using two temporally continuous rectangular parallelepiped blocks. However, in order to further reduce the estimation error in the motion amount estimation. , Three consecutive in time
A second embodiment of the present invention in which a motion amount is estimated using two rectangular parallelepiped blocks will be described below.

FIG. 3 is a block diagram showing a second embodiment of the moving picture motion amount estimating apparatus according to the present invention. In FIG. 3, reference numeral 4 denotes a rectangular parallelepiped block extraction circuit, 5 denotes a matching evaluation function value operation circuit (I), 6 denotes a matching evaluation function value operation circuit (II), and 7 denotes a motion estimation circuit.

The most different point of the present embodiment from the first embodiment is that, as can be seen from FIG. 3, the device according to the second embodiment includes one frame among three temporally continuous rectangular parallelepiped blocks. 2 that are temporally consecutive with a period shift
Two matching evaluation function value calculation circuits (I) for calculating a matching evaluation function value between two rectangular parallelepiped blocks
And (II) (shown in FIG. 3 as 5 and 6, respectively)
It is equipped with. Hereinafter, as in the case of the first embodiment, the operation of the present invention will be described by dividing it into first to third stages.

First, as a first step, a rectangular parallelepiped block is extracted by a rectangular parallelepiped block extracting circuit 4.
The different n-2, n-1, n-th,
From the images of the (n + 1) th and (n + 2) th frames, a rectangular parallelepiped block (first rectangular parallelepiped block) of 5 pixels × 5 pixels × 5 frames at a spatial position (x, y) centering on the nth frame, the nth, and the nth −2, n−1, nth and nth
From the image of the (+1) th frame to the (n-1) th frame
The spatial position (x + Δx,
y + Δy) and the spatial position (x−Δx, y) centered on the (n + 1) th frame from the images of the (n−1) th, (n) th, (n + 1) th, (n + 2) th and (n + 3) th frames. −Δy), each of which extracts a cuboid block (third cuboid block). Note that all of the extracted first to third rectangular parallelepiped blocks have the same size (the same size). In the second embodiment, motion estimation is performed by performing block matching using the three rectangular parallelepiped blocks.

In the second stage, the matching evaluation function value calculation circuits 5 and 6 are provided with the first rectangular parallelepiped block at the spatial position (x, y) centered on the n-th frame extracted by the rectangular parallelepiped block extraction circuit 4. Φ and n-th
-1 and the (n + 1) th frame as the center, and the spatial position is (Δx, Δy), (−Δx, −) from the position (x, y).
Δy) The difference sum of each pixel value is calculated between the second and third rectangular parallelepiped blocks displaced by pixels.

The calculation method in the present embodiment will be described below. For a rectangular parallelepiped block Φ, an evaluation function defined by the following equation for a plurality of rectangular blocks having the same size within the search range for each of two rectangular blocks having n-1 and n + 1 frames as a center frame

[Equation 11] And the evaluation function defined by

(Equation 12) Is calculated by the following equations (3) and (4), respectively.
Output the evaluation function value of the operation result.

(Equation 13)

As a third stage in the second embodiment, the motion estimating circuit 7 calculates the calculation results of the matching evaluation function value calculating circuits 5 and 6 respectively.

[Equation 14] and

(Equation 15) Is used to estimate the motion vector. The estimation method in the present embodiment is shown in the following equation (5).

(Equation 16) That is, one set of values that gives the smallest matching evaluation function value in the search range

[Equation 17] Of the n-th frame located at the center of the rectangular parallelepiped block Φ
It is estimated to be a motion vector of a dimensional block.

[0027]

As described above, the motion estimation apparatus according to the present invention, which is constructed by using a plurality of frames in consideration of the continuity of the motion in the time direction, employs two rectangular parallelepiped blocks. Estimation errors are reduced, and more accurate motion vectors can be obtained.

Further, by using three temporally continuous rectangular parallelepiped blocks, it is possible to further reduce errors in the estimation of the amount of motion, and to more accurately and smoothly estimate the amount of motion.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of a motion estimation device for a moving image according to the present invention.

FIG. 2 is a diagram showing a rectangular parallelepiped block when K = 2.

FIG. 3 is a block diagram showing a second embodiment of a motion image motion estimation device according to the present invention.

[Explanation of symbols]

 1, 4 rectangular parallelepiped block extraction circuit 2 matching evaluation function value calculation circuit 3, 7 motion estimation circuit 5 matching evaluation function value calculation circuit I 6 matching evaluation function value calculation circuit II

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Shinichi Sakai 1-10-11 Kinuta, Setagaya-ku, Tokyo Inside the Japan Broadcasting Corporation Research Institute (72) Inventor Yutaka Tanaka 1-10-11 Kinuta, Setagaya-ku, Tokyo No. Japan Broadcasting Corporation Broadcasting Research Institute

Claims (2)

[Claims] 1. A first rectangular parallelepiped block at a spatial position (x, y) centered on an n-th frame and a rectangular parallelepiped block having the same size as the first rectangular parallelepiped block, A plurality of second rectangular parallelepiped blocks deviated by (Δx, Δy) pixels with respect to the (x, y) using the Δx and Δy as parameters with respect to the (x, y), respectively. A rectangular parallelepiped block extraction circuit that extracts from an input moving image signal; a difference calculation between corresponding pixel values between the first rectangular block and the plurality of second rectangular blocks extracted from the rectangular block extraction circuit; A matching evaluation function value calculation circuit for performing a calculation for summing the difference calculation results and outputting a matching evaluation function value for each of the second rectangular parallelepiped blocks; Among the matching evaluation function values output from the valence function value calculation circuit, the value was obtained from the pixel deviation (Δx, Δy) of the second rectangular parallelepiped block that gives the minimum evaluation function value. And a motion estimating circuit for estimating and outputting a motion vector of a two-dimensional block of an n-th frame located at the center of the first rectangular parallelepiped block of the moving image. Quantity estimation device. 2. A first rectangular parallelepiped block at a spatial position (x, y) centered on an n-th frame as a time position, and both rectangular blocks having the same size as the first rectangular parallelepiped block, The (n-1) th and (n + 1) th frames are the centers of the time positions, and the spatial position is (x, y) with respect to the (x, y) by using x and y as parameters.
y), a rectangular parallelepiped block extraction circuit for extracting three types of rectangular parallelepiped blocks each composed of a second and a third rectangular parallelepiped block deviated by (−Δx, −Δy) pixels from the input video signal, respectively. The first rectangular parallelepiped block and the plurality of second rectangular parallelepiped blocks extracted from the block extracting circuit, and the pixel values between the corresponding pixel values between the first rectangular parallelepiped block and the plurality of third rectangular parallelepiped blocks, respectively. A first and second matching evaluation function value operation circuit for performing a difference operation and an operation for summing up the difference operation results and outputting a matching evaluation function value for each of the second and third rectangular parallelepiped blocks; Before giving the minimum evaluation function value among the matching evaluation function values output from the first and second matching evaluation function value calculation circuits. The pixel offset of the second and third rectangular solid block ([Delta] x, delta
y), (−Δx, −Δy) And a motion estimating circuit for estimating and outputting a motion vector of a two-dimensional block of an n-th frame located at the center of the first rectangular parallelepiped block of the moving image. Quantity estimation device.