JP2869142B2 - Image motion detection method - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to signal processing of a moving image, and more particularly to a method of detecting a motion of an image for motion compensation in inter-frame prediction.

2. Description of the Related Art Conventionally, in high-efficiency coding of a moving image, inter-frame prediction is sometimes used to improve coding efficiency. The encoding and transmission of a moving image by simple inter-frame prediction is a method of encoding and transmitting a difference from a previous screen, that is, a changed part of an image. Therefore, the coding efficiency of the inter-frame predictive coding for a still image is very high, but the coding efficiency for an image with a large motion is greatly reduced. In order to overcome this, prediction adapted to the speed and direction of motion, that is, motion compensated inter-frame prediction is required.

In order to perform motion-compensated inter-frame prediction, it is necessary to detect the motion of the input image, that is, how much the subject in the screen has moved in which direction, that is, to detect a motion vector. For example, "High Efficiency Coding Technology in Image Transmission" (Trickeps Co., Ltd. (Showa 62-3-31 P.217-2)
21) describes the “block matching method” for detecting a motion vector in block units and the “principle of multi-step motion vector detection” for the purpose of reducing the amount of calculation.

The detection of a motion vector by the block matching method is to find a block A in a frame to be coded and a block B that is the same as a block B in a coded previous frame, as shown in FIG. . Here, assuming that the block size is m lines × n pixels and the motion vector search range is ± J lines / frame · ± K pixels / frame, the total number of (2J + 1) · (2K + 1) trial vectors is The absolute value or the square value of the inter-frame difference of the pixel is obtained, and the sum thereof is evaluated from the magnitude. That is, for each trial vector, the absolute value or the sum of the square values of the inter-frame differences of the total pixels in the block are compared, and the block of the trial vector that is the smallest is determined to be the “best matching block”.

As described above, the total number of operations for obtaining a motion vector per block by the block matching method requires the difference operation, the absolute value operation of the difference result, and the accumulation operation, and requires (2J + 1) × (2K + 1) × (m × n ) × 3 times.

Here, an example of the configuration of the motion detection circuit using the block matching method is shown in FIG. 1 is a memory circuit for storing a luminance signal of the current frame (hereinafter referred to as a current frame memory), 2 is another memory circuit for storing a previous frame luminance signal one frame time earlier (hereinafter referred to as a previous frame memory), and 3 is a current frame memory. A difference calculation circuit for obtaining the difference between the luminance signals P and Q read from the memory 1 and the previous frame memory 2 for each pixel, 4 is an absolute value circuit for obtaining the absolute value of the difference R, and 5 is 1 of the absolute value S. This is an accumulation circuit for accumulating the sum of blocks. Reference numeral 6 denotes a register for storing the cumulative result T. The clock generation circuit 8 determines the previous result and the magnitude comparison circuit 7 for each trial vector. Then, the contents of the register 6 are updated to the minimum value.

Reference numeral 9 denotes an address generation circuit which generates an address for reading data of all pixels in a block from which a motion vector is to be detected and supplies the address to the current frame memory 1 and a trial vector generator 10 to the previous frame memory 2. The address shifted by the address shift circuit 11 is supplied by the amount of the generated trial vector.

When the sum of the absolute values of the inter-frame differences is calculated for several trial vectors including the 0 (zero) vector, clock generation is performed only when the cumulative value is smaller than that for the previous trial vector as described above. Since the circuit 8 generates the write clock, finally, the best matching vector is stored in the optimum vector storage register 12, and
13 outputs a motion vector.

In this way, motion detection can be performed by the block matching method. However, if the search is blindly performed for all possible trial vectors within the search range, the motion vector search range becomes ± J lines / frame. In the case of ± K pixels / frame, the total number (2J + 1) · (2K +
Even if 1) different trial vectors are present and the search range is limited to ± 7 lines / frame · ± 7 pixels / frame, the number of trials becomes 225.

The multi-step vector detection method is one method for reducing the number of trials, and FIG. 4 shows the principle of multi-step vector detection using three-step detection as an example.

First, in the first stage, for example, 0 is equivalent to no motion compensation.
Eight representative trial vectors arranged as shown in the figure around the (zero) vector are determined, and a vector having the highest similarity is detected from the trial vectors by the above-described block matching, and this is determined as a tentative motion. Vector (▲
▼). In the second stage, it is determined that the true motion vector to be obtained should exist around this temporary motion vector, and eight more spatially dense representative trial vectors are centered on the temporary motion vector. Then, a vector having the highest similarity is detected by block matching, and this is set as a second temporary motion vector (▲). In the third stage, the third vector (▲
▼) can be detected to obtain a true motion vector () to be obtained.

In this example, it is possible to detect a motion vector within a search range of ± 7 lines / frame and ± 7 pixels / frame with 25 trials, and as described above, the number of trials is smaller than that of blind trial. Was reduced to 1/9.

(Problems to be Solved by the Invention) However, as described above, the method of detecting a motion vector by block matching requires an enormous amount of computation, and the time required for reading from the memory is reduced to the real time. However, there is a disadvantage that the search range or the number of trial vectors is limited.

Also, in the multi-stage vector detection method for reducing the amount of computation described above, there is no guarantee that the optimal vector detected in the first stage is closer to the true vector than the other eight vectors in the first stage. There may be no true vector around the vector detected in the first stage. For example, in the case of an image quality having a wide spatial frequency band where black and white and black and white are repeated for each pixel or line,
Even if the trial vector in the first stage is closest to the true vector, a large inter-frame difference is detected because it happened to be off by one pixel or line from the true vector,
The trial vector is determined to have low similarity,
Another trial vector is chosen. In that case, there is a drawback that the true motion vector cannot be detected without the search in the second and third stages.

The present invention provides a method for detecting a motion vector by block matching as described above, which requires an enormous amount of computation, and the range that can be searched in real time or the number of trial vectors is limited due to the time required for reading from memory. And the "multi-step vector detection method for reducing the amount of computation, for example, in the case of three-step detection, a problem that a true vector cannot be detected due to erroneous determination in the first and second steps" It is an object of the present invention to provide a method for detecting a motion of an image which solves the problem at the same time and enables accurate detection of a motion vector at high speed.

(Means for Solving the Problems) In order to achieve the above object, according to the present invention, as a first step, a temporary vector having a higher similarity than a spatially sparsely arranged trial vector is detected by block matching. In the two steps, a temporary vector closer to the true vector is detected by block matching around the temporary vector detected in the first step than in the trial vectors arranged more densely in the first step, and so on. In an image motion detection method using a multi-stage detection method for detecting a true vector in an N-th stage, a filter for limiting the spatial frequency band of input image data as the spatial arrangement of trial vectors in the corresponding stage becomes sparser And sub-sampling by reducing the number of sub-samples as the spatial arrangement of the trial vectors at the corresponding stage of the image data from the filter becomes sparser. Sampling means and a memory for storing the sub-sampled image data as image data for block matching are provided for each stage except the N-th stage,
A memory for storing the input image data as image data for block matching is provided corresponding to the Nth stage, and the frequency band is limited by each of the filters, the subsampling is performed by each subsampling unit, and the image data is stored in each memory. Writing and reading are performed in parallel at each stage, and block matching is performed.

(Operation) The input image data is subjected to block matching by limiting the spatial frequency band according to the spatial interval of the trial vector at the corresponding stage by the filter at each stage. This reduces the risk of selecting the wrong vector and improves the accuracy of detecting the true vector.

The image data whose spatial frequency band is limited by the filters at each stage is subjected to sub-sampling by each sub-sampling unit, and is subjected to block matching. At this time, since the number of subsamples is reduced according to the spatial interval of the trial vector at the corresponding stage, the memory capacity for storing the subsampled image data can be reduced, and the vector search at each stage is performed. Can be greatly reduced.

Further, since each memory is provided independently for each stage, vector search at each stage can be executed in parallel, so that vector detection can be speeded up or the vector search range can be expanded.

(Embodiment) FIG. 1 is a system diagram showing an embodiment of the present invention,
The video signal input from the “video signal input terminal” 100 is sent to the “current frame memory (1)” 102 for the first stage via the “spatial filter (1)” 101 and the “spatial filter (2)” 2
"Current frame memory (2)" 202 for the second stage via 01
Is written to the “current frame memory (3)” 302 for the third stage. The data of the “current frame memory (1)” 102, the “current frame memory (2)” 202 and the “current frame memory (3)” 302 before being written are simultaneously “previous frame memory (1)” 103,
A data path is provided to shift to “previous frame memory (2)” 203 and “previous frame memory (3)” 303.

The “block matching circuit (1)” 104, the “block matching circuit (2)” 204, and the “block matching circuit (3)” 304 determine the data read from the “current frame memory” for each stage and the “previous frame”. In response to the data read from the "memory", a cumulative total operation for blocks of absolute values of inter-frame differences is performed for similarity evaluation for block matching.

Reference numeral 300 denotes a "control circuit for detecting a motion vector", which provides a read address to each memory circuit for reading data of each block from the "current frame memory" and the "previous frame memory" for each stage. That is, the current frame read address for each stage is output from A1, A2, and A3 of the "control circuit for motion vector detection" 300, and the previous frame read address for each stage is output from B1, B2, and B3.

Read address (B1), (B2), (B
3) is output by shifting the address of the trial vector by the trial vector for each search of the motion vector at each stage of each block, and outputs the “block matching circuit (1)” 104 and the “block matching circuit (2). ) "20
4, and the “block matching circuit (3)” 304 calculates the cumulative value of the block of the absolute value of the inter-frame difference for each trial in C1, C2, C3 of the “control circuit for motion vector detection” 300
To enter. "Control circuit for motion vector detection" 30
A value of 0 performs a so-called block matching in which the total value is compared for each trial, a trial vector having the smallest analog value is detected, and a vector having the highest similarity is determined.

In this embodiment, a motion vector is detected by three-stage vector detection. In the first stage, a motion vector is detected by block matching for a tentative trial vector that is spatially sparsely arranged.

The arrangement of trial vectors is shown in the explanatory diagram of FIG. As shown in FIG. 5, 49 trial vectors of the first stage are arranged at 0 pixels (zero) at intervals of 4 pixels and 4 lines.

In the second stage, nine trial vectors including the center, which are arranged at 2-pixel / 2-line intervals and centered on the temporary vector determined to have the highest similarity in the first stage, are further similarized by block matching. A temporary vector that can be determined to be high is detected, and it is determined that a true vector should exist around Q.

In the third stage, nine vectors are evaluated with one-pixel / one-line accuracy including the center with respect to Q. For example, in the case of R, the similarity is determined to be the highest, and a true vector is determined.

In this embodiment, in order to improve the conventional disadvantage that an erroneous determination may be made in the first stage or the second stage, the pixel data for calculating the inter-frame difference for the first stage and the second stage is improved. To limit the spatial frequency band of
A “spatial filter (1)” 101 and a “spatial filter (2)” 201 are inserted between the “video signal input terminal” 100 and the “current frame memory”, respectively, and the trial vectors are spatially sparsely arranged. The number of samples of the pixel data is reduced in accordance with the spatial distance of the trial vector while limiting the band as much as possible.

FIG. 6 is an explanatory diagram of the spatial arrangement of sample points of video data written in the "current frame memory" or the "previous frame memory" for the first stage, the second stage, and the third stage.

First, as the data for the first stage, one sample data of the square mark is representative of the sample points of the original signal indicated by the crosses 16 around 16. That is, after limiting the spatial frequency band to satisfy the sampling theorem, the original signal 4
For the pixel, 16 points consisting of 4 lines of the original signal in the vertical method,
Subsample at a rate of one point.

Similarly, as the data for the second stage, one sample data of a circle is represented by a sample point of the original signal indicated by a cross in the vicinity 4. That is, similarly, after limiting the spatial frequency band to satisfy the sampling theorem, the original signal is composed of two pixels of the original signal in the horizontal direction and two lines of the original signal in the vertical direction.
Subsample one point at a time.

 For the third stage, the original signal indicated by the symbol x is used.

For block matching of spatially sparsely arranged trial vectors in the first and second stages, the spatial frequency band of the data is limited according to the accuracy of the vector arrangement. Vector false positives due to unbalanced spatial frequency bands of the data should be reduced. The black-and-white image for each pixel described above has the meaning of gray data as a result of the limited spatial frequency band, so that no large inter-frame difference is detected even if the pixel is shifted by one pixel from the true vector.

Also, thanks to the reduced number of samples per block in the first stage and the second stage, the frame memories for the first stage and the second stage may have a small capacity, and it is necessary to provide independent memory circuits. There is not much price, and in this embodiment, an independent frame memory is provided for each stage, and as a result, the search of each stage can be performed in parallel.

The reduction in the number of samples per block, at the same time, from the frame memory for evaluation of block matching,
The number of times of reading required per trial vector is reduced.

Therefore, in this embodiment, the search range is greatly extended by using the fact that the reading time per trial vector in the first stage is 1/16 of the conventional one, and even if the number of trial vectors is increased to 49 points, The search time is shorter than before. FIG. 7A shows the time required for vector detection in this embodiment, where w1, w2,... Are the times for writing video data to the frame memory, and a1, a2,. For reading data from the frame memory, as well as b1, b2, ... and c1,
.., c2,... are time periods for reading data from the frame memory for block matching in the second and third stages.
In the case of a pixel x 16 lines, a1 is 16 x 49 clocks and b1 is
With 64 × 9 clocks, c1 is for 256 × 9 clocks.

Further, as shown in FIG.
The block matching of the third and third stages can be temporally parallel-processed, and the motion vector can be detected at a higher speed.

In this case, as shown in the system diagram of FIG. 8, the "delay circuits" 205 and 305 are used to delay the data to be written into the frame memory for the second and third stages by d1 and d2, respectively. Insert between the “video signal input terminal” 100 and the “current frame memory”.

(Effects of the Invention) As described above in detail, according to the present invention, (1) in detecting a motion vector by block matching, a multi-stage detection method is improved and spatially sparsely arranged trial vectors In the case of evaluating the similarity of, video data for obtaining the difference between frames is also subjected to subsampling by limiting the spatial frequency band,
Since the risk of selecting an erroneous vector in the first stage and the second stage is reduced and the accuracy of detecting a true vector is improved, a moving image height is detected using the motion vector detected by the motion vector detecting method according to the present invention. Performing motion compensation inter-frame prediction in efficiency coding improves the efficiency of inter-frame coding (improves detection accuracy).

(2) Since the number of video data samples used for block matching in the first stage is 1/16,
The number of trial vectors that can be searched for in the first stage is compared with the case of the multi-stage detection method described in the related art in the same time.
It became possible to increase 16 times. Or, if the number of vectors is the same, search can be performed in 1/16 of the time. In this example, even if the trial vector in the first stage is set to 49, even if the first stage, the second stage, and the third stage are sequentially processed, the memory read cycle which is the total processing time is a1 + b1 + c1. Thus, 16 × 49 + 64 × 9 + 256 × 9 = 3664 (cycles). Compared with the memory read cycle of 256 × (9 + 8 + 8) = 6400 (cycles) in the case of the first-stage trial vector number of 9, the vector search range Can be expanded four times in area, and the search processing time has been reduced to about 57% (higher speed).

(3) By providing a memory for block matching independently for each stage, as described in FIG. 7B, the first block of the next block is executed in parallel with the vector search in the third stage. Pipeline processing, such as performing a vector search in the first and second stages, can be performed, and further speedup of vector detection or expansion of a vector search range can be realized.

(4) Since the number of read cycles of the memory for the first and second stages of block matching can be reduced, it is useful for reducing the required memory capacity and speed, and in terms of cost and high integration. It is informative.

Note that the above is for multi-stage block matching.
Although an example of three steps has been described, it is needless to say that the same effect can be expected even if the steps are realized in two steps, four steps, and any other steps.

[Brief description of the drawings]

FIG. 1 is a system diagram showing an embodiment of the present invention, FIG. 2 is an explanatory diagram of motion vector detection, FIG. 3 is a conventional motion detection circuit, FIG.
FIG. 5 is a diagram illustrating the principle of multi-stage motion vector detection, FIG. 5 is a diagram illustrating three-stage vector detection, FIG. 6 is a diagram illustrating sample points of pixel data for each stage, and FIG. 7 is a diagram illustrating the time required for vector detection. FIG. 8 is a system diagram showing a second embodiment of the present invention. 100 video signal input terminal 101, 201 spatial filter (1), (2) 102, 202, 302 current frame memory (1), (2), (3), 103, 203, 303 previous frame memory (1), ( 2), (3), 104, 204, 304 ... block matching circuits (1), (2), (3), 300 ...
Control circuits for motion vector detection, 205, 305... Delay circuits.

Continuation of front page (56) References JP-A-2-224490 (JP, A) IEEE ICC'87 International Conference on Communication (1987) p. 0151-0156 (A 64KBI T / S MOTION COMPENS ATED TRANSFORMCODE R USING VECTOR QUA NTIZATION WITH SCE NE ADAPTIVE CODEBO OK) (58) investigated the field (Int.Cl. ^6, DB name) H04N 7/24 - 7/68

Claims (1)

(57) [Claims] In a first step, a temporary vector having a higher similarity is detected by block matching from trial vectors spatially sparsely arranged, and a second step is performed around the temporary vector detected in the first step. A multi-step detection of detecting a tentative vector closer to the true vector than in the trial vectors arranged more densely than in the first step by block matching, and detecting the true vector in the Nth step in the same manner. A filter for limiting the spatial arrangement of the trial vectors in the step corresponding to the spatial frequency band of the input image data, and a step of corresponding to the image data from the filter. A sub-sampling means for reducing the number of sub-samples to perform sub-sampling as the spatial arrangement of the trial vectors becomes sparser; A memory for storing image data as image data for block matching is provided for each stage except for the Nth stage, and a memory for storing the input image data as image data for block matching is provided in the Nth stage It is provided in correspondence with the above, performing the block matching by performing the frequency band limitation by each of the filters, the sub-sampling by each sub-sampling unit, and the writing and reading of the image data to and from each memory in parallel at each stage. A special motion detection method.