JP2937213B2 - Motion vector detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector detecting device, and more specifically, to a TV camera, a video camera,
The present invention relates to a motion vector detection device in an imaging device such as an electronic still camera.

[0002]

2. Description of the Related Art A motion vector detecting method in image processing includes a cross-correlation method using a cross-correlation coefficient and a matching method. In the matching method, the sum e of the absolute value of the difference between each pixel in the range of interest B between two images g0 (x, y) and g1 (x, y) continuous in time series is calculated by the following equation.
(Ε, η) (hereinafter referred to as a residual) is determined, and a position (ε, η) at which the residual is minimized is determined. That is, e (ε, η) = {b | g0 (x−ε, y−η) −g1 (x, y) |} b indicates the summation operation in the attention area B. Note that the square of the difference may be used instead of the absolute value of the difference.

However, when the above equation is actually calculated,
For example, the search range of (ε, η) is 5
Even if the size of the region B for calculating the pixel and the residual is about 8 × 8 pixels, the difference sum operation for 64 (= 8 × 8) pixels is 121 {= (5 × 2 + 1) × (5 × 2 + 1)}. Since it is necessary to perform the calculation twice, it is necessary to reduce the amount of calculation.

The following two methods have been proposed as methods for reducing the amount of computation. The first method is a residual sequential test method (for example, DI Barnea and HFS).
ilverman; "Sequential Simi
latitude Detection Argorith
m "IEEE Trans. Compt, Vol.
C-21, pp179-186 (Feb., 1972)
See). In this residual sequential test method (hereinafter abbreviated as SSDA method), attention is paid to the feature that when a shift occurs between images consecutive in time series, the residual value increases sharply.
If the residual exceeds a certain threshold during the difference sum calculation for obtaining the residual, the difference sum calculation is terminated and the process proceeds to the next calculation.

Another method is described in T.W. Koga, K .;
Iinuma, A .; Hirano, Y .; Iijim
a and T.A. Ishiguro; "Motion-
compensated interframe co
Ding for video conference
g ", Nat'l Telecommu., Con
f. G. FIG. 5.3 (Dec., 1981).

In this three-step search method, first, residuals are roughly calculated for several places (for example, 9 places) within the range of interest, and then, a step slightly smaller than the previous time is performed centering on the position where the residual is the smallest. Similarly, the residual is calculated at several places (for example, nine places). Finally, focusing on the position where the residual is the smallest,
Residuals are calculated at several places (for example, nine places) in finer steps, and the position where the residual is minimum at this stage is evaluated as a target corresponding point.

[0007]

However, the above-mentioned SSD
In the method A, if the threshold as a reference for terminating the difference sum calculation is too large, the amount of calculation increases, and if it is too small, the possibility of calculating an erroneous motion vector increases. Therefore,
The threshold value must be appropriately set, and for that purpose, it is necessary to know the statistical properties of the target image in advance.

In the above-described three-step search method, since the search within the search range B is roughly performed, there is a high possibility that the search will be drawn to the local minimum value of the residual. That is, there is a disadvantage that a global minimum value cannot always be searched.

An object of the present invention is to provide a motion vector detecting device which solves these problems.

[0010]

According to a first aspect of the present invention, there is provided a motion vector detecting apparatus for extracting a predetermined spatial frequency component from an input image, and extracting a predetermined spatial frequency component from an output amplitude of the filter and an extraction frequency of the filter. It is characterized by comprising a threshold value determining means for determining a threshold value for the inter-image calculation, and a motion vector calculating means for controlling a threshold value determined by the threshold value determining means and calculating a motion vector from an output of the filter means.

According to a second aspect of the present invention, there is provided a motion vector detecting apparatus, comprising: a filter for extracting a predetermined spatial frequency component from an input image; a threshold determining unit for determining a threshold for an inter-image operation from the extraction frequency of the filter; A binarizing unit for binarizing an output of the filter unit; and a motion vector calculating unit controlled by a threshold determined by the threshold determining unit and calculating a motion vector from an output of the binarizing unit. It is characterized by.

According to a third aspect of the present invention, there is provided a motion vector detecting apparatus, comprising: a filter for extracting a predetermined spatial frequency component from an input image; a motion vector calculating means for calculating a motion vector from an output of the filter; And a calculation position specifying means for specifying a calculation position between images based on the output amplitude of the filter and the extraction frequency of the filter means.

[0013]

With the above means, the detection range for detecting the motion vector of the input image can be known in advance. Therefore,
The possibility of detecting an erroneous motion vector can be reduced. In addition, since an appropriate threshold can be set, the amount of calculation can be reduced. That is, an accurate motion vector can be detected in a short time.

[0014]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a circuit configuration of an embodiment of the present invention in which the above-mentioned disadvantages of the SSDA method are improved. Reference numeral 10 denotes an input terminal of a video signal, and reference numeral 12 denotes a two-dimensional band-pass filter (BPF) for extracting a specific frequency component of a video signal input to the input terminal 10 in accordance with a filter coefficient set by a filter coefficient setting circuit 14. is there. 16
Is an amplitude calculation circuit for calculating the amplitude of the output of the two-dimensional BPF 12, and 18 is the period T of the image extracted by the two-dimensional BPF 12.
, An extracted image period calculating circuit 20 for calculating
6 is a threshold value determination circuit that determines a threshold value for two-dimensional phase difference detection according to the amplitude A obtained in 6 and the period T obtained in the extracted image period calculation circuit 18

Reference numeral 22 denotes a two-dimensional phase difference detection circuit for detecting a two-dimensional phase difference (ie, image shift or motion) from the output of the two-dimensional BPF 12 in accordance with the threshold Vth determined by the threshold determination circuit 20; Reference numeral 24 denotes an output terminal of the motion vector detected by the two-dimensional phase difference detection circuit 22.

The input terminal 10 has an NTSC system,
Only the luminance component of the PAL or SECAM video signal needs to be input.

FIG. 2 shows an example of a circuit configuration when the two-dimensional BPF 12 is configured by a digital circuit. In FIG. 2, H is a line memory, D is a pixel memory for providing a delay between adjacent one pixels, and a1 to an, b1 to bn, c1... Are multiplications of coefficients set by the filter coefficient setting circuit 14. Σ is an addition circuit. In the circuit of FIG. 2, a specific frequency component is extracted by two-dimensional convolution (mask processing).

FIG. 3 shows a waveform of a luminance signal input to the input terminal 10, that is, an input signal of the two-dimensional BPF 12, and FIG. 3 and 4
In the graph, the horizontal axis represents the coordinates in the x direction, and the vertical axis represents the density of the image, which indicates one section in a two-dimensional space. Curves 30, 32, and 34 in FIG. 3 are luminance signal waveforms when the same subject is photographed under different illumination conditions. Curve 3 in FIG.
8, 40 and 42 are curves 30, 32 and 3 in FIG.
4 is an output waveform of the BPF 12 corresponding to FIG. T in FIG.
Indicates the period of the output waveform of the BPF 12.

The output waveform (density) of the BPF 12 is
(Ωx), where the image shift amount is α and the residual for the shift amount α is ρ (α), ρ (α) = Σ | Asin (ωx) −Asin (ω (x−α)) | If the addition area for obtaining the residual is equal to one cycle T of the waveform, ρ (α) = (4AT / π) sin (πα / T), and thereafter, this is repeated with the cycle T. This ρ (α)
5 is shown in FIG. In FIG. 5, the vertical axis represents the residual ρ
(Α), the horizontal axis is the image shift amount α.

As can be seen from FIG. 5, -T / 2 <α <T
In the range of (/ 2) (detection range), it is guaranteed that the correct minimum value of the residual ρ (α), that is, the position where ρ (α) = 0 can be correctly detected. Therefore, the amplitude calculation circuit 16,
The extracted image period calculation circuit 18 and the threshold value determination circuit 20 can determine an appropriate threshold value Vth within this detection range and in the range of 0 to 4 AT / π. It is known in advance that the output of the BPF 12 is a sine waveform having a period T and an amplitude A, and the peak value of ρ (α) is also known to be 4AT / π.
Can be set. As a result, the amount of calculation can be significantly reduced.

FIG. 6 is a block diagram showing the configuration of a modified embodiment of FIG. Reference numeral 50 denotes a moving image video signal input terminal, and reference numeral 52 denotes a two-dimensional band-pass filter (BP) for extracting a specific frequency component of the video signal input to the input terminal 50 in accordance with the filter coefficient set by the filter coefficient setting circuit 54.
F). 56 is a binarizing circuit for binarizing the output of the two-dimensional BPF 12, 58 is an extracted image period calculating circuit for obtaining the period T of the image extracted by the two-dimensional BPF 52, 60 is
This is a threshold value determination circuit that determines a threshold value for two-dimensional phase difference detection in accordance with the period T obtained by the extracted image period calculation circuit 18.

Reference numeral 62 denotes a two-dimensional phase difference detection circuit for detecting a two-dimensional phase difference from the output of the binary circuit circuit 56 in accordance with the threshold value Vth determined by the threshold value determination circuit 60; Is an output terminal of the motion vector detected by.

As in the case of FIG. 1, only the luminance component of the video signal of the NTSC system, the PAL system or the SECAM system needs to be input to the input terminal 50. 2D BP
F52 has the same circuit configuration as the two-dimensional BPF 12, and its input waveform and output waveform are as shown in FIGS. 3 and 4, respectively.

In the conventional binarization matching method, a constant threshold level as indicated by reference numeral 36 is set for the luminance signal waveform as shown in FIG.
When the density changes under illumination conditions as in 32 and 34, the binarized pattern naturally changes. On the other hand, in the present embodiment, a specific frequency component is extracted by the two-dimensional BPF 52, and a motion vector is detected from phase information of the specific frequency component.

Specifically, as shown in FIG.
In the output waveforms 38, 40, and 42 of the BPF 12, the zero-cross point is constant regardless of the lighting conditions. Since the zero-cross point moves according to the movement of the subject, a motion vector can be detected from the zero-cross point.
Therefore, a zero-crossing point of the output signal of the two-dimensional BPF 52 is detected by the binarization circuit 56, and the output of the BPF 52 is binarized using the zero level as a threshold.

The operation of the two-dimensional phase difference detection circuit 62 will be described with reference to FIGS. FIG. 7 shows a cross section in the x direction of the output (binary image) of the binarization circuit 56, where the vertical axis represents the density,
The horizontal axis is the x coordinate. T is the period of the binary image, and its magnitude is equal to the reciprocal of the extraction frequency of the BPF 52.

FIG. 8 shows a residual obtained when a matching operation is performed on a binary image having a period T. The horizontal axis is the image shift amount, and the vertical axis is the residual error. Here, the area width for calculating the residual is T, but it is not necessary to be limited to T. . The purpose of the matching operation here is to detect the position where the residual becomes minimum (ideally zero). As can be seen from FIG. 8, the image shift amount is within the range of ± T / 2 (detection It is guaranteed that a point of residual = 0 can be detected within (range). Further, the distribution diagram of the residual shown in FIG. 8 has a unique value having a peak value (here, T) determined by the period T of the image extracted by the BPF 52 and the area width (here, T) for calculating the residual. It is a waveform and is not affected at all by changes in image contrast or illumination.

Therefore, the extracted image period calculation circuit 58 and the threshold value determination circuit 60 provide an appropriate threshold value Vt in the range of 0 to T.
h can be determined. In the embodiment of FIG. 6, there is no need to know the image amplitude A of the output of the BPF 52, and the circuit and signal processing are simplified.

In the embodiment shown in FIGS. 1 and 6, the range (detection range) in which the residual is minimized can be set in advance.
The possibility of detecting an erroneous minimum residual position is reduced. In addition, since the distribution of the residual can be predicted in advance, an appropriate threshold value for the residual can be set, and the amount of calculation can be significantly reduced.

FIG. 9 is a block diagram showing a configuration of a modification of the embodiment shown in FIG. The same circuit elements as those in FIG. 1 are denoted by the same reference numerals. That is, 66 is a two-dimensional phase difference detection circuit for detecting a two-dimensional phase difference (that is, a motion vector) from the output of the two-dimensional BPF 12, and 68 is the output (image amplitude A) of the two-dimensional amplitude calculation circuit 16 and the extracted image Periodic operation circuit 18
Output (image period T) and two-dimensional phase difference detection circuit 66
From the residual ρ (x, y) from the two-dimensional phase difference detection circuit 6
6 is an operation position specifying circuit for specifying the operation position (x, y). The motion vector detected by the two-dimensional phase difference detection circuit 66 is supplied from an output terminal 70 to an external circuit.

The output of the BPF 12 is a signal having a constant period, and the correlation coefficient obtained by the correlation operation and the residual ρ (α) obtained by the matching operation have the same period.
If the frequency characteristics of the stop band of the BPF 12 are good, the BP
Since the output waveform of F12 can be approximated by a trigonometric function, the residual ρ
(Α) (or correlation coefficient) can also be expressed by a trigonometric function. Before calculating the displacement between the images, the outline of the residual ρ (α) (or correlation coefficient) will be known. Therefore, the range (detection range) for obtaining the correct shift amount α is -T / 2 to T / 2.
Performs an operation at any two adjacent places in
The amount of change in residual ρ (α) (or correlation coefficient) between points,
That is, if the gradient information is used, the calculation over the entire search range becomes unnecessary.

A specific description will be given with reference to FIG. FIG. 6 shows a change Δρ of the residual ρ (α) when α is slightly changed.
(Α), that is, the distribution of the absolute value of the first derivative. The vertical axis is | Δρ (α) |, and the horizontal axis is α. Detection range-T / 2
Within the range of ~ T / 2, the smaller the absolute value of the shift amount α is,
| Δρ (α) | increases, and conversely, as the absolute value of the shift amount α increases, | Δρ (α) | decreases.

The operation position designating circuit 68 performs matching operation (or correlation operation) at any two points within the search range (detection range), and sends a two-dimensional phase difference detection circuit 66 to the two-dimensional phase difference detection circuit 66 to determine the remaining position between the two points. If the amount of change in the difference ρ (α) (or the gradient of the correlation coefficient) is small, the movement is coarse, and if it is large, the movement is fine and the next calculation is performed. By doing so, a correct motion vector can be detected at high speed. Note that the moving direction can be determined by the sign of the amount of change in the residual ρ (α) (or the slope of the correlation coefficient).

As can be seen from FIG. 10, the differential of the residual is known in advance to have a peak value of 4 A and a minimum value of 0. Therefore, the amount of movement (ie, the step width) in the repetitive operation is It can be automatically determined from the relationship between the derivative of the residual at the measurement point and the peak value and the minimum value. For example, assuming that the maximum step width is Smax, the minimum step width is Smin, and the range of 0 to 4 A is linearly cut by the differentiation of the residual, the next step width Snext is given by the following equation. That is, Snext = Δρ (α) (Smax−Smin) / 4A + Smax Of course, according to the distribution form of Δρ (α), the above-mentioned Snext
May be determined nonlinearly.

According to the embodiment of FIG. 9, the detection range in which the minimum value of the residual can be correctly calculated can be known in advance. Further, since the residual can be approximated by a trigonometric function, the residual has no local minimum value. As a result, the possibility of detecting an erroneous motion vector is greatly reduced. Further, the distribution of the residual can be predicted, the minimum position of the residual can be detected relatively easily, the amount of computation can be reduced, and the computation time can be reduced.

[0037]

As can be easily understood from the above description, according to the present invention, a correct motion vector can be detected with a small amount of calculation.

[Brief description of the drawings]

FIG. 1 is a configuration block diagram of an embodiment of the present invention.

FIG. 2 is a circuit configuration example of a two-dimensional BPF 12;

FIG. 3 is an input waveform diagram of a two-dimensional BPF 12;

FIG. 4 is an output waveform diagram of the two-dimensional BPF 12.

FIG. 5 is a diagram illustrating a change in a residual at the time of a matching calculation.

FIG. 6 is a configuration block diagram of another embodiment.

7 is an output waveform diagram of a binarization circuit 56. FIG.

FIG. 8 is a diagram showing a change in a residual at the time of matching calculation in the embodiment of FIG. 6;

FIG. 9 is a configuration block diagram of a third embodiment.

FIG. 10 is a diagram showing an absolute value of a change in a residual in the embodiment of FIG. 9;

[Explanation of symbols]

10: input terminal 12: two-dimensional BPF 14: filter coefficient setting circuit 16: image amplitude calculation circuit 18: extracted image period calculation circuit 20: threshold value determination circuit 22: two-dimensional phase difference detection circuit 24: output terminal 50: input terminal 5
2: Two-dimensional BPF 54: Filter coefficient setting circuit 56: Binarization circuit 5
8: Extracted image cycle calculation circuit 60: Threshold value determination circuit 62: Two-dimensional phase difference detection circuit 6
4: output terminal 66: two-dimensional phase difference detection circuit 68: operation position designation circuit 70: output terminal

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁶ , DB name) H04N 5/232 H04N 7/18 H04N 7/32

Claims (3)

(57) [Claims] 1. A filter means for extracting a predetermined spatial frequency component from an input image, a threshold value determining means for determining a threshold value of an inter-image operation from an output amplitude of the filter means and an extraction frequency of the filter means, A motion vector calculating unit that is controlled by a threshold value determined by the unit and calculates a motion vector from an output of the filter unit. 2. A filter means for extracting a predetermined spatial frequency component from an input image, a threshold value determining means for determining a threshold value of an inter-image operation from an extraction frequency of the filter means, and binarizing an output of the filter means. A motion vector detecting device comprising: a binarizing unit; and a motion vector calculating unit that is controlled by a threshold value determined by the threshold value determining unit and calculates a motion vector from an output of the binarizing unit. 3. A filter for extracting a predetermined spatial frequency component from an input image, a motion vector calculator for calculating a motion vector from an output of the filter, an output amplitude of the filter and an extraction frequency of the filter. And a calculation position designating means for designating a calculation position between images.