JPH057327A - Motion vector detecting device - Google Patents

Abstract

PURPOSE:To effectively utilize spatial information a picture has by providing a simple evaluation standard or evaluation function to show the spatial frequency of an input picture and setting automatically the extracting area of the maximum size and shape adaptive to the spatial frequency of each part of the input picture. CONSTITUTION:A picture signal inputted to an input terminal 10 is passed through a pre-processing filter 20 before the calculation of a motion vector, and spatial frequency components unsuitable to correlation calculation or matching calculation, for instance, an extremely high spatial frequency component and a low frequency component close to a DC component are excluded. Next, the picture signal after being passed through the filter 20 is supplied to a space gradient arithmetic circuit 30 and a delay circuit 50 after dividing it into two passages. The circuit 30 calculates space gradients in a vertical direction and a horizontal direction respectively, and stores a position at which calculated luminance difference is '0' or below a prescribed threshold, and forms one closed area, i.e., a block by connecting these positions, and controls so that difference sum calculation for the correlation calculation in an analog switch 60 is executed within the block determined by a block boundary determination circuit 40, and operates a motion vector arithmetic circuit 70.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector detecting device, and more particularly to a device for detecting a motion vector from an image signal.

[0002]

2. Description of the Related Art In recent years, in the field of image pickup devices such as cameras, automatic focus control, automatic exposure control, automatic image stabilization control, etc.
Its multi-functionality and automation are remarkable. In addition, the method is also changed to a method in which a signal component serving as a parameter used for various controls is extracted from a video signal based on a captured image to perform various controls, from a system having an independent sensor system for each function. It's starting. This means that the system is made intelligent by performing all control using the video signal output from the image pickup means.

Against such a background, in an image pickup apparatus such as a camera, taking automatic image stabilization control as an example, it is necessary to detect the movement of the image accurately and highly accurately. This is not limited to automatic image stabilization, and even in automatic focus control, image motion detection is important for improving accuracy.

Various methods are conceivable as means for detecting the motion of an image, but the motion is detected from the image signal corresponding to the detection area (called block) for detecting the motion vector set in the screen. The method of detecting a vector has been attracting attention in recent years because it can detect a complicated image movement with high precision and fine detail.

As a motion vector detection method by image signal processing, a cross-correlation method for calculating a cross-correlation coefficient h (ξ, η) according to a definition formula and a sum of absolute values of differences between two images (hereinafter,
There is a matching method for calculating the residual).

The definition equation for calculating the cross-correlation coefficient is represented by h (ξ, η) = ∬g ₀ (x −ξ, y −η) · g ₁ (x, y) dxdy. However, g ₀ (x, y) · g ₁ (x, y)
Represents two images, and ξ and η represent the deviation amounts of the two images. The cross-correlation coefficient h (ξ, η) becomes maximum when the two images completely match, and
If there is a gap between the images, it becomes exponentially small.

In the matching method, the residual e
Calculate (ξ, η).

E (ξ, η) = Σ _B | g ₀ (x −ξ, y −η) · g ₁ (x, y) | Here, the symbol B indicating the accumulation range is called a block, and a block unit In order to calculate the degree of similarity and the degree of matching between two images, the method is also called a block matching method or a template matching method. Note that the definition of the degree of mismatch between the two images and the residual indicating the degree of mismatch may be not only the sum of absolute values of the differences between the pixels described above, but also the sum of squares of the differences. The size and shape of the block are fixed regardless of the input image.

[0009]

However, according to the above method, since the size and shape of the detection block are fixed, in the region where the spatial frequency of the image is low,
There is a drawback that the image information necessary for the correlation calculation is insufficient in the block and the accuracy is remarkably deteriorated. On the contrary, in the region where the number of spatial species of the image is high, the detection range is narrow due to the influence of the fine period of the image, and although the large block is not necessarily required, the size and shape of the fixed block are fixed. Therefore, there is a drawback that the spatial resolution for detecting the motion vector is limited.

Therefore, an object of the present invention is to provide a motion vector detecting device capable of adaptively and automatically determining an optimum block according to a spatial frequency of an image by a simple method.

[0011]

SUMMARY OF THE INVENTION The present invention has been made for the purpose of solving the above-mentioned problems, and is characterized by a device for detecting a motion vector in an image signal. Area setting means for adaptively and automatically setting the size and shape of the unit operation area for outputting the motion vector according to the spatial frequency of the input image, and within the unit operation area set by the area setting means. A motion vector detecting device provided with a motion vector detecting means for detecting a motion vector of an image by performing a correlation calculation or a matching calculation between consecutive images in time series from a corresponding image signal.

[0012]

With this, by providing a simple evaluation criterion (or evaluation function) indicating the spatial frequency of the input image, the detection area of the optimum size and shape adapted to the spatial frequency of each part of the input image is automatically determined. It can be set, and as a result, the spatial information of the image can be effectively used.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a motion vector detecting device according to the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of a motion vector detecting device of the present invention, which is suitable for use in an image pickup device such as a video camera or an electronic camera.

Reference numeral 10 is an image signal input terminal, 20 is a preprocessing filter before motion vector calculation, 30 is a circuit for calculating the spatial gradient of the image, 40 is a boundary determining circuit for determining the boundary of the block, and 50 is spatial gradient calculation. A delay circuit for adjusting the operation time by the circuit 30 and the block boundary determination circuit 40,
Reference numeral 60 is an analog switch whose opening and closing is controlled by the output of the block boundary determination circuit 40, 70 is a motion vector calculation circuit by correlation calculation, and 80 is an output terminal which outputs a signal indicating the final motion vector amount.

The image signal input to the input terminal 10 passes through the pre-processing filter 20 and has a spatial frequency component (for example, a very high spatial frequency component or a low DC component that is not suitable for correlation calculation (or matching calculation)). Frequency components) are removed. The image signal is divided into two paths after passing through the preprocessing filter 20, and is supplied to the spatial gradient calculation circuit 30 and the delay circuit 50.

The spatial gradient calculation circuit 30 calculates the spatial gradients in the vertical and horizontal directions, respectively. Figure 2
Is a simple circuit configuration example for calculating the spatial gradient in the horizontal direction. In FIG. 2, the image signal input from the image signal input terminal 90 is a latch 1 that causes a delay of one pixel time.
The image signal after passing 00 is subtracted by the subtractor 100, and the difference is output from the output terminal 120. This means that the brightness difference between the pixels adjacent in the horizontal direction is obtained.

FIG. 3 shows a simple circuit configuration example for calculating the spatial gradient in the vertical direction. Image signal input terminal 13
The image signal input from 0 is subtracted by the subtracter 150 from the image signal that has passed through the line memory 140 that causes a delay of one scanning line time, and the difference is output from the output terminal 160. This means that the brightness difference between the pixels adjacent in the vertical direction is obtained.

The block boundary determining circuit 40 stores the position where the brightness difference calculated by the spatial gradient calculating circuit 30 becomes 0 or less than a predetermined threshold value, and connects this position to one closed region, that is, a block. To form. The analog switch 60 controls so that the difference-sum calculation for correlation calculation is performed within the block determined by the block boundary determination circuit 40.

Under the control of the analog switch 60, the motion vector calculation circuit 70 performs a correlation calculation or a matching calculation to calculate a motion vector. The calculated motion vector is output from the output terminal 80.

The motion vector device according to the present invention is constructed as described above, and its unique operation will be described below.

FIG. 4 shows an arbitrary section of the image. The vertical axis represents the density value of the image, and the horizontal axis represents the position in the image. X ₁ x ₂ , ..., X ₈ on the horizontal axis are points at which the image signal g ′ (x ₁ ) ≈g ′ (x ₂ ) ≈ …… ≈g ′ (x ₈ ) ≈0. That block 1 between x ₁ ~x ₂ be determined points spatial gradient is zero or nearly zero and the boundary of the block of the image, x
An optimum block size and a two-dimensional shape are automatically set according to the spatial frequency of the image, such as block 2 between _{2 and} x ₃ .

In FIG. 4, 1/2 of the image period T is set to the block size, but the image period T may be set to the block size as it is.

By doing so, the block size is increased in the low frequency region where the image information required for the correlation calculation (or the matching calculation) is scarce, so that the problem occurs from the low frequency region which has been a problem in the past. It is possible to remove a large error vector that does. On the contrary, in the high frequency area where information is sufficiently present, it is possible to reduce the block size, and it is possible to reduce the redundancy of spatial gradient information in the block generated by the conventional method in which the block size is fixed. As a result, high spatial resolution can be obtained.

In the above embodiment, the case where the spatial gradient of the image is used as the means for determining the boundary of the block has been described. However, in the second embodiment of the present invention, the density difference between consecutive images will be described below. That is, a method of automatically setting the size and shape of the input image signal by using the time gradient will be described as in the above embodiment.

FIG. 5 is a diagram showing a second embodiment of the present invention. In the figure, the only difference from the first embodiment shown in FIG. 1 is the operation of the image density difference calculation circuit 190 which is the reference for determining the block boundaries by the block boundary determination circuit 200.

FIG. 6 is a simple configuration example of a circuit for obtaining a density difference between two images which are continuous in time series, that is, a time gradient. The image signal input from the input terminal 250 is stored in the memory 26.
The image signal that has passed 0 and is delayed by one screen is subtracted by the subtractor 270, and the difference is output from the output terminal 280. The data holding time of the memory 260 is, for example, NT.
It is 1/60 second for SC field processing and 1/30 second for frame processing. The density difference is 0
Alternatively, an adaptive block shape corresponding to the spatial frequency of each part of the input image is automatically set by setting a block boundary at a position equal to or less than the set threshold value.

FIG. 7 is a diagram showing the state of this embodiment. The vertical axis shows the image density value, and the horizontal axis shows the position in the image. X ₁ , x ₂ , ..., x ₈ entered on the horizontal axis
Is the point where the density difference between images that are continuous in time, that is, the time gradient d becomes zero. Each point (x ₁ , x ₂ , ..., x
_{If 8} ) is defined as the block boundary, the optimum block size and the two-dimensional shape are automatically set according to the spatial frequency of the image. In FIG. 7, the block is divided for each point with the time gradient d = 0, but the point with d = 0 is skipped once as a block boundary, and the size of one block is approximately one cycle T of the image. You may make it equal.

Next, a case where the above-mentioned motion vector detecting device is actually applied to the image stabilizing device of a video camera will be described with reference to FIGS. 8 and 9.

FIG. 8 shows a shake correction means for detecting a shake of an image by using the motion vector detecting device according to the present invention and correcting the shake by changing the optical axis of the photographing lens to optically shake the shake. A variable apex angle prism for correction is used.

In the figure, reference numeral 101 denotes a variable apex prism for varying the direction of the optical axis of the photographic lens optical system, that is, the apex angle. Reference numeral 102 denotes a taking lens, 103 denotes an image pickup device such as a CCD for photoelectrically converting a subject image formed by the taking lens 102 and outputting an image pickup signal, 10
Reference numeral 4 is a preamplifier, 105 is a camera signal processing circuit for performing various processing such as blanking processing, addition of a synchronization signal, and gamma correction on an image pickup signal output from an image pickup element to output a standardized video signal, 106 is the above-mentioned FIG.
A motion vector detection circuit including the motion vector detection circuit described in each embodiment of FIG. 5, 107 is a variable that takes in the motion vector information of the image supplied from the motion vector detection circuit 106 and cancels the motion of the image due to camera shake. A system control circuit that calculates information on the driving direction of the apex angle prism and a drive amount necessary for shake correction, 108 controls the drive of the actuator 109 based on the information calculated by the system control circuit 107, and sets the variable apex angle prism 101. It is a driving circuit for driving.

Accordingly, the motion vector detection circuit 106 for detecting the motion vector in each of the above-described embodiments detects the motion vector based on the image blur (camera blur), and based on this motion vector, the variable The driving direction and the driving amount of the apex angle prism are calculated, and the variable apex angle prism is driven by the output of this calculation to perform the blur correction. The operation of the motion vector detection circuit itself is the same as that described in the above embodiment, and the description thereof is omitted.

In FIG. 9, an optical system is not used for the blur correction means,
The image is once taken in the memory, and the movement of the image is corrected by changing the read range from the memory.

Regarding the same components as those of the embodiment of FIG.
The same reference numerals are used and the description thereof is omitted.

The image pickup signal output from the preamplifier 104 is processed by the camera signal processing circuit 110 such as AGC and gamma, and then converted into a digital signal by the A / D converter 111, which will be described later in memory control. Circuit 116
The image memory 112 is loaded at a rate set by. The digital image signal read from the memory 112 is further processed by the digital camera signal processing circuit 11
3, the predetermined processing which is performed in the form of an analog signal in the example of FIG. 8 is performed. The memory control circuit 116 controls the rate and timing of A / D conversion for capturing an image in the memory at this time, and the writing timing and address in the memory. The control of the address and timing of reading from the memory is also controlled by this memory control circuit.

The image signal output from the camera signal processing circuit 113 is shake-corrected through a field angle correction circuit 114, which will be described later, and then converted into an analog signal by the D / A converter 115 and output. Note that the digital video signal output may be output as it is.

On the other hand, in the motion vector detection circuit 117,
A motion vector due to camera shake is detected and supplied to the system control circuit 118 as in the embodiment of FIG.
Then, the motion direction and the motion amount of the image are calculated based on the motion vector detected by the motion vector detection circuit 117, and based on this, the memory control circuit 116.
To control the read range of the memory. That is, an image is captured in the memory at an angle of view wider than the angle of view output in advance, and when the memory is read out, the read range is changed on the memory to correct the motion. That is, the movement of the image can be corrected by shifting the read range in the movement direction.

When the image data taken in the memory is read, the read range is set smaller than the memory space,
Since the movement of the image is canceled by moving the reading range in the direction of movement on the memory, the size of the angle of view is different before and after taking in the memory. The angle-of-view correction circuit 114 corrects the size of the angle of view by this shake correction. For example, the read-out image information is enlarged,
The processing is performed to make the size the same as the angle of view before shake correction.

In the above configuration, the camera signal processing circuit may be arranged after the D / A converter 115 to perform analog signal processing, but digital signal processing facilitates the processing, and It is also advantageous in terms of noise.

As described above, the motion vector detecting circuit according to the present invention can perform the shake correction on the video camera. In addition, as motion detection, not only blur correction but also many other applications such as camera panning detection can be applied.

As a result, it is possible to realize a high-performance video camera with a shake correction function, which has an extremely wide motion detection range, can detect and correct various large and small motions.

[0042]

As described above, according to the motion vector detecting apparatus of the present invention, the size and shape of the block as the motion vector detecting area is adaptively and automatically set according to the spatial frequency of the input image. By doing so, it is possible to remove a large error vector in a pattern region having a low spatial frequency and few features, and by effectively utilizing the spatial gradient information that the image has, a high spatial resolution can be obtained, and Since the boundary of the block can be determined by a simple evaluation function, the block setting process can be speeded up.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a first embodiment of a motion vector detection device according to the present invention.

FIG. 2 is a block diagram showing a circuit configuration for obtaining a horizontal gradient.

FIG. 3 is a block diagram showing a circuit configuration for obtaining a vertical gradient.

FIG. 4 is a diagram for explaining a method of determining a block boundary using a spatial gradient of an image.

FIG. 5 is a configuration block diagram in the second embodiment of the present invention.

FIG. 6 is a circuit configuration diagram for obtaining a density difference between images.

FIG. 7 is a diagram for explaining a method of determining a block boundary by using a density difference between images.

FIG. 8 is a block diagram showing an embodiment of a video camera to which the motion vector detection circuit of the present invention is applied.

FIG. 9 is a block diagram showing another embodiment of a video camera to which the motion vector detection circuit of the present invention is applied.

Claims (3)

[Claims] 1. An apparatus for detecting a motion vector from an image signal, wherein the size and shape of a unit calculation area for outputting one motion vector is adaptively and automatically set according to the spatial frequency of an input image. Motion for detecting a motion vector of an image by performing a correlation operation or a matching operation between images that are continuous in time series from an image signal corresponding to the unit operation area set by the setting means and the area setting means. A motion vector detection device comprising: a vector detection means. 2. The motion vector detection device according to claim 1, wherein the region setting means is configured to determine a boundary of the unit calculation region based on a spatial gradient of an image. 3. The area setting means is configured to determine a boundary of the unit operation area based on a time gradient represented by a density difference between temporally consecutive images. The motion vector detection device according to claim 1, wherein the motion vector detection device is a motion vector detection device.