JPH05257196A - Motion vector detection device - Google Patents

Abstract

PURPOSE:To improve the detection precision of a motion vector by calculating a motion vector value by performing a weighting process corresponding to the extent of the spatial or time contiguity of each block. CONSTITUTION:A signal which is sent to a brightness signal processing part is inputted to a low-pass filter(LPF) 100 first and the LPF 100 removes a carrier component from a color difference sequential signal to separate a brightness signal. A following enhancer 110 emphasizes a high frequency component such as the edge of a subject for picture quality improvement and a gamma corrector 120 prevents saturation at a highlight part to widen the dynamic range. Then a microcomputer 140 extracts a block which outputs a motion vector based upon a hand shake to be corrected from the distribution (optical flow) of motion vectors detected by a motion vector detecting circuit 130, and further weights the output signal from the block corresponding to how much the block is surrounded to calculate the momentary motion quantity of the image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement of a motion vector detecting device arranged in a moving image signal coding device or an image blur correcting device.

[0002]

2. Description of the Related Art Conventionally, as a motion vector detecting method required for an image encoding device and an image shake correcting device, a spatiotemporal gradient method disclosed in US Pat. No. 3,890,462, Japanese Patent Publication No. 60-46878, and the like, Alternatively, a correlation method and a block matching method based on a correlation calculation are known. For the spatiotemporal gradient method, see BKP, Horn et al., Artificial Int.
elligence 17, p.185-203 (1981), and the matching operation is discussed in detail by Morio Onoue et al. in Information Processing Vol.17, No.7, p634-640 July 1976. There is.

The spatio-temporal gradient method is a method in which the amount of motion of an image is expressed by "d / Δ" from the difference d in brightness between frames (or fields) and the difference Δ in brightness between pixels in the screen. is there. This is because the signal obtained from the camera is the time average of the field period, and the edge becomes dull as the amount of movement of the image becomes larger, and the brightness difference Δ between pixels becomes smaller. The brightness difference d between fields is normalized by the signal Δ.

On the other hand, the block matching method (template
The input image signal is divided into blocks of an appropriate size (for example, 8 pixels × 8 lines), and the pixels of a predetermined range of the previous frame (or field) are divided into blocks. And the block of the frame (or field) before the sum of the absolute values of the differences is minimized is searched. The relative shift of the block represents the motion vector of the block.

[0005]

However, in the above-mentioned conventional example, when there are a plurality of different movements in the screen, the blocks extending over the boundaries of the different movements are
Only the unreliable motion vector can be detected due to the mismatching of corresponding points due to the appearance of the object, the concealment, or the effect of different motion of other objects, and the coding is performed by the incorrect motion vector detected from the block. There is a problem that the coding accuracy of the apparatus and the correction accuracy of the image blur correction apparatus are significantly deteriorated.

In view of the above points, it is an object of the present invention to provide a motion vector detecting device capable of improving the motion vector detection accuracy.

[0007]

According to the present invention, an input image is divided into a plurality of blocks, and a motion vector detecting means for detecting a motion vector for each block, and the same motion from the motion vector from the motion vector detecting means. Motion vector calculating means for extracting a region in which the motion vector value is calculated and performing a weighted average in accordance with the position information of each of the blocks in the region, or the motion vector A motion in which a region having the same motion is extracted from the motion vector from the detection means, and a motion vector value in the region is calculated by performing weighted averaging according to position information of each of the blocks in the input image. And vector calculation means.

[0008]

The motion vector calculation means is the position information of each block in the same moving area, or the position information of each block in the input image,
That is, the motion vector value is calculated by performing weighting processing according to the spatial or temporal adjacency of each block.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below based on the illustrated embodiments.

FIG. 1 is a block diagram showing the structure of a video camera incorporating an image shake correction apparatus to which the first embodiment of the present invention is applied.

In FIG. 1, 10 is a subject, 20 is a taking lens, 30 is an image pickup device composed of a two-dimensional CCD, for example.
Numeral 40 is a sample-hold (S / H) circuit for holding an output signal from the image pickup device 30 such as a two-dimensional CCD, 5
0 is an auto gain control (AGC) circuit, 60 is an analog-digital (A / D) converter, and 70 is a color difference line sequential signal from the image pickup device 30 such as a two-dimensional CCD for two horizontal scanning periods. A delay circuit for delaying only the color signal, a color signal processing circuit 80, and a digital-analog (D
/ A) converter, and 95 is a color signal output terminal.

Reference numeral 100 is a low-pass filter (LPF) for removing color signals mixed in the luminance signal, 110 is an enhancer for emphasizing high frequency components of the luminance signal, 120 is a gamma corrector, 130 is a motion vector detection circuit, 140 is a microcomputer for judging the reliability of each block,
Reference numeral 150 is a memory read control circuit, 160 is a memory for storing a field or frame, and 170 is a luminance signal D.
A / A converter, 180 is a luminance signal output terminal.

Next, the operation of the video camera having the above configuration will be described.

The subject 10 is projected onto the image pickup device 30 by the taking lens 20 and photoelectrically converted there. The sample hold 40 holds the output signal of the image sensor 30, and the subsequent AGC circuit 50 automatically controls the gain of the signal.
The A / D converter 60 converts the output signal of the AGC circuit 50 to A / D.
D-convert. The 2-horizontal scanning period delay circuit 70 outputs the color difference line sequential signal from the image sensor 30 to the “1H” delay signal and the “0H” signal.
The signals are separated into + 2H ”delayed signals and sent to the luminance signal processing unit (circuits after LPF 100) and the color signal processing circuit 80, respectively.

A color signal is generated in the color signal processing circuit 80, and this color signal is converted into an analog signal in the subsequent D / A converter 90 and output from the color signal output terminal 95.

On the other hand, the signal sent to the luminance signal processing section is
First, it is input to the LPF 100. This LPF100 is
The luminance signal is separated by removing the carrier component from the color difference sequential signal. The next enhancer 110 performs processing for emphasizing high-frequency components such as edges of the subject in order to improve image quality. Usually, the second derivative of the video signal is added to the original signal. The gamma corrector 120 prevents saturation in the highlight part and widens the dynamic range. Motion vector detection circuit 130
May be composed of a circuit based on the space-time gradient method,
Similarly, a circuit based on the matching calculation may be used, but in the present embodiment, a detection method capable of real-time processing is required. The memory 160 is a delay circuit that delays the luminance signal by a predetermined time (one field time in this embodiment), stores the luminance signal one field before and calculates the space-time gradient method with the current field. , Or enables correlation calculation. The microcomputer 140 includes a motion vector detection circuit 13
From the distribution of the motion vector detected at 0 (optical flow), first, the block outputting the motion vector due to the camera shake to be corrected is extracted, and the output signal from the block further indicates how the block is surrounded. According to the weighting, the amount of motion of the image at that moment is calculated.
The memory read control circuit 150 is a microcomputer.
The read position of the memory 160 is controlled so as to cancel (or correct) the final motion vector calculated by the controller 140. The luminance signal read from the memory 160 is D / A
It is converted into an analog signal by the converter 170 and output from the luminance signal output terminal 180.

Next, a method of evaluating the reliability of a motion vector performed in the microcomputer 140 will be described with reference to FIGS.
Will be explained.

FIG. 2 is a diagram showing the flow from the input of an image signal to the determination of an area to be shake-stabilized (hereinafter referred to as a shake-proof area) at that moment.

FIG. 2A shows an input image, "8 × 6".
It is divided into blocks. FIG. 2B shows the result of detecting the motion vector for each block. Such a motion vector distribution diagram is generally called an optical flow diagram. In FIG. 2 (b), it can be seen that there are two largely different regions in the central part and the peripheral part of the screen. FIG. 2C shows the result of extracting the image stabilization area from the optical flow diagram.

As a method for extracting the image stabilization area from the motion vector, there is a time direction addition method of motion vectors for each block described in Japanese Patent Application No. 63-269554 already filed.

FIG. 3 is a binary image stabilization area map (effective area) in which "1 value" is assigned to the extracted area and "0 value" is assigned to the excluded area.

Usually, the motion vector of the image is multiplied by the detected motion vector to extract the effective region, and the motion vectors in the effective region are simply averaged with a uniform weight to obtain the motion of the image. Find the amount.

FIG. 4 shows an example of a convolution mask applied to the image stabilization area map as shown in FIG.

The meaning of the mask in FIG. 4 is that if there is a block adjacent to the block of interest now in the vertical and horizontal directions, point 1 is given to each of the blocks of interest and the diagonal direction If there is a block in contact with, a point of 0.5 is given, and the point is increased as the block of interest is surrounded by other blocks. The point in the diagonal direction is low because the image signal generally has few diagonal components. Note that the convolution is performed only on the effective block (the block in which the value "1" is inserted), or the result of convolution over the entire screen is ANDed with the image stabilization area map (AND operation), The value of the non-shake area (invalid area) is left as "0".

FIG. 5 shows an example in which the convolution mask is expanded from “3 × 3” shown in FIG. 4 to “5 × 5”.

The size of the convolution mask and the weighting coefficient in the mask (that is, how to give points) may be switched according to the photographing conditions.

FIG. 6 shows the result of convolution of the mask of FIG. 4 in the effective area of the image stabilization area map of FIG.

The coefficient values shown in FIG. 6 represent the reliability of the motion vector detected from the corresponding block, and the motion vector averaging may be performed according to this weight.

The position where the image is read from the memory is determined by using the motion vector obtained by the above processing, and the image in which the unpleasant camera shake component is corrected is output to the luminance signal output terminal 18
Output from 0.

(Second Embodiment) In the first embodiment, the motion vector weighted average processing is performed on the effective area of the image stabilization area map according to the result of convolution of a certain mask. However, in this embodiment, the reliability of the motion vector itself obtained in the first embodiment is re-evaluated using the sum of the weighting factors obtained as a result of convolving the mask, that is, the sum of the points. However, it can be used in combination with the first embodiment.

FIG. 7 shows the result of applying the convolution mask of FIG. 4 when it is determined that all “8 × 6” blocks in the screen are valid.

In this case, the sum of all the weighting factors in FIG. 7 is "282", which is the upper limit of the sum of the weighting factors. The larger the area of the image stabilization area is, the higher the reliability of the motion vector detected from the area is. Therefore, the reliability of the motion vector can be evaluated again from the magnitude of the sum of the weighting factors.

FIG. 8 shows a reliability evaluation function according to the sum of the weighting factors.

If the total number of points is small, the extracted regions are small or scattered, and it is highly possible that the motion vector is erroneously detected.
A small coefficient less than “1” is multiplied to suppress the influence on subsequent processing. When the number of blocks in the screen is “M × N”, the distribution of points is represented by 0 ≦ (total number of points) ≦ 7MN-4M-4N + 2.

The above processing is carried out by the micro computer of FIG.
Data 140.

(Third Embodiment) In the first and second embodiments, the reliability is evaluated by using the information on the spatial arrangement of the block of interest as a clue.
In this embodiment, this concept is extended in the time axis direction.

Generally, the correlation of moving images in the time direction is high,
For example, when a subject is being tracked and photographed, the position occupied by the subject on the screen is strongly restricted to the position on the previous screen (front field), and there is no correlation for each field. It rarely appears. That is, it is highly possible that the effective area on the current screen appears near the area considered to be the effective area on the previous screen. Therefore, a map having a gentle weighting coefficient distribution is formed with the area determined as the effective area on the current screen as the center of the peak, and the logical product (AND) is taken with the image stabilization area map obtained on the next screen. , It is possible to evaluate the reliability by making use of the correlation in the time direction of the position on the screen of a certain area.

The implementation procedure will be described with reference to FIGS. 9 and 10.

FIG. 9 shows a convolution mask applied to the image stabilization area map obtained on the current screen. Since it is considered that the displacement of the position of the subject between the images occurs in all directions of 360 degrees with the same probability, the weighting coefficient distribution of the mask is also isotropic.

FIG. 10 shows the result of convolving the mask shown in FIG. 9 with the image stabilization area map shown in FIG.

The reliability of each block can be easily evaluated by ANDing (weighting) the weighting coefficient distribution shown in FIG. 10 and the image stabilization area map obtained on the next screen.

The above processing is performed by the microcomputer 140 shown in FIG.

According to each of the above-described embodiments, the reliability of the detected motion vector is evaluated based on the spatial or temporal adjacency (proximity) of each block, and the weighting process is performed accordingly. Since it is applied to the vector, the accuracy of the moving picture coding device and the image blur correction device that had been significantly deteriorated by the unreliable motion vector detected from the block including the boundary where originally different motion exists Can be improved.

[0044]

As described above, according to the present invention,
The input image is divided into a plurality of blocks, a motion vector detecting means for detecting a motion vector for each block, and an area having the same motion are extracted from the motion vector from the motion vector detecting means, Motion vector calculation means for calculating a motion vector value by performing weighted averaging according to position information occupying each area of each of the blocks, or the same motion from the motion vector from the motion vector detection means. A motion vector calculating unit that extracts a region and calculates a motion vector value in the region by performing weighted averaging according to position information of each of the blocks in the input image; Position information in the area that is displayed, or position information in the input image of each block,
That is, the motion vector value is calculated by performing weighting processing according to the spatial or temporal adjacency of each block.

Therefore, it is possible to improve the detection accuracy of the motion vector.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a video camera incorporating an image blur correction device to which a first embodiment of the present invention is applied.

FIG. 2 is a diagram showing a flow from image signal input to image stabilization area determination.

FIG. 3 is a diagram showing a binary map showing an image stabilization area in the first embodiment of the present invention.

FIG. 4 is a diagram showing an example of a convolution mask applied to the image stabilization area of FIG.

5 is a diagram showing another example of a convolution mask applied to the image stabilization area in FIG.

FIG. 6 is a weight coefficient distribution diagram showing reliability in the image stabilization area according to the first exemplary embodiment of the present invention.

FIG. 7 is a weight coefficient distribution diagram when the entire screen in the second embodiment of the present invention is the image stabilization area.

FIG. 8 is a diagram showing a reliability evaluation function based on a sum of weighting factors according to the second embodiment of the present invention.

FIG. 9 is a diagram showing an example of a convolution mask applied to an image stabilization area in a third embodiment of the present invention.

10 is a distribution diagram of weighting factors when the mask of FIG. 9 is incorporated into the image stabilization area map of FIG.

[Explanation of sign]

 30 Image sensor 100 LPF 110 Enhancer 120 Gamma corrector 130 Motion vector detection circuit 140 Microcomputer 150 Memory read control circuit 160 Memory

Claims (2)

[Claims] 1. An input image is divided into a plurality of blocks, and a motion vector detecting means for detecting a motion vector for each block and an area having the same motion are extracted from the motion vector from the motion vector detecting means. , A motion vector calculating device for calculating a motion vector value in the area by performing weighted averaging according to position information of each of the blocks in the area. 2. An input image is divided into a plurality of blocks, a motion vector detecting means for detecting a motion vector for each block, and a region having the same motion are extracted from the motion vector from the motion vector detecting means. , A motion vector calculating device for calculating a motion vector value in the area by performing weighted averaging according to position information of each of the blocks in the input image.