JPH0591393A - Image pickup device - Google Patents

Abstract

PURPOSE:To obtain the image pickup device capable of obtaining the sufficient blurring correction result in case of the movement in the constant direction and providing a natural movement picture after correction. CONSTITUTION:The image pickup device is provided with an integrated part 11 integrating the speed vector and outputting the displacement vector, a high- pass filter (H.P.F) part 13 suppressing the low-pass component of the displacement vector outputted from the integrated part 11 and outputting the correction vector, a remaining range detection part 14 detecting the correction vector value outputted from the H.P.F. part 13 and the difference between the maximum and minimum values of the correction vector, and a gain control part 12 multiplying the prescribed value by the displacement vector outputted from the integrated part 11 based on the detection result of the remaining range detection part 14 to be outputted to a filter.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup device capable of eliminating screen shake.

[0002]

2. Description of the Related Art In a conventional image pickup apparatus, a charge-coupled device (hereinafter referred to as CCD) is used in order to eliminate a screen shake that occurs when the image pickup apparatus is held by hand and an image is taken.
A video signal from the image sensor such as the above is converted into a digital video signal and stored in a memory, and the moving speed of the imaging device is detected based on the digital video signal. Then, the movement amount of the image pickup device is obtained based on the detection result, and the address when reading the video signal from the memory is changed according to the movement amount to correct the shake of the screen due to the swing of the image pickup device. It is configured.

FIG. 7 shows an example of the configuration of the above-mentioned conventional image pickup apparatus.

The image pickup apparatus shown in FIG. 7 has a CCD light receiving element 40, A
/ D converter 41, motion detector 42, calculator 43, interpolation enlarger 44
And a field memory 45.

Next, the operation of each of the above components will be described.

The CCD light receiving element 40 converts the light from the subject inputted through the lens into an electric signal and outputs it as a video signal.

The A / D converter 41 digitally converts the video signal from the CCD light receiving element 40. Field memory 45
Receives a digital video signal from the A / D converter 41 and stores the received digital video signal.

The motion detector 42 receives a luminance signal of the digital video signal output from the A / D converter 41, and a certain image is detected from the luminance signals of two consecutive images.
A velocity vector that represents in what direction and by how much the previous image is moved is obtained. As a method for obtaining the velocity vector, any method can be used as long as the velocity vector can be obtained at constant time intervals such as between fields or frames, and a gradient method, a matching method, a phase correlation method, or the like is used.

The calculation unit 43 obtains a displacement vector by integrating the velocity vector from the motion detection unit 42, and further obtains a displacement correction vector (hereinafter simply referred to as a correction vector) by suppressing the low frequency component. .. The interpolation enlarging unit 44 determines the cut-out position when reading the video signal from the field memory 45 based on the correction vector obtained by the calculating unit 43, and an image at a constant magnification with respect to the video signal read from the field memory 45. And outputs the video signal in which the motion of the image is corrected.

[0010]

FIG. 8A shows a state before performing panning photography, in which a region surrounded by a dotted line in a substantially central portion of the right drawing is cut out and is subjected to motion compensation. The image looks like the one on the right. When the imaging device is panned to the left from this state, the cutout range moves to the right, and when the panning speed is fast, the cutout range sticks to the right end of the memory area as shown in FIG. 8B. Become. After that, as shown in FIG. 8C, the cutout range remains attached to the right end of the memory area, and it is impossible to effectively prevent the screen from shaking.

The filter provided in the calculation unit 43 is provided to alleviate such a problem and cuts the DC component contained in the displacement vector obtained by integrating the velocity vector, so that the correction vector is It is not fixed to a fixed value. However, when the panning or tilting speed is high or the hand shake is large, it is not possible to sufficiently cope with the problem as described above.

Therefore, when the cut-out position of the corrected image reaches the end of the correction range, the velocity vector is forcibly set to 0.
The method to However, in this method, the velocity vector is set to 0 even when the image pickup apparatus is actually moving, so that a jump is generated in the image at the moment when the cutout position of the corrected image reaches the end of the correction range.

Also, a "pure electronic image shake correction system"
(ITEJ TECHNICAL REPORT Vol.15, No.7, pp.43-48) states that the value of the correction vector does not exceed the correction range, the attenuation factor that changes the value based on the remaining correction range in the integrated displacement vector. A method has been proposed in which the correction vector does not exceed the correction range by multiplying K (K ≦ 1) and changing the frequency characteristic of the filter (see FIG. 9). However, in this method, when panning photography starts from the stopped state, the velocity vector is constant, and the value of the correction vector gradually increases, the frequency band of the high-pass filter narrows when the value becomes more than half of the correction range. , The speed at which the value of the correction vector increases becomes dull, and eventually saturates to a constant value.

The saturation value is expressed by the following equation showing the integration effect of the velocity vector going outward and the damping effect by the coefficient K.

[0015]

[Equation 1]

The coefficient K changes depending on the saturation value, so that a loop is formed and an oscillation phenomenon occurs. FIG. 10 is a graph showing the situation, where the horizontal axis represents time and the vertical axis represents the value of the correction vector.

Then, when the value of the correction vector increases with time, the increasing speed becomes slower as the value increases, but after reaching the maximum value, the oscillation state occurs as shown in FIG. As a result, the corrected image has an unnatural movement. In the figure, the dotted line shows the change in the correction vector value when the frequency characteristic of the filter is not changed.

Therefore, in the above-described conventional image pickup apparatus, when panning photographing is performed in which the image pickup apparatus is moved from the stopped state to the left or right while moving at a constant speed, or when the image pickup apparatus is fixed to the upper or lower side. When performing tilting shooting in which shooting is performed while moving at a speed, there is a problem in that the value of the correction vector is fixed to the maximum value or the minimum value, and the effect of preventing screen shake is not exhibited.

In view of the above-mentioned problems in the conventional image pickup apparatus, the present invention provides a sufficient shake correction effect even when the image pickup apparatus is moved in a fixed direction, and the corrected image is unnatural. An object is to provide an imaging device that does not move.

[0020]

DISCLOSURE OF THE INVENTION The present invention comprises a calculating means for integrating a velocity vector and outputting a displacement vector, and a filter for suppressing a low frequency component of the displacement vector outputted from the calculating means and outputting a correction vector. A detecting means for detecting a difference between the value of the correction vector output from the filter and the maximum value or the minimum value of the correction vector, and a predetermined value based on the detection result of the detecting means for the displacement vector output from the calculating means. And a control means for multiplying by and outputting to a filter.

[0021]

The calculating means integrates the velocity vector and outputs the displacement vector, the filter suppresses the low frequency component of the displacement vector output from the calculating means and outputs the correction vector, and the detecting means outputs from the filter. The difference between the value of the correction vector and the maximum value or the minimum value of the correction vector is detected, and the control means multiplies the displacement vector output from the calculation means by a predetermined value based on the detection result of the detection means and outputs the result to the filter. ..

[0022]

Embodiments of the image pickup apparatus of the present invention will be described below with reference to the drawings.

FIG. 1 shows the configuration of an embodiment of a calculation unit which is a main part of the image pickup apparatus of the present invention.

The calculation unit 10 in FIG. 1 is an integration unit which is a calculation means.
11, a gain control unit 12 as a control means, a high-pass filter (high-pass filter, hereinafter referred to as H.264).
P. (Referred to as F) section 13 and a remaining correction range detecting section 14 which is a detecting means.

Next, each of the above components will be described.

The integrator 11 integrates the velocity vector from the motion detector described later and outputs a displacement vector.

H. P. The F unit 13 suppresses the low frequency component of the displacement vector and outputs the result as a correction vector.

The gain control unit 12 multiplies the displacement vector output from the integration unit 11 by a constant value to determine the H.264 level. P. F
The value to be output to the unit 13 is changed based on the detection result of the remaining correction range detection unit 14, which will be described later.

More specifically, the gain controller 12 changes the value (gain) by which the displacement vector is multiplied, as shown in FIG. That is, the value of the correction vector is-(M
When in the range of (AX-n) to (MAX-n),
In the range where the gain is 1, the value of the correction vector exceeds (MAX-n) and reaches MAX (maximum value), the gain is linearly decreased from 1 to 0, and the value of the correction vector is −.
Even in the range from (MAX-n) to -MAX (minimum value), the gain is linearly reduced from 1 to 0.

The remaining correction range detection unit 14 is a H. P. F part 13
The correction vector output from is received, and the difference between the received correction vector value and the maximum or minimum value of the correction vector is detected.

The characteristics of the calculation unit 10 described above are expressed by the following equations.

[0032]

[Equation 2]

Since the value of the filter coefficient Kx is changed according to the detection result of the remaining correction range detecting section 14, the characteristic as shown in FIG. 2 is realized.

The velocity vector input for each field is integrated by the integrator 11, and the value passed through the integrator 11 is used for the gain control unit 12, H.264. P. It is output as a correction vector via the F unit 13. At this time, the value of the correction vector remains and the correction range detection unit 14 determines the remaining pixels whose correction vector value is within the correction range and adjusts the filter coefficient. It is the gain control section 12 that actually performs the adjustment.

The gain control of the filter has a value (MA) obtained by subtracting a certain value (n) from the maximum value (MAX) of the correction range when the center of the correction range is 0 as shown in FIG.
X-n) to-(MAX-n) to (M
If it is up to AX-n), the coefficient of the high-pass filter is set to 1, and the maximum value (MAX) of the correction range to (MAX-
Between n), the value is 0 or more and 1 or less. In the case of FIG.
As the maximum value is approached, the filter coefficient decreases proportionally, and when the correction vector reaches the maximum value, the filter coefficient becomes zero.

In a normal state in which panning photography or the like is not performed, the integration unit 11 integrates the velocity vector output from the motion detection unit (described later) and outputs the displacement vector.

The gain control section 12 uses the displacement vector output from the integrating section 11 as it is (that is, by multiplying it by 1).
H. P. Output to F section 13.

H. P. F part 13 is a gain control part
The low frequency component of the displacement vector from 12 is suppressed and output as a correction vector. Based on the value of this correction vector, the image cutout range in the field memory (described later) is adjusted, and the image shake associated with the shake of the image pickup apparatus is corrected.

In this state, the value of the correction vector is not so large, and normally,-(MAX-n) to (MAX-n).
Since it is within the range, the difference between the value of the correction vector and its maximum value or minimum value is large, so the gain control unit 12 sets the value to be multiplied by the displacement vector to 1, as described above.

On the other hand, for example, panning photography is performed,
H. P. When the value of the correction vector output from the F unit 13 increases and the difference from the maximum value MAX decreases, the gain control unit 12 multiplies the displacement vector by 1 or less based on the detection result of the remaining correction range detection unit 14. And As a result, the value of the correction vector is suppressed, and it is possible to prevent the cut-out range of the correction image from sticking to the end of the memory area in the field memory (described later).

Next, the operation of the calculator 10 of FIG. 1 will be described with reference to the flowchart of FIG.

First, the velocity vector obtained in the current field and the velocity vector obtained in the previous field are added (step S1), and the added result is subjected to high-pass filtering to calculate a correction vector (step S1). S2), it is determined whether or not the obtained correction vector is out of the correction range (step S3). If YES in step S3, the filter coefficient is set to 0 and the correction vector is set to the maximum value. (Step S4), the correction vector is output (step S5).

On the other hand, if NO in step S3, it is determined whether or not the correction vector is within the correction range and within the range obtained by subtracting a certain value from the correction range maximum value (step S6). If YES in step S6, the filter coefficient is set to 1 (step S7), and the correction vector is output (step S8).

If NO in step S6, the filter coefficient is varied depending on how close the correction vector is to the maximum value of the correction range (step S9), and the correction vector is output (step S10).

FIG. 4 shows the configuration of an embodiment of an image pickup apparatus having the calculation unit 10 of FIG.

The image pickup apparatus shown in FIG. 4 has a CCD light receiving element 15, A
/ D converter 16, motion detector 17, calculator 10, interpolation enlarger 18
And a field memory 19.

Next, the operation of each of the above components will be described.

The CCD light receiving element 15 converts the light from the subject inputted through the lens into an electric signal and outputs it as a video signal, and the A / D converter 16 digitally converts the video signal from the CCD light receiving element 15. To do.

The field memory 19 receives the digital video signal from the A / D converter 16 and stores the received digital video signal. The motion detecting section 17 uses the A / D converter.
A speed that receives the luminance signal of the digital video signal output from the converter 16 and indicates in what direction and in what direction a certain image has moved from the luminance signals of two consecutive images. Find the vector. As a method for obtaining the velocity vector, any method can be used as long as the velocity vector can be obtained at constant time intervals such as between fields or frames, and a gradient method, a matching method, a phase correlation method, or the like is used.

The calculation unit 10 includes the motion detection unit 17 as described above.
The displacement vector is obtained by integrating the velocity vector from, and the displacement correction vector (hereinafter simply referred to as the correction vector) is obtained by suppressing the low-frequency component by the high-pass filter. The low frequency component is suppressed because the correction vector is returned to the center of the correction range.

The interpolation enlarging unit 18 determines the cutout position when the video signal is read from the field memory 19 based on the correction vector obtained by the calculating unit 10, and determines the cutout position.
The image signal read out from 19 is subjected to image enlargement interpolation processing at a constant magnification, and the image signal in which the motion of the image has been corrected is output.

FIG. 5 shows a concrete configuration example of the calculation unit 10 of FIG.

The calculation section of FIG. 5 includes multipliers 20 and 25 and an adder 2
2, flip-flop 24, absolute value differencer 27, 28, comparator 2
9. It is composed of a selector 30 and a ROM 31.

Next, the operation of the calculation section of FIG. 5 will be described.

First, the following equation

[0056]

[Equation 3]

To calculate the velocity vector (Xn: l
Bit) and output data from ROM31 Kx: 6 bits)
By multiplying by and truncating the lower 6 bits and extracting the upper 1 bits, the second term {Xn * (Kx
/ 64)} can be calculated.

The output is input to the adder 22, and addition is performed with the calculation result of the first term {Y _n-1 * (63/64)} of the equation (1). Round down the lower 6 bits of one of the addition results
The bit is taken out and output as a correction vector. The other of the addition results is multiplied by 63 by a multiplier 25 that performs 63 times, the lower 6 bits are truncated, and the upper 1 bits are taken out to calculate the first term of the equation (1).

The output correction vector is the absolute value differencer 2
7 and 28, the positive maximum value (a) and the negative maximum value (-a) of the correction range are respectively input to the terminal A of the absolute value differentiator 27 and the terminal A of the absolute value differentiator 28. The input values are input, and the calculated results are compared by the comparator 29, and the smaller value is selected by the selector 30. The output of the selector 30 is R
Output data Kx by table conversion etc. inside OM31
The value of is output.

Table 1 shows an example of the above data conversion.

[0061]

[Table 1]

FIG. 6 conceptually shows how a video signal is read from the field memory 19.

The area surrounded by the solid line shown in FIG.
It is the entire image represented by the video signal stored in the field memory 19, and the area surrounded by the dotted line is the image portion extracted from the field memory 19.

The interpolation enlarging unit 18 filters the video signal of the image in the area surrounded by the dotted line as the cut-out position of the image, that is, the upper left position of the image from which the point P determined by the correction vector given by the calculating unit 10 is read. -Read from the field memory 19.

The interpolation enlarging unit 18 performs enlarging and interpolating processing on the read video signal so that the image is magnified as shown in FIG. 6B, and the result is used as a motion-corrected video signal. Output. In this way, the position of the point P is changed according to the amount of movement of the imaging device, and the shaking of the screen due to the shaking of the imaging device is corrected.

Possible values of the point P are the field memory 19
It depends on the size of the image represented by the video signal read from the whole. When the horizontal length of the image is X, the vertical length is Y, the horizontal length of the image to be cut out is x, and the vertical length is y, the value that the point P can take is the horizontal direction ( X-
x) and the vertical method (Y-y). The enlargement ratio is X / x in the horizontal direction and Y / y in the vertical direction.

The case where the motion detecting section 17 obtains the velocity vector based on the correlation between the two images has been described, but the velocity vector may be obtained using an angular velocity sensor. That is, an angular velocity sensor is attached to the main body of the image pickup apparatus, and its output is used as a velocity vector. In that case, the A / D converter 16
The synchronizing signal included in the video signal output by the is supplied to the angular velocity sensor, and the velocity vector is output in synchronization with the synchronizing signal.

[0068]

The image pickup device of the present invention comprises an arithmetic means for integrating a velocity vector and outputting a displacement vector, and a filter for suppressing a low frequency component of the displacement vector output from the integrating means and outputting a correction vector. , A detection means for detecting the difference between the value of the correction vector output from the filter and the maximum value or the minimum value of the correction vector, and a predetermined value based on the detection result of the detection means for the displacement vector output from the integration means. Since the control means for multiplying and outputting to the filter is provided, if the predetermined value is reduced according to the difference between the value of the correction vector and the maximum value or the minimum value of the correction vector becoming smaller, The value can be suppressed, and a sufficient shake correction effect can be obtained even when the image pickup means is moved in a certain direction. As a result, the correction vector value does not change rapidly and is corrected. The image is a natural movement of.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a configuration of a calculation unit which is a main part in an image pickup apparatus of the present invention.

FIG. 2 is a graph showing characteristics of a gain control unit that constitutes the calculation unit of FIG.

FIG. 3 is a flowchart for explaining the operation of the calculation unit in FIG.

4 is a diagram showing a configuration of an embodiment of an image pickup apparatus including the calculation unit in FIG.

5 is a diagram showing a specific configuration of a calculation unit in FIG.

FIG. 6 is a diagram for explaining how a video signal is read from a field memory unit included in the imaging device of FIG.

FIG. 7 is a block diagram showing an example of a conventional imaging device.

FIG. 8 is a diagram for explaining a problem of the image pickup apparatus of FIG.

9 is a graph showing the relationship between the characteristics of a high-pass filter forming the image pickup apparatus of FIG. 7 and image shake correction.

FIG. 10 is a graph for explaining a problem in another conventional imaging device.

[Explanation of symbols]

 10 Calculation unit 11 Integration unit 12 Gain control unit 13 H. P. F section 14 Remaining correction range detection section 15 CCD image sensor 16 A / D converter 17 Motion detection section 18 Interpolation enlargement section 19 Field memory

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Yasukuni Yamane 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka

Claims (1)

[Claims] 1. A calculating means for integrating a velocity vector to output a displacement vector, a filter for suppressing a low frequency component of the displacement vector output from the calculating means and outputting a correction vector, and an output from the filter. Detecting means for detecting the difference between the value of the corrected vector and the maximum value or the minimum value of the corrected vector, and the displacement vector output from the calculating means a predetermined value based on the detection result of the detecting means. An image pickup apparatus comprising: a control unit that multiplies by and outputs to the filter.