JPH07107368A - Image processor - Google Patents

Abstract

PURPOSE:To provide an image processor which is capable of suppressing the disturbance of a corrected image to a minimum without imparting a feeling of incongruity to a photographer. CONSTITUTION:In an image processor detecting the motion vectors between images from image signals and correcting the shake in real time, the motion vector between images is detected by performing a correlation calculation between images which are continuous in time series by a motion vector detection circuit 34, the shake state of an image is detected by a logical controller 120 based on the absolute values, spatial distribution and timewise distribution of the plural detected motion vectors and the correction of shake is interrupted when the shake amount of the image is decided to exceed a preliminarily set prescribed value.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus,
In particular, the present invention relates to an image processing apparatus having a function of correcting blur of an image captured by a video camera.

[0002]

2. Description of the Related Art In recent years, a camera-integrated VTR has been greatly reduced in weight and size due to the rear focus of the lens, the miniaturization of parts used, the high-density mounting, and the high magnification of the lens itself has been advanced. Therefore, it is often the case that handheld and high-magnification photography is performed without using a tripod or the like. For this reason, a slight camera shake during use appears as a large image blur in a captured image and is recorded, so that it is necessary to prevent the image blur.

As a device for preventing such image blurring, an inertial pendulum type image blurring preventing device is disclosed in US Pat.
959,088 and U.S. Pat. No. 2,829,557. In this inertial pendulum type image blur prevention device,
The correction optical system is movably held by a gimbal mechanism to prevent image blur caused by camera shake due to its inertia. In addition, there is also a variable vertical angle type image blur prevention device that arranges a variable vertical angle prism in front of the front lens of the video lens, detects vibration by a vibration sensor, and controls the vertical angle of the prism to prevent image blur. Proposed. Furthermore, the video signal from the image sensor is stored in an image memory or the like,
A pure electronic image blur prevention device that corrects image blur by detecting a shift (movement) between consecutive images in time series based on the information and shifting the read address of the image memory according to the detected shift amount is provided. JP-A-61-248
No. 681 is proposed. This purely electronic image blurring prevention device has attracted attention in recent years because it does not require a mechanical mechanism for image blurring correction, can be made compact and lightweight, and can reduce the manufacturing cost.

[0004]

However, in the conventional pure electronic image blur prevention device, the maximum blur amount that can be detected by the image blur detection unit is smaller than that in the above-described sensor detection method, etc. Image blurring may exceed the maximum detectable range (hereinafter referred to as the detection range). In such a case, correction becomes impossible, and if the correction is continued, the image is distorted and an unsightly image is taken. Will end up. The present invention has been made in view of the current state of image blur prevention of a conventional pure electronic image blur prevention device,
It is an object of the present invention to provide an image processing device capable of minimizing the disturbance of a corrected image without giving a sense of discomfort to a photographer.

[0005]

In order to achieve the above object, a first invention according to claim 1 detects a motion vector between images in an image signal and corrects the blur in real time. In the image processing device, a vector detection unit that performs a correlation calculation between images that are continuous in time series to detect a motion vector between the images, an absolute value of a plurality of motion vectors obtained by the vector detection unit, a spatial distribution, A blur detection unit that detects a blur state of an image based on information such as time distribution, and a control unit that suspends the blur correction when the blur amount of the image detected by the blur detection unit exceeds a predetermined value. It is characterized by having.

Similarly, in order to achieve the above object, a second aspect of the present invention is an image processing apparatus for detecting a motion vector between images in an image signal and correcting the blur in real time. , A vector detecting means for performing a correlation calculation between consecutive images in time series to detect a motion vector between the images, and an absolute value of a plurality of motion vectors obtained by the vector detecting means, a spatial distribution, a temporal distribution, etc. Blur detection means for detecting the blur state of an image based on motion information,
And a warning unit for issuing a warning when the blur amount of the image detected by the blur detection unit exceeds a predetermined value.

[0007]

According to the first aspect of the invention, in the image processing apparatus for detecting the motion vector between the images in the image signal and correcting the blur in real time, the vector detecting means allows the correlation between the images continuous in time series. Calculation is performed to detect the motion vector between the images. Then, the blurring state of the image is detected by the blurring detecting means on the basis of the absolute values, the spatial distribution and the temporal distribution of the plurality of motion vectors obtained by the vector detecting means, and the blurring amount of the image detected by the blurring detecting means. When exceeds the predetermined value, the blur correction is suspended by the suspending means.

According to the second invention, in the image processing apparatus for detecting the motion vector between the images in the image signal and correcting the blur in real time, the vector detecting means allows
Correlation calculation between images continuous in time series is performed to detect a motion vector between the images. Then, the blurring state of the image is detected by the blurring detecting means on the basis of the absolute values, the spatial distribution and the temporal distribution of the plurality of motion vectors obtained by the vector detecting means, and the blurring amount of the image detected by the blurring detecting means. When exceeds the predetermined value, the warning means issues a warning.

[0009]

Embodiments of the present invention will be sequentially described below with reference to the drawings. [First Embodiment] FIG. 1 is a block diagram showing the arrangement of the first embodiment of the present invention. In FIG. 1, a focusing lens 100 for focus adjustment, a zoom lens 101 for changing a focal length, and a zoom lens are used. A correction lens 102 for correcting the focused surface and a diaphragm 103 for adjusting the light amount
Are arranged in this order, and an image pickup element 104 formed of, for example, a two-dimensional CCD is arranged in the subsequent stage of the diaphragm 103. In the subsequent stage of the image sensor 104, a sample hold (S /
H) circuit 16, automatic gain control (AGC) circuit 1
8. An analog-to-digital (A / D) converter 20, a delay circuit (2HDLY) 22 for delaying the color difference line sequential signal from the image sensor by two horizontal scanning periods, and a color signal generation circuit 24 for generating a color signal (C). Are connected in this order. A brightness signal processing unit B is connected to the delay circuit 22. The luminance signal processing unit B includes a low-pass filter (LPF) 26 that removes color signals mixed in luminance, an enhancer 28 that emphasizes high-frequency components, a gamma correction circuit 30, and a two-dimensional band-pass filter (BPF) 32. , A motion vector detection circuit 34 for detecting a motion vector from the video signal
A field memory 36 and a logic control device 120 for performing various signal processings are connected in this order.
On the other hand, the color signal generation circuit 24 includes the field memory 3
8, an electronic zoom circuit 38 for electronically enlarging an image, and an analog-to-digital (A / D) converter 44 are sequentially connected, and a memory read control circuit 42 is connected between the field memory 38 and the logic controller 120. Has been done. In FIG. 1, 10 is a subject, and 48 is an output terminal for the color signal (C) and the luminance signal (y). Next, the operation of the configuration of FIG. 1 will be described. The light flux from the subject 10 passes through the diaphragm 103 via the focusing lens group 100, the zoom lens group 101, and the correction lens group 102, and is imaged on the image pickup surface of the image pickup element 104. In the image sensor 104, the formed optical image is photoelectrically converted to obtain an image signal, the sample signal is sample-held by the sample-hold circuit 16, and the auto-gain control circuit 18 keeps the signal level constant. The gain is adjusted and converted into AD by the AD converter 20, and the 2 horizontal scanning period delay circuit 2
2 is supplied.

In the 2-horizontal scanning period delay circuit 22, the color difference line sequential signal obtained from the image signal is combined with the 1H delay signal (0
(H + 2H) delayed signal, which is input to the luminance signal processing unit B and the color signal processing circuit 24, which are respectively connected to the delay circuit 22 for two horizontal scanning periods, and so-called YC separation is performed. Then, the color signal processing circuit 24 creates a color signal based on the input signal, and the created color signal is stored in the field memory 38. On the other hand, in the luminance signal processing unit B, the output signal from the delay circuit 22 is input to the low-pass filter 26, the carrier component is removed from the color difference line sequential signal, and the luminance signal is separated. The luminance signal thus obtained is input to the enhancer 28, and a process of emphasizing a high-frequency component by adding a quadratic differential of the image signal to an edge of a subject or the like for improving image quality is performed.

The luminance signal thus enhanced is input to the gamma processing circuit 30, which is subjected to gamma processing for preventing saturation in the highlight portion and expanding the dynamic range, and then to the field memory 28. , And the band pass filter 32. Then, the bandpass filter 32 extracts a spatial frequency component effective for detecting the motion vector. That is, the band-pass filter 32 removes low-frequency components and high-frequency components in the image signal that are unsuitable for motion vector detection. Further, in the present embodiment, it is assumed that only the sign bit of the output signal of the bandpass filter 32 is output, and specifically, the luminance signal is binarized with the DC level as a threshold, and the luminance signal in the stage subsequent to the bandpass filter 32 is It is 1 bit. The luminance signal from the bandpass filter 32 is temporarily stored in the field memory 36 and delayed by one field time. The motion vector detection circuit 34 matches the luminance signal of one field before and the luminance signal of the current field. Process operations in real time.

The motion vector detection circuit 34 outputs signals corresponding to the horizontal direction component and the vertical direction component of the motion vector, these signals are input to the logic control unit 120, and the logic control unit 120 outputs these signals. Based on, the deviation of the image from the reference position is calculated every moment. The deviation signal calculated in this manner is input to the memory read control circuit 42, and the memory read control circuit 4
2 so that the calculated deviation position becomes the center,
That is, the read position of the field memory 38 connected to the memory read control circuit 42 is controlled so as to cancel the motion vector. Then, in the electronic zoom circuit 40,
Conversion to a desired size by linear interpolation is performed, and the image signal obtained by this conversion is converted into an analog signal by the DA converter 44 and output from the output terminal 48 as an image signal with image blur correction. It

Next, the operation of the logic control device 120 of this embodiment will be described with reference to the flowchart of FIG. FIG. 2 is a flowchart showing the operation of the logic control device 120 of this embodiment. First, in step S202 of FIG.
The logic controller 120 causes the motion vector detection circuit 3
From 4, the signals corresponding to the horizontal and vertical components of the motion vector are fetched for each field, and the process proceeds to step S204. In step S204, the logic controller 120 integrates the motion vector at the predetermined screen position, calculates the deviation from the reference position on the screen, and obtains the image blur correction signal based on the obtained deviation. In this deviation calculation, the reliability of each motion vector is also evaluated, and this is reflected in the integration processing as well, so that a more accurate deviation calculation is performed.

In the next step S206, the logic controller 120 determines whether or not the current image blur exceeds the detection range of the motion vector detection circuit 34 based on the motion vector captured in step S202. Done.
Then, when it is determined that the image blur exceeds the detection range of the motion vector detection circuit 34, it is determined that the image blur correction is impossible, the process proceeds to step S214, and the image blur correction is interrupted. When it is determined in step S206 that the image blur does not exceed the detection range of the motion vector detection circuit 34, the process proceeds to step S208 and the logic control device 1
20. Based on the motion vector captured in step S202 and the spatial distribution of the integrated value of the motion vector obtained in step S204, or the temporal variation, the blurring area, that is, the correction target area is detected. The determination is made, and the process proceeds to step S210.

In step S210, the logic controller 12
According to 0, the motion vector integral value of the correction target area determined in step S208 is extracted, the final image blur correction value is obtained, and whether or not the obtained image blur correction value exceeds the correctable area When the image blur correction value exceeds the correctable area, the blur correction cannot be completely performed, and therefore necessary processing such as stopping the correction operation or slowing the response is performed. . Step S210
From step S212, the image blur correction value obtained in step S210 is calculated by the logic control device 120.
It is converted into the read address of the field memory 38, and a command for actually controlling the memory is issued, and the read range of the image from the memory is shifted based on the image blur correction value to cancel the movement of the image. Then, an instruction is given to the electronic zoom circuit 40 to give a desired enlargement or reduction ratio according to the size of the read range from the memory, and the angle of view of the final output image is returned to the normal angle of view.

[0016] With this, in order to shift the reading range on the memory, it is possible to correct the angle of view, which was previously small compared to the entire screen, to the normal angle of view, and to electronically display the screen. The expansion has been done. Next, the image blur determination operation and the correction interruption operation of this embodiment will be described with reference to FIGS. 3 is a flowchart showing the image blur determination operation of the present embodiment, FIG. 4 is a flowchart showing the correction interruption operation of the present embodiment, FIG. 5 is an explanatory diagram of the motion vector of the present embodiment, and FIG. 6 is centering of the present embodiment. It is explanatory drawing of operation. In step S302 of FIG. 3, the logic controller 120 counts the number of locations in the screen where the motion vectors detected at a plurality of locations in the screen exceed a predetermined amount, and the detected screen position. i and j are reset, the process proceeds to step S304, and the X direction motion vector MVX (i,
j) and the Y-direction motion vector MVY (i, j) are determined whether or not they respectively exceed predetermined threshold values TH1 and TH2. These thresholds TH1 and TH2 are set to the maximum amount of motion that can be detected by the motion vector detection circuit 34. In step S304, the X-direction motion vector MVX (i, j) at the position i, j, the Y-direction motion vector MVY
For any one of (i, j), the corresponding threshold value TH1, T
If it is determined that the value exceeds H2, the process proceeds to step S306, the space counter is incremented, and the process proceeds to step S308. In step S304, both the X-direction motion vector MVX (i, j) and the Y-direction motion vector MVY (i, j) at the positions i and j are set to the threshold values TH1 and TH2.
If it is determined that the value does not exceed the limit, the step S308 is directly executed.
Proceed to.

In step S308, it is determined whether or not the processing of all motion vectors in the screen is completed. If it is determined that the processing is not completed, the process returns to step S304,
The processes of steps S304 and S306 are repeated, and when it is determined in step S308 that the processes of all the motion vectors in the screen are completed, the process proceeds to step S310. In step S310, it is determined whether or not the number of locations in the screen where the motion vector exceeds the threshold value exceeds a preset predetermined value TH3, and the motion vector is equal to the threshold value TH.
1, the number of points exceeding TH2 is a preset value T
If it exceeds H3, the process proceeds to step S312, the time counter is incremented, and the preset predetermined value TH
If it does not exceed 3, the process proceeds to step S316 and the time counter is reset.

Step S312 or step S316
Then, the process proceeds to step S314, it is determined whether or not the count value of the time counter exceeds a predetermined value TH4 set in advance, and the count value of the time counter is set in a predetermined value TH set in advance.
If it exceeds 4, it is finally determined that the image blur is too large to be corrected and the image blur correction is interrupted.
In addition, the count value of the time counter is set to a predetermined value TH
If it does not exceed 4, it is determined that the image blur is normal, and the correction of the image blur is continued.

When it is determined in step S314 that the image blur cannot be corrected, the routine of step S214 shown in FIG.
In step S402, the logical control device 120 interrupts the normal image blur correction operation, controls the read address of the field memory 38, and displays the corrected image P in the full screen PT as shown in FIG. Centering correction for moving to the center O of the screen is performed over a predetermined time.

In general, when the amount of image blur is large, the spatial distribution of the detected motion vector does not always increase with the amount of image blur, and the detected motion vector is largely shaken as shown in FIG. In many cases, the spatial distribution is detected unevenly. On the other hand, in the present embodiment, the spatial distribution and the temporal distribution of the motion vector can be accurately evaluated to accurately identify that it is difficult to correct the image blur. Then, in the present embodiment, when it is determined that the image blur exceeds the predetermined value, the image blur correction is not continued carelessly, the image blur correction is interrupted, and the centering correction is performed to perform the correction. The image position is corrected, the corrected image position is corrected, and then the normal image blur correction is performed.

Thus, according to the first embodiment,
When blurring an image is corrected, the occurrence of uncorrectable blur is accurately detected, and when blurring that cannot be corrected occurs, the normal image blurring correction operation is interrupted and the corrected image is moved to the center of the screen. It is possible to minimize the disturbance in the corrected image without giving a sense of discomfort.

[Second Embodiment] Next, a second embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a block diagram showing the configuration of the second embodiment, FIG. 9 is a flow chart showing the operation of the logic control device of the second embodiment, and FIG.
FIG. 8 is an explanatory diagram showing a warning operation of the second embodiment. As shown in FIG. 8, in the present embodiment, as compared with the first embodiment already described with reference to FIG. 1, the logic controller 120A is warned to the operator that the image blur cannot be corrected. The warning generation circuit 130 is connected, and the warning generation circuit 130 is connected to the warning electric viewfinder (EVF).
To the EVF display circuit 132 for controlling the display of
The EVF 134 is connected to the EVF display circuit 132. Then, when the logic control device 120A determines that the image blur cannot be corrected, it uses the EVF 134 to display a warning to the operator that the image blur cannot be corrected. The configuration of the other parts of this embodiment is the same as that of the first embodiment already described, and therefore the description is omitted.

As shown in FIG. 9, in the flow chart of the present embodiment, except for step S214A, the same processing as that of each step of the flow chart of the first embodiment already described is executed. That is, in this embodiment, if the logic control device 120A determines that the image blur cannot be corrected in step S206, the process proceeds to step S214A.
A warning signal is output from the warning generation circuit 130, and the EVF display circuit 132 outputs the EVF based on the warning signal.
As shown in FIG. 10, a warning message AR “Camera shake has occurred!” Is displayed at 134. The other operations of this embodiment in the flowchart of FIG. 9 are the same as those of the first embodiment already described, and the image blur determination operation of this embodiment is the same as the first operation already described with reference to FIG. Since this is the same as the image blur determination operation of the embodiment, the description of these operations will be omitted.

As described above, according to the second embodiment, the occurrence of the uncorrectable blur is accurately detected during the image blur correction, and when the uncorrectable blur occurs, the EVF 134 displays the image blur uncorrectable warning. However, since the operator is notified, the disturbance of the corrected image can be minimized without giving the photographer a feeling of strangeness, and it is possible to prevent the unsightly image from which the image blur cannot be corrected.

Although FIG. 7 is a flow chart showing another example of the image blur determination operation of the first embodiment, it is also applicable to the second embodiment, and this flow chart is shown in FIG.
Only the step S314M is different from the flowchart shown in FIG. That is, in each of the above-described embodiments, the count value of the time counter is the predetermined value T set in advance as in the case of FIG.
When H4 is exceeded, it is not finally determined that the image blur cannot be corrected, but the count value of the time counter of the past predetermined field is set to the predetermined value TH in step S314M.
If the number exceeds 4, it is possible to finally determine that the image blur cannot be corrected. Further, in the second embodiment, the case where the image blur uncorrectable warning is displayed on the EVF 134 has been described, but the present invention is not limited to the second embodiment, and the image blur uncorrectable is not displayed. It is also possible to give the warning by voice or other alarm sound.

[0026]

According to the first aspect of the present invention, the motion vector between images is detected by the correlation calculation between images that are continuous in time series, and the absolute value, space distribution and time distribution of the detected plurality of motion vectors are obtained. Based on this, the blurring state of the image is detected, and when the amount of blurring of the image exceeds a predetermined value, the blurring correction is interrupted, so that the disturbance of the corrected image is minimized without giving a sense of discomfort to the photographer. It will be possible.

According to the second aspect of the invention, the motion vector between the images is detected by the correlation calculation between the images which are consecutive in time series, and based on the absolute value, the spatial distribution and the time distribution of the plurality of detected motion vectors. , The image blur condition is detected,
When the image blur amount exceeds a predetermined value, a warning is issued, so that the same effect as the first invention can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a flow chart showing an operation of the logic control device of the embodiment.

FIG. 3 is a flowchart showing an image blur determination operation of the embodiment.

FIG. 4 is a flowchart showing a correction interruption operation of the same embodiment.

FIG. 5 is an explanatory diagram of motion vectors according to the embodiment.

FIG. 6 is an explanatory diagram of centering operation of the embodiment.

FIG. 7 is a flowchart showing another example of the image blur determination operation of the same embodiment.

FIG. 8 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 9 is a flowchart showing an operation of the logic control device of the embodiment.

FIG. 10 is an explanatory diagram of a warning operation of the embodiment.

[Explanation of symbols]

 22 2 horizontal scanning period delay circuit 24 color signal processing circuit 28 enhancer 30 gamma correction circuit 32 bandpass filter 34 motion vector detection circuit 120 logic control circuit 130 warning generation circuit 132 EVF display circuit 134 EVF

Claims (2)

[Claims] 1. An image processing apparatus for detecting a motion vector between images in an image signal and correcting the blur in real time, performing a correlation calculation between images continuous in time series to obtain a motion vector between the images. Based on the vector detection means for detecting and the motion information such as the absolute value of the plurality of motion vectors obtained by the vector detection means, the spatial distribution, the time distribution,
An image comprising: a blur detecting unit that detects a blur state of the image; and a control unit that interrupts the blur correction when the blur amount of the image detected by the blur detecting unit exceeds a predetermined value. Processing equipment. 2. An image processing apparatus which detects a motion vector between images in an image signal and corrects the blur in real time, performs a correlation calculation between images continuous in time series, and calculates a motion vector between the images. Based on the vector detection means for detecting and the motion information such as the absolute value of the plurality of motion vectors obtained by the vector detection means, the spatial distribution, the time distribution,
An image processing apparatus comprising: a blur detection unit that detects a blur state of an image; and a warning unit that issues a warning when the blur amount of the image detected by the blur detection unit exceeds a predetermined value.