JP2022516481A - Imaging device, image stabilization device, imaging method and image stabilization method - Google Patents

Abstract

Provided is an image pickup device capable of effectively applying optical image stabilization and electronic image stabilization in a hybrid image stabilization system for each frequency band. The image pickup device (1) includes a vibration detection unit that detects the vibration of the image pickup device and outputs a detection signal, and a high-pass filter (HPF) and a low-pass filter (LPF), and outputs a detection signal output from the vibration detection unit at a high frequency. A signal processor that separates the vibration signal and a low-frequency vibration signal, a first correction unit that performs optical camera shake correction based on the high-frequency vibration signal separated by the signal processor, and a low-frequency vibration signal separated by the signal processor. An image pickup unit (2) including a second correction unit that performs electronic camera shake correction based on the above, and an image processor (3) that processes an image corrected by the first correction unit and / or the second correction unit. And, including.

Description

The present invention relates to an image pickup device, a camera shake correction device, an image pickup method, and a camera shake correction method having a mechanism for correcting image blur due to vibration such as a handshake.

Optical image stabilization (OIS) and electronic image stabilization (EIS) are known as image stabilization (IS) techniques for correcting image blur. Optical image stabilization corrects image blur by detecting the vibration of an imaging device such as a camera with a gyro sensor, and image stabilization is performed by moving the lens to cancel the movement caused by the vibration. .. Electronic image stabilization (EIS), on the other hand, corrects image blur by shifting the image capture range of the image sensor so as to cancel the amount of motion detected by the gyro sensor.

One form of the hybrid image stabilization system that uses both the above optical image stabilization and electronic image stabilization is a system that performs vibration correction on the OIS side, passes the amount of movement to the EIS side, and processes the residue with EIS. be.

An object of the present invention is to provide an image pickup device, an image stabilization device, an image pickup method, and an image stabilization method capable of effectively applying optical image stabilization and electronic image stabilization in a hybrid image stabilization system for each frequency band. It is to be.

The first aspect is an imaging device, wherein the image pickup device is used.
It ’s an imaging unit,
A vibration detection unit that detects the vibration of the imaging device and outputs the detection signal,
A signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal.
A first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor, and
An imaging unit including a second correction unit that performs electronic image stabilization based on a low frequency vibration signal separated by a signal processor.
Includes a first correction unit and / or an image processor that processes the image corrected by the second correction unit.
The signal processor provides an imaging device including a cutoff frequency controller that variably sets the cutoff frequencies of the HPF and LPF in the frequency band.

The first aspect can provide an imaging device that effectively applies optical image stabilization and electronic image stabilization.

According to the first possible embodiment of the first aspect, the cutoff frequencies of the HPF and LPF are the same as each other.

In the first possible embodiment of the first aspect or the first aspect, according to the second possible embodiment of the first aspect, the cutoff frequency is included in the control band of the detection signal.

In the first or second possible embodiment of the first aspect or the first aspect, according to the third possible embodiment of the first aspect, the cutoff frequency controller cuts according to the ambient brightness. Change the off frequency.

In any one of the first to third possible embodiments of the first aspect or the first aspect, according to the fourth possible embodiment of the first aspect, the cutoff frequency controller is exposed. Change the cutoff frequency over time.

In any one of the first to fourth possible embodiments of the first aspect or the first aspect, according to the fifth possible embodiment of the first aspect, the imaging device is an imaging device. It further includes a pan-tilt signal processor that determines whether or not it is in a pan-tilt state and operates a first correction unit and a second correction unit based on the result of the determination.

The second aspect is an image stabilization device, which is a camera shake correction device.
A vibration detection unit that detects the vibration of the imaging device and outputs the detection signal,
A signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal.
A first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor, and
It includes a second correction unit that performs electronic image stabilization based on the low frequency vibration signal separated by the signal processor.
The signal processor provides a camera shake correction device including a cutoff frequency controller that variably sets the cutoff frequencies of the HPF and LPF in the frequency band.

A second aspect can provide an image stabilization device that effectively applies optical image stabilization and electronic image stabilization.

According to the first possible embodiment of the second aspect, the cutoff frequencies of the HPF and the LPF are the same as each other.

In the first possible embodiment of the second aspect or the second aspect, according to the second possible embodiment of the second aspect, the cutoff frequency is included in the control band of the detection signal.

In the first or second possible embodiment of the second aspect or the second aspect, according to the third possible embodiment of the second aspect, the cutoff frequency controller cuts according to the ambient brightness. Change the off frequency.

In any one of the first to third possible embodiments of the second aspect or the second aspect, according to the fourth possible embodiment of the second aspect, the cutoff frequency controller is exposed. Change the cutoff frequency over time.

In any one of the second aspect or the first to fourth possible embodiments of the second aspect, according to the fifth possible embodiment of the second aspect, the image pickup device is an image pickup device. It further includes a pan-tilt signal processor that determines whether or not it is in a pan-tilt state and operates a first correction unit and a second correction unit based on the result of the determination.

The third aspect is an image pickup method in an image pickup device, wherein the image pickup method is used.
Detecting the vibration of the imaging device and outputting the detection signal,
Using a high-pass filter (HPF) and a low-pass filter (LPF) to separate the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal,
While electronic image stabilization is performed based on the low-frequency vibration signal, optical image stabilization is performed based on the separated high-frequency vibration signal.
Includes processing images corrected by optical image stabilization and / or electronic image stabilization.
The cutoff frequencies of the HPF and LPF are variably configurable in the frequency band to provide an imaging method.

A third aspect can provide an imaging method that effectively applies optical image stabilization and electronic image stabilization.

According to the first possible embodiment of the third aspect, the cutoff frequencies of the HPF and LPF are the same as each other.

In the first possible embodiment of the third aspect or the third aspect, according to the second possible embodiment of the third aspect, the cutoff frequency is included in the control band of the detection signal.

In the first or second possible embodiment of the third aspect or the third aspect, according to the third possible embodiment of the third aspect, the cutoff frequency controller cuts according to the ambient brightness. Change the off frequency.

According to the fourth possible embodiment of the third aspect in any one of the third aspect or the first to third possible embodiments of the third aspect, the cutoff frequency controller is exposed. Change the cutoff frequency over time.

According to the fifth possible embodiment of the third aspect, in any one of the first to fourth possible embodiments of the third aspect or the third aspect, based on the low frequency vibration signal. While performing electronic image stabilization, performing optical image stabilization based on the separated high-frequency vibration signal is performed according to the result of determination of whether the imaging device is in the pan-tilt state.

The fourth aspect is a camera shake correction method.
Detecting the vibration of the imaging device and outputting the detection signal,
Using a high-pass filter (HPF) and a low-pass filter (LPF) to separate the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal,
Includes electronic image stabilization based on low-frequency vibration signals, while optical image stabilization based on separated high-frequency vibration signals.
The cutoff frequencies of the HPF and LPF provide a camera shake correction method that can be variably set in the frequency band.

A fourth aspect can provide an image stabilization method that effectively applies optical image stabilization and electronic image stabilization.

According to the first possible embodiment of the fourth aspect, the cutoff frequencies of the HPF and LPF are the same as each other.

In the first possible embodiment of the fourth aspect or the fourth aspect, according to the second possible embodiment of the fourth aspect, the cutoff frequency is included in the control band of the detection signal.

In the first or second possible embodiment of the fourth aspect or the fourth aspect, according to the third possible embodiment of the fourth aspect, the cutoff frequency controller cuts according to the ambient brightness. Change the off frequency.

In any one of the fourth aspect or the first to third possible embodiments of the fourth aspect, according to the possible embodiment of the fourth aspect of the fourth aspect, the cutoff frequency controller. , Change the cutoff frequency according to the exposure time.

According to the fifth possible embodiment of the fourth aspect in any one of the fourth aspect or the first to fourth possible embodiments of the fourth aspect, based on the low frequency vibration signal. While performing electronic image stabilization, performing optical image stabilization based on the separated high-frequency vibration signal is performed according to the result of determination of whether the imaging device is in the pan-tilt state.

In order to more clearly explain the technical solution in the embodiment, the accompanying drawings necessary for explaining the embodiment will be briefly described below. Obviously, the attached drawings in the following description show only some of the possible embodiments, and one of ordinary skill in the art will still derive other drawings from these attached drawings without creative effort. Sometimes.

It is a figure which shows the functional structure of the image pickup device by one Embodiment of this disclosure. It is a figure which shows the structure of the camera shake correction unit by one Embodiment of this disclosure. It is a figure for demonstrating optical image stabilization. It is a figure for demonstrating electronic image stabilization. It is a flowchart which shows the operation procedure of the image stabilization device. It is a figure for demonstrating the setting of the cutoff frequency by one Embodiment of this disclosure. It is a figure for demonstrating the setting of the cutoff frequency by one Embodiment of this disclosure. It is a figure which shows the structure of the camera shake correction unit by one Embodiment of this disclosure. It is a figure which shows the schematic structure of the smartphone which carries out the image pickup device by one Embodiment of this disclosure. It is a figure which shows the structure of the camera shake correction unit by one Embodiment of this disclosure.

In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present disclosure. .. Obviously, the embodiments described are only a portion, but not all, of the embodiments of the present disclosure. All other embodiments acquired by one of ordinary skill in the art based on the embodiments of the present disclosure without creative effort shall be within the scope of protection of the invention.

(First Embodiment)
FIG. 1 is a diagram showing a functional configuration of an image pickup device 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the image pickup device 1 includes an image pickup unit 2 and an image processor 3 that processes an image corrected through image blur correction. The image pickup unit 2 includes a camera shake correction unit 100 for correcting image blur. The configuration of the image stabilization unit 100 will be described in detail later with reference to FIG. 2 and the like.

Further, the image pickup device 1 includes a display unit 4, a storage unit 5, and an input unit 6, and is configured so that these components can communicate with each other. The image pickup device 1 may be configured as an electronic device such as a smartphone, a digital camera, a personal digital assistant, or a game device.

The image pickup unit 2 is configured to include a lens unit having a lens, an autofocus (AF) mechanism, and an image stabilization mechanism, and accumulates electric charges according to an optical image formed on an image sensor through the lens. .. The image pickup unit 2 converts the accumulated charge into a voltage, performs predetermined signal processing on the converted voltage, and outputs the signal obtained through the signal processing to the image processor 3. The signal processing performed here may include various types of signal processing such as optical image stabilization or image stabilization based on electronic image stabilization, automatic white balance processing, automatic exposure processing, noise reduction, and the like. The image stabilization unit 100 may be configured by a processor such as a central processing unit (CPU), for example.

The image processor 3 executes image processing using the signal output from the image pickup unit 2. The image processing executed by the image processor 3 may include, for example, demographic processing on the signal output from the image pickup unit 2, control of displaying the image data on the display unit 4, and storage of the image data on the storage unit 5. The image processor 3 may be configured as, for example, an image signal processor (ISP).

The display unit 4 is a display device that displays an image under the control of the image processor 3. The display unit 4 may be composed of, for example, a liquid crystal display (LCD), an organic light emitting display (OECD), or the like.

The storage unit 5 stores an image captured by the image pickup unit 2 and processed by the image processor 3. The image stored by the storage unit 5 may be displayed on the display unit 4 according to the user's operation on the image pickup device 1. The storage unit 5 may include the form of a computer-readable storage medium such as volatile memory, random access memory (RAM) and / or non-volatile memory, such as flash memory.

Computer-readable storage media may include volatile or non-volatile types, removable or non-removable media that may store information using any method or technique. The information may include computer-readable instructions, data structures, program modules, or other data.

The input unit 6 is configured to input user commands and image data to the image pickup device 1. Specifically, the input unit 6 is composed of a touch panel, a switch, a button, and the like. User instructions include image capture, image data editing, recording, deletion, and the like. The input unit 6 may be configured as a communication interface for exchanging data with another electronic device such as a personal computer. In this case, user commands and data input from another electronic device are executed.

Next, the configuration of the image stabilization unit 100 according to the present embodiment will be described with reference to FIG. 2.

The image stabilization unit 100 is
A gyro sensor 102 as a vibration detection unit that detects the vibration of the image pickup device 1 and outputs a detection signal,
A signal processor 104 that includes a low-pass filter (LPF) 106 and a high-pass filter (HPF) 108 and separates the detection signal output from the gyro sensor 102 into a high-frequency vibration signal and a low-frequency vibration signal.
An optical image stabilization (OIS) signal processor 110 as a first correction unit that performs optical image stabilization based on a high frequency vibration signal separated by the signal processor 104, and
It is configured to include an electronic image stabilization (EIS) signal processor 112 as a second image stabilization unit that performs electronic image stabilization based on the low frequency vibration signal separated by the signal processor 104.

The image stabilization unit 100 is further configured to include a driver (DRV) 114, a lens unit 115, an actuator 116, and an image sensor 117. Hereinafter, each of these components will be described.

The gyro sensor 102 detects the posture of the image pickup device 1 by measuring the angular velocity and outputs the detected signal, that is, functions as a vibration detection unit.

The signal processor 104 includes a low-pass filter 106, a high-pass filter 108, and a cutoff frequency (fc) controller 109. The low-pass filter 106 performs a filtering process for passing a low-frequency vibration signal in the detection signal. The high-pass filter 108 performs a filtering process for passing a high-frequency vibration signal in the detection signal. The cutoff frequency (fc) controller 109 controls to set and change the cutoff frequency of the low-pass filter 106 and the high-pass filter 108.

The OIS signal processor 110 outputs a drive signal for controlling the actuator 116 to the driver 114. The driver 114 drives the actuator 116 according to a drive signal from the OIS signal processor 110. The EIS signal processor 112 corrects the blur of the captured image. The lens unit 115 operates under the control of the driver 114 and forms an optical image on the image sensor 117.

Next, with reference to FIG. 3, the optical image stabilization performed by the OIS signal processor 110 will be described. In the figure, the lens unit 115 includes a focal lens 115-1, a correction lens 115-2, and another lens 115-3. The light beam incident from the left side of the figure in which the subject is placed passes through the focal lens 115-1, the correction lens 115-2, and another lens 115-3 along the axis 301 shown by the solid line, and passes through the image sensor 117. Form an optical image on top. The posture of the image pickup device 1 is tilted, the light ray enters along the axis 301'indicated by the broken line, and the arrival point of the light ray shifts from the point P on the image sensor 117 to which the light ray should reach to the point P'. When done, this shift becomes a handshake. This shift is offset by moving the correction lens 115-2 to the driver 114 in the vertical direction indicated by arrow A or in the direction perpendicular to the surface of the sheet indicated by arrow B.

Next, with reference to FIG. 4, the electronic image stabilization performed by the EIS signal processor 112 will be described. EIS is camera shake correction executed through inter-frame image processing based on the posture information of the image pickup device 1 detected by the gyro sensor 102, and correction is performed by shifting continuous image data in a moving image or the like on a frame-by-frame basis. conduct. As shown in FIG. 4, when an image is output from the image sensor 117 in the order of frames 402, 404, 406, the EIS signal processor 112 uses the frame based on the attitude information to correct the blur of the image 408. Shift the relative positions of frame 404 and frame 406 with respect to 402. EIS, such as OIS, is a technology implemented in cameras mounted on smartphones and the like.

With respect to the aforementioned OIS and EIS, firstly, handshake oscillations generally tend to have large amplitudes at low frequencies and small amplitudes at high frequencies.

Since OIS is a technique for performing image stabilization at the time of imaging, it is possible to effectively perform image stabilization essentially without depending on the exposure time. However, since OIS uses a mechanism for physically moving the lens, the movable range of the lens is limited by the size of this mechanism. Therefore, it is impractical to implement a mechanism that allows the OIS to deal with relatively large handshakes. On the other hand, unlike OIS, EIS is not restricted by a physical mechanism, so that EIS can perform image stabilization to cope with a large handshake in the low frequency region. However, in the case of an imaging device that uses a CMOS sensor as an image sensor, a phenomenon called a rolling shutter may occur in which the captured image is deformed by a handshake or a fast movement of the subject. This phenomenon is caused by the fact that blurring occurs in one frame of the image because the exposure start timing is different for each line to be imaged. As described above, it is not easy for EIS to perform image stabilization by itself when blurring occurs within one frame of an image. Subject handshakes and fast movements are more likely to occur in the high frequency band. Therefore, EIS is inherently prone to poor performance in the high frequency range.

From the above, according to the present embodiment, the handshake detection signal by the gyro sensor 102 is separated into a high frequency signal and a low frequency signal, and the image stabilization based on the high frequency signal is performed by EIS while the image stabilization is performed based on the high frequency signal. Is executed by OIS.

Next, the operation procedure in the image stabilization unit 100 will be described with reference to the flowchart of FIG.

In step S502, the vibration of the image pickup device 1 is detected, and the detection signal is output. Information about vibration due to changes caused by the handshake in the posture of the image pickup device 1 is detected by the gyro sensor 102. The detection signal from the gyro sensor 102 is output to the signal processor 104.

In step S504, the high-pass filter 108 and the low-pass filter 106 are used to separate the output detection signal into a high-frequency signal and a low-frequency signal. The detection signal from the gyro sensor 102 is processed by the low-pass filter 106 to generate a low frequency vibration signal. In parallel with this processing, the signal input to the signal processor 104 is processed by the high-pass filter 108 to generate a high frequency vibration signal. The cutoff frequency of the high-pass filter 108 and the low-pass filter 106 set by the cutoff frequency (fc) controller 109 can be variably set in the frequency band.

In step S506, the EIS image stabilization is performed based on the separated low frequency vibration signal, while the OIS image stabilization is performed based on the separated high frequency vibration signal. The high frequency vibration signal output from the high-pass filter 108 is processed by the OIS signal processor 110. This signal is input to the driver 114, which converts the input signal into a signal for driving the actuator 116. The lens unit 115 is controlled to be moved by the actuator 116 and cancels out the blur of the image generated by the high frequency vibration.

On the other hand, the low frequency vibration signal output from the low-pass filter 106 is input to the EIS signal processor 112. The EIS signal processor 112 processes a plurality of frames of the moving image data provided by the image sensor 117 based on the input signal to correct the image. Next, the moving image data that has undergone image stabilization performed by the EIS signal processor 112 is output to the image processor 3.

Next, in step S508, the image corrected by OIS and / or EIS is processed. The image processor 3 performs various types of processing such as demosaic processing for an input signal, display control of image data in a display unit 4, and storage of image data in a storage unit 5.

According to the above-mentioned operation, the signal output from the gyro sensor 102 is processed by the high-pass filter 108 to generate a signal in the high frequency region in the control band, and this signal is used for OIS signal processing.

Further, the signal output from the gyro sensor 102 is processed by the low-pass filter 106 to generate a signal in the low frequency region of the control band, and this signal is used for EIS signal processing.

Next, the setting of the cutoff frequency (fc) according to the present embodiment will be described with reference to FIGS. 6 and 7.

FIG. 6 is a diagram showing an example of the cutoff frequency of the low-pass filter 106 and the high-pass filter 108. In FIG. 6, the cutoff frequency of the low-pass filter 106 coincides with the cutoff frequency of the high-pass filter 108. This makes it possible to completely flatten the frequency characteristics of the image stabilization process. That is, the image stabilization can be seamlessly performed over the entire control band of the detection signal. FIG. 7 shows another example of the cutoff frequency setting position. In FIG. 7, the cutoff frequency of the low-pass filter 106 coincides with the cutoff frequency of the high-pass filter 108. As shown in FIGS. 6 and 7, the cutoff frequency should preferably be set within the control band.

In the present embodiment, the cutoff frequencies of the high frequency vibration signal and the low frequency vibration signal can be variably set in the frequency band. The signal processor 104 detects the ambient brightness of the image pickup device and the amplitude of the handshake vibration, and sets the value of the cutoff frequency fc according to the detection result. This control is performed in the cutoff frequency (fc) controller 109 through the implementation of an algorithm for performing the following operations.

In the determination based on the detection signal from the gyro sensor 102, the cutoff frequency (fc) controller 109 determines that OIS-based camera shake correction is possible when the amplitude of the handshake vibration is less than a specific value. However, the cutoff frequency is reduced to near the lower limit of the control band. Execution of such control can perform OIS-based image stabilization without depending on the exposure time.

On the other hand, in the determination based on the detection signal from the gyro sensor 102, when the amplitude of the handshake vibration is equal to or more than the specific value, the cutoff frequency (fc) controller 109 can perform OIS-based image stabilization. It is determined that the state is out of a certain range. Next, the cutoff frequency (fc) controller 109 shifts the cutoff frequency fc to the high frequency side in the control band. That is, the cutoff frequency (fc) controller 109 sets the cutoff frequency fc so that the OIS is applied to a frequency band having an amplitude less than a specific value. Execution of such control allows the OIS to compensate only for high frequencies with relatively small amplitudes and allows the EIS to cope with large amplitudes and low frequency handshake vibrations.

In addition, when the exposure time becomes long due to imaging in a dark environment and handshake vibration occurs during the exposure time, the cutoff frequency fc is set near the lower limit of the control frequency band, regardless of the magnitude of the amplitude. Make sure to implement OIS-based camera shake correction. The cutoff frequency fc for performing OIS-based image stabilization may be set according to the determination as to whether the exposure time is longer than a specific value. Alternatively, the imaging device is provided with a measuring unit that measures the brightness of the surrounding environment, with a cutoff frequency fc depending on whether the brightness measured by the measuring unit is less than a certain value. It may be set.

(Second embodiment)
Next, a second embodiment of the present disclosure will be described with reference to FIG. FIG. 8 shows the functional configuration of the image stabilization unit 700 according to the present embodiment. The image stabilization unit 700 differs from the image stabilization unit 100 in that an accelerometer 118 is added to the configuration shown in FIG. Since the other components shown in FIG. 8 have the same functions as the components in FIG. 2 having the same reference numerals, detailed description of the components will be omitted.

The accelerometer 118 is connected to the signal processor 104, detects the vibration of the image pickup device 1, and outputs the detected signal to the signal processor 104. In this embodiment, the gyro sensor 102 and the accelerometer 118 function as a vibration detection unit. The signal processor 104 processes the detection signal output from the gyro sensor 102 and the accelerometer 118.

FIG. 9 is a diagram showing a schematic configuration of a smartphone 800 on which the imaging device according to the present embodiment is implemented. When the gyro sensor 102 is configured as a biaxial angular velocity sensor, the gyro sensor 102 is centered on rotation about the X-axis (pitch) indicated by arrow 801 and centered on the Y-axis (yaw) indicated by arrow 802. Detects the rotation. When the gyro sensor 102 is configured as a triaxial angular velocity sensor, the gyro sensor 102 further detects rotation about the Z axis (roll) indicated by arrow 805. In this embodiment, the accelerometer 118 is used to further ensure the detection of translation of the imaging device. Therefore, the movement in the X-axis direction indicated by the arrow 803 in FIG. 9 and the movement in the Y-axis direction indicated by the arrow 804 can also be detected.

The operation of the image stabilization unit 700 according to the present embodiment can be described with reference to FIG. In step S502, information about the handshake vibration is detected by the gyro sensor 102 and the accelerometer 118. The detection signal from the gyro sensor 102 and the accelerometer 118 is output to the signal processor 104. These signals may be coupled and processed in the signal processor 104 or may be processed separately. The processing after step S504 is the same as the processing of the above-described embodiment.

(Third embodiment)
Next, a third embodiment of the present disclosure will be described with reference to FIG. FIG. 10 shows the image stabilization unit 900 according to the present embodiment. The image stabilization unit 900 differs from the image stabilization unit 700 in that the pan-tilt signal processor 120 is added to the configuration shown in FIG.

When the user performs an image pickup (pan-tilt) while significantly changing the direction of the camera or moving the camera in an arbitrary direction, the image pickup device may mistakenly recognize the pan-tilt as a handshake. In this case, the image stabilization device performs image stabilization in consideration of pan-tilt as a handshake. As a result, image stabilization attempts to make the screen stand still in a temporarily stable state, but when the stabilization operation exceeds the controllable range of movement, it suddenly returns to an unstable state. offer. Therefore, when the vibration of the image pickup device 1 is detected, the pan-tilt signal processor 120 determines whether the vibration is a handshake or a pan-tilt. Hereinafter, the state of the imaging device during pan-tilt imaging is referred to as a pan-tilt state.

The pan-tilt signal processor 120 is configured to receive a detection signal from the signal processor 104. The signal received here may be a detection signal that has not been filtered, or may be a detection signal that has been filtered by the low-pass filter 106 or the high-pass filter 108. For example, when a user who is photographing a landscape on the right side of the user tries to photograph a person on the left side of the user, the user may significantly change the direction of the image pickup device 1 in the horizontal direction. In this case, the pan-tilt signal processor 120 determines that the state of the image pickup device is the pan-tilt state. Alternatively, the direction of the camera may be rotated to a range of 180 ° or 360 ° for panoramic photography, in which case the pan-tilt signal processor 120 also determines that the state of the imaging device is in the pan-tilt state. ..

Specifically, the pan-tilt signal processor 120 holds as a first designated value a value indicating a rotation range that can be corrected by camera shake correction. The pan-tilt signal processor 120 further holds a value for ending the pan-tilt state and restarting the image stabilization as a second designated value. When the detection signal in the detection signal output from the gyro sensor 102 indicating rotation about the X-axis or the Y-axis becomes equal to or higher than the first specified value, the pan-tilt signal processor 120 states that the image pickup device is in the pan-tilt state. Judge that there is. The pan-tilt signal processor 120 outputs a signal instructing the OIS signal processor 110 and the EIS signal processor 112 to stop the image stabilization. As a result, the image stabilization by the OIS signal processor 110 and the EIS signal processor 112 is stopped, and the correction lens 115-2 of the lens unit 115 returns to the original position (center). In this way, OIS and EIS are executed except when the image pickup device 1 is in the pan-tilt state.

Further, when the image stabilization is stopped, the pan-tilt signal processor 120 may instruct the signal processor 104 not to perform the filter processing by the low-pass filter 106 and the high-pass filter 108.

When the detection signal indicating rotation about the X-axis or the Y-axis becomes equal to or less than the second designated value, the pan-tilt signal processor 120 determines that the pan-tilt state has ended. The pan-tilt signal processor 120 outputs a signal instructing the OIS signal processor 110 and the EIS signal processor 112 to start image stabilization. As a result, the image stabilization by the OIS signal processor 110 and the EIS signal processor 112 is restarted.

The second designated value may be set to a value larger than the first designated value.

When the image stabilization is resumed, the pan-tilt signal processor 120 may instruct the signal processor 104 to resume the filtering process stopped by the low-pass filter 106 and the high-pass filter 108.

In the present embodiment, the cutoff frequencies of the low-pass filter 106 and the high-pass filter 108 are variably set within the frequency band based on the signal output from the pan-tilt signal processor 120 or the situation determined based on this signal. obtain.

For example, the cutoff frequency is set or changed by the pan-tilt signal processor 120 based on determining how close the OIS or EIS is currently to the limit of the range of motion. When the state of the image pickup device is the pan-tilt state in the determination based on the detection signal from the gyro sensor 102, the pan-tilt signal processor 120 controls the cutoff frequency fc with respect to the cutoff frequency (fc) controller 109 at a high frequency within the control band. Order to shift to the side. That is, the cutoff frequency fc is set to apply OIS to a frequency band having an amplitude less than a specified value. Such control allows the OIS to compensate only for high frequencies with relatively small amplitudes and allows the EIS to cope with large amplitudes and low frequency handshake vibrations.

EIS has been described, for example, as electronic image stabilization in the aforementioned embodiments, but digital image stabilization (DIS) may be used in place of or in conjunction with EIS. In the case of DIS, the capture range of the video image from the image sensor is shifted, and a plurality of continuously captured images are compared with each other to determine the deviation caused by the vibration of the captured image, and the determination is made. Camera shake correction is performed by shifting the area of a plurality of image data based on the deviation.

The above description is merely a specific method of carrying out the present invention, but is not intended to limit the scope of protection of the present invention. Modifications or alternatives readily apparent by one of ordinary skill in the art within the disclosed technical scope are within the scope of protection of the invention. Therefore, the scope of protection of the present invention shall be in accordance with the scope of protection of the claims.

Claims (24)

It ’s an imaging device,
It ’s an imaging unit,
A vibration detection unit that detects the vibration of the image pickup device and outputs a detection signal,
A signal processor including a high-pass filter (HPF) and a low-pass filter (LPF) that separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal.
A first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor, and a first correction unit.
An imaging unit including a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor.
Including an image processor that processes an image corrected by the first correction unit and / or the second correction unit.
The signal processor is an imaging device including a cutoff frequency controller that variably sets the cutoff frequencies of the HPF and the LPF in a frequency band. The imaging device according to claim 1, wherein the HPF and the LPF have the same cutoff frequency. The imaging device according to claim 1 or 2, wherein the cutoff frequency is included in the control band of the detection signal. The imaging device according to any one of claims 1 to 3, wherein the cutoff frequency controller changes the cutoff frequency according to the ambient brightness. The imaging device according to any one of claims 1 to 4, wherein the cutoff frequency controller changes the cutoff frequency according to an exposure time. Claims 1 to 5 further include a pan-tilt signal processor that determines whether or not the image pickup device is in a pan-tilt state and operates the first correction unit and the second correction unit based on the result of the determination. The imaging device according to any one of the above. It ’s an image stabilization device,
A vibration detection unit that detects the vibration of the image pickup device and outputs a detection signal,
A signal processor including a high-pass filter (HPF) and a low-pass filter (LPF) that separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal.
A first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor, and a first correction unit.
Includes a second correction unit that performs electronic image stabilization based on the low frequency vibration signal separated by the signal processor.
The signal processor is a camera shake correction device including a cutoff frequency controller that variably sets the cutoff frequencies of the HPF and the LPF in a frequency band. The image stabilization device according to claim 7, wherein the HPF and the LPF have the same cutoff frequency. The image stabilization device according to claim 7, wherein the cutoff frequency is included in the control band of the detection signal. The image stabilization device according to any one of claims 7 to 9, wherein the cutoff frequency controller changes the cutoff frequency according to the ambient brightness. The image stabilization device according to any one of claims 7 to 10, wherein the cutoff frequency controller changes the cutoff frequency according to an exposure time. Claims 7 to 11 further include a pan-tilt signal processor that determines whether or not the image pickup device is in a pan-tilt state and operates the first correction unit and the second correction unit based on the result of the determination. The image stabilization device according to any one of the above. It is an imaging method in an imaging device.
To detect the vibration of the image pickup device and output the detection signal,
Using a high-pass filter (HPF) and a low-pass filter (LPF) to separate the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal,
While electronic image stabilization is performed based on the low-frequency vibration signal, optical image stabilization is performed based on the separated high-frequency vibration signal.
Including processing the image corrected by the optical image stabilization and / or the electronic image stabilization.
An imaging method in which the cutoff frequencies of the HPF and the LPF can be variably set in a frequency band. 13. The imaging method according to claim 13, wherein the HPF and the LPF have the same cutoff frequency. The imaging method according to claim 13, wherein the cutoff frequency is included in the control band of the detection signal. The imaging method according to any one of claims 13 to 15, wherein the cutoff frequency can be variably set according to the ambient brightness. The imaging method according to any one of claims 13 to 16, wherein the cutoff frequency can be variably set according to the exposure time. Performing electronic image stabilization based on the low-frequency vibration signal while performing optical image stabilization based on the separated high-frequency vibration signal is the result of determining whether or not the image pickup device is in the pan-tilt state. The imaging method according to any one of claims 13 to 17, which is carried out according to the above. It ’s an image stabilization method.
Detecting the vibration of the imaging device and outputting the detection signal,
Using a high-pass filter (HPF) and a low-pass filter (LPF) to separate the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal,
It includes performing electronic image stabilization based on the low frequency vibration signal, while performing optical image stabilization based on the separated high frequency vibration signal.
A camera shake correction method in which the cutoff frequencies of the HPF and the LPF can be variably set in a frequency band. The image stabilization method according to claim 19, wherein the cutoff frequencies of the HPF and the LPF are the same as each other. The image stabilization method according to claim 19, wherein the cutoff frequency is included in the control band of the detection signal. The image stabilization method according to any one of claims 19 to 21, wherein the cutoff frequency can be variably set according to the ambient brightness. The image stabilization method according to any one of claims 19 to 22, wherein the cutoff frequency can be variably set according to the exposure time. Performing electronic image stabilization based on the low-frequency vibration signal while performing optical image stabilization based on the separated high-frequency vibration signal is the result of determining whether or not the image pickup device is in the pan-tilt state. The image stabilization method according to any one of claims 19 to 23, which is executed according to the above.

Images