WO2020132917A1

WO2020132917A1 - Imaging device, image stabilization device, imaging method and image stabilization method

Info

Publication number: WO2020132917A1
Application number: PCT/CN2018/123758
Authority: WO
Inventors: Yamamoto Takashi; Ariga HIROSHI; Ito Yuichi; Takizawa NARUO
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2018-12-26
Filing date: 2018-12-26
Publication date: 2020-07-02
Also published as: CN113243103A; CN113243103B; JP2022516481A; JP7324284B2

Abstract

An imaging device capable of effectively applying optical image stabilization and electronic image stabilization in a hybrid image stabilization system for each frequency band is provided. The imaging device (1) includes: an imaging unit (2) including a vibration detection unit that detects vibration and outputs a detection signal, a signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal, a first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor, and a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor; and an image processor (3) that processes an image corrected by the first correction unit and/or the second correction unit. The cutoff frequencies of the HPF and the LPF are variably settable in a frequency band.

Description

IMAGING DEVICE, IMAGE STABILIZATION DEVICE, IMAGING METHOD AND IMAGE STABILIZATION METHOD

Technical Field

The present invention relates to an imaging device, an image stabilization device, an imaging method and an image stabilization method that have a mechanism of correcting image blur due to vibration such as handshaking.

Background Art

Optical image stabilization (OIS) and electronic image stabilization (EIS) are known as image stabilization (IS) techniques for correcting image blur. Optical image stabilization corrects image blur by detecting vibration of an imaging device such as a camera by a gyro sensor, and the image shake correction is performed by moving a lens so as to cancel the movement caused by the vibration. On the other hand, electronic image stabilization (EIS) corrects image blur by shifting the image capturing range of an image sensor so as to cancel the amount of movement detected by the gyro sensor.

One form of a hybrid image stabilization system using both the optical image stabilization and the electronic image stabilization described above is a system that performs the correction of vibration on the OIS side, passes the amount of movement to the EIS side, and processes the residue with the EIS.

Summary of Invention

It is an objective of the invention to provide an imaging device, image stabilization device, imaging method and image stabilization method that can effectively apply optical image stabilization and electronic image stabilization in a hybrid image stabilization system for each frequency band.

A first aspect provides an imaging device including:

an imaging unit including:

a vibration detection unit that detects vibration of the imaging device and outputs a detection signal;

a signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal;

a first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor; and

a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor; and

an image processor that processes an image corrected by the first correction unit and/or the second correction unit,

wherein the signal processor includes a cutoff frequency controller that variably sets cutoff frequencies of the HPF and the LPF in a frequency band.

The first aspect can provide an imaging device that effectively applies optical image stabilization and electronic image stabilization.

According to a first possible implementation of the first aspect, the cutoff frequencies of the HPF and the LPF are identical to each other.

In the first aspect or the first possible implementation of the first aspect, according to a second possible implementation of the first aspect, the cutoff frequencies fall within a control band of the detection signal.

In the first aspect or the first or second possible implementation of the first aspect, according to a third possible implementation of the first aspect, the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.

In the first aspect or any one of the first to third possible implementations of the first aspect, according to a fourth possible implementation of the first aspect, the cutoff frequency controller changes the cutoff frequencies according to an exposure time.

In the first aspect or any one of the first to fourth possible implementations of the first aspect, according to a fifth possible implementation of the first aspect, the imaging device further includes a pan-tilt signal processor that determines whether the imaging device is in a pan-tilt state, and operates the first correction unit and the second correction unit according to a result of the determination.

A second aspect provides an image stabilization device including:

a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor,

The second aspect can provide an image stabilization device that effectively applies optical image stabilization and electronic image stabilization.

According to a first possible implementation of the second aspect, the cutoff frequencies of the HPF and the LPF are identical to each other.

In the second aspect or the first possible implementation of the second aspect, according to a second possible implementation of the second aspect, the cutoff frequencies fall within a control band of the detection signal.

In the second aspect or the first or second possible implementation of the second aspect, according to a third possible implementation of the second aspect, the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.

In the second aspect or any one of the first to third possible implementations of the second aspect, according to a fourth possible implementation of the second aspect, the cutoff frequency controller changes the cutoff frequencies according to an exposure time.

In the second aspect or any one of the first to fourth possible implementations of the second aspect, according to a fifth possible implementation of the second aspect, the imaging device further includes a pan-tilt signal processor that determines whether the imaging device is in a pan-tilt state, and operates the first correction unit and the second correction unit according to a result of the determination.

A third aspect provides an imaging method in an imaging device, including:

detecting vibration of the imaging device and outputting a detection signal;

separating the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal by using a high-pass filter (HPF) and a low-pass filter (LPF) ;

performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal; and

processing an image corrected by the optical image stabilization and/or the electronic image stabilization,

wherein cutoff frequencies of the HPF and the LPF are variably settable in a frequency band.

The third aspect can provide an imaging method that effectively applies optical image stabilization and electronic image stabilization.

According to a first possible implementation of the third aspect, the cutoff frequencies of the HPF and the LPF are identical to each other.

In the third aspect or the first possible implementation of the third aspect, according to a second possible implementation of the third aspect, the cutoff frequencies fall within a control band of the detection signal.

In the third aspect or the first or second possible implementation of the third aspect, according to a third possible implementation of the third aspect, the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.

In the third aspect or any one of the first to third possible implementations of the third aspect, according to a fourth possible implementation of the third aspect, the cutoff frequency controller changes the cutoff frequencies according to an exposure time.

In the third aspect or any one of the first to fourth possible implementations of the third aspect, according to a fifth possible implementation of the third aspect, the performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal is carried out depending on a result of determination as to whether the imaging device is in a pan-tilt state.

A fourth aspect provides an image stabilization method including:

detecting vibration of an imaging device and outputting a detection signal;

separating the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal by using a high-pass filter (HPF) and a low-pass filter (LPF) ; and

performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal,

The fourth aspect can provide an image stabilization method that effectively applies optical image stabilization and electronic image stabilization.

According to a first possible implementation of the fourth aspect, the cutoff frequencies of the HPF and the LPF are identical to each other.

In the fourth aspect or the first possible implementation of the fourth aspect, according to a second possible implementation of the fourth aspect, the cutoff frequencies fall within a control band of the detection signal.

In the fourth aspect or the first or second possible implementation of the fourth aspect, according to a third possible implementation of the fourth aspect, the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.

In the fourth aspect or any one of the first to third possible implementations of the fourth aspect, according to a fourth possible implementation of the fourth aspect, the cutoff frequency controller changes the cutoff frequencies according to an exposure time.

In the fourth aspect or any one of the first to fourth possible implementations of the fourth aspect, according to a fifth possible implementation of the fourth aspect, the performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal is carried out depending on a result of determination as to whether the imaging device is in a pan-tilt state.

Brief Description of Drawings

To describe the technical solutions in the embodiments more clearly, the following briefly describes the accompanying drawings required for describing the embodiments. Apparently, the accompanying drawings in the following description depict merely some of the possible embodiments, and a person of ordinary skill in the art may still derive other drawings, without creative efforts, from these accompanying drawings, in which:

[Fig. 1] Fig. 1 is a diagram showing the functional configuration of an imaging device according to an embodiment of the present disclosure.

[Fig. 2] Fig. 2 is a diagram showing the configuration of an image stabilization unit according to an embodiment of the present disclosure.

[Fig. 3] Fig. 3 is a diagram for describing optical image stabilization.

[Fig. 4] Fig. 4 is a diagram for describing electronic image stabilization.

[Fig. 5] Fig. 5 is a flowchart illustrating procedures of the operation of an image stabilization device.

[Fig. 6] Fig. 6 is a diagram for describing setting of cutoff frequencies according to an embodiment of the present disclosure.

[Fig. 7] Fig. 7 is a diagram for describing setting of cutoff frequencies according to an embodiment of the present disclosure.

[Fig. 8] Fig. 8 is a diagram showing the configuration of an image stabilization unit according to an embodiment of the present disclosure.

[Fig. 9] Fig. 9 is a diagram showing an overview configuration of a smartphone in which an imaging device according to an embodiment of the present disclosure is implemented.

[Fig. 10] Fig. 10 is a diagram showing the configuration of an image stabilization unit according to an embodiment of the present disclosure.

Description of Embodiments

To make persons skilled in the art understand the technical solutions in the present disclosure better, the following clearly and completely describes the technical solutions in the embodiments of the present disclosure with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are merely some rather than all of the embodiments of the present disclosure. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present invention.

(First Embodiment)

Fig. 1 is a diagram showing the functional configuration of an imaging device 1 according to an embodiment of the present disclosure. As shown in Fig. 1, the imaging device 1 includes an imaging unit 2, and an image processor 3 that processes an image corrected through image blur correction. The imaging unit 2 includes an image stabilization unit 100 for correcting image blur. The configuration of this image stabilization unit 100 will be described in detail later with reference to Fig. 2 and so on.

The imaging device 1 further includes a display unit 4, a storage unit 5 and an inputting unit 6, and is configured so that those components can communicate with one another. The imaging device 1 may be configured as an electronic device, such as a smartphone, a digital camera, a personal digital assistant, or a game device.

The imaging unit 2 is configured to include a lens unit including a lens, an auto-focus (AF) mechanism, and an image stabilization mechanism, and stores charges according to an optical image formed on an image sensor through the lens. The imaging unit 2 converts the stored charges to a voltage, performs predetermined signal processing on the converted voltage, and outputs a signal obtained through the signal processing to the image processor 3. The signal processing that is performed here may include various kinds of signal processing, such as image blur correction based on optical image stabilization or electronic image stabilization, automatic white balance processing, automatic exposure processing, and noise reduction. The image stabilization unit 100 may be configured by a processor like a central processing unit (CPU) , for example.

The image processor 3 executes image processing by using signals output from the imaging unit 2. The image processing executed by the image processor 3 may include, for example, de-mosaic processing on the signal output from the imaging unit 2, control on displaying image data to the display unit 4, and storage of image data into the storage unit 5. The image processor 3 may be configured as, for example, an image signal processor (ISP) .

The display unit 4 is a display device that displays an image under control of the image processor 3. The display unit 4 may be configured by, for example, a liquid crystal display (LCD) , an organic light-emitting display (OLED) or the like.

The storage unit 5 stores an image captured by the imaging unit 2 and subjected to image processing by the image processor 3. The image stored by the storage unit 5 may be displayed on the display unit 4 according to a user's operation on the imaging device 1. The storage unit 5 may include a form of a computer readable storage medium, such as a volatile memory, random access memory (RAM) and/or a non-volatile memory, for example, flash memory.

The computer readable storage medium may include a volatile or non-volatile type, removable or non-removable medium that may store information using any method or technique. Information may include computer readable instructions, a data structure, a program module, or other data.

The inputting unit 6 is configured to input a user instruction and image data to the imaging device 1. Specifically, the inputting unit 6 is configured by a touch panel, a switch, and buttons or the like. User instructions include capturing of an image, and edition, recording, deletion and the like of image data. The inputting unit 6 may be configured as a communication interface for exchanging data with another electronic device such as a personal computer (PC) . In this case, inputting of a user instruction and data is carried out from another electronic device.

Next, referring to Fig. 2, the configuration of the image stabilization unit 100 according to the present embodiment is described.

The image stabilization unit 100 is configured to include:

a gyro sensor 102 as a vibration detection unit that detects vibration of the imaging device 1 and outputs a detection signal;

a signal processor 104 that includes a low-pass filter (LPF) 106 and a high-pass filter (HPF) 108 and separates the detection signal output from the gyro sensor 102 into a high-frequency vibration signal and a low-frequency vibration signal;

an optical image stabilization (OIS) signal processor 110 as a first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor 104; and

an electronic image stabilization (EIS) signal processor 112 as a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor 104.

The image stabilization unit 100 is further configured to include a driver (DRV) 114, a lens unit 115, an actuator 116, and an image sensor 117. The following describes each of those components.

The gyro sensor 102 detects the posture of the imaging device 1 by measuring the angular velocity and outputs a detected signal, and thus serves as the vibration detection unit.

The signal processor 104 includes the low-pass filter 106, the high-pass filter 108, and a cutoff frequency (fc) controller 109. The low-pass filter 106 performs a filtering process of passing a low-frequency vibration signal in the detection signal. The high-pass filter 108 performs a filtering process of passing a high-frequency vibration signal in the detection signal. The cutoff frequency (fc) controller 109 performs control to set and change the cutoff frequencies of the low-pass filter 106 and the high-pass filter 108.

The OIS signal processor 110 outputs a drive signal for controlling the actuator 116 to the driver 114. The driver 114 drives the actuator 116 according to the drive signal from the OIS signal processor 110. The EIS signal processor 112 corrects blurring of a captured image. The lens unit 115 operates under the control of the driver 114 to form an optical image on the image sensor 117.

Next, optical image stabilization that is performed by the OIS signal processor 110 is described with reference to Fig. 3. In the figure, the lens unit 115 includes a focus lens 115-1, a correction lens 115-2, and another lens 115-3. A ray of light incident from the left-hand side in the figure where a subject is positioned passes through the focus lens 115-1, correction lens 115-2, and another lens 115-3 along an axis 301 indicated by a solid line to form an optical image on the image sensor 117. When the posture of the imaging device 1 tilts and a ray of light enters along an axis 301' indicated by a broken line so that the arrival point of the ray is shifted to a point P' from a point P on the image sensor 117 where the ray is supposed to arrive, this shift becomes handshaking. This shift is canceled out by causing the driver 114 to move the correction lens 115-2 in an up-down direction indicated by an arrow A, or in a direction perpendicular to the surface of the sheet indicated by an arrow B.

Next, electronic image stabilization that is performed by the EIS signal processor 112 is described with reference to Fig. 4. EIS is image stabilization carried out through interframe image processing based on posture information of the imaging device 1 that is detected by the gyro sensor 102, and performs stabilization by shifting consecutive image data in a moving picture or the like frame by frame. When images are output from the image sensor 117 in the order of

frames

402, 404 and 406, as shown in Fig. 4, the EIS signal processor 112 shifts the relative positions of the frame 404 and the frame 406 to the frame 402 based on posture information to correct blurring of an image 408. EIS like OIS is a technique which is implemented on a camera mounted in a smartphone or the like.

Regarding the foregoing OIS and EIS, first, handshake vibration generally tends to have a large amplitude at low frequencies and have a small amplitude at higher frequencies.

Since OIS is a technique for performing image stabilization at the time of imaging, OIS can effectively perform image stabilization inherently without depending on the exposure time. Since OIS uses a mechanism for physically moving the lenses, however, the movable range of the lenses is limited by the size of this mechanism. It is not therefore practical to implement a mechanism for allowing OIS to cope with relatively large handshaking. On the other hand, EIS unlike OIS is not subject to restriction by a physical mechanism, so that EIS can perform image stabilization to cope with large handshaking in a low-frequency region. In case of an imaging device that uses a CMOS sensor as an image sensor, however, a phenomenon called rolling shutter may occur in which a captured image is deformed by handshaking or quick motion of a subject. This phenomenon is originated from the occurrence of blurring within one frame of images due to the exposure initiation timing differing line by line to be imaged. When blurring occurs within one frame of images in this way, it is not easy for EIS to perform image stabilization itself. It is likely that handshaking or quick motion of a subject occurs in a high-frequency band. Accordingly, EIS is likely to inherently have a poor performance in a high-frequency region.

From the above, according to the present embodiment, a detection signal of handshaking by the gyro sensor 102 is separated into a high-frequency signal and a low-frequency signal, and image stabilization based on the high-frequency signal is carried out by OIS while image stabili zation based on the low-frequency signal is carried out by EIS.

Next, the operational procedures in the image stabilization unit 100 are described with reference to the flowchart in Fig. 5.

At step S502, the vibration of the imaging device 1 is detected and a detection signal is output. Information on the vibration based on a handshake-originated change in the posture of the imaging device 1 is detected by the gyro sensor 102. The detection signal from the gyro sensor 102 is output to the signal processor 104.

At step S504, the output detection signal is separated into a high-frequency signal and a low-frequency signal by using the high-pass filter 108 and the low-pass filter 106. The detection signal from the gyro sensor 102 is processed by the low-pass filter 106 to generate a low-frequency vibration signal. In parallel to this processing, the signal input to the signal processor 104 is processed by the high-pass filter 108 to generate a high-frequency vibration signal. The cutoff frequencies of the high-pass filter 108 and the low-pass filter 106 that are set by the cutoff frequency (fc) controller 109 are variably settable in the frequency band.

At step S506, image stabilization by OIS is performed based on the separated high-frequency vibration signal while image stabilization by EIS is performed based on the separated low-frequency vibration signal. The high-frequency vibration signal output from the high-pass filter 108 is processed by the OIS signal processor 110. This signal is input to the driver 114, which in turn converts the input signal to a signal for driving the actuator 116. The lens unit 115 is controlled to be moved by the actuator 116 to cancel out image blur generated by high-frequency vibration.

On the other hand, the low-frequency vibration signal output from the low-pass filter 106 is input to the EIS signal processor 112. The EIS signal processor 112 processes multiple frames of moving picture data provided from the image sensor 117 based on the input signal to correct the image. Then, the moving picture data subjected to image stabilization performed by the EIS signal processor 112 is output to the image processor 3.

Then, at step S508, the image corrected by OIS and/or EIS is processed. The image processor 3 performs various kinds of processing, such as de-mosaic processing on the input signal, control on displaying image data to the display unit 4, and storage of image data into the storage unit 5.

According to the foregoing operation, the signal output from the gyro sensor 102 is processed by the high-pass filter 108 to generate a signal in a high-frequency region in the control band, and this signal is used in the OIS signal processing.

Moreover, the signal output from the gyro sensor 102 is processed by the low-pass filter 106 to generate a signal in a low-frequency region in the control band, and this signal is used in the EIS signal processing.

Next, setting of the cutoff frequency (fc) according to the present embodiment is described with reference to Figs. 6 and 7.

Fig. 6 is a diagram showing an example of the cutoff frequencies of the low-pass filter 106 and the high-pass filter 108. In Fig. 6, the cutoff frequency of the low-pass filter 106 matches the cutoff frequency of the high-pass filter 108. This can completely flatten the frequency characteristic of the image stabilization processing. That is, image stabilization can be performed seamlessly over the entire control band of the detection signal. Fig. 7 shows another example of set positions of the cutoff frequencies. In Fig. 7, the cutoff frequency of the low-pass filter 106 also matches the cutoff frequency of the high-pass filter 108. As shown in Figs. 6 and 7, the cutoff frequency should desirably be set within the control band.

In the present embodiment, the cutoff frequencies for the high-frequency vibration signal and the low-frequency vibration signal can variably be set in the frequency band. The signal processor 104 detects the surrounding brightness of the imaging device or the amplitude of the handshake vibration, and sets the value of the cutoff frequency fc according to the detection result. This control is carried out through implementation of an algorithm for executing the following operations in the cutoff frequency (fc) controller 109.

When the amplitude of the handshake vibration is less than a specified value in the determination based on the detection signal from the gyro sensor 102, the cutoff frequency (fc) controller 109 determines that OIS-based image stabilization can be implemented, and reduces the cutoff frequency to near the lower limit of the control band. Execution of control in such a way can implement OIS-based image stabilization without depending on the exposure time.

When the amplitude of the handshake vibration is equal to or greater than the specified value in the determination based on the detection signal from the gyro sensor 102, on the other hand, the cutoff frequency (fc) controller 109 determines that the state is out of the range over which OIS-based image stabilization can be implemented. Then, the cutoff frequency (fc) controller 109 shifts the cutoff frequency fc to the high-frequency side within the control band. That is, the cutoff frequency (fc) controller 109 sets the cutoff frequency fc so that OIS is applied to the frequency band which has an amplitude less than the specified value. Execution of control in such a way allows OIS to correct only high frequencies with a relatively small amplitude, and allows EIS to cope with large-amplitude and low-frequency handshake vibration.

Also, when the exposure time becomes longer due to imaging under a dark environment so that handshake vibration occurs during the exposure time, the cutoff frequency fc may be set to near the lower limit of the control frequency band so as to implement OIS-based image stabilization regardless of the magnitude of the amplitude. The cutoff frequency fc for performing OIS-based image stabilization may be set according to the determination as to whether the exposure time is longer than the specified value. Alternatively, the imaging device may be provided with a measuring unit for measuring the brightness of the surrounding environment so that the cutoff frequency fc may be set depending on whether the brightness measured by the measuring unit is less than the specified value.

(Second Embodiment)

Next, a second embodiment of the present disclosure is described with reference to Fig. 8. Fig. 8 shows the functional configuration of an image stabilization unit 700 according to the present embodiment. The image stabilization unit 700 differs from the image stabilization unit 100 in that an accelerometer 118 is added to the configuration shown in Fig. 2. Since the other components shown in Fig. 8 have same functions as those components in Fig. 2 which respectively have same reference numerals, the detailed description of the components is omitted.

The accelerometer 118 is connected to the signal processor 104, and detects the vibration in the imaging device 1 and outputs a detection signal to the signal processor 104. In the present embodiment, the gyro sensor 102 and the accelerometer 118 serve as the vibration detection unit. The signal processor 104 processes the detection signals output from the gyro sensor 102 and the accelerometer 118.

Fig. 9 is a diagram showing an overview configuration of a smartphone 800 in which the imaging device according to the present embodiment is implemented. When the gyro sensor 102 is configured as a biaxial angular velocity sensor, the gyro sensor 102 detects the rotation about an X axis (pitch) indicated by an arrow 801, and the rotation about a Y axis (yaw) indicated by an arrow 802. When the gyro sensor 102 is configured as a triaxial angular velocity sensor, the gyro sensor 102 further detects the rotation about a Z axis (roll) indicated by an arrow 805. In the present embodiment, the use of the accelerometer 118 further ensures detection of a parallel movement of the imaging device. It is therefore possible to also detect the movement along the X-axial direction indicated by an arrow 803 in Fig. 9, and the movement along the Y-axial direction indicated by an arrow 804.

The operation of the image stabilization unit 700 according to the present embodiment can be described with reference to Fig. 5. At step S502, information on handshake vibration is detected by the gyro sensor 102 and the accelerometer 118. The detection signals from the gyro sensor 102 and the accelerometer 118 are output to the signal processor 104. These signals may be combined and processed in the signal processor 104, or may be processed separately. The processing following step S504 is the same as the processing in the foregoing embodiment.

(Third Embodiment)

Next, a third embodiment of the present disclosure is described with reference to Fig. 10. Fig. 10 shows an image stabilization unit 900 according to the present embodiment. The image stabilization unit 900 differs from the image stabilization unit 700 in that a pan-tilt signal processor 120 is added to the configuration shown in Fig. 8.

When a user performs imaging (panning-tilting) while significantly changing the direction of a camera, or moving the camera, in any direction, the imaging device may erroneously recognize panning-tilting as handshaking. In this case, the image stabilization device performs image stabilization considering the panning-tilting as handshaking. As a result, the image stabilization provides a video image such that the screen attempts to temporarily stay still at stabilized condition, but abruptly returns to the unstable condition when the stabilizing motion exceeds controllable movement range. Accordingly, when vibration of the imaging device 1 is detected, the pan-tilt signal processor 120 determines whether the vibration is handshaking or panning-tilting. Hereinafter the state of the imaging device during panning-tilting imaging is called a pan-tilt state.

The pan-tilt signal processor 120 is configured to receive a detection signal from the signal processor 104. The signal to be received here may be non-filtered detection signal or a detection signal undergone the filtering in the low-pass filter 106 or the high-pass filter 108. When a user who has been shooting a landscape on the user's right side attempts to shoot a person on the user's left side, for example, the user may significantly change the direction of the imaging device 1 in a horizontal direction. In this case, the pan-tilt signal processor 120 determines that the state of the imaging device is in the pan-tilt state. Alternatively, the direction of the camera may be turned over a range of 180° or 360° for panoramic shooting, in which case the pan-tilt signal processor 120 also determines that the state of the imaging device is in the pan-tilt state.

Specifically, the pan-tilt signal processor 120 holds a value indicating the turning range which can be corrected by image stabilization as a first specified value. The pan-tilt signal processor 120 further holds a value for terminating the pan-tilt state and resuming image stabilization as a second specified value. When a detection signal in detection signals output from the gyro sensor 102 which indicates rotation about the X axis or the Y axis becomes equal to or greater than the first specified value, the pan-tilt signal processor 120 determines that the state of the imaging device is in the pan-tilt state. The pan-tilt signal processor 120 outputs a signal instructing stopping of image stabilization to the OIS signal processor 110 and the EIS signal processor 112. As a result, image stabilization by the OIS signal processor 110 and the EIS signal processor 112 is stopped, so that the correction lens 115-2 of the lens unit 115 returns to the original position (center) . In this way, OIS and EIS are carried out except in the case where the imaging device 1 is in the pan-tilt state.

Also, at the time image stabilization is stopped, the pan-tilt signal processor 120 may instruct the signal processor 104 not to perform filtering by means of the low-pass filter 106 and the high-pass filter 108.

When the detection signal indicating rotation about the X axis or the Y axis becomes equal to or smaller than the second specified value, the pan-tilt signal processor 120 determines that the pan-tilt state has ended. The pan-tilt signal processor 120 outputs a signal instructing initiation of image stabilization to the OIS signal processor 110 and the EIS signal processor 112. As a result, image stabilization by the OIS signal processor 110 and the EIS signal processor 112 resumes.

The second specified value may be set larger than the first specified value.

At the time image stabilization is resumed, the pan-tilt signal processor 120 may instruct the signal processor 104 to resume filtering which has been stopped by the low-pass filter 106 and the high-pass filter 108.

In the present embodiment, the cutoff frequencies of the low-pass filter 106 and the high-pass filter 108 can be variably set in the frequency band based on the signal output from the pan-tilt signal processor 120 or the situation that is determined based on this signal.

For example, the cutoff frequencies are set or changed by the pan-tilt signal processor 120 based on the determination of the situation as to how close to the limit in the movable range OIS or EIS is currently approaching. When the state of the imaging device is in the pan-tilt state in the determination based on the detection signal from the gyro sensor 102, the pan-tilt signal processor 120 instructs the cutoff frequency (fc) controller 109 to shift the cutoff frequency fc to the high-frequency side within the control band. That is, the cutoff frequency fc is set so as to apply OIS to the frequency band having an amplitude smaller than the specified value. Such control allows OIS to correct only high frequencies with a relatively small amplitude, and allows EIS to cope with large-amplitude and low-frequency handshake vibration.

Although EIS has been described by way of example as electronic image stabilization in the foregoing embodiments, digital image stabilization (DIS) may be used in place of EIS or together with EIS. In the case of DIS, the capturing range of a video image from the image sensor is shifted to implement image stabilization by comparing a plurality of images consecutively shot with one another to determine a vibration-originated deviation in shot images, and shifting regions of the plurality of image data based on the determined deviation.

The foregoing descriptions are merely specific implementation manners of the present invention, but are not intended to limit the protection scope of the present invention. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed shall fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

An imaging device comprising:

an imaging unit including:

a vibration detection unit that detects vibration of the imaging device and outputs a detection signal;

a signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal;

a first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor; and

a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor; and

an image processor that proces ses an image corrected by the first correction unit and/or the second correction unit,

wherein the signal processor includes a cutoff frequency controller that variably sets cutoff frequencies of the HPF and the LPF in a frequency band.
The imaging device according to claim 1, wherein

the cutoff frequencies of the HPF and the LPF are identical to each other.
The imaging device according to claim 1 or 2, wherein the cutoff frequencies fall within a control band of the detection signal.
The imaging device according to any one of claims 1 to 3, wherein

the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.
The imaging device according to any one of claims 1 to 4, wherein

the cutoff frequency controller changes the cutoff frequencies according to an exposure time.
The imaging device according to any one of claims 1 to 5, further comprising:

a pan-tilt signal processor that determines whether the imaging device is in a pan-tilt state, and operates the first correction unit and the second correction unit according to a result of the determination.
An image stabilization device comprising:

a vibration detection unit that detects vibration of the imaging device and outputs a detection signal;

a signal processor that includes a high-pass filter (HPF) and a low-pass filter (LPF) and separates the detection signal output from the vibration detection unit into a high-frequency vibration signal and a low-frequency vibration signal;

a first correction unit that performs optical image stabilization based on the high-frequency vibration signal separated by the signal processor; and

a second correction unit that performs electronic image stabilization based on the low-frequency vibration signal separated by the signal processor,

wherein the signal processor includes a cutoff frequency controller that variably sets cutoff frequencies of the HPF and the LPF in a frequency band.
The image stabilization device according to claim 7, wherein

the cutoff frequencies of the HPF and the LPF are identical to each other.
The image stabilization device according to claim 7 or 8, wherein

the cutoff frequencies fall within a control band of the detection signal.
The image stabilization device according to any one of claims 7 to 9, wherein

the cutoff frequency controller changes the cutoff frequencies according to a surrounding brightness.
The image stabilization device according to any one of claims 7 to 10, wherein

the cutoff frequency controller changes the cutoff frequencies according to an exposure time.
The image stabilization device according to any one of claims 7 to 11, further comprising:

a pan-tilt signal processor that determines whether the imaging device is in a pan-tilt state, and operates the first correction unit and the second correction unit according to a result of the determination.
An imaging method in an imaging device, comprising:

detecting vibration of the imaging device and outputting a detection signal;

separating the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal by using a high-pass filter (HPF) and a low-pass filter (LPF) ;

performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal; and

processing an image corrected by the optical image stabilization and/or the electronic image stabilization,

wherein cutoff frequencies of the HPF and the LPF are variably settable in a frequency band.
The imaging method according to claim 13, wherein

the cutoff frequencies of the HPF and the LPF are identical to each other.
The imaging method according to claim 13 or 14, wherein

the cutoff frequencies fall within a control band of the detection signal.
The imaging method according to any one of claims 13 to 15, wherein

the cutoff frequencies are variably settable according to a surrounding brightness.
The imaging method according to any one of claims 13 to 16, wherein

the cutoff frequencies are variably settable according to an exposure time.
The imaging de method vice according to any one of claims 13 to 17, wherein

the performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal is carried out depending on a result of determination as to whether the imaging device is in a pan-tilt state.
An image stabilization method comprising:

detecting vibration of an imaging device and outputting a detection signal;

separating the output detection signal into a high-frequency vibration signal and a low-frequency vibration signal by using a high-pass filter (HPF) and a low-pass filter (LPF) ; and

performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal,

wherein cutoff frequencies of the HPF and the LPF are variably settable in a frequency band.
The image stabilization method according to claim 19, wherein

the cutoff frequencies of the HPF and the LPF are identical to each other.
The image stabilization method according to claim 19 or 20, wherein

the cutoff frequencies fall within a control band of the detection signal.
The image stabilization method according to any one of claims 19 to 21, wherein

the cutoff frequencies are changed according to a surrounding brightness.
The image stabilization method according to any one of claims 19 to 22, wherein

the cutoff frequencies are changed according to an exposure time.
The image stabilization method according to any one of claims 19 to 23, wherein

the performing optical image stabilization based on the separated high-frequency vibration signal while performing electronic image stabilization based on the low-frequency vibration signal is carried out depending on a result of determination as to whether the imaging device is in a pan-tilt state.