WO2022138122A1 - Information processing device, information processing method, and program - Google Patents

Abstract

The present technique relates to an information processing device, an information processing method and a program that make it possible to reduce power consumption during image blur correction. Detection data indicating the shake of an image capturing unit is acquired at predetermined time intervals. Control information for controlling a blur correcting unit that corrects image blur caused by said shake of the image capturing unit is generated on the basis of the acquired detection data. Said time intervals at which the detection data is acquired are changed in accordance with the state of said image blur.

Description

Information processing equipment, information processing methods, and programs

The present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program capable of reducing power consumption when image blur correction is performed.

Patent Documents 1 and 2 disclose a technique for automatically stopping image blur correction to reduce power consumption when it is not necessary to correct image blur in the photographing apparatus.

Japanese Unexamined Patent Publication No. 9-5913 Japanese Unexamined Patent Publication No. 4-20947

In Patent Documents 1 and 2, it is not possible to reduce the power consumption when image blur correction is performed in a situation where image blur is occurring.

This technology was made in view of such a situation, and makes it possible to reduce the power consumption when performing image stabilization.

The information processing device or program of the present technology acquires detection data indicating blurring of the imaging unit at predetermined time intervals, and corrects image blurring caused by the blurring of the imaging unit based on the acquired detection data. An information processing device that is a processing unit that generates control information for controlling the blur correction unit, and has a processing unit that changes the time interval for acquiring the detection data according to the image blur state, or an information processing unit thereof. It is a program for operating a computer as such an information processing device.

In the information processing method of the present technology, the processing unit of the information processing apparatus having the processing unit acquires detection data indicating blurring of the imaging unit at predetermined time intervals, and based on the acquired detection data, the information processing unit of the imaging unit. This is an information processing method that generates control information for controlling a blur correction unit that corrects an image blur caused by the blur, and changes the time interval for acquiring the detection data according to the state of the image blur.

In the present technique, detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, a blur correction unit that corrects image blurring caused by the blurring of the imaging unit is controlled. The control information is generated, and the time interval for acquiring the detection data is changed according to the state of the image blur.

It is a block diagram which showed the structural example of the embodiment of the camera to which this technique is applied. It is a figure explaining the blur correction processing by a blur correction part. It is a figure which showed the relationship between the degree of blurring and the sampling frequency in a sampling part. It is a flowchart which exemplifies the procedure of the setting process of the sampling frequency of IMU data performed by a blur correction device. It is a block exemplifying the hardware configuration of the blur correction device of FIG. It is a block diagram which shows the configuration example of the hardware of the computer which executes a series of processing by a program.

Hereinafter, embodiments of this technique will be described with reference to the drawings.

<Embodiment of a camera to which this technique is applied>
FIG. 1 is a block diagram showing a configuration example of an embodiment of a camera to which the present technology is applied.

In FIG. 1, the camera 1, which is an embodiment of a camera (imaging device) to which the present technology is applied, displays an image of a subject captured by the imaging unit 11 on a display unit 14 included in the camera 1, or a storage unit. Store in 15. The image captured by the camera 1 may be a moving image composed of frames having a predetermined period, or may be a still image.

The camera 1 has an image pickup unit 11, an image processing unit 12, an output control unit 13, a display unit 14, a storage unit 15, and a blur correction device 21.

The image pickup unit 11 captures an image of the subject. The image pickup unit 11 has an optical system 11A and an image pickup element 11B. The optical system 11A collects light from the subject and forms an image (optical image) of the subject on the light receiving surface of the image pickup device 11B. The image pickup device 11B photoelectrically converts the image formed on the light receiving surface and supplies it to the image processing unit 12 as an electric signal (image signal).

The image processing unit 12 performs general image processing such as white balance, interpolation processing (demosaiking processing), color conversion, gradation conversion, noise reduction, etc., on the image from the image pickup unit 11 (imaging element 11B), and , The process for blur correction (image blur correction) by the blur correction unit 36 described later is performed. The image processing unit 12 supplies the processed image to the output control unit 13.

The output control unit 13 generates an output image according to the output destination based on the image from the image processing unit 12, and outputs the output image as a signal in a format according to the output destination. For example, when displaying the image captured by the imaging unit 11 on the display unit 14, the output control unit 13 generates an output image in which predetermined information is appropriately added to the image from the image processing unit 12. , A standard signal corresponding to the display unit 14 is output to the display unit 14. When the image captured by the image pickup unit 11 is stored in the storage unit 15, the image from the image processing unit 12 is supplied to the storage unit 15 with a signal of a predetermined standard.

The display unit 14 is, for example, a display installed on the back surface of the camera 1. The display unit 14 displays the output image from the output control unit 13 and presents it to the user.

The storage unit 15 stores the image from the output control unit 13 in the storage medium.

(Blur correction device 21)
The image stabilization device 21 (image stabilization device) corrects (reduces) image blur (image blur) caused by blur such as camera shake that occurs in the camera 1 (imaging unit 11). The image blur means that the position of the image of the still subject in the image captured by the image pickup unit 11 fluctuates due to the blur of the camera 1. A stationary subject is a subject that is stationary in real space.

The image stabilization device 21 includes an IMU sensor 31, a sampling unit 32, a buffer memory 33, an image stabilization unit 34, an image stabilization table unit 35, an image stabilization unit 36, and an image stabilization level determination processing unit 37.

The IMU sensor 31 is a blur detection sensor that detects blurring of the camera 1 (imaging unit 11). Specifically, the IMU sensor 31 includes an acceleration sensor and an angular velocity sensor. The accelerometer detects acceleration in each of the three orthogonal axes fixed to the camera 1 (optical system 11A). The angular velocity sensor detects the angular velocity in the direction around each of the three orthogonal axes.

The IMU sensor 31 includes acceleration signals indicating acceleration in each axis of the three orthogonal axes detected by the acceleration sensor and angular velocities in the directions around each axis of the three orthogonal axes detected by the angular velocity sensor as detection signals indicating blurring of the camera 1. Is supplied to the angular velocity signal indicating the above and the sampling unit 32.

The sampling unit 32 samples (extracts) the detection signals (acceleration signal and angular velocity signal) from the IMU sensor 31 at predetermined time intervals and converts them from analog signals to digital signals. The sampling unit 32 supplies the detection data (acceleration data and angular velocity data) acquired at predetermined time intervals by sampling from the detection signals (acceleration signal and angular velocity signal) of the IMU sensor 31 to the buffer memory 33. Hereinafter, the detection data acquired from the IMU sensor 31 at a predetermined time interval (sampling cycle) is also referred to as IMU data. The IMU sensor 31 may output the detection signal as a digital signal. In that case, the sampling unit 32 does not need to convert the detection signal from the IMU sensor 31 into a digital signal.

The time interval (sampling cycle) in which the sampling unit 32 acquires the detection data from the IMU sensor 31 is changed based on the determination result of the image stabilization level determination processing unit 37 described later. The determination result of the image stabilization level determination processing unit 37 represents the level of the magnitude of image stabilization. In the present embodiment, the time interval for acquiring the detection data by the sampling unit 32 is changed by changing the number of times per unit time (sampling frequency) of sampling performed by the sampling unit 32.

The buffer memory 33 temporarily stores the detection data from the sampling unit 32. In the buffer memory 33, the detection data from the sampling unit 32 from the time when the latest detection data is stored to the time when a predetermined time goes back in the past is accumulated. The detection data stored in the buffer memory 33 is supplied (read) to the blur amount calculation unit 34 in order from the detection data at the oldest time. When the detection data stored in the buffer memory 33 is supplied to the blur amount calculation unit 34, the detection data is erased from the buffer memory 33.

The blur amount calculation unit 34 calculates the blur amount α generated in the camera 1 based on the detection data read from the buffer memory 33, and supplies it to the blur correction table unit 35. The amount of blur α represents, for example, the magnitude of blur in each of the three orthogonal axes fixed to the camera 1 in each axial direction and in the direction around each axis. The amount of blur α will be described later.

Here, the blur amount calculation unit 34 calculates the blur amount α at predetermined time intervals. Therefore, when the blur amount calculation unit 34 calculates the blur amount α at the next time point (post-point point) after calculating the blur amount α at a certain time point (previous time point), it is between the previous time point and the later time point. The detection data acquired from the IMU sensor 31 is used to calculate the amount of blur α at a later point in time. Since the calculation of the blur amount α at the later time point is started after a predetermined time has elapsed with respect to the previous time point, the detection data acquired at least during that period is accumulated in the buffer memory 33.

The time interval for the sampling unit 32 to acquire the detection data becomes shorter as the sampling frequency in the sampling unit 32 is higher (larger). Therefore, the higher the sampling frequency in the sampling unit 32, the higher the calculation accuracy of the blur amount α and the higher the accuracy of the blur correction. On the other hand, the higher the sampling frequency in the sampling unit 32, the larger the amount of detection data stored in the buffer memory 33. At the same time, the data amount and the processing amount of the detection data used for calculating the blur amount α increase. Therefore, the power consumption required for processing such as calculation of the blur amount α increases.

On the contrary, the time interval for the sampling unit 32 to acquire the detection data becomes longer as the sampling frequency in the sampling unit 32 is lower (smaller). Therefore, the lower the sampling frequency in the sampling unit 32, the lower the calculation accuracy of the blur amount α and the lower the accuracy of the blur correction. On the other hand, the lower the sampling frequency in the sampling unit 32, the smaller the amount of detection data stored in the buffer memory 33. At the same time, the amount of detection data and the amount of processing used to calculate the amount of blur α are reduced. Therefore, the power consumption required for processing such as calculation of the blur amount α is reduced.

When the magnitude and frequency of image blurring are small, the change in the amount of blurring α over time is small. In that case, even if the sampling frequency in the sampling unit 32 is low, the influence on the calculation accuracy of the blur amount α is small, and the accuracy of the blur correction is not significantly different from the case where the sampling frequency is high.

Therefore, the smaller the image blur or the smaller (lower) the frequency of the image blur, the lower the sampling frequency in the sampling unit 32, so that the blur correction is performed without deteriorating the accuracy of the blur correction. In some cases, it is possible to reduce power consumption. In the present embodiment, a case where the sampling frequency of the sampling unit 32 is changed in consideration of only the magnitude of the image blur will be described. However, the sampling frequency in the sampling unit 32 may be changed in consideration of only the frequency of image blur, or may be changed in consideration of both the magnitude and frequency of image blur. May be good. That is, in this technique, by changing the sampling frequency in the sampling unit 32 according to the state of image blurring, the consumption required for blurring correction even when blurring correction is performed without deteriorating the accuracy of blurring correction. Power can be reduced.

The blur correction table unit 35 is a blur correction amount as control information for controlling the blur correction unit 36 that corrects the image blur caused by the blur of the camera 1 with respect to the blur amount α of the camera 1 from the blur amount calculation unit 34. Calculate γ. A blur correction table in which an appropriate blur correction amount γ is associated with the blur correction amount α is used for calculating the blur correction amount γ. The image stabilization table is created in advance. The image stabilization table unit 35 supplies the calculated image stabilization amount γ to the image stabilization unit 36. However, the image stabilization table unit 35 may be calculated using a predetermined relational expression for calculating an appropriate image stabilization amount γ with respect to the image stabilization amount α as the image stabilization amount calculation unit. The blur correction amount γ will be described later.

The image stabilization unit 36 controls the range (shooting range) of the subject to be imaged by the image pickup unit 11 based on the image stabilization amount γ. As a method of image stabilization, an electronic method and an optical method are well known. The image stabilization device 21 of the present embodiment employs electronic image stabilization. The image stabilization unit 36 cuts out a part of a range (region) of the image (original image) supplied from the image pickup unit 11 to the image processing unit 12 as an effective image. The blur correction unit 36 controls the position of the cutout range and the like based on the blur correction amount γ, with the range for cutting out the effective image from the original image (cutout range) as the control target. As a result, the range of the subject captured within the range of the effective image is stabilized against the blur of the camera 1, and the image blur in the effective image is corrected.

Note that this technology is also effective for image stabilization devices that employ optical image stabilization instead of electronic image stabilization. The image stabilization unit 36 in the image stabilization device 21 when the optical image stabilization is adopted uses the correction optical system (correction lens) arranged in the optical system 11A of the image pickup unit 11 or the image pickup element 11B as a control target. It has a mechanism to change the position of the controlled object. The change in the position of the controlled object includes the change in the posture of the controlled object (rotation, etc.). In this case, the blur correction table unit 35 calculates the blur correction amount γ as control information for designating the position of the control target for correcting the image blur and supplies it to the blur correction unit 36. The image stabilization unit 36 controls the position of the control target according to the image stabilization amount γ.

The image stabilization level determination processing unit 37 determines the image stabilization level based on the image stabilization amount γ calculated by the image stabilization table unit 35. The image stabilization level is composed of a plurality of levels that represent the magnitude of the image stabilization amount γ by dividing it into a plurality of stages. In the case of the image stabilization unit 36 that performs electronic image stabilization according to the present embodiment, the image stabilization level is an image because the magnitude of the image stabilization amount γ directly represents the magnitude of image stabilization as described later. Represents the level of blur (degree of blur).

The image stabilization level consists of four levels of image stabilization (image stabilization amount γ), in descending order: large level, medium level, small level, and no level. The image stabilization level determination processing unit 37 determines which of the four levels the image stabilization magnitude belongs to based on the image stabilization amount γ. The image stabilization level is not limited to four levels, and may be any plurality of levels. The image stabilization level will be described later.

The image stabilization level determination processing unit 37 supplies the image stabilization level determined based on the image stabilization amount γ to the sampling unit 32. The sampling unit 32 changes the sampling frequency based on the image stabilization level from the image stabilization level determination processing unit 37. As a general rule, the sampling frequency in the sampling unit 32 is set to a higher value as the blur correction level is larger (higher), and is set to a lower value as the blur correction level is smaller (lower). A large image stabilization level means a large level of image stabilization, and a small image stabilization level means a small level of image stabilization.

By changing the sampling frequency in the sampling unit 32 according to the blur correction level, the accuracy of the blur correction does not decrease, and the power consumption required for the blur correction is reduced even when the blur correction is performed.

(Blur correction processing of the shake correction device 21)
Next, the image stabilization process of the image stabilization device 21 will be described in detail.

In the image stabilization device 21, camera shake and other blurring that occurs in the camera 1 (imaging unit 11) is detected by the acceleration sensor and the angular velocity sensor included in the IMU sensor 31. The accelerometer detects the acceleration of the camera 1 generated in each of the three orthogonal axes fixed to the camera 1 (optical system 11A). The angular velocity sensor detects the angular velocity of the camera 1 generated in the direction around each of the three orthogonal axes.

It is assumed that the three orthogonal axes fixed to the camera 1 are the x-axis, the y-axis, and the z-axis that are orthogonal to each other. The x-axis is an axis along the lateral direction (horizontal direction) of the camera 1. The y-axis is an axis along the vertical direction (vertical direction) of the camera 1. The z-axis is an axis along the optical axis direction (front-back direction of the camera 1) of the optical system 11A of the image pickup unit 11.

From the IMU sensor 31, the angular velocity signal indicating the acceleration in each axis of the three orthogonal axes detected by the acceleration sensor and the angular velocity in the direction around each axis detected by the angular velocity sensor are used as detection signals indicating the blurring of the camera 1. The indicated angular velocity signal is output.

The detection signal (acceleration signal and angular velocity signal) output from the IMU sensor 31 is sampled by the sampling unit 32 at a predetermined sampling frequency, and is temporarily stored in the buffer memory 33 as detection data. As a result, the buffer memory 33 stores the detection data (acceleration in each axis direction and angular velocity in each axis direction) indicating the shake of the camera 1 acquired by the sampling unit 32 at the time interval (sampling cycle) corresponding to the sampling frequency. The indicated detection data) is accumulated.

The sampling frequency in the sampling unit 32 is set to a value corresponding to the image stabilization level determined by the image stabilization level determination processing unit 37 (described later).

The detection data stored in the buffer memory 33 is read out by the blur amount calculation unit 34, and the blur amount calculation unit 34 calculates the blur amount α generated in the camera 1. When the detection data temporarily stored in the buffer memory 33 is read by the blur amount calculation unit 34, it is erased from the buffer memory 33.

Here, the amount of blur α represents the magnitude of blur in each of the three orthogonal axes (x-axis, y-axis, z-axis) and in the direction around each axis. Camera 1 shakes include angle shake in the x-axis direction (pitch direction), angle shake in the y-axis direction (yaw direction), rotation shake in the z-axis direction (roll direction), and x-axis direction (lateral direction). There are shift blurs in the y-axis direction (vertical direction), and shift blurs in the z-axis direction (front-back direction).

The angular velocity (magnitude) θx in the pitch direction can be calculated by the first-order integral of the angular velocity ωx in the direction around the x-axis. The angular velocity (magnitude) θy in the yaw direction can be calculated by the first-order integral of the angular velocity ωy in the direction around the y-axis. The rotational shake (magnitude) θz in the roll direction can be calculated by the first-order integral of the angular velocity ωz in the direction around the z-axis.

The shift blur (magnitude) Dx in the x-axis direction can be calculated by the second-order integral of the acceleration ax in the x-axis direction. The shift blur Dy in the y-axis direction can be calculated by the second-order integral of the acceleration ay in the y-axis direction. The shift blur Dz in the z-axis direction can be calculated by the second-order integral of the acceleration az in the z-axis direction.

The amount of blur α is a value having angular blur θx, θy, rotational blur θz, shift blur Dx, Dy, and Dz as components, and is represented by the amount of blur α (θx, θy, θz, Dx, Dy, Dz). The blur amount α (each component) is calculated by the blur amount calculation unit 34 at predetermined time intervals based on the detection data read from the buffer memory 33.

When the blur amount calculation unit 34 calculates the latest (current time) blur amount α, the detection data stored in the buffer memory 33 between the previous calculation time of the blur amount α and the current time is read out in order. , The amount of blur α at the current time is calculated by integration processing or the like. Of the components of the amount of blur α (θx, θy, θz, Dx, Dy, Dz), θx, θy, and θz are the angular velocity data in the x-axis direction and the y-axis direction in the detection data, respectively. It is calculated by processing the first-order integration using the angular velocity data and the angular velocity data in the direction around the z-axis. Dx, Dy, and Dz are calculated by processing the second-order integration using the acceleration data in the x-axis direction, the acceleration data in the y-axis direction, and the acceleration data in the z-axis direction among the detected data, respectively. To.

Note that the lower the sampling frequency in the sampling unit 32, the smaller the number of detection data used for calculating the blur amount α for one time, so that the processing amount is small and the power consumption is reduced.

The method of calculating the amount of blur α is not limited to a specific method. For example, shift blurs in the x-axis direction and the y-axis direction are the yaw direction (direction around the y-axis) centered on a position distant from the detection position of the shift blurs (for example, the main point position of the optical system 11A). It may be considered to be caused by angular deviation in the pitch direction (direction around the x-axis). In this case, the blur amount calculation unit 34 determines the distance between the detection position of the shift blur in the x-axis direction and the y-axis direction and the center of rotation, and the angular blur (magnitude) θy and θx in the yaw direction and the pitch direction, respectively. It is also possible to calculate the shift blur (magnitude) Dx and Dy in the x-axis direction and the y-axis direction, respectively.

Of the (θx, θy, θz, Dx, Dy, Dz) components of the amount of blur α, only (θx, θy, θz, Dx, Dy) is the amount of blur without considering the shift blur Dz in the z-axis direction. It may be calculated as a component of α. In this case, the acceleration sensor of the IMU sensor 31 does not have to detect the acceleration in the z-axis direction. Image blur is generally affected by angular blur θx in the pitch direction (direction around the x-axis) and yaw direction (direction around the y-axis), and θy, so it is a component of the amount of blur α (θx, θy). ) May be calculated. In this case, the acceleration sensor of the IMU sensor 31 is unnecessary, and the angular velocity sensor does not have to detect the angular velocity around the z-axis.

The blur amount calculation unit 34 may perform high-pass filter processing on the detection data read from the buffer memory 33 before calculating the blur amount α. That is, the frequency of blurring is generally about 1 Hz to 15 Hz. Therefore, the blur amount calculation unit 34 processes the detection data from the buffer memory 33 with a high-pass filter (digital filter), so that, for example, a noise component having a low frequency lower than 1 Hz from the detection signal of the IMU sensor 31 is used. , Drift component, detection data when the signal component caused by the image angle change operation by the user or the like is removed may be acquired.

The blur amount α (θx, θy, θz, Dx, Dy, Dz) calculated by the blur amount calculation unit 34 is supplied to the blur correction table unit 35, and the blur correction amount γ is calculated in the blur correction table unit 35. To.

The image stabilization table unit 35 has an image stabilization table in which an appropriate image stabilization amount γ is associated with the image stabilization amount α of the camera 1 as control information for controlling the image stabilization unit 36. The image stabilization table unit 35 calculates an appropriate image stabilization amount γ corresponding to the image stabilization amount α from the image stabilization unit 34 using the image stabilization table. The process of the image stabilization table unit 35 may be replaced with the process of calculating an appropriate image stabilization amount γ with respect to the image stabilization amount α from the image stabilization unit 34 by a predetermined relational expression.

The image stabilization amount γ calculated by the image stabilization table unit 35 is supplied to the image stabilization unit 36, and the image stabilization unit 36 is controlled based on the image stabilization amount γ.

The image stabilization unit 36 controls a range (cutout range) of a part of the image supplied from the image pickup unit 11 to the image processing unit 12 as an effective image. The position of the cutout range, which is the control target of the image stabilization unit 36, is changed based on the image stabilization amount γ.

FIG. 2 is a diagram illustrating a shake correction process by the shake correction unit 36.

In FIG. 2, the image 61 represents an image imaged by the image sensor 11B and supplied from the image pickup unit 11 to the image processing unit 12. Since the image 61 represents an image of each frame supplied from the image pickup unit 11 to the image processing unit 12 at predetermined time intervals, the position of the image of the same subject (subject image) reflected in the image 61 is the blur of the camera 1. It changes due to changes in the shooting direction and movement of the subject. The horizontal and vertical directions of the image 61 are the x-axis direction and the y-axis direction on the image, respectively.

The image range 61A represents the entire range (contour) of the image 61. The image range 61A of the image 61 may be the entire range of the image that can be captured by the light receiving surface of the image sensor 11B, or may be a partial range thereof.

The effective image 62 represents an image cut out by the blur correction unit 36 from the image range 61A of the image 61 supplied from the image pickup unit 11 to the image processing unit 12. The image of the portion other than the valid image in the image range 61A is removed as an invalid image. The effective image 62 cut out by the image stabilization unit 36 is supplied to the output control unit 13 after other processing by the image processing unit 12 is performed. The effective image 62 can also be interpreted as an image in the range of the image captured by the image pickup device 11B that is not removed and is output to the display unit 14 and the storage unit 15.

The cutout range 62A is the entire range (contour) of the effective image 62, and represents the range in which the effective image 62 is cut out from the image range 61A by the image stabilization unit 36. The cutout range 62A has, for example, a rectangular shape, has a shorter horizontal width and vertical width than the image range 61A, and is included in the image range 61A. The cutout range 62A is a control target in the image stabilization unit 36, and the position and inclination of the cutout range 62A are changed based on the image stabilization amount γ.

The cutout range 62A is set to the reference state, for example, at the start of image stabilization. In the reference state, the cutout range 62A is set to, for example, an inclination in which the center position coincides with the center position of the image range 61A and the lateral direction of the cutout range 62A is parallel to the x-axis direction on the image.

The subject image 63 represents an example of an image of a still subject captured in the effective image 62 when the cutout range 62A is set to the reference state. A still subject represents a subject that is stationary in real space.

Under such circumstances, blurring occurs in the camera 1, and the position and tilt of the still subject in the image range 61A changes from the position and tilt of the subject image 63 to the position and tilt of the subject image 63-1 due to the image blurring. Suppose you did.

Assuming that the amount of image blur at this time is the image blur amount β, the image blur amount β is the amount of change Δx representing the magnitude of the image blur in the x-axis direction on the image, and the magnitude of the image blur in the y-axis direction on the image. It is represented by the amount of image blur β (Δx, Δy, Δθ) with the amount of change Δy representing the amount of change and the amount of change Δθ representing the magnitude of image blurring in the rotation direction on the image as components.

The amount of change Δx is the x-axis direction (first) on the image of the position of the subject image 63-1 changed due to image blurring with respect to the position of the subject image 63 when the cutout range 62A is set to the reference state. Represents the magnitude of displacement in the linear direction).

The amount of change Δy is the y-axis direction (second) on the image at the position of the subject image 63-1 changed due to image blurring with respect to the position of the subject image 63 when the cutout range 62A is set to the reference state. Represents the magnitude of displacement in the linear direction).

The amount of change Δθ is the change in the angle of the rotation direction on the image to the inclination of the subject image 63-1 changed by the image blur with respect to the inclination of the subject image 63 when the cutout range 62A is set to the reference state. Represents the size. However, in FIG. 2, the amount of change Δθ is represented as 0.

For such an image blur of the image blur amount β (Δx, Δy, Δθ), the cutout range 62A has the same change as the image blur amount β (Δx, Δy, Δθ) with respect to the position and inclination of the reference state. It is assumed that the amount changes in the x-axis direction, the y-axis direction, and the rotation direction, and the position and inclination of the cutting range 62A-1 are changed. At this time, the position and inclination of the subject image 63-1 in the effective image 62-1 cut out from the cutout range 62A-1 coincide with the position and inclination of the subject image 63 in the effective image 62 cut out from the cutout range 62A in the reference state. do. Therefore, the effective image 62-1 is an image in which image blur is corrected.

Therefore, the image stabilization table unit 35 has an image blur amount β due to the blur with respect to the blur amount α (θx, θy, θz, Dx, Dy, Dz) of the camera 1 supplied from the blur amount calculation unit 34. (Δx, Δy, Δθ) is calculated as the blur correction amount γ (Δx, Δy, Δθ). The image stabilization unit 36 changes the position and inclination of the cutting range 62A with respect to the position and inclination in the reference state according to the image stabilization amount γ (Δx, Δy, Δθ) supplied from the image stabilization table unit 35. This corrects the image blur.

Here, the change amount Δx, which is a component of the blur correction amount γ (Δx, Δy, Δθ), represents the change amount from the position of the reference state of the cutout range 62A in the x-axis direction on the image. The change amount Δy, which is a component of the blur correction amount γ (Δx, Δy, Δθ), represents the change amount from the position of the reference state of the cutout range 62A in the y-axis direction on the image. The change amount Δθ, which is a component of the blur correction amount γ (Δx, Δy, Δθ), represents the change amount from the inclination of the reference state of the cutout range 62A in the rotation direction on the image. The change amounts Δx, Δy, and Δθ, which are components of the image stabilization amount γ (Δx, Δy, Δθ), are equivalent to the change amounts Δx, Δy, and Δθ, which are components of the image blur amount β (Δx, Δy, Δθ), respectively. be.

The amount of change Δx, Δy, which is a component of the amount of blur correction γ (Δx, Δy, Δθ), and Δθ, that is, the amount of change Δx, Δy, and the amount of change, which is a component of the amount of image blur β (Δx, Δy, Δθ). Δθ has a relationship as shown in the following equations (1) to (3) with respect to the amount of blur α (θx, θy, θz, Dx, Dy, Dz).

Δx ＝ a1 ・ θy ＋ b1 ・ Dx ・ ・ ・ (1)
Δy ＝ a2 ・ θx ＋ b2 ・ Dy ・ ・ ・ (2)
Δθ ＝ a3 ・ θz ・ ・ ・ (3)
However, a1, b1, a2, b2, and a3 are predetermined constants.

The shift blur Dz in the z-axis direction of the camera 1, which is a component of the blur amount α (θx, θy, θz, Dx, Dy, Dz), mainly affects the image blur due to the fluctuation of the focus position, and the focus adjustment of the optical system 11a. Blurring correction is performed by. When the camera 1 has an autofocus function as focus adjustment, it is automatically corrected by continuous autofocus. Therefore, in the above equations (1) to (3), the shift blur Dz does not contribute to the calculation of the blur correction amount γ (Δx, Δy, Δθ). In the image stabilization, it is not always necessary to detect the shift image Dz, and the IMU sensor 31 as the image detection sensor may not detect the acceleration of the camera 1 in the z-axis direction.

The angle blur θy in the yaw direction (direction around the y-axis) of the camera 1 greatly contributes to the calculation of the change amount Δx (change amount Δx) in the x-axis direction of the cutout range (image blur) in the equation (1). The contribution of the shift blur Dx in the x-axis direction of 1 is small. Therefore, it is not always necessary to detect the shift blur Dx of the camera 1 in the blur correction, and the acceleration in the x-axis direction of the camera 1 may not be detected by the IMU sensor 31 as the blur detection sensor.

Similarly, in the calculation of the change amount Δy (change amount Δy) in the y-axis direction of the cutout range (image blur) in the equation (2), the angle blur θx in the pitch direction (x-axis direction) of the camera 1 is large. The contribution of the shift blur Dy in the y-axis direction of the camera 1 is small. Therefore, it is not always necessary to detect the shift blur Dy of the camera 1 in the blur correction, and the acceleration in the y-axis direction of the camera 1 may not be detected by the IMU sensor 31 as the blur detection sensor.

The change amount Δθ (change amount Δθ) of the cutout range (image blur) in the equation (3) does not necessarily have to be calculated because it has a small effect on the image blur. Therefore, only the change amounts Δx and Δy may be calculated as the blur correction amount γ (Δx, Δy, Δθ), and the cutting range may be displaced only in the x-axis direction and the y-axis direction. In this case, it is not always necessary to detect the rotational blur θz in the roll direction (direction around the z-axis) of the camera 1 in the blur correction, and the angular velocity of the camera 1 in the z-axis direction by the IMU sensor 31 as the blur detection sensor. It may be the case that the detection is not performed.

As described above, when the blur correction amount γ (image blur amount β) is calculated by setting the equation (1) as Δx = a1 · θy and the equation (2) as Δy = a2 · θx, the IMU sensor 31 as the blur correction sensor 31. May include only an angular velocity sensor without an accelerometer.

(Sampling frequency of sampling unit 32)
Next, the sampling frequency of the sampling unit 32 will be described.

In FIG. 1, the sampling frequency of the sampling unit 32 is changed according to the image stabilization level which is the determination result of the image stabilization level determination processing unit 37. The image stabilization level determination processing unit 37 calculates the size (comprehensive size) of the image shake (image stabilization amount γ) based on the image stabilization amount γ calculated by the image stabilization table unit 35, and the image stabilization level. To judge.

The magnitude of the blur correction amount γ represents the comprehensive magnitude of the operating range of the controlled object of the blur correction unit 36 operated to correct the image blur. At the same time, the magnitude of the blur correction amount γ means the magnitude (comprehensive magnitude) of the image blur (image blur amount β) caused by the blur of the camera 1.

That is, when electronic image stabilization is adopted as in the image stabilization device 21 of the present embodiment, the image stabilization amount γ is equivalent to the image image stabilization amount β as described above, so the calculation is as follows. The magnitude of the blur correction amount γ to be performed directly represents the magnitude of image blur. When the image stabilization device employs optical image stabilization, the image stabilization amount γ is an amount according to the operation content of the control target as control information for operating the control target of the image stabilization unit, so it is not necessarily an image. It does not match the amount of blur β. However, since the blur correction amount γ is indirectly related to the image blur amount β, the magnitude of the blur correction amount γ means the magnitude of the image blur even in the case of optical blur correction. In the case of optical image stabilization, the image stabilization amount β may be calculated from the image stabilization amount α instead of the magnitude of the image stabilization amount γ, and the magnitude of the image blur amount β may be calculated.

In the following, the magnitude of image blur is referred to instead of the magnitude of the image stabilization amount γ. Since the magnitude of the blur (blur amount α) of the camera 1 is also related to the magnitude of the image blur, the magnitude of the image blur also means the magnitude of the blur of the camera 1. Therefore, it is also referred to as the degree of blurring, including the meanings of both the magnitude of image blurring and the magnitude of blurring of the camera 1.

Since the blur correction amount γ, that is, the image blur amount β has a plurality of components, the blur correction level determination processing unit 37 determines the comprehensive magnitude of the image blur (blurring degree G) as the magnitude of the image blur. It is calculated by, for example, the following equations (4) and (5) based on the image stabilization amount γ (Δx, Δy, Δθ).

g ＝ a4 ・ (Δx) ² ＋ b4 ・ (Δy) ² ＋ c4 ・ (Δθ) ²・ ・ ・ (4)
G = max {g}} ・ ・ ・ (5)
However, a4, b4, and c4 are predetermined constants.

According to the equation (4), the value of the variable g is calculated based on the change amounts Δx, Δy, and Δθ. Since the change amounts Δx, Δy, and Δθ are oscillating values, the value of the variable g also fluctuates. According to the equation (5), the maximum value of the vibrating variable g is calculated as the degree of blurring G.

However, the degree of blurring G may be the average value of the variables g. The variable g in the equation (4) may be, for example, a value calculated by the changes Δx and Δy excluding the third term on the right side. In this case, the constants a4 and b4 may be set to the same value, and the size of the range in which the image is displaced due to image blurring on the image may be set to the degree of blurring G. The degree of blurring G does not have to be the value calculated by the equations (4) and (5), and may be a value for evaluating the comprehensive magnitude of image blurring. The blur condition G may be a value calculated based on the blur amount α calculated by the blur amount calculation unit 34, or a value calculated based on the detection data acquired from the IMU sensor 31. May be good.

The image stabilization level determination processing unit 37 determines the calculated image stabilization level of the degree G. As described above, the image stabilization level has four levels, that is, a large level, a medium level, a small level, and no level in descending order of the degree of blur G.

The image stabilization level 37 determines the image stabilization level of the image stabilization G by comparing the image stabilization level G with the predetermined threshold values H1, H2, and H3 (however, 0 <H1 <H2 <H3). Is determined. When the blur condition G is less than the threshold value H1, it is determined that the blur correction level of the blur condition G is no level. No level indicates a case where image blur is hardly generated to the extent that image stabilization is unnecessary. When the blur condition G is equal to or higher than the threshold value H1 and less than the threshold value H2, the blur correction level of the blur condition G is determined to be a small level. When the blur condition G is equal to or higher than the threshold value H2 and less than the threshold value H3, the blur correction level of the blur condition G is determined to be a medium level. When the blur condition G is equal to or higher than the threshold value H3, the blur correction level of the blur condition G is determined to be a large level.

The blur correction level of the blur condition G determined by the blur correction level determination processing unit 37 is supplied to the sampling unit 32.

The sampling unit 32 sets the sampling frequency when acquiring the detection data (IMU data) based on the image stabilization level of the image stabilization G, which is the determination result from the image stabilization level determination processing unit 37.

The sampling unit 32 sets, for example, the sampling frequency to one of three predetermined frequencies f1, f2, and f3 (however, 0 <f1 <f2 <f3).

FIG. 3 is a diagram showing the relationship between the degree of blurring G and the sampling frequency in the sampling unit 32.

Situations A to D in FIG. 3 represent cases where the magnitude of image blur is different. In the situation A, the image blur is larger than the standard (normal), and the blur correction level determination processing unit 37 determines that the blur correction level of the blur condition G is a large level. In this case, the sampling frequency in the sampling unit 32 is set to the maximum frequency f3 among the selectable frequencies f1 to f3.

In situation B, the image shake is a standard (normal) size, and the shake correction level determination processing unit 37 determines that the shake correction level of the shake condition G is a medium level. In this case, the sampling frequency in the sampling unit 32 is set to the intermediate frequency f2 among the selectable frequencies f1 to f3.

In situation C, the image shake is smaller than the standard (normal), and the shake correction level determination processing unit 37 determines that the shake correction level of the shake condition G is a small level. In this case, the sampling frequency in the sampling unit 32 is set to the smallest frequency f1 among the selectable frequencies f1 to f3.

In the situation D, almost no image blurring occurs, and the blurring correction level determination processing unit 37 determines that the blurring correction level of the blurring condition G is no level. In this case, the sampling frequency in the sampling unit 32 is set to the smallest frequency f1 among the selectable frequencies f1 to f3.

In the situation A, when the sampling frequency of the sampling unit 32 is set to the maximum frequency f3, the accumulated amount of the detection data (IMU data) stored in the buffer memory 33 is larger than that in the other situations B to D, and is the maximum. It becomes. At this time, since the data amount of the detection data used for calculating the blur amount α in the blur amount calculation unit 34 becomes large, the processing amount required for the blur correction (the processing amount of the DSP (Digital Signal Processor) described later) becomes large. It is the maximum compared to other situations B to D. Therefore, although the power consumption is maximized, the blur amount α and the blur correction amount γ are calculated with appropriate accuracy for the magnitude of the image blur.

In situation B, when the sampling frequency of the sampling unit 32 is set to the intermediate frequency f2, the amount of detection data (IMU data) stored in the buffer memory 33 is compared with the other situations A, C, and D. Medium (normal). At this time, since the data amount of the detection data used for the calculation of the blur amount α by the blur amount calculation unit 34 is medium, the processing amount (DSP processing amount) required for the blur correction is the other situation A. It becomes normal compared with C and D. Therefore, the power consumption is also normal, and the blur amount α and the blur correction amount γ are calculated with appropriate accuracy for the magnitude of the image blur.

In situation C, when the sampling frequency of the sampling unit 32 is set to the minimum frequency f1, the amount of detection data (IMU data) stored in the buffer memory 33 becomes the minimum as compared with the situations A and B. At this time, since the data amount of the detection data used for calculating the blur amount α in the blur amount calculation unit 34 is reduced, the processing amount (DSP processing amount) required for the blur correction is the situations A and B. Is the smallest compared to. Therefore, the power consumption is also the minimum as compared with the situations A and B. Since the image blur is small, even if the amount of detection data used for calculating the blur amount α in the blur amount calculation unit 34 is small, the blur amount α and the blur correction amount γ can be calculated with appropriate accuracy. Will be.

In the situation D, when the sampling frequency of the sampling unit 32 is set to the minimum frequency f1, the accumulated amount of the detection data (IMU data) stored in the buffer memory 33 becomes the minimum as compared with the situations A and B, and the situation C Is equivalent to. Further, as in the situation D, when the blur correction level of the blur condition G is determined to be no level, it is not necessary to perform the blur correction, so the DSP that performs the blur correction processing is put into the sleep state and the blur amount α or the blur is set. It is possible to stop the image stabilization without calculating the correction amount γ. Therefore, the power consumption is minimized as compared with other situations A to C.

<Sampling frequency setting process in the image stabilizer 21>
FIG. 4 is a flowchart illustrating the procedure for setting the sampling frequency of the IMU data performed by the image stabilization device 21.

In step S11, when the image stabilization device 21 starts image stabilization, the sampling unit 32 sets the sampling frequency to a predetermined initial value (for example, the maximum frequency f3), and the image stabilization table unit 35 sets the image stabilization amount γ. Is calculated. The process proceeds from step S11 to step S12. It should be noted that the calculation of the blur amount α in the blur amount calculation unit 34, the calculation of the blur correction amount γ in the blur correction table unit 35, the control of the cutting range in the blur correction unit 36, etc. are appropriate as the blur correction processing below. It is assumed that it has been done and is not specified.

In step S12, the image stabilization level determination processing unit 37 determines whether or not the image stabilization level of the image stabilization G is a large level (the image stabilization level G is large) based on the image stabilization amount γ calculated by the image stabilization table unit 35. To judge. That is, it is determined whether or not the blur condition G calculated by the above equations (4) and (5) is equal to or higher than the threshold value H3 based on the blur correction amount γ.

If it is determined in step S12 that the degree of blurring G is large, the process proceeds from step S12 to step S13.

In step S13, the sampling unit 32 sets the sampling frequency to the maximum (frequency f3). The process proceeds from step S13 to step S14.

In step S14, the image stabilization process of the image stabilization device 21 is performed, and the processing amount of the DSP is maximized. The process returns from step S14 to step S12.

If it is determined in step S12 that the degree of blurring G is not large, the process proceeds from step S12 to step S15.

In step S15, the image stabilization level determination processing unit 37 determines whether or not the image stabilization level of the image stabilization G is a medium level (normal image stabilization G) based on the image stabilization amount γ calculated by the image stabilization table unit 35. To judge. That is, it is determined whether or not the blur condition G calculated by the above equations (4) and (5) based on the blur correction amount γ is equal to or more than the threshold value H2 and is less than the threshold value H3.

If the degree of blurring G is determined to be normal in step S15, the process proceeds from step S15 to step S16.

In step S16, the sampling unit 32 sets the sampling frequency to normal (frequency f2). The process proceeds from step S16 to step S17.

In step S17, the amount of processing in the DSP after the image stabilization processing in the image stabilization device 21 is performed becomes normal. The process returns from step S17 to step S12.

If it is determined in step S15 that the degree of blurring G is not normal, the process proceeds from step S15 to step S18.

In step S18, the image stabilization level determination processing unit 37 determines whether or not the image stabilization level of the image stabilization G is a small level (the image stabilization is small) based on the image stabilization amount γ calculated by the image stabilization table unit 35. judge. That is, it is determined whether or not the blur condition G calculated by the above equations (4) and (5) based on the blur correction amount γ is equal to or more than the threshold value H1 and is less than the threshold value H2.

If it is determined in step S18 that the degree of blurring G is small, the process proceeds from step S18 to step S19.

In step S19, the sampling unit 32 sets the sampling frequency to the minimum (frequency f1). The process proceeds from step S19 to step S20.

In step S20, the image stabilization process of the image stabilization device 21 is performed, and the amount of processing by the DSP is minimized. The process returns from step S20 to step S12.

In step S18, when it is determined that the blur condition G is not small, that is, when the blur correction level determination processing unit 37 determines that the blur correction level of the blur condition G is no level, the process starts from step S18. The process proceeds to step S21.

In step S21, the sampling unit 32 sets the sampling frequency to the minimum (frequency f1). The process proceeds from step S21 to step S22.

In step S22, the DSP is set to the sleep state, and the processing by the DSP for blur correction in the blur correction device 21 is stopped. The process returns from step S22 to step S12. When the DSP is set to the sleep state, the calculation of the blur amount α and the blur correction amount γ is stopped, so that the determination of the blur correction level in steps S12, S15, and S18 is also stopped. Therefore, when the DSP is set to the sleep state, the sleep state of the DSP is released after a certain period of time so that the blur amount α and the blur correction amount γ are calculated, and steps S12, S15, and , The blur correction level in step S18 may be determined. Alternatively, when the DSP is set to the sleep state, the DSP is released from the sleep state when a predetermined change (value increase, etc.) occurs in the detected data, and the shake of step S12, step S15, and step S18 occurs. The correction level may be determined.

Note that the image stabilization process may be performed by a processing unit other than the DSP, and even in that case, power consumption is reduced by changing the sampling frequency according to the image stabilization level.

By the above processing procedure, the detection data (IMU data) is sampled at the sampling frequency according to the blur correction level of the blur condition G. As a result, the lower (smaller) the blur correction level, the lower the sampling frequency, so that the power consumption required for the blur correction process is reduced. When the blur correction level is low, that is, when the image blur is small, the change in the blur amount α over time is small, so even if the sampling frequency in the sampling unit 32 is low, the effect on the calculation accuracy of the blur amount α is large. Few. Therefore, the accuracy of image stabilization hardly decreases as compared with the case where the sampling frequency is high.

In Patent Document 1 (Japanese Patent Laid-Open No. 9-5830) and Patent Document 2 (Japanese Patent Laid-Open No. 4-20947), power consumption is reduced by not performing image stabilization in a situation where image stabilization is not required. However, the power consumption is not reduced under the condition that the image stabilization is required. On the other hand, this technique is different from the techniques of Patent Documents 1 and 2 because it is possible to reduce the power consumption according to the state of image blur while performing blur correction.

<Hardware configuration>
Next, a hardware configuration example of the image stabilization device 21 of FIG. 1 will be described. FIG. 5 is a block illustrating the hardware configuration of the image stabilization device 21 of FIG.

The camera 1 in FIG. 5 is a camera equipped with the image stabilization device 21 in FIG. Since the process for acquiring an image by the camera 1 is the same as that in FIG. 1, the description thereof will be omitted.

The camera 1 has a system CPU (Central Processing Unit) 81, a DSP (Digital Signal Processor) 82, an IMU sensor 83, and a battery 84 in relation to blur correction. The IMU sensor 83 corresponds to the IMU sensor 31 in FIG.

The system CPU 81 is a one-chip CPU (microprocessor) in which a CPU, RAM, ROM, various input / output devices, etc. are mounted on one IC chip. The system CPU 81 samples the detection signal output from the IMU sensor 83 and acquires the detection data (IMU data). The system CPU 81 supplies the IMU data from the IMU sensor 83 to the DSP 82, and causes the DSP 82 to perform a blur correction amount γ calculation process for image blur correction and a blur correction level determination process for the blur condition G. The system CPU 81 acquires the image stabilization amount γ calculated by the DSP 82 and supplies it to the image stabilization unit (abrasion correction unit 36 in FIG. 1) of the camera 1. The system CPU 81 acquires the image stabilization level of the degree of blur G from the DSP 82, and sets the sampling frequency when sampling the detection signal from the IMU sensor 83 based on the image stabilization level.

The DSP 82 acquires the IMU data from the system CPU 81 and calculates the blur amount α and the blur correction amount γ. The DSP 82 determines the blur correction level of the blur condition G based on the blur correction amount γ. The DSP 82 supplies the calculated image stabilization amount γ and the determination result of the image stabilization level to the system CPU 81.

The battery 84 is, for example, a secondary battery, and supplies electric power for driving a circuit, an actuator, or the like mounted on the camera 1. However, the battery 84 may be a primary battery, or may be a case where a commercial power source is used as the power source of the camera 1 instead of the battery 84.

The system CPU 81 has a software execution unit 91, a storage unit 92, a RAM 93, and a RAM 94.

The software execution unit 91 is an arithmetic processing unit that executes a program read from the storage unit 92 and temporarily stored in the RAM 93. The software execution unit 91 executes a process related to control of each unit of the camera 1 by executing a program.

The storage unit 92 is a non-volatile memory (ROM), and stores programs executed by the software execution unit 91 and various data referred to by the software execution unit 91.

The RAM 93 temporarily stores a program to be read and executed by the software execution unit 91 from the storage unit 92, data to be referred to, and the like.

The RAM 94 temporarily stores the detection data (IMU data) obtained by sampling the detection signal output from the IMU sensor 83 by a sampling unit (sampling unit 32 in FIG. 1) (not shown). The RAM 94 corresponds to the buffer memory 33 in FIG.

The software execution unit 91 executes processing as the camera control unit 101, the charge management unit 102, the IMU data acquisition unit 103, and the DSP control unit 104 by executing the program.

The camera control unit 101 controls the operation of each unit of the camera 1. For example, the camera control unit 101 controls the operations of the image pickup unit 11, the image processing unit 12 (including the image stabilization unit 36), the output control unit 13, the display unit 14, the storage unit 15, and the like in FIG.

The charge management unit 102 controls charging of the battery 84 when the camera 1 is connected to a commercial power source.

The IMU data acquisition unit 103 samples the detection signal from the IMU sensor 83, temporarily stores the detection data (IMU data) in the RAM 94, and reads and acquires the IMU data stored in the RAM 94. The IMU data acquisition unit 103 also controls sampling at the time of IMU data acquisition, such as setting a sampling frequency according to the degree of blur (shake correction level).

The DSP control unit 104 controls the DSP 82, causes the DSP 82 to execute a predetermined process, supplies data necessary for the process, and acquires a process result. For example, the DSP control unit 104 supplies the IMU data acquired by the IMU data acquisition unit 103 to the DSP 82, and executes the calculation process of the blur amount α and the blur correction amount γ, the determination process of the blur correction level, and the like to the DSP 82. Let me. When the blur correction level of the blur condition is no level, the DSP control unit 104 sets the DSP 82 to a sleep state for a certain period of time, for example, to reduce the power consumption of the DSP 82.

The DSP 82 has a software execution unit 111 and a RAM 112.

The software execution unit 111 is an arithmetic processing unit that executes a program supplied from the DSP control unit 104 of the system CPU 81 and temporarily stored in the RAM 112. By executing the program, the software execution unit 111 performs, for example, calculation processing of the image stabilization amount γ, determination processing of the image stabilization level, and the like, and supplies the processing result to the DSP control unit 104.

The RAM 112 temporarily stores a program supplied and executed by the software execution unit 111 from the DSP control unit 104, data referred to by the software execution unit 111, and the like.

The software execution unit 111 executes processing as the image stabilization table unit 121, the image stabilization level determination processing unit 122, and the DSP processing unit 123 by executing the program.

Since the image stabilization table unit 121 corresponds to the image stabilization table unit 35 in FIG. 1, the description of the process will be omitted. The image stabilization amount γ calculated by the image stabilization table unit 121 is supplied to the camera control unit 101 via the DSP control unit 104 of the system CPU 81. The camera control unit 101 supplies the image stabilization amount γ to the image stabilization unit (corresponding to the image stabilization unit 36 in FIG. 1) of the image processing unit (corresponding to the image processing unit 12 in FIG. 1) (not shown).

Since the image stabilization level determination processing unit 122 corresponds to the image stabilization level determination processing unit 37 in FIG. 1, the description of the processing will be omitted. The image stabilization level of the image stabilization G determined by the image stabilization level determination processing unit 122 is supplied to the IMU data acquisition unit 103 via the DSP control unit 104 of the system CPU 81. The IMU data acquisition unit 103 sets the sampling frequency at the time of IMU data acquisition based on the blur correction level.

Since the DSP processing unit 123 corresponds to the blur amount calculation unit 34 in FIG. 1, the description of the processing will be omitted. The DSP processing unit 123 supplies the calculated blur amount α to the blur correction table unit 121.

As described above, in the blur correction device 21 of FIG. 1, the sampling frequency (detection data) in the sampling unit 32 when acquiring the detection data from the IMU sensor 31 according to the blur correction level of the image blur magnitude (blurring degree G). Although the mode of changing the time interval for acquiring the data) has been described, the present technology is not limited to this.

This technique includes a case where the image stabilization device 21 directly and steplessly changes the sampling frequency in the sampling unit 32 according to the image stabilization level G instead of the image stabilization level. In this case, the image stabilization device 21 increases the sampling frequency as the degree of blur G increases. There may be a range in which the sampling frequency is constant even if the magnitude of the degree of blur G changes.

In this technique, instead of the image stabilization device 21 calculating the magnitude of the image blur as the blur condition G indicating the state of the image blur, the higher the frequency of the image blur, the larger the image blur G. Includes the case where the frequency (height) of is calculated as the degree of blurring G. In this case, the image stabilization device 21 increases the sampling frequency in the sampling unit 32 as the image stabilization G increases (or the image stabilization level of the image stabilization G increases). Even if the magnitude of the degree of blur G (or the blur correction level) changes, there may be a range in which the sampling frequency is constant. When the degree of blurring is small (when the frequency of image blurring is low), the change in the amount of blurring α over time becomes small, so even if the sampling frequency is lowered, the accuracy of blurring correction does not decrease, and power consumption is reduced. To. The present technique includes a case where the image stabilization device 21 calculates a value considering both the magnitude and frequency of image blur as the degree of blur G. That is, the case where the blur condition G is calculated so that the larger the image blur, the larger the blur condition G, and the higher the image blur frequency, the larger the blur condition G is included.

This technique includes the case where the image stabilization device 21 acquires the detection data indicating the image of the camera 1 (imaging unit 11) from other than the IMU sensor 31. For example, when the image shake is directly detected from the temporal change of the position of the subject image in the image captured by the image pickup unit 11, the image shake correction device 21 uses the detection data indicating the image shake of the camera 1. This includes the case of acquiring as detection data indicating blurring.

The present technique includes a case where the image stabilization device 21 calculates the image blur degree G for each direction of the image blur as the blur condition G indicating the image blur state (magnitude, frequency, etc.). That is, the image blur includes two linear image blurs having different directions such as the x-axis direction and the y-axis direction on the image, and a rotation direction image blur on the image. The image stabilization device 21 calculates the degree of blur G, which represents the state of image blur in each direction. In this case, the image stabilization device 21 changes the sampling frequency for each detection data related to (contribution to) image blur in each direction according to the degree of blur G in each direction. For example, the detection data related to the image blur in the x-axis direction on the image are the acceleration data in the x-axis direction of the camera 1 and the angular velocity data in the direction around the y-axis. The sampling frequency of the detected data is changed according to the degree of blurring G indicating the state of image blurring in the x-axis direction. Similarly, the detection data related to the image blur in the y-axis direction on the image are the acceleration data in the y-axis direction of the camera 1 and the angular velocity data in the direction around the x-axis. The sampling frequency for these detected data is changed according to the degree of blurring G, which represents the state of image blurring in the y-axis direction. The detection data related to the image blur in the rotation direction on the image is the angular velocity data in the z-axis direction of the camera 1. The sampling frequency of the detected data is changed according to the degree of blurring G indicating the state of image blurring in the rotation direction.

The image stabilization device 21 calculates the degree of blur G in only one or two of the three directions of image blur, and the detection data is related (contributed) to those one or two directions. The sampling frequency may be changed according to the degree of blurring G for each direction. Calculate the linear blur condition G that comprehensively represents the state of the two linear image blurs on the image, and set the sampling frequency of the detection data related to the two linear image blurs to the linear blur condition G. It may be changed to the same value accordingly.

The image stabilization device to which this technology is applied can record images taken by a camera by installing the camera on a moving object that moves violently, for example, when a camera is attached to an animal to observe the ecology of the animal. In some cases, it may be mounted on the camera. According to this, even when the blurring that occurs in the camera is severe, a stable image in which the image blurring is corrected can be obtained, and at the same time, the power consumption of the battery is reduced, so that the image can be recorded for a long time.

<Program>
The series of processes in the camera 1 (shake correction device 21) described above can be executed by hardware or by software. When a series of processes are executed by software, the programs constituting the software are installed in the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 6 is a block diagram showing a configuration example of computer hardware when the computer executes each process executed by the camera 1 (shake correction device 21) by a program.

In a computer, a CPU (Central Processing Unit) 201, a ROM (ReadOnlyMemory) 202, and a RAM (RandomAccessMemory) 203 are connected to each other by a bus 204.

The input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

The input unit 206 includes a keyboard, a mouse, a microphone, and the like. The output unit 207 includes a display, a speaker, and the like. The storage unit 208 includes a hard disk, a non-volatile memory, and the like. The communication unit 209 includes a network interface and the like. The drive 210 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 201 loads the program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executes the above-mentioned series. Is processed.

The program executed by the computer (CPU201) can be recorded and provided on the removable media 211 as a package media or the like, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 208 via the input / output interface 205 by mounting the removable media 211 in the drive 210. Further, the program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. In addition, the program can be pre-installed in the ROM 202 or the storage unit 208.

The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in the present specification, in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

The present technology can also take the following configurations.
(1)
Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image blurring caused by the blurring of the imaging unit is generated. It ’s a processing unit,
An information processing device having a processing unit that changes the time interval for acquiring the detection data according to the state of the image blur.
(2)
The information processing apparatus according to (1), wherein the processing unit changes the time interval according to the magnitude of the image blur.
(3)
The information according to (1) or (2) above, wherein the processing unit determines which of the plurality of levels the image blur is, and changes the time interval according to the determined level. Processing equipment.
(4)
The information processing apparatus according to any one of (1) to (3), wherein the processing unit shortens the time interval as the image blur increases.
(5)
The information processing apparatus according to any one of (1) to (4), wherein the processing unit changes the time interval according to the frequency of the image blur.
(6)
The information processing apparatus according to any one of (1) to (5), wherein the processing unit shortens the time interval as the frequency of the image blur increases.
(7)
The processing unit detects the state of the image blur in each of a plurality of different directions, and causes the blur of the image pickup unit related to the image blur in each direction according to the state of the image blur in each direction. The information processing apparatus according to any one of (1) to (6), which changes the time interval for acquiring the detected detection data.
(8)
The information processing apparatus according to (7), wherein the plurality of directions include two linear directions on an image captured by the image pickup unit, or a linear direction and a rotation direction on the image.
(9)
The information processing apparatus according to any one of (1) to (8) above, wherein the detected data includes at least the angular velocity of the angular velocity and acceleration of the blur of the imaging unit.
(10)
The detection data is an axis with respect to a first axis along the optical axis direction of the optical system of the imaging unit and a second axis and a third axis orthogonal to the first axis and orthogonal to each other. The information processing apparatus according to any one of (1) to (9) above, which includes any of an angular velocity in the circumferential direction and an acceleration in the axial direction.
(11)
The information processing apparatus according to any one of (1) to (10), wherein the processing unit detects a state of image blur based on the detection data or the control information.
(12)
The blur correction unit targets a cutout range for cutting out an effective image from the image captured by the image pickup unit, and the control information includes information for controlling the position of the cutout range (1) to (11). ) Is described in any of the information processing devices.
(13)
The blur correction unit targets a correction optical system or an image pickup element arranged in the optical system of the image pickup unit, and the control information includes information for controlling the position of the control target (1) to (11). ) Is described in any of the information processing devices.
(14)
The processing unit of the information processing device having the processing unit
Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image stabilization caused by the blurring of the imaging unit is generated. ,
An information processing method that changes the time interval for acquiring the detection data according to the state of the image blur.
(15)
Computer,
Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image blurring caused by the blurring of the imaging unit is generated. It ’s a processing unit,
A program for functioning as a processing unit that changes the time interval for acquiring the detection data according to the state of the image blur.

1 camera, 11 image pickup unit, 12 image processing unit, 21 image stabilization unit, 31 IMU sensor, 32 sampling unit, 33 buffer memory, 34 image stabilization unit, 35 image stabilization table unit, 36 image stabilization unit, 37 image stabilization level. Judgment processing unit

Claims (15)

1. Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image blurring caused by the blurring of the imaging unit is generated. It ’s a processing unit,
   An information processing device having a processing unit that changes the time interval for acquiring the detection data according to the state of the image blur.
2. The information processing apparatus according to claim 1, wherein the processing unit changes the time interval according to the magnitude of the image blur.
3. The information processing apparatus according to claim 2, wherein the processing unit determines which of the plurality of levels the image blur is, and changes the time interval according to the determined level.
4. The information processing apparatus according to claim 1, wherein the processing unit shortens the time interval as the image blur increases.
5. The information processing apparatus according to claim 1, wherein the processing unit changes the time interval according to the frequency of the image blur.
6. The information processing apparatus according to claim 1, wherein the processing unit shortens the time interval as the frequency of the image blur increases.
7. The processing unit detects the state of the image blur in each of a plurality of different directions, and causes the blur of the image pickup unit related to the image blur in each direction according to the state of the image blur in each direction. The information processing apparatus according to claim 1, wherein the time interval for acquiring the detected detection data is changed.
8. The information processing apparatus according to claim 7, wherein the plurality of directions include two linear directions on an image captured by the image pickup unit, or a linear direction and a rotation direction on the image.
9. The information processing apparatus according to claim 1, wherein the detected data includes at least the angular velocity of the angular velocity and acceleration of the blur of the imaging unit.
10. The detection data is an axis with respect to a first axis along the optical axis direction of the optical system of the imaging unit and a second axis and a third axis orthogonal to the first axis and orthogonal to each other. The information processing apparatus according to claim 1, further comprising either an angular velocity in the circumferential direction or an acceleration in the axial direction.
11. The information processing device according to claim 1, wherein the processing unit detects the state of the image blur based on the detection data or the control information.
12. The information according to claim 1, wherein the image stabilization unit targets a cutout range for cutting out an effective image from the image captured by the image pickup unit, and the control information includes information for controlling the position of the cutout range. Processing equipment.
13. The information according to claim 1, wherein the blur correction unit targets a correction optical system or an image pickup element arranged in the optical system of the image pickup unit, and the control information includes information for controlling the position of the control target. Processing equipment.
14. The processing unit of the information processing device having the processing unit
    Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image stabilization caused by the blurring of the imaging unit is generated. ,
    An information processing method that changes the time interval for acquiring the detection data according to the state of the image blur.
15. Computer,
    Detection data indicating blurring of the imaging unit is acquired at predetermined time intervals, and based on the acquired detection data, control information for controlling the image stabilization unit that corrects image blurring caused by the blurring of the imaging unit is generated. It ’s a processing unit,
    A program for functioning as a processing unit that changes the time interval for acquiring the detection data according to the state of the image blur.

Images