JP2010220002A - Image pickup device and image pickup method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To readily generate panoramic images. <P>SOLUTION: A filter controller 517 resets an analog high-pass filter (HPF) 513 with a predetermined cycle. The HPF 513 filters a movement detection signal supplied from a movement detection sensor 42. A shake amount calculator 515 performs extrapolation by using an output signal when the HPF 513 is not in a reset state and generates a signal in the reset state. The shake amount calculator 515 adapts the generated signal to be continuously followed by the output signal which is not in the reset state so as to calculate the shake amount from the signal. A displacement controller 516 displaces the relative positional relation between a lens portion and an image pickup element, according to the shake amount with respect to an optical axis and performs shake correction. Thus, even if the image pickup direction is swept during the generation of a panoramic image, a picked up image which is not blurred accompanying the sweeping operation is readily obtained. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method. Specifically, a plurality of captured images are connected to obtain a panoramic image.

  Conventionally, for example, in order to obtain a panoramic image using an imaging device, as in Patent Document 1, imaging is performed by moving an imaging region little by little, and frame (or field) images are continuously recorded from the imaging start point. To do. Then, overlapping portions of the captured images that are adjacent to each other are extracted from the recorded captured images, and a predetermined calculation is performed on the overlapping portions, so that the respective captured images are seamlessly combined and panoramic. An image is generated.

  For example, the user performs imaging while moving the imaging direction in the horizontal direction, and records a plurality of captured images. A horizontally long panoramic image can be obtained by seamlessly connecting the captured images while applying appropriate processing to the overlapped portions of the captured images.

  In addition, the optical axis variable element is changed in the direction opposite to the moving direction of the imaging direction at the time of imaging, and the shutter is opened while the optical axis is linearly moved, and the optical axis is quickly changed while the shutter is closed. A plurality of captured images are recorded by repeating the operation of restoring the original. In this way, even if the imaging apparatus is moved quickly, the resolution is not deteriorated, and a good panoramic image can be generated without increasing the shutter speed.

JP 11-88754 A

  By the way, the imaging apparatus is provided with a camera shake correction function for preventing image blurring by changing the optical axis in accordance with camera shake. However, the camera shake correction function is a function for preventing image blur due to camera shake, and is not a function for generating a captured image used for a panoramic image.

  Accordingly, it is an object to provide an imaging apparatus and an imaging method that can easily generate a panoramic image.

  According to a first aspect of the present invention, a motion detection sensor, an analog high-pass filter unit that performs filtering of a motion detection signal output from the motion detection sensor, and the analog high-pass filter unit are reset at a predetermined cycle. And generating a signal when the analog high-pass filter unit is in a reset state by extrapolation using an output signal when the analog high-pass filter unit is not in a reset state, The output signal when the reset state is no longer in the generated signal is continued, the shake amount calculation unit that calculates the shake amount from the continuous signal, and the relative relationship between the lens unit and the image sensor according to the shake amount The image pickup apparatus includes a displacement control unit that performs shake correction by displacing the target positional relationship with respect to the optical axis.

  In this invention, a filter control part short-circuits the capacitor | condenser which comprises an analog high-pass filter part with a predetermined period, and resets an analog high-pass filter part with a predetermined period. The shake amount calculation unit determines a linear function or a higher-order function from the output signal when not in the reset state, performs extrapolation processing using the determined function, and generates a signal when in the reset state. Further, the shake amount calculation unit calculates an offset amount with respect to the output signal when the reset state is lost from the determined function, and adds the calculated offset amount to the output signal when the reset state is lost. The signal generated by the extrapolation process is made to continue the output signal when the reset state is lost, and the shake amount is calculated from the continuous signal. The displacement control unit displaces the relative positional relationship between the lens unit and the image sensor with respect to the optical axis according to the shake amount calculated by the shake amount calculation unit when the analog high-pass filter unit is not in the reset state. Perform shake correction. The exposure period of the image sensor is set within a period during which shake correction is performed. Further, the relative positional relationship is returned to the correction operation start position after the exposure period.

  According to a second aspect of the present invention, there is provided a step of performing motion detection by a motion detection sensor, a step of performing filtering processing of a motion detection signal output from the motion detection sensor by an analog high-pass filter unit, and a filter control unit Resetting the analog high-pass filter unit at a predetermined cycle, and by an extrapolation process using an output signal when the analog high-pass filter unit is not in a reset state, A step of generating a signal when the pass filter unit is in a reset state, and continuing the output signal when the generated filter is not in a reset state, and calculating a shake amount from the continuous signal; and a displacement The control unit performs shake correction by displacing the relative positional relationship between the lens unit and the image sensor with respect to the optical axis according to the shake amount. In the imaging method and a-up.

  According to this invention, the filtering process of the motion detection signal obtained by performing the motion detection by the motion detection sensor is performed by the analog high-pass filter unit that is reset at a predetermined cycle. Further, the shake amount calculation unit performs extrapolation processing using the output signal when the analog high-pass filter unit is not in the reset state, and generates a signal when the analog high-pass filter unit is in the reset state. The The output signal when the reset state is not generated is continued to the generated signal, and the shake amount is calculated from the continuous signal, and the lens unit and the image sensor are relative to each other according to the calculated shake amount. The target positional relationship is displaced with respect to the optical axis, and shake correction is performed. Therefore, even if the imaging direction is moved in a predetermined direction, it is possible to perform shake correction suitable for generating a panoramic image by calculating a shake amount, and it is possible to easily generate a panoramic image.

It is a figure which shows the structure of an imaging device. It is a figure which shows the structure of a shake correction control part. It is the figure which illustrated the composition of the amplification part, the analog low-pass filter part, and the analog high-pass filter part. It is the figure which illustrated the appearance of the imaging device. It is a figure for demonstrating the case where an electric charge is saturated with the capacitor | condenser in an analog high-pass filter part. It is a figure for demonstrating the case where the electric charge of the capacitor | condenser in an analog high-pass filter part is discharge | released periodically. It is a figure for demonstrating an extrapolation process and an offset process. It is a figure which shows the case where an extrapolation process and an offset process are performed. It is a figure for demonstrating the case where a quadratic function is used by the extrapolation process. It is a figure which shows the specific example at the time of performing an extrapolation process and an offset process. It is a figure which shows operation | movement at the time of using CCD as an image pick-up element. It is a figure which shows operation | movement at the time of using a CMOS sensor as an image pick-up element. It is a figure which shows the electric charge read-out timing in an image pick-up element. It is a figure which shows the specific example of the instruction | indication position with respect to a correction lens, and the position of a correction lens. It is a figure for demonstrating a return period and a run-up period. It is a flowchart which shows operation | movement of an imaging device. It is a figure which shows the state by which menu display was performed. It is the figure which illustrated the case where the right direction sweep mode was selected. It is a figure for demonstrating the case where normal mode is selected. It is a figure which shows the case where a panoramic image is produced | generated in right direction sweep mode. It is a figure for demonstrating operation | movement when set to the right direction sweep mode. It is a figure for demonstrating a captured image generation process. It is a figure for demonstrating the captured image produced | generated by the right direction sweep mode.

Hereinafter, modes for carrying out the invention will be described. The description will be given in the following order.
1. Configuration of imaging apparatus 2. Configuration of shake correction control unit 3. Appearance of imaging device 4. Extrapolation processing performed by the shake correction control unit 5. Shake correction operation in shake correction control unit 6. Operation of imaging device Example of panorama image generation (in right sweep mode)

<1. Configuration of Imaging Device>
FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to the present invention. The imaging apparatus 10 includes an imaging optical system block 11, a driver 12, a position sensor 13, an imaging device 21, a timing signal generation (TG) unit 22, an analog front end (AFE) unit 23, a signal processing unit 24, and a detection unit 25. ing. Further, the imaging apparatus 10 includes an image output unit 31, a display unit 32, a recording / playback unit 33, an operation unit 41, a motion detection sensor 42, and a control unit 50.

  The imaging optical system block 11 includes a lens unit 11 a and a diaphragm mechanism 11 b that adjusts the amount of light of an optical image formed on the imaging surface of the imaging element 21. The lens unit 11a is, for example, a zoom lens 111 that performs zooming, a focus lens 112 that performs focusing, and a correction lens unit 113 that moves the position of an optical image formed on the imaging surface of the imaging device 21 described later on the imaging surface. It is configured.

  The correction lens unit 113 includes, for example, a correction lens provided so that the optical axis coincides with the optical axis of the imaging optical system, and an actuator that displaces the correction lens in a direction orthogonal to the optical axis of the imaging optical system. It is configured. The correction lens unit 113 having such a configuration moves the correction lens in a direction orthogonal to the optical axis of the image pickup optical system, so that the relative positional relationship between the lens unit 11a and the image pickup element 21 is set with respect to the optical axis. Displace.

  The correction lens unit 113 may use a variable apex angle prism unit. The variable apex angle prism unit is provided with a translucent entrance end plate and an exit end plate on the end face of a bendable cylinder such as a bellows, and a translucent liquid having a desired refractive index is enclosed in the cylinder. Is. When the variable apex angle prism unit is used, one of the entrance end plate and the exit end plate is fixed, and the other is driven by an actuator to form an optical wedge. The correction lens unit having such a configuration moves the position of the optical image formed on the imaging surface on the imaging surface by, for example, displacing the inclination angle of the exit end plate with respect to the incident end plate.

  The driver 12 drives actuators of the zoom lens 111, the focus lens 112, and the correction lens unit 113 based on a lens control signal from the control unit 50 described later. The driver 12 drives the aperture mechanism 11b based on the aperture control signal from the control unit 50.

  The position sensor 13 detects the position of the zoom lens 111 and the focus lens 112, the displacement state of the correction lens unit 113 (equivalent to the displacement position and correction angle of the correction lens unit 113), and the set position of the diaphragm mechanism 11b. The signal is supplied to the control unit 50.

  For example, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor is used as the image sensor 21. The imaging element 21 converts the optical image formed on the imaging surface by the imaging optical system block 11 into an electrical signal and outputs the electrical signal to the AFE unit 23.

  The TG unit 22 generates various drive pulses necessary for outputting an electrical signal indicating a captured image by the image sensor 21 and an electronic shutter pulse for controlling the charge accumulation time of the image sensor 21.

  The AFE unit 23 performs noise removal processing such as CDS (Correlated Double Sampling) processing on the electrical signal (image signal) output from the image sensor 21, and AGC (Automatic Gain Control) using the imaging signal as a desired signal level. Process. Further, the AFE unit 23 converts an analog image pickup signal subjected to noise removal processing and gain control into a digital signal and outputs the digital signal to the signal processing unit 24.

  The signal processing unit 24 performs camera signal preprocessing, camera signal processing, resolution conversion processing, compression / decompression processing, and the like. In the camera signal preprocessing, the image signal supplied from the AFE unit 23 is subjected to a defect correction process for correcting a signal of a defective pixel in the image sensor 21, a shading correction process for correcting a peripheral light loss of the lens, and the like. In camera signal processing, processing such as white balance adjustment and brightness correction is performed. In addition, a digital camera or the like may be configured to obtain red, green, and blue signals with a single image sensor by providing a color filter array in front of the image sensor. is there. In such a case, a demosaic process is performed in the camera signal process, and a missing color signal is generated in each pixel by interpolation or the like using signals of surrounding pixels. In the resolution conversion process, an image signal that has been subjected to camera signal processing or an image signal that has been subjected to decompression decoding is converted to a predetermined resolution. In the compression / decompression process, an image signal after camera signal processing or an image signal subjected to resolution conversion processing is compression-encoded to generate, for example, a JPEG encoded signal. In the compression / decompression process, a JPEG encoded signal is decompressed and decoded. Note that the compression / decompression process may be performed by compressing and encoding the image signal of a still image by a method different from the JPEG method. In the compression / decompression process, a moving image signal may be compressed and encoded by a moving image compression method.

  Further, when the panorama image is generated by the imaging apparatus 10, the signal processing unit 24 calculates a motion vector using the captured image, and a plurality of images of the same subject are overlapped using the calculated motion vector. A panoramic image is generated by combining the captured images. In addition, for the synthesis of the captured image, a motion detection signal from the motion detection sensor may be used. In this case, even if the motion vector is not obtained correctly, it is possible to synthesize a plurality of captured images so that the images of the same subject overlap using the motion detection signal. Note that the captured image used for generating the panoramic image is written to a recording medium by a memory (not shown) or a recording / reproducing unit 33 described later. Further, the panorama image may be generated at the time of imaging or reproduction, and the panorama image may be generated by an external device different from the imaging device 10, such as a computer. When generating a panoramic image during playback or with an external device, identification information that enables generation of the panoramic image during playback or with the external device (for example, information indicating a series of captured images or information indicating the imaging order) Are provided for a captured image used for generating a panoramic image. By providing identification information in this way, it is possible to generate a panoramic image by synthesizing captured images in the correct order.

  The detection unit 25 detects the brightness level and the focus state of the captured image using the imaging signal supplied to the signal processing unit 24, generates a detection signal indicating the brightness level and the focus state, and generates the detection signal. To supply.

  The image output unit 31 converts the image signal processed by the signal processing unit 24 into an image signal having a format corresponding to an external device connected to the imaging apparatus 10 and outputs the image signal.

  The display unit 32 displays an image captured by the imaging device 10 and a captured image reproduced by the recording / reproducing unit 33. The display unit 32 also displays a menu for setting the imaging device 10 and the like.

  In the recording / reproducing unit 33, for example, a recording medium such as a flash memory, an optical disk, or a magnetic tape is used. The recording / reproducing unit 33 records the image signal and the encoded signal of the captured image output from the signal processing unit 24 on a recording medium. The recording / reproducing unit 33 reads the image signal recorded on the recording medium and supplies the image signal to the image output unit 31 or the display unit 32, or reads the encoded signal recorded on the recording medium to read the signal processing unit. The process which supplies to 24 is performed. The recording / reproducing unit 33 is not limited to the configuration in which the recording medium is detachable. For example, a hard disk device or the like may be incorporated as the recording / reproducing unit 33.

  The operation unit 41 includes operation buttons and a touch panel provided on the screen of the display unit 32. The operation unit 41 generates an operation signal corresponding to a user operation and supplies the operation signal to the control unit 50.

  The motion detection sensor 42 is configured using, for example, a gyro sensor that detects the motion of the imaging device 10. The motion detection sensor includes a yawing angular velocity sensor that detects, for example, an angular velocity according to a shake in the yawing direction, and a pitching angular velocity sensor that detects, for example, an angular velocity according to a shake in the pitching direction. Note that the motion detection sensor 42 is not limited to being configured using an angular velocity sensor. For example, the motion may be detected using an acceleration sensor or the like. When an acceleration sensor is used, the speed can be calculated by integrating the output of the acceleration sensor.

  The control unit 50 includes a CPU (Central Processing Unit), a memory, and the like. The memory stores programs executed by the CPU and various data. As this memory, for example, a nonvolatile memory such as an EEPROM (Electronically Erasable and Programmable ROM) or a flash memory is used. The CPU of the control unit 50 executes a program stored in the memory, and the operation of the imaging device 10 corresponds to a user operation based on various data stored in the memory and operation signals supplied from the operation unit 41. Each part is controlled so that it becomes operation. For example, when the user performs a shutter operation, the control unit 50 controls the operation of the TG unit 22 and the like, and an encoded signal of a still image captured at a desired shutter speed is recorded on the recording medium of the recording / reproducing unit 33. Let me record. When a moving image recording start operation is performed, a moving image encoded signal or the like is recorded on the recording medium of the recording / reproducing unit 33.

  In addition, when the user performs a mode selection operation, the control unit 50 performs an imaging operation or the like in the mode selected by the user.

  Further, the control unit 50 generates a lens control signal and an aperture control signal based on the position signal supplied from the position sensor 13 and the detection signal supplied from the detection unit 25 and supplies the lens control signal and the aperture control signal to the driver 12. Accordingly, the focus lens 112 and the aperture mechanism 11b are driven by the driver 12 so that a captured image with a desired brightness can be obtained. In addition, when the user performs a zoom operation, the control unit 50 generates a lens control signal and supplies it to the driver 12 to drive the zoom lens 111 so that a captured image with a desired zoom ratio is obtained.

  In the imaging apparatus 10 configured as described above, a shake correction control unit 51 is provided in the control unit 50. The shake correction control unit 51 performs shake correction of the captured image by displacing the relative positional relationship between the lens unit 11a and the image sensor 21 with respect to the optical axis in accordance with the motion detected by the motion detection sensor 42. Do. For example, the shake correction control unit 51 generates a lens control signal for driving the correction lens of the correction lens unit 113 based on the detection signal from the motion detection sensor 42 so that the shake of the captured image does not occur, and the driver 12. To supply. The driver 12 supplies a drive signal for driving the correction lens to the correction lens unit 113 based on the lens control signal. The correction lens unit 113 moves the correction lens with an actuator based on the drive signal. In this manner, by driving the correction lens unit 113 based on the motion detection signal from the motion detection sensor 42, the relative positional relationship between the lens unit 11a and the image sensor 21 is displaced with respect to the optical axis, and imaging is performed. The shake correction of the optical image formed on the imaging surface of the element is performed.

  Alternatively, the shake correction may be performed by moving the image sensor 21 instead of the correction lens. In this case, as indicated by a dotted line in FIG. 1, an actuator 21a that moves the image sensor 21 in a direction orthogonal to the optical axis is provided. Furthermore, by moving the position of the image sensor 21 by the actuator 21a based on the motion detection signal, the relative positional relationship between the lens unit 11a and the image sensor 21 is displaced with respect to the optical axis, and shake correction is performed.

  In other words, the imaging apparatus 10 can be shaken by driving at least one of the lens unit 11a and the imaging element 21 based on the motion detection signal and displacing the relative positional relationship between the lens unit and the imaging element with respect to the optical axis. Correction can be performed. In the following description, a case where shake correction is performed by moving the correction lens to displace the relative positional relationship between the lens unit and the image sensor with respect to the optical axis will be described.

  In addition, when the imaging mode is a mode for generating a panoramic image (hereinafter referred to as “panoramic mode”), the shake correction control unit 51 determines the relative relationship between the lens unit and the imaging element according to the motion detected by the motion detection sensor. Displace the positional relationship. Furthermore, by shifting the relative positional relationship between the lens unit and the image sensor in accordance with the motion detected by the motion detection sensor, a plurality of captured images that are not shaken even if the imaging direction is moved in a predetermined direction are generated. Let Further, the shake correction control unit 51 returns the displacement to the correction operation start state between the generation of the captured image and the generation of the next captured image. By performing such processing, it is possible to repeatedly generate a captured image without blurring even if the imaging direction is moved in a certain direction. Also, a plurality of captured images generated while moving the imaging direction in a predetermined direction can be combined to generate an image having a wider imaging range than one captured image.

<2. Configuration of shake correction control unit>
The shake correction control unit 51 performs filter processing for removing unnecessary signal components from the motion detection signal. For example, the shake correction control unit 51 performs a low-pass filter process for removing a high-frequency noise component or the like and a high-pass filter process for removing a DC component or the like. In addition, the shake correction control unit 51 is configured such that when the high-pass filter process is performed, the analog high-pass filter unit prevents the charge from being saturated by the capacitor constituting the analog high-pass filter unit. Is reset at a predetermined cycle. That is, the analog high-pass filter unit is reset at a predetermined cycle to discharge the charge accumulated in the capacitor. Further, the shake correction control unit 51 performs extrapolation processing (extrapolation processing) using an output signal when the analog high-pass filter unit is not in the reset state, and when the analog high-pass filter unit is in the reset state. Generate a signal. Furthermore, the shake amount is calculated from a signal obtained by continuing the output signal when the generated signal is no longer in the reset state, and the relative positional relationship between the lens unit and the image sensor is set on the optical axis according to the shake amount. Displacement is performed and shake correction is performed.

  FIG. 2 shows a configuration of the shake correction control unit 51. The motion detection signal is supplied to the analog low-pass filter (LPF) unit 512 via the amplification unit 511. The analog low-pass filter unit 512 removes a noise component having a frequency higher than that of the motion detection signal and supplies it to the analog high-pass filter (HPF) unit 513. The analog high-pass filter unit 513 removes a direct current component or the like and supplies it to the A / D conversion unit 514. Further, the analog high-pass filter unit 513 is reset at a predetermined period based on a control signal from a filter control unit 517, which will be described later, and discharges the electric charge accumulated in the capacitor constituting the analog high-pass filter unit 513. . For example, when the imaging mode is set to the panorama mode and the imaging direction is moved in a predetermined direction, the motion detection signal becomes a signal with a low frequency, so that the capacitor constituting the analog high-pass filter unit 513 saturates the charge. End up. For this reason, the shake correction control unit 51 resets the analog high-pass filter unit 513 at a predetermined period to release electric charges. Further, in the shake correction control unit 51, when the imaging mode is set to the panorama mode and the imaging direction is moved in a predetermined direction, the charge is saturated by the capacitor constituting the analog high-pass filter unit 513 with the motion detection signal at this time A predetermined cycle is set so as not to cause a failure.

  FIG. 3 shows circuit examples of the amplifying unit 511, the analog low-pass filter unit 512, and the analog high-pass filter unit 513. The amplifying unit 511 and the analog low-pass filter unit 512 perform, for example, a low-pass filtering process and an amplifying process on the motion detection signal using a differential amplifier. The analog high-pass filter unit 513 includes a capacitor 513a, a resistor 513b, and a short-circuit switch 513c.

  The filter control unit 517 controls the short-circuit switch 513c according to the filter control signal to reset the analog high-pass filter unit 513 at a predetermined cycle. That is, the filter control unit 517 outputs the signal after the filter processing from the analog high-pass filter unit 513 by turning off the short-circuit switch 513c. Also, the filter control unit 517 resets the analog high-pass filter unit 513 by turning on the short-circuit switch 513c, and sets the output of the analog high-pass filter unit 513 as the reference voltage Vref. Further, the short circuit switch 513c is turned on to reset the analog high-pass filter unit 513, thereby discharging the charge accumulated in the capacitor 513a.

  The A / D conversion unit 514 converts the filtered motion detection signal into a digital motion detection signal and supplies the digital motion detection signal to the shake amount calculation unit 515.

  The shake amount calculation unit 515 calculates the shake amount based on the motion detection signal. Here, the analog high-pass filter unit 513 is reset at a predetermined cycle so that the charge accumulated in the capacitor 513a is not saturated. Therefore, the motion detection signal supplied from the A / D conversion unit 514 becomes the reference voltage Vref when the analog high-pass filter unit 513 is in a reset state. Further, the motion detection signal supplied from the A / D conversion unit 514 changes in signal level according to the motion from the reference voltage Vref when the analog high-pass filter unit 513 is not in the reset state. For this reason, the motion detection signal supplied from the A / D conversion unit 514 is different from a signal obtained by digitizing the motion detection signal output from the motion detection sensor 42. For this reason, the shake amount calculation unit 515 performs extrapolation processing using an output signal when the analog high-pass filter unit 513 is not in the reset state, and outputs a signal when the analog high-pass filter unit 513 is in the reset state. Generate. Further, the shake amount calculation unit 515 makes the output signal when the reset state disappears to the signal generated by the extrapolation process, calculates the shake amount from the continuous signal, and outputs the shake amount to the displacement control unit 516.

  The displacement control unit 516 moves the imaging direction in a predetermined direction by displacing the relative positional relationship between the lens unit and the imaging element with respect to the optical axis in accordance with the shake amount calculated by the shake amount calculation unit 515. However, shake correction is performed so that a captured image without blurring can be generated. That is, the displacement control unit 516 calculates the position of the correction lens based on the shake amount calculated by the shake amount calculation unit 515. Further, the displacement control unit 516 generates a correction lens drive signal based on the position of the correction lens detected by the position sensor 13 and the position calculated based on the shake amount so that the correction lens becomes the calculated position. To the driver 12.

  The filter control unit 517 controls the short-circuit switch 513c to reset the analog high-pass filter unit 513 at a predetermined cycle so that the charge is not saturated by the capacitor 513a of the analog high-pass filter unit 513.

<3. Appearance of imaging device>
FIG. 4 illustrates the appearance of the imaging apparatus 10. A display unit 32 is provided on the back surface of the casing of the imaging apparatus 10, and an operation unit 41 is provided in the vicinity of the display unit 32. The operation unit 41 includes a plurality of operation keys and the like. For example, the menu key 411 is an operation key for displaying a menu on the display unit 32. The direction keys 412a to 412d are operation keys that are operated when menu items are selected. A determination key 413 provided at the center of the direction keys 412a to 412d is an operation key that is operated when performing a determination operation or the like of a selected item. Furthermore, a shutter key 415 provided on the upper surface of the housing is an operation key for performing a shutter operation. Note that the operation unit 41 illustrated in FIG. 4 is an example, and the position and type of the key are not limited to this example. Further, a touch panel may be provided on the screen of the display unit 32 so that various operations can be set and instructions for execution can be performed by touching a predetermined position of the display unit 32.

<4. Extrapolation process performed by shake correction control unit>
Next, after describing the extrapolation processing performed by the shake correction control unit 51, the operation of the shake correction control unit 51 will be described.

  FIG. 5 shows the operation of the analog high-pass filter unit 513. When the signal shown in FIG. 5A is input to the analog high-pass filter unit 513, the charge is accumulated in the capacitor 513a and then saturated, and the output of the analog high-pass filter unit 513 is (B) in FIG. ). For this reason, when the imaging mode is set to the panorama mode and the imaging direction is moved in a predetermined direction, and the motion detection signal becomes the signal shown in FIG. 6A, the motion detection signal is output from the analog high-pass filter unit 513. Cannot be output.

  Therefore, the short circuit switch 513c of the analog high-pass filter unit 513 is turned on at a predetermined cycle as shown in FIG. 6C to reset the analog high-pass filter unit 513 before the capacitor 513a is saturated. , To release the accumulated charge. In this case, as shown in FIG. 6B, the output of the analog high-pass filter unit 513 is input after that, when the short-circuit switch 513c is in the OFF state, with the start time of the OFF state as the reference voltage Vref. Output according to the signal. Further, the output of the analog high-pass filter unit 513 becomes the reference voltage Vref when the short-circuit switch 513c is on.

  The shake amount calculation unit 515 performs extrapolation processing using the output signal from the analog high-pass filter unit 513 when the short-circuit switch 513c is in an off state, and generates a signal for a period during which the short-circuit switch 513c is in an on state. To do. Further, the shake amount calculation unit 515 makes the signal generated by the extrapolation process continuous with the signal when the short-circuit switch 513c is next turned off, so that the motion detection signal is a signal having a low frequency. To obtain a motion detection signal.

  FIG. 7 is a diagram for explaining extrapolation processing and offset processing. The black circles indicate data values when the output of the analog high-pass filter unit 513 is converted into digital signals by the A / D conversion unit 514, and the square frame display indicates the data values generated by the extrapolation processing. Yes.

  The shake amount calculation unit determines a linear function or a higher-order function from the output signal when not in the reset state, performs extrapolation processing using the determined function, and generates a signal when in the reset state. The shake amount calculation unit 515 uses the data values DMa to DMa + 3 when the short-circuit switch 513c is in an on state (the analog high-pass filter unit is in a reset state), and the off-state immediately before (the analog high-pass filter unit is in a non- Extrapolation processing is performed using data values in the period of the reset state.

  Next, an example of extrapolation processing using a linear function will be described. The shake amount calculation unit 515 uses the data value DMa-2 immediately before the data value DM-1 and the data value DMa-2 immediately before the data value DM-1 in the immediately previous off-state period. And the slope AS-1 of the straight line connecting the data value DMa-1. Further, the shake amount calculation unit 515 sets a straight line that passes the data value DMa-1 with the calculated slope AS-1, and sets the data value of each sampling timing when the short-circuit switch 513c is in the ON state. Data values DMa to DMa + 3 are determined so as to be on a straight line.

  The slope may be calculated using data values of three or more points in the immediately preceding off period. For example, in the immediately preceding off period, the data value DMa-1 immediately preceding the data value DMa, the data value DMa-2 immediately preceding the data value DM-1, and the data value DMa-3 immediately preceding the data value DM-2 are used. . Further, the slope AS-2 of the straight line connecting the data value DMa-3 and the data value DMa-2 and the slope AS-1 of the straight line connecting the data value DMa-2 and the data value DMa-1 are calculated. The shake amount calculation unit 515 calculates a change amount DAS of the inclination from the two inclinations AS-2 and AS-1, and calculates the inclination AS0 by adding the change amount DAS to the inclination AS-1. Further, the shake amount calculation unit 515 sets a straight line having the calculated slope AS0 and passing through the data value DMa-1, and the data value of each sampling timing when the data value is DMa to DMa + 3 is set. Data values DMa to DMa + 3 are determined so as to be on the straight line.

  In this way, the shake amount calculation unit 515 performs extrapolation processing using a linear function, and obtains the data values DMa to DMa + 3 when the short-circuit switch 513c is in the on state (the analog high-pass filter unit is in the reset state). Generate. Further, the shake amount calculation unit 515 determines that the data values DMa to DMa + 3 and the next off-state data value (the data value when the analog high-pass filter unit is no longer in the reset state) are continuous. Calculate the offset amount for the output signal when the function is no longer in the reset state. The shake amount calculation unit 515 performs offset processing by adding the calculated offset amount to the next off-state data value.

  The shake amount calculation unit 515 calculates an offset value DFA to be added to the next data value in the off state from, for example, the linear function used in the extrapolation process and the timing at which the shorting switch 513c is turned off. The shake amount calculation unit 515 uses the data value at the timing when the short-circuit switch 513c is turned off in the linear function used for the extrapolation process as the offset value DFa. For example, when the short-circuit switch 513c is turned off immediately after the timing of the data value DMa + 3, the data value DMa + 3 is set as the offset value DFa. Further, when the short-circuit switch 513c is turned off immediately before the timing of the data value DMa + 4, the data value at the timing of the data value DMa + 4 in the linear function used for the extrapolation process is set as the offset value DFa.

  If such extrapolation processing and offset processing are performed, the capacitor 513a is short-circuited by the short-circuit switch 513c and the analog high-pass filter unit 513 is reset at a predetermined cycle, and a motion detection signal is obtained as shown in FIG. be able to.

  Next, an example of extrapolation processing using a quadratic function will be described. As shown in FIG. 9, the shake amount calculation unit 515 performs a data value DMa-1 immediately before the data value DMa, a data value DMa-2 immediately before the data value DM-1, and a data value in the immediately preceding off period. A quadratic function G is calculated using the data value DMa-3 immediately before DM-2.

Here, the quadratic function G is represented by equation (1). In Equation (1), Ka, Kb, and Kc indicate coefficients determined based on the data values DMa-1, DMa-2, and DMa-3.
G = Kat ² + Kbt + Kc (1)

When the sampling interval “T = 1 / f” is constant and the data value DMa-3 is set to the reference time t = 0, the timing of the data value DMa-2 is t = (1 / f), and the data value DMa The timing of −1 is t = (2 / f). Therefore, the coefficients Ka, Kb, Kc are
Kc = DMa-3
Kb = f / 2 (4DMa-2-DMa-1)
^{Ka = f 2/2 (DMa} -3-2DMa-2 + DMa-1)
It becomes.

Therefore, when obtaining the data value DMa, t = (3 / f) in equation (1). When obtaining the data value DMa + 1, t = (4 / f) in the equation (1), and when obtaining the data value DMa + 2, t = (5 / f) in the equation (1), the data value DMa +. When determining 3, t = (6 / f) in equation (1).
DMa = Ka × (3 / f) ² + Kb × (3 / f) + Kc
DMa + 1 = Ka × (4 / f) ² + Kb × (4 / f) + Kc
DMa + 2 = Ka × (5 / f) ² + Kb × (5 / f) + Kc
DMa + 3 = Ka × (6 / f) ² + Kb × (6 / f) + Kc

  Further, the shake amount calculation unit 515 calculates the offset value DFA from the timing when the quadratic function G used for the extrapolation process and the short-circuit switch 513c are turned off, and adds them to the data value of the next off state. For example, when the short-circuit switch 513c is turned off immediately after the timing of the data value DMa + 3, the data value DMa + 3 is added to the data value DMa + 4 as the offset value DFa to obtain the data value DMa + 4 ′. .

  Further, by adding the offset value DFA to other data values in this off period to obtain data values with white circles, the data value when the short-circuit switch 513c is in the on state is changed to the next off state. Make data values continuous.

  In this way, the analog high-pass filter unit 513 is reset at a predetermined cycle, and the charge accumulated in the capacitor 513a is discharged. Furthermore, by performing extrapolation processing and offset processing on the output from the analog high-pass filter unit 513, a signal obtained by performing filter processing on the motion detection signal from the motion detection sensor 42 can be obtained. . In addition, if a high-order function is used in the extrapolation process, it is possible to improve the accuracy of the extrapolation process compared to the case where a linear function is used. A specific example when the extrapolation process and the offset process are performed is shown in FIG.

<5. Operation of shake correction control unit>
Next, the operation of the shake correction control unit when the imaging mode is set to the panorama mode will be individually described, for example, when a CCD is used as an imaging element and when a CMOS sensor is used. FIG. 11 is a diagram for explaining the operation when a CCD is used as the image sensor.

  11A shows the operation of the short-circuit switch 513c, FIG. 11B shows the output of the analog high-pass filter unit 513, and FIG. 11C shows the filter generated by performing extrapolation processing and offset processing. The motion detection signals after processing are shown.

  When the imaging mode is the panorama mode, the shake correction control unit 51 moves the imaging direction in a predetermined direction by displacing the relative positional relationship between the lens unit and the imaging element in accordance with the motion detection signal after the filtering process. A captured image without blurring is generated.

  Here, when the analog high-pass filter unit is not in the reset state, that is, when the short-circuit switch 513c is in the off period, the shake correction control unit 51 corrects the correction lens according to the shake amount calculated from the motion detection signal after the filter process. To correct shake. In addition, the control unit 50 sets the exposure period during the period when shake correction is performed, and generates an image signal based on the charge accumulated in the CCD during the exposure period, thereby moving the imaging direction in a predetermined direction. Even so, a captured image without blurring can be generated. As described above, if the shake correction is performed when the short-circuit switch 513c is in the off period, the shake correction according to the signal generated by the extrapolation process is not performed, and thus the shake correction is generated by the extrapolation process. It is possible to perform shake correction that is less affected by data errors.

  FIG. 11D shows a shake correction flag. The shake correction flag indicates that shake correction is performed based on the motion detection signal after the filter process when in the on state, and indicates that shake correction is not performed when in the off state. Yes.

  FIG. 11E shows the position of the correction lens. In FIG. 11E, a position Qps indicates a correction range correction operation start position in the panorama mode. The shake amount calculation unit 515 performs shake correction when the shake correction flag is turned on, and calculates an instruction value so that image blur does not occur even when the imaging direction is moved in a predetermined direction. That is, the shake amount calculation unit 515 calculates the instruction value so that the correction lens is moved from the correction operation start position Qps according to the motion detection signal after the filter process. Therefore, the correction lens is moved so as not to cause image blur even when the imaging direction is moved in a predetermined direction. Further, a return process is performed in which the correction lens is set to the correction operation start position Qps after the exposure period so as not to affect the captured image.

  FIG. 11F shows the exposure period of the CCD image sensor. The exposure period is set within a period during which shake correction is performed so that image blurring does not occur even when the imaging direction is moved in a predetermined direction. FIG. 11F shows a case where the period during which the shake correction flag is on and the shake correction is performed is the exposure period.

  In the CCD image pickup device, the accumulated charges are simultaneously read out from the transfer register in each line. Therefore, the control unit 50 controls the TG unit 22 to read out the charge of each pixel accumulated in the exposure period set during the period in which the shake correction is performed, and captures an image signal based on the read out charge. It is generated by the element 21. In this way, by setting the exposure period within the period during which shake correction is performed, a plurality of captured images that are free from blurring due to movement in the imaging direction are generated while moving the imaging direction in a predetermined direction. it can.

  FIG. 12 is a diagram for explaining the operation when a CMOS sensor is used as the image sensor. Note that FIG. 12A to FIG. 12E correspond to FIG. 11A to FIG. 11E. 12A shows the operation of the short-circuit switch 513c, FIG. 12B shows the output of the analog high-pass filter unit 513, and FIG. 12C shows the filter generated by performing extrapolation processing and offset processing. The motion detection signals after processing are shown. 12D shows a shake correction flag, and FIG. 12E shows the position of the correction lens.

  FIG. 12F shows the exposure period of the CMOS sensor. The exposure period is set within a period during which shake correction is performed so that image blurring does not occur even when the imaging direction is moved in a predetermined direction. FIG. 12F shows a case where the period during which the shake correction flag is on and the shake correction is performed is the exposure period.

  In the CMOS sensor, charge is read for each line. For example, as shown in FIG. 13A, the uppermost line “Line-S” is selected, the charges of each pixel in the selected line are read simultaneously, and the pixel signal of each pixel is output to the multiplexer in parallel. To do. The multiplexer converts the parallel pixel signals into series and outputs them, thereby generating an image signal for the uppermost line. Further, by sequentially selecting such processing in the downward direction from the uppermost line, an image signal of one captured image can be generated. As described above, since the charge reading for each line is sequentially performed from the uppermost line “Line-S” to the lowermost line “Line-E”, the charge reading timing is sequentially delayed for each line. Therefore, the exposure period of the CMOS sensor when generating one captured image is as shown in FIG. For this reason, if the exposure period of the CCD is set within each period during which shake correction is performed, the influence of errors due to extrapolation processing is reduced, and the imaging direction is changed while moving in the predetermined direction. It is possible to generate a plurality of captured images that are not shaken due to movement.

  FIG. 13B shows the charge readout timing when a CCD image sensor is used. In the CCD image pickup device, the accumulated charges are simultaneously read out from the transfer register in each line. Therefore, the charge read timing of each line from the uppermost line “Line-S” to the lowermost line “Line-E” is equal.

  FIG. 14 shows a specific example of the indicated position with respect to the correction lens and the position of the correction lens when shake correction is performed. In FIG. 14, a position Qps indicates a correction operation start position when the imaging mode is set to the panorama mode, and a position Q0 indicates a correction operation start when the imaging mode is different from the panorama mode, for example, the normal mode. Yes.

  The displacement control unit 516 performs a return process after setting the correction lens to the correction operation start position Qps after the exposure period so as not to affect the captured image. Further, when the return process is performed in a short period, as shown in FIG. 15A, a so-called overshoot may occur in which the correction lens passes the correction operation start position Qps. In FIG. 15A, the actual position of the correction lens is indicated by a solid line. Therefore, the displacement control unit 516 sets the return process period RP in consideration of overshoot as shown in FIG. Further, when the correction lens is moved according to the motion detection signal, it may take time until the correction lens moves from the state of the correction operation start position Qps to the state where the correction lens moves according to the motion detection signal. Therefore, as shown in FIG. 15C, the control unit 50 provides a run-up period at the beginning of the shake correction period CP, and exposure is performed after the correction lens moves according to the motion detection signal. An exposure period may be provided after the run-up period to start.

<6. Operation of Imaging Device>
FIG. 16 is a flowchart illustrating the operation of the imaging apparatus. In step ST1, the control unit 50 determines whether or not the imaging mode is a panorama mode that generates a panoramic image. When determining that the menu key 411 of the operation unit 41 has been operated, the control unit 50 performs the menu display GA on the display unit 32. FIG. 17 illustrates a state in which menu display is performed on the display unit 32. Thereafter, the control unit 50 switches modes in accordance with the operation of the direction key 412a indicating the upward direction or the direction key 412c indicating the downward direction, and displays the selected imaging mode in an identifiable manner. For example, the control unit 50 provides a cursor display GB, and moves the position of the cursor display GB up and down in accordance with the operation of the direction keys 412a and 412c, so that the selected imaging mode can be identified. Further, the control unit 50 sets the imaging mode selected when the enter key 413 is operated to the imaging mode of the imaging device 10. Here, the control unit 50 proceeds to step ST2 when the panorama mode is selected as the imaging mode, and proceeds to step ST9 when the imaging mode is set to an imaging mode different from the panorama mode.

  In step ST2, the control unit 50 determines the sweep mode. When the panorama mode is selected, the control unit 50 displays the sweep mode selection image GC as shown in FIG. Thereafter, the control unit 50 switches the sweep mode in accordance with the operation of the direction key 412b indicating the left direction or the direction key 412d indicating the right direction, and displays the selected sweep mode in an identifiable manner. For example, the selected sweep mode can be easily identified by displaying the arrow of the selected sweep mode in a color, brightness, or the like different from the arrow display of the other sweep modes.

  After the sweep mode selection image GC is displayed, the control unit 50 determines that the right sweep mode is selected when the direction key 412d indicating the right direction is operated. When the right direction sweep mode is selected, the control unit 50 makes the arrow display indicating the right direction different from the arrow display indicating the other direction in the sweep mode selection image GC as shown in FIG. . Furthermore, when the direction key 412d indicating the right direction is operated in this state, the control unit 50 determines that the left sweep mode has been selected. When the left sweep mode is selected, the control unit 50 sets the arrow display indicating the left direction in the sweep mode selection image GC to be different from the arrow display in the other sweep modes. When the direction key 412d indicating the right direction is further operated, the control unit 50 determines that the upward sweep mode is selected. When the upward sweep mode is selected, the control unit 50 changes the arrow display indicating the upward direction in the sweep mode selection image GC to a display different from the arrow display in the other sweep modes. When the direction key 412d indicating the right direction is further operated, the control unit 50 determines that the downward sweep mode has been selected. When the downward sweep mode is selected, the control unit 50 causes the arrow display indicating the downward direction in the sweep mode selection image GC to be different from the arrow display in the other sweep modes. When the direction key 412b indicating the left direction is operated, the control unit 50 switches the sweep mode to the sweep mode located on the left side.

  As described above, the control unit 50 switches the sweep mode in accordance with the operation of the direction keys 412b and 412d, and sets the sweep mode selected when the enter key 413 is operated to the sweep mode in the panorama mode. .

  In step ST3, the control unit 50 performs optical position offset setting according to the sweep direction. In the optical position offset setting, when the left sweep mode is selected, when the imaging direction of the imaging apparatus 10 is moved to the right, the lens unit and the imaging are set so that the correction range of shake correction with respect to the right sweep is widened. Set the relative positional relationship with the element. Here, the control unit 50 sets the lens unit and the imaging device to a predetermined relative positional relationship when the imaging mode is different from the imaging mode for generating a panoramic image. In addition, the control unit 50 offsets a predetermined relative positional relationship according to the sweep mode in the imaging mode for generating a panoramic image. For example, when a right sweep is performed during imaging and the imaging direction is moved to the right, the imaging apparatus 10 moves the correction lens to the left so that the position of the subject in the optical image on the imaging surface is in the right direction. It can be corrected so as not to move in accordance with the sweep. The control unit 50 offsets the lens unit and the image sensor to the right as shown in FIG. 18B from a center position (hereinafter referred to as “predetermined position”) at which the lens unit and the image sensor have a predetermined relative positional relationship. That is, the control unit 50 outputs a lens control signal to the driver 12 to drive the actuator 113b, thereby offsetting the correction lens 113a of the correction lens unit 113 to the right from the predetermined position. Therefore, the range in which correction can be performed by moving the correction lens is wider than that before the offset. Further, the offset position is set as the correction operation start position Qps, and the process proceeds to step ST4. In FIG. 18B, the position indicated by the dotted line AR indicates the position of the correction control end of the correction lens 113a.

  When another sweep mode is selected, when the imaging direction is moved in accordance with the selected sweep mode, a position offset so as to widen the shake correction range is set as a correction operation start position Qps. For example, when the left sweep mode is selected, a position offset in the left direction so that the correction range of shake correction is widened is set as a correction operation start position.

  In step ST4, the control unit 50 issues a sweep direction instruction. The control unit 50 displays the sweep instruction image indicating the sweep direction on the display unit 32 so that the imaging direction is moved in the direction corresponding to the sweep mode, notifies the user of the movement direction, and proceeds to step ST5.

  In step ST5, the control unit 50 starts a shutter operation and proceeds to step ST6. The shutter operation is started based on the operation of the shutter key 415 or a motion detection signal. For example, when the control unit 50 detects that the shutter key 415 has been pressed, the control unit 50 starts the shutter operation.

  Further, the shutter operation can be automatically started by using the motion detection signal. For example, the control unit 50 starts the shutter operation when detecting that the direction of the imaging device 10 has moved in the moving direction notified in step ST4 based on the motion detection signal. In this way, the user can start the shutter operation only by moving the direction of the imaging device 10 in the notified direction without operating the shutter key 415. When the shutter operation is started when it is detected that the shutter key 415 has been pressed, if the camera apparatus shakes due to the operation of the shutter key 415, the influence of the shake may appear in the panoramic image. is there. However, if the shutter operation is started using the motion detection signal, it is not necessary to operate the shutter key 415. Therefore, it is possible to easily obtain a panoramic image that is not affected by the blur due to the shutter operation.

  In step ST6, the control unit 50 performs a captured image generation process. The control unit 50 drives the correction lens 113a, which is the correction operation start position, according to the motion detection signal, and performs correction (sweep correction) for shake caused by the sweep operation. In other words, the control unit 50 generates a captured image in which no shake due to the sweep operation occurs even when the imaging device 10 is swept, and the process proceeds to step ST7. Here, the correction operation start position Qps is offset so that the range that can be corrected according to the sweep direction is widened. Therefore, when the imaging apparatus 10 is swept, the correction possible period is longer than when the correction operation start position is not offset, so that the exposure time can be extended. Further, in the captured image generation processing, the extrapolation processing and the offset processing are performed on the output of the analog high-pass filter unit 513 as described above, so that even if the imaging direction is moved in the sweep direction, the blur due to the sweep operation is performed. It is possible to generate a captured image in which no occurrence occurs.

  In step ST7, the control unit 50 determines whether or not the shutter operation is finished. The controller 50 returns to step ST6 when determining that the shutter operation has not been completed, and proceeds to step ST8 when determining that the shutter operation has been completed. When returning to step ST6, the control unit 50 returns the correction lens 113a to the correction operation start position Qps, and then drives the correction lens 113a again according to the motion detection signal to sweep the image pickup apparatus 10. Generation of an image in which no occurrence occurs.

  For example, the control unit 50 ends the shutter operation when the movement amount reaches a predetermined amount set in advance. The amount of movement can be calculated using, for example, a motion vector. Further, the control unit 50 may determine the sweep amount from the motion detection signal, and end the shutter operation when the sweep amount reaches a predetermined amount. Further, when it is detected that the shutter key 415 has been pressed and the shutter operation is started, the shutter operation may be terminated when it is detected that the shutter key 415 has not been pressed. it can. Further, when the control unit 50 detects that the direction of the imaging device 10 has moved in the notified movement direction based on the motion detection signal and starts the shutter operation, the direction of the imaging device 10 moves in the notified movement direction. When it stops, the shutter operation can be ended.

  In step ST8, the control unit 50 performs a captured image composition process. The control unit 50 controls the signal processing unit 24 to calculate a motion vector using a plurality of captured images generated by the processes in steps ST6 and ST7 in the generation order. Further, the control unit 50 combines a plurality of captured images so that the images of the same subject overlap based on the calculated motion vector, and generates a plurality of panoramic images having a wider imaging range than one captured image. It produces | generates from a captured image, and progresses to step ST10.

  In step ST1, when it is determined that the imaging mode is set to a mode different from the panorama mode, for example, the normal mode, and the process proceeds to step ST9, the control unit 50 performs a captured image generation process. Since the controller 50 does not need to perform shake correction corresponding to the sweep operation, the control lens 113a is set to the correction operation start position Q0. The correction operation start position Q0 is a predetermined position where no offset corresponding to the sweep operation is performed. In addition, the control unit 50 drives the correction lens 113a according to the motion detection signal to perform correction (camera shake correction) for camera shake during imaging. Further, the control unit 50 generates an image that is not shaken by the camera shake correction in accordance with the operation of the shutter key 415, and proceeds to step ST10.

  FIG. 19A shows a case where the normal mode is selected, for example. FIG. 19B shows the correction operation start position Q0 of the correction lens unit 113 when the normal mode is set. In addition, the control unit 50 generates a predetermined number of captured images according to the operation of the shutter key 415.

  In the normal mode, it is not necessary to perform shake correction corresponding to the sweep operation unlike the panoramic mode, and shake correction corresponding to camera shake is performed. Here, since the camera shake direction is not limited to a specific direction, the correction operation start position is set to a position that is not offset so that the camera shake correction can be performed regardless of the direction of camera shake. For example, the center position of the movable range of the correction lens unit 113 is set as a correction operation start position Q0. In addition, the camera shake direction is not limited to a specific direction, and the motion detection signal is a signal having a higher frequency than that in the panoramic mode in which the imaging direction is moved in the sweep direction, and the analog high-pass filter unit 513 It is difficult for the capacitor 513a to cause charge saturation. Therefore, the filter control unit 517 maintains the short-circuit switch 513c of the analog high-pass filter unit 513 in the OFF state so as not to perform the reset of a predetermined cycle.

  In step ST10, the control unit 50 causes the recording / playback unit 33 to record the panoramic image generated in step ST8 or the captured image generated in step ST9 on a recording medium.

  The control unit 50 records the captured image as described above, and the operation mode is switched to another mode, for example, a playback mode for reproducing the recorded captured image, or when the operation of the image capturing apparatus is ended. Then, the imaging mode is terminated.

  In this way, the control unit 50 resets the analog high-pass filter unit at a predetermined cycle when in the imaging mode in which a plurality of captured images generated while moving the imaging direction are combined to generate a panoramic image. Further, a signal when the analog high-pass filter unit is in the reset state is generated by extrapolation processing using the output signal when the analog high-pass filter unit is not in the reset state. Further, the output signal when the reset state is lost is continued to the generated signal, and the shake amount is calculated from the continuous signal. Further, shake correction is performed by displacing the relative positional relationship between the lens unit and the image sensor with respect to the optical axis in accordance with the shake amount. For this reason, even if the imaging direction is moved in the sweep direction, it is possible to perform shake correction with respect to the movement in the sweep direction, and it is possible to easily generate a captured image without blurring.

  Since the correction operation start position is offset according to the sweep direction, the shake correction performance with respect to the imaging direction is improved, and shake correction suitable for generating a panoramic image is performed. For example, the exposure time is increased. Even in this case, it is possible to generate a captured image in which no blur occurs. In addition, even if the sweep speed is high, it is possible to generate a captured image that is not shaken. Therefore, when a panoramic image is generated, restrictions such as the brightness of the subject and the sweep speed are reduced, and the panoramic image can be easily generated.

<7. Example of panorama image generation (right sweep mode)>
Next, a specific operation when generating a panorama image, for example, when generating a panorama image in the right sweep mode will be described.

  When the imaging mode of the imaging device 10 is the panorama mode and the right sweep mode, the user sweeps the orientation (imaging direction) of the imaging device 10 in the right direction as indicated by an arrow A as shown in FIG.

  FIG. 21 is a diagram for explaining the operation when the right sweep mode is set. When the imaging mode is the panorama mode and the right sweep mode is set, the control unit 50 displays a sweep direction instruction GE indicating a right sweep operation as shown in FIG. By displaying in 32, the user is notified of the sweep direction. Since the right sweep mode is set, the control unit 50 offsets the position of the correction lens 113a to the right from the center position as shown in FIG. The shake correction is started from this correction operation start position.

  Thereafter, the control unit 50 starts a shutter operation and performs captured image generation processing. FIG. 22 shows captured image generation processing. FIG. 22A shows the movement of the correction lens 113a. The control unit 50 drives the correction lens 113a, which is the correction operation start position, according to the motion detection signal. Therefore, the correction lens 113a moves to the left from the correction operation start position in order to prevent image shake due to the right sweep. Here, the analog high-pass filter unit is reset in a predetermined cycle, and the analog high-pass filter unit is in the reset state by extrapolation using the output signal when the analog high-pass filter unit is not in the reset state. Signal is generated. Further, the output signal when the reset state is no longer in the generated signal is continued, and the shake amount is calculated from the continuous signal. Accordingly, it is possible to calculate the shake amount even when the right sweep is performed, and the correction lens 113a is moved in accordance with the calculated shake amount. Therefore, as shown in FIG. It is possible to generate a captured image PG1 that is not shaken even if the direction is swept. In addition, since the correction operation start position of the correction lens 113a is offset rightward from the center position, the correction range for shake correction with respect to movement in the imaging direction is wider than when the correction operation start position is not offset. Become. Therefore, when the correction operation start position is offset and when the correction operation start position is not offset, when the sweep speed is the same, the correction operation start position is offset so that no blurring occurs even if the exposure time is extended. A captured image can be generated. In addition, when the correction operation start position is offset and when the correction operation start position is not offset and the exposure time is the same, by offsetting the correction operation start position, imaging with no blurring even if the sweep speed is high An image can be generated.

  When the shutter operation is not completed when one captured image is generated, the control unit 50 returns the correction lens 113a to the correction operation start position, and again moves the correction lens 113a from the correction operation start position to the motion detection signal. Drive according to. In FIG. 22A, a period during which the process of returning the correction lens 113a to the correction operation start position Qps is performed as a period RP. In this way, by returning the correction lens 113a to the correction operation start position and then driving again according to the motion detection signal, it is possible to generate a captured image PG2 that is not shaken even when the imaging device 10 is swept.

  Such processing is repeatedly performed, and when the control unit 50 determines that the shutter operation is finished when the captured image PG5 is generated, for example, the control unit 50 ends the captured image generation processing. Therefore, the sequentially generated captured images PG1 to PG5 are images in which the imaging direction is sequentially switched to the right as shown in FIG.

  The signal processing unit 24 calculates a motion vector from the sequentially generated captured images PG1 to PG5 or detects a motion vector based on a motion detection signal. Further, the signal processing unit 24 aligns the captured images PG1 to PG5 based on the motion vector so that the images of the subject overlap each other, and synthesizes the captured images PG1 to PG5 so that the captured image is more than one captured image. A panoramic image shown in FIG. 22C with a wide imaging range can be generated.

  As described above, in the present invention, the analog high-pass filter section that performs the filtering process of the motion detection signal is reset at a predetermined period. Further, extrapolation processing is performed using an output signal when the analog high-pass filter unit is not in the reset state, and a signal when the analog high-pass filter unit is in the reset state is generated. In addition, the output signal when the reset state is lost is continued to the generated signal, and the shake amount is calculated from the continuous signal.

  If such processing is performed, even when the motion detection signal is filtered by the analog high-pass filter unit, even if the imaging mode is set to the panorama mode and the imaging direction is moved to the sweep direction, the sweep direction The amount of shake according to the movement of can be calculated. Therefore, it is possible to perform shake correction suitable for generating a panoramic image by displacing the relative positional relationship between the lens unit and the image sensor with respect to the optical axis in accordance with the calculated shake amount. Can be easily generated.

  Also, when the analog high-pass filter unit is not in the reset state, shake correction is performed by displacing the relative positional relationship between the lens unit and the image sensor with respect to the optical axis in accordance with the calculated shake amount. An exposure period is provided within the period during which the process is performed. By providing the exposure period in this way, it is possible to obtain a captured image without blurring even if an error or the like occurs in the signal generated by the extrapolation process. In addition, since the process of returning the relative positional relationship to the correction operation start position is performed after the exposure period, it is possible to repeatedly obtain captured images without blurring. Furthermore, by offsetting the correction operation start position according to the sweep direction, it is possible to widen the shake correction range when the imaging direction is moved in the sweep direction.

  Furthermore, the present invention should not be construed as being limited to the above-described embodiments. For example, any configuration that can detect the shake of the imaging apparatus is not limited to the configuration using the angular velocity sensor or the acceleration sensor as described above. For example, you may make it detect the shake of an imaging device from a captured image.

  Further, in the case of an imaging device capable of exchanging lenses, the correction lens may be provided on the lens side, or the correction lens may be provided on the main body side of the imaging device. Further, for example, a correction lens or a driving unit that drives the correction lens may be provided on the lens side, and other components may be provided on the main body side of the imaging apparatus. Further, as described above, the image sensor provided on the main body side of the image capturing apparatus may be moved according to the motion detection signal.

  The embodiments of the present invention disclose the present invention in the form of examples, and it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. That is, in order to determine the gist of the present invention, the claims should be taken into consideration.

  In the imaging apparatus and imaging method of the present invention, the filtering process of the motion detection signal obtained by performing motion detection by the motion detection sensor is performed by the analog high-pass filter unit that is reset at a predetermined period. Further, the shake amount calculation unit performs extrapolation processing using the output signal when the analog high-pass filter unit is not in the reset state, and generates a signal when the analog high-pass filter unit is in the reset state. The The output signal when the reset state is not generated is continued to the generated signal, and the shake amount is calculated from the continuous signal, and the lens unit and the image sensor are relative to each other according to the calculated shake amount. The target positional relationship is displaced with respect to the optical axis, and shake correction is performed. Therefore, even if the imaging direction is moved in a predetermined direction, it is possible to perform shake correction suitable for generating a panoramic image by calculating a shake amount, which is suitable for a digital camera or a video camera.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Imaging optical system block, 11a ... Lens part, 11b ... Diaphragm mechanism, 12 ... Driver, 13 ... Position sensor, 21 ... Imaging device, 21a, 113b ... Actuator, 22 ... TG part, 23 ... AFE part, 24 ... Signal processing part, 25 ... Detection part, 31 ... Image output part, 32 ... Display , 33... Recording / playback unit, 41... Operation unit, 42... Motion detection sensor, 50... Control unit, 51. ..Focus lens, 113... Correction lens section, 113a... Correction lens, 411... Menu key, 412a to 412d .. direction key, 413. 511... Amplifier 512 ..Analog low pass filter (LPF) section, 513... Analog high pass filter (HPF) section, 513a... Capacitor, 513b... Resistor, 513c. A / D conversion unit, 515 ... shake amount calculation unit, 516 ... displacement control unit, 517 ... filter control unit

Claims (11)

A motion detection sensor;
An analog high-pass filter unit that performs filtering of a motion detection signal output from the motion detection sensor;
A filter control unit that resets the analog high-pass filter unit at a predetermined period;
By performing extrapolation using an output signal when the analog high-pass filter unit is not in the reset state, a signal is generated when the analog high-pass filter unit is in the reset state, and the generated signal is reset. A shake amount calculation unit that continuously outputs an output signal when it is no longer, and calculates a shake amount from the continuous signal;
An image pickup apparatus comprising: a displacement control unit that performs shake correction by displacing a relative positional relationship between the lens unit and the image pickup element with respect to the optical axis according to the shake amount. The imaging apparatus according to claim 1, wherein the filter control unit resets the analog high-pass filter unit at a predetermined cycle when in an imaging mode for generating a panoramic image. The shake amount calculation unit determines a linear function or a high-order function from an output signal when not in a reset state, performs extrapolation using the determined function, and generates a signal when in the reset state The imaging device according to claim 2. The shake amount calculation unit calculates an offset amount with respect to the output signal when the reset state is lost from the determined function, and adds the calculated offset amount to the output signal when the reset state is lost. The imaging apparatus according to claim 3, wherein an output signal when the signal generated by the extrapolation process is not in a reset state is continued. The displacement control unit is configured to change a relative positional relationship between the lens unit and the imaging element with respect to an optical axis according to a shake amount calculated by the shake amount calculation unit when the analog high-pass filter unit is not in a reset state. The imaging apparatus according to claim 4, wherein shake correction is performed by displacing the imaging apparatus. The imaging apparatus according to claim 5, wherein an exposure period of the imaging element is within a period during which the shake correction is performed. The imaging apparatus according to claim 6, wherein the displacement control unit performs processing for returning the relative positional relationship to a correction operation start position after the exposure period. The imaging device according to claim 5, wherein the driving unit drives at least one of the lens unit or the imaging device to displace a relative positional relationship between the lens unit and the imaging device with respect to an optical axis. When the control unit is in an imaging mode that generates the panoramic image with the lens unit and the imaging element being in a predetermined relative positional relationship when the imaging mode is different from the imaging mode for generating the panoramic image, the predetermined unit The imaging apparatus according to claim 8, wherein the relative positional relationship is offset to widen a correction range of shake correction with respect to movement in the imaging direction. The imaging apparatus according to claim 2, wherein the filter control unit resets the analog high-pass filter unit by short-circuiting a capacitor used in the analog high-pass filter unit. Performing a motion detection by a motion detection sensor;
Performing a filtering process of a motion detection signal output from the motion detection sensor in an analog high-pass filter unit;
Resetting the analog high-pass filter unit at a predetermined period in a filter control unit;
A shake amount calculation unit generates a signal when the analog high-pass filter unit is in a reset state by extrapolation using an output signal when the analog high-pass filter unit is not in a reset state, The output signal when the generated signal is no longer in the reset state is continued, and the amount of shake is calculated from the continuous signal;
And a step of performing shake correction by displacing a relative positional relationship between the lens unit and the imaging device with respect to the optical axis in accordance with the amount of shake.