JP2009177332A - Blurring detection apparatus, blurring correction apparatus, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a data size of blurring correction information of a correction object image. <P>SOLUTION: A hand blurring amount (S<SB>L</SB>) of a normal exposure image is estimated from a focal length (f<SB>D</SB>) and an exposure time (T<SB>EP</SB>) when imaging the normal exposure image as a correction target image. When the hand blurring amount is comparatively large, the normal exposure image and a short exposure image which is photographed in a shorter exposure time than the normal exposure time, are continuously obtained, image data of a small image in the short exposure image are recorded in a recording medium together with the image data of the normal exposure image. At a time of image reproduction, a point spread function of the normal exposure image is obtained from both the image data, and blurring correction is made by image restoration processing. Meanwhile, when the presumed hand blurring amount is comparatively small, an edge emphasizing flag is recorded on the recording medium together with the image data of the normal exposure image as information for correction. In this case, the blurring correction at image reproduction is obtained by edge emphasizing processing. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a shake detection device that detects a shake of a still image derived from hand shake and the like, and a shake correction device that corrects the shake. The present invention also relates to an imaging apparatus having a shake detection function.

  The camera shake correction technology is a technology for reducing image blur caused by camera shake, and is regarded as important as a differentiation technology in an imaging apparatus such as a digital still camera.

  As a method for correcting camera shake, a method for recording camera shake information for correcting shake of a correction target image by image processing at the time of shooting and reading the camera shake information at the time of image reproduction and correcting the shake of the correction target image. Has been proposed. For example, the angular velocity of the housing of the imaging device during the exposure period of the correction target image is measured by the angular velocity sensor, and the output data of the angular velocity sensor is recorded on the recording medium together with the image data of the correction target image. Then, at the time of image reproduction or the like, a point spread function (PSF) of the correction target image is obtained from the recorded output data of the angular velocity sensor, and a blur correction image is generated by image restoration processing (Patent Document 1 below). And 2).

  An operator or a spatial filter in which an ideal point image is weighted in accordance with a trajectory drawn on an image due to image blur is called a PSF and is generally used as a mathematical model for camera shake. The PSF is represented by a matrix, and if the correction target image is filtered using an image restoration filter having each element of the inverse matrix of the PSF as each filter coefficient, an image without blur is restored. FIG. 33 shows the relationship between the camera shake trajectory, the PSF expressed in a matrix and the image restoration filter, and how the restored image is generated from the camera shake image as the correction target image. In the PSF, the hatched portion represents a weighted matrix element.

  There has also been proposed a method for removing the blur of the correction target image based only on the output signal of the image sensor without using a camera shake detection sensor such as an angular velocity sensor. For example, a sub image for blur correction of the main image is photographed before and after the main image as the correction target image is photographed, and the image data of the sub image is recorded together with the image data of the main image. Then, at the time of image reproduction or the like, the state of blurring of the main image is estimated from the image data of the sub-image to correct the blurring of the main image (see Patent Document 3 below).

  As described above, regardless of whether or not a camera shake detection sensor such as an angular velocity sensor is used, in order to perform camera shake correction by image processing after shooting a correction target image, correction information for performing the camera shake (the above-described conventional technique) is used. In the example, it is necessary to record the output data of the angular velocity sensor or the image data of the sub-image on a recording medium. On the other hand, since the recording capacity of the recording medium is finite, it is obvious that the smaller the data size of the correction information is, the better.

  A method is also proposed in which camera shake information (PSF) representing camera shake during shooting is estimated from one camera shake image (correction target image), and a restored image free from shake is generated by image processing from the camera shake information and the camera shake image. Has been. Among these types of methods, a method using the Fourier iteration method is disclosed in Non-Patent Document 1 below. FIG. 34 shows a block diagram of a configuration for realizing the Fourier iteration method. In the Fourier iteration method, the final restored image is estimated from the degraded image by repeatedly executing Fourier transform and inverse Fourier transform while correcting the restored image and PSF. In order to execute the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image), and a random image or a degraded image as a camera shake image is used as the initial restored image.

  If the above-mentioned Fourier iteration method is used, it is possible to remove the blur of the correction target image from only the image data of the correction target image. However, the Fourier iteration method is a nonlinear optimization method, and a large number of iterations are required to obtain an appropriate restored image. That is, the processing time required for camera shake detection and correction becomes very long. For this reason, it is difficult to put it into practical use with a digital still camera or the like, and it is realistic to record and use correction information other than the image data of the correction target image.

JP 2006-129236 A JP 2004-88567 A JP 2001-346093 A G. R. Ayers and J. C. Dainty, "Iterative blind deconvolution method and its applications", OPTICS LETTERS, 1988, Vol. 13, No. 7, p.547-549

  Therefore, an object of the present invention is to provide a shake detection device, a shake correction device, and an imaging device that contribute to reducing the data size of correction information for correcting shake of a correction target image.

  The blur detection device according to the present invention includes a recording control unit that records correction information for correcting blur of a correction target image, which is a still image obtained by an imaging device by photographing, by image processing. In the shake detection device, the recording control unit can generate a plurality of types of correction information, and the correction control unit records in the recording unit according to the movement of the imaging device during the exposure period of the correction target image. The type of information is selected.

  For example, when the movement of the imaging device during the exposure period of the correction target image is small or when the movement can be approximated by a simple motion, even if the correction information is simplified, a blur correction effect that does not cause a problem in practice is obtained. be able to. Taking this into consideration, it is possible to generate a plurality of types of correction information (the data size of certain types of correction information is relatively large and the data size of other types of correction information is relatively small), and the imaging apparatus The type of correction information to be recorded on the recording medium is selected according to the movement of the image. As a result, the data size of the correction information can be reduced.

  Specifically, for example, the shake detection device further includes device motion estimation means for estimating the magnitude of motion of the imaging device during an exposure period of the correction target image, and the recording control means Based on the size, the type of the correction information to be recorded in the recording means is selected.

  More specifically, for example, the recording control means determines whether the estimated size belongs to one of a first range to which a relatively large size should belong and a second range to which a relatively small size should belong. Whether or not the estimated size belongs to the first range and the second range is generated as different types of correction information, and the estimated size corresponds to the second range. When belonging to the range, information indicating a correction method of the correction target image is recorded on the recording medium as the correction information.

  And, for example, the recording control means, when the estimated size belongs to the first range, is an image shot before or after shooting of the correction target image, and from the exposure period of the correction target image Also, part or all of the image data of a short exposure image taken in an exposure period with a short length is recorded on the recording medium as the correction information.

  More specifically, for example, the apparatus motion estimation means estimates the size based on the shooting condition of the correction target image.

  The shooting condition of the correction target image is determined before (or during) the shooting of the correction target image. For this reason, if the magnitude of the movement of the imaging apparatus during the exposure period of the correction target image is estimated from the photographing conditions of the correction target image, the estimation is completed in a short time. As a result, it is possible to avoid or suppress a decrease in continuous shooting performance of a still image that should be a correction target image.

  Alternatively, specifically, for example, the apparatus motion estimation unit is an image that is captured before or after the correction target image is captured and is captured in an exposure period that is shorter than the exposure period of the correction target image. The size is estimated based on the edge intensity of the short-exposure image and the edge intensity of the correction target image.

  Also by this, the estimation can be completed in a relatively short time, and it is possible to avoid or suppress a decrease in continuous shooting performance of a still image that should be a correction target image.

  Further, for example, the apparatus motion estimation means treats each of a plurality of still images arranged in time series photographed by the imaging apparatus as a correction target image, and detects the movement of the imaging apparatus during the exposure period of each correction target image. The size to be estimated and the size to be estimated for the focused correction target image is the image data of the past correction target image of the focused correction target image, and before the past correction target image is captured. Alternatively, it is estimated using image data of a short exposure image which is an image shot after shooting and is shot in an exposure period shorter than the exposure period of the past correction target image.

  Further, for example, the apparatus motion estimation means pays attention to an image sequence captured before capturing the correction target image, and the size of a motion vector between temporally adjacent images forming the image sequence is a predetermined value or less. During the exposure period of the correction target image, based on the motion vector, the frame rate at the time of shooting the image sequence, and the length of the exposure period of the correction target image. The magnitude of the movement of the imaging device is estimated.

  If the magnitude of the movement of the imaging device during the exposure period of the correction target image is estimated from the image sequence captured before the correction target image is shot, the estimation is completed before or near the time of shooting the correction target image. can do. As a result, it is possible to avoid or suppress a decrease in continuous shooting performance of a still image that should be a correction target image.

  Further, for example, the recording control unit determines whether or not the blur of the correction target image belongs to a predefined type of blur based on an output signal of a sensor that measures the movement of the imaging device, and the determination result Based on this, the type of the correction information to be recorded in the recording means is selected.

  When the movement of the imaging device during the exposure period of the correction target image is relatively simple and the blur of the correction target image can be approximated by a predefined type blur (for example, linear and constant speed blur), Even if the correction information is simplified, it is possible to obtain a shake correction effect with no practical problem. Considering this, it is possible to generate a plurality of types of correction information (the data size of one type of correction information is relatively large and the data size of other types of correction information is relatively small), and the correction target The type of correction information to be recorded on the recording medium is selected according to whether the image blur belongs to a type blur. As a result, the data size of the correction information can be reduced.

  More specifically, for example, when the recording control unit determines that the blur of the correction target image belongs to the type of blur, the correction data is converted to the formula data indicating the trajectory shape of the type of blur and the speed state of drawing the trajectory. While recording in the recording means as information for use, when it is determined that the blur of the correction target image does not belong to the type blur, data representing the output signal of the sensor during the exposure period of the correction target image is used for the correction. Information is recorded in the recording means.

  The shake correction apparatus according to the present invention includes the shake detection apparatus, and a correction processing unit that corrects a shake of the correction target image based on the correction information recorded in the recording unit. To do.

  An image pickup apparatus according to the present invention includes the shake detection device and acquires an image by photographing.

  According to the present invention, it is possible to provide a shake detection device, a shake correction device, and an imaging device that contribute to reducing the data size of correction information for correcting shake of a correction target image.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

<< First Embodiment >>
First, a first embodiment of the present invention will be described. FIG. 1 is an overall block diagram of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 is a digital still camera capable of capturing and recording still images, or a digital video camera capable of capturing and recording still images and moving images.

  The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, and an operation unit 17. The operation unit 17 is provided with a shutter button 17a.

  FIG. 2 shows an internal configuration diagram of the imaging unit 11. The imaging unit 11 drives and controls the optical system 35, the diaphragm 32, the imaging element 33 including a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like, and the optical system 35 and the diaphragm 32. And a driver 34. The optical system 35 is formed from a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 drives and controls the positions of the zoom lens 30 and the focus lens 31 and the opening degree of the diaphragm 32 based on the drive control signal from the main control unit 13, so that the focal length (angle of view) and the imaging unit 11 are controlled. The focal position and the amount of light incident on the image sensor 33 are controlled.

  The image sensor 33 photoelectrically converts an optical image representing a subject incident through the optical system 35 and the diaphragm 32 and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and in each photographing, each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time. An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1. In the following description, “exposure” means exposure of the image sensor 33.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and functions as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image”). The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15. The main control unit 13 outputs an exposure time control signal for controlling the exposure time of the image sensor 33 to the imaging unit 11.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device including a liquid crystal display panel and the like, and displays a photographed image, an image recorded on the recording medium 16, and the like under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and stores captured images and the like under the control of the main control unit 13.

  The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing photographing and recording of a still image. By pressing the shutter button 17a, the photographing and recording of a still image is instructed.

  The operation modes of the imaging apparatus 1 include a shooting mode capable of shooting and recording still images or moving images, and a playback mode for playing back and displaying still images or moving images recorded on the recording medium 16 on the display unit 15. included. Transition between the modes is performed according to the operation on the operation unit 17.

  The photographed image can include a camera shake caused by camera shake. The main control unit 13 includes a blur correction unit (blur correction device) that executes a blur correction process, and has a function of correcting blur of a still image by image processing. Hereinafter, a captured image as a still image whose blur is to be corrected by the blur correction unit is particularly referred to as a “correction target image”, and an image after the blur correction is referred to as a “blur correction image”.

  When the shutter button 17a is pressed in the shooting mode, exposure for shooting the correction target image is performed, and a signal representing the correction target image is output from the image sensor 33. A period in which one or a plurality of photographed images including the correction target image is exposed is referred to as an actual photographing period. In the shooting mode, the imaging unit 11 sequentially performs shooting at a predetermined frame period (for example, 1/30 second), and each shooting obtained by shooting during a period other than the actual shooting period (hereinafter referred to as a preview period). The image is called a preview image. The preview images obtained one after another are sequentially updated and displayed on the display unit 15.

  FIG. 3 shows a temporal relationship between the preview period and the actual shooting period in the shooting mode. In a state where the preview image sequence is acquired by periodic shooting during the preview period, when the shutter button 17a is pressed, the preview period shifts to the actual shooting period, and the correction target image is exposed during that period. The When necessary exposure including exposure of the correction target image is completed, the preview period is resumed from the actual photographing period. Thereafter, each time the shutter button 17a is pressed, the transition from the preview period to the actual shooting period and the transition from the actual shooting period to the preview period occur in the same manner as described above.

  FIG. 4 shows a partial block diagram of the imaging apparatus 1. The main control unit 13 is a camera shake amount estimation unit 51 for estimating the amount of camera shake that has acted on the imaging apparatus 1 during the exposure period of the correction target image, and a shooting control that performs shooting control including control over the exposure time of the image sensor 33. Unit 52, a recording control unit 53 that generates data to be recorded on the recording medium 16 and records the data on the recording medium 16, and receives the recording data of the recording medium 16 and executes blur correction processing on the correction target image And a correction processing unit 54.

In this specification, data representing an image is referred to as image data, and in a sentence explaining that some processing (recording, saving, reading, etc.) is performed on the image data of a certain image, the description is simplified. The description of the image data may be omitted. For example, the expression of recording still image data is synonymous with the expression of recording still images. In this specification, by describing a symbol, the name of an image or the like corresponding to the symbol may be simplified. For example, the normal exposure image _GL and the image _GL indicate the same thing. In the present specification, an image sequence refers to a collection of a plurality of images arranged in time series.

  Hereinafter, first to seventh examples will be described as examples for explaining in detail the operation of the imaging apparatus 1 (particularly, the operation of blur correction processing). The matters described in one embodiment can be applied to other embodiments as long as no contradiction arises.

[First embodiment]
First, the first embodiment will be described as an embodiment for realizing the basic operation of the processing shown in the second and subsequent embodiments. Matters described in the first embodiment are appropriately referred to in the description of the second and subsequent embodiments. FIG. 5 is a flowchart illustrating the flow of the shooting operation and the correction operation of the imaging apparatus 1 according to the first embodiment. In the first embodiment, the camera shake amount estimation unit 51 of FIG. 4 does not function.

Each processing content of steps S1 to S5 executed in the shooting mode will be described. Acquired when the shutter button 17a is pressed in shooting mode, in step S1, the main control unit 13, the data representing the focal length f _D of the imaging unit 11 based on the position and the known information of the lens in the optical system 35 while, the exposure amount of the image sensor 33 to calculate the exposure time T _EP to be appropriate based on the brightness information obtained from the photometric circuit for measuring the brightness of an object (not shown). The exposure time is the length of the exposure period for taking a certain still image.

Thereafter, in step S2, the imaging apparatus 1 acquires a normal exposure image and a short exposure image by continuously executing normal exposure shooting and short exposure shooting. Short exposure photography is photography performed with exposure time shorter than exposure time of normal exposure photography. _TEP is used as the exposure time for normal exposure photography, and _TEP / 4 is used as the exposure time for short exposure photography. The shooting control unit 52 generates an exposure time control signal for normal exposure shooting and short exposure shooting based on the exposure time _TEP and supplies it to the image sensor 33, whereby normal exposure shooting and short exposure shooting are executed. Also, usually the focal length of the imaging unit 11 at the time of execution of exposure shooting and short-exposure shooting is f _D.

The normal exposure image means a photographic image as a still image obtained by normal exposure photography, and the short exposure image means a photographic image as a still image obtained by short exposure photography. The normal exposure shooting and the short exposure shooting in step S2 are performed during the actual shooting period (see FIG. 3), and the short exposure shooting is performed immediately before or after the shooting of the normal exposure image. A normal exposure image and the short-exposure image obtained here are represented by the symbol G _L and G _S. Usually the image data of the exposure image G _L and the short-exposure image G _S is applied to the recording control unit 53 of FIG. Normal image size of the exposure image G _L and the short-exposure image G _S (i.e., number of pixels in the horizontal direction and the vertical direction) are equal.

After step S2, at step S3, the recording control unit 53 extracts a characteristic small area from a short-exposure image G _S. The image in the extracted small area is represented by a small image _GSA . For example, a small area of 128 × 128 pixels is extracted as a characteristic small area by using a Harris corner detector. A characteristic small region refers to a region having a relatively large number of edge components (in other words, a relatively strong contrast) in the extraction source image, and is, for example, a region including a characteristic pattern. A characteristic pattern has, for example, a luminance change in two or more directions such as a corner of an object, and the position of the pattern (position on the image) can be easily detected by image processing based on the luminance change. Meaning a simple pattern.

Thereafter, in step S4, the recording control unit 53 compresses the image data of the normal exposure image G _L and the small image G _SA according to the compression method of JPEG (Joint Photographic Experts Group). Hereinafter, this compression is referred to as JPEG compression. It is assumed that the image size does not change by this JPEG compression. Images obtained by applying this compression to the normal exposure image G _L and the small image G _SA are denoted by symbols J _L and J _SA, respectively. Since the image G _L and J _L is the presence or absence of compression only difference is, hereinafter, sometimes image J _L is also referred to as a normal exposure image. Similarly, the image J _SA may be called a small image or a small image in a short exposure image.

In step S5, the recording control unit 53 creates an image file for recording the image data in the recording medium 16, and records the image data of the normal exposure image J _L in the main body area of the image file. The image data of the small image _JSA in the short exposure image is recorded in the header area of the image file. At this time, the recording control unit 53, the position data representing the position of the small image J _SA, also data representing the data and the exposure time T _EP which represents the focal length f _D is recorded in the header area. The position data represents the coordinate value of the central pixel of the small image G _{SA on} the short exposure image G _S. Image data of the image J _SA is a correction information used to correct the blur of the normal exposure image J _L during execution of the motion compensation process, the position data is also included in the correction information.

  For example, the image file is created so as to conform to the file format of Exif (Exchangeable image file format). FIG. 6 shows the structure of one image file. The image file is formed of a main body area and a header area. When conforming to the Exif file format, the header area is also referred to as an Exif tag or an Exif area. In the following description, an image file refers to an image file recorded in the recording medium 16 unless otherwise specified.

  After the above-described photographing operation, when a correction instruction is given to the imaging apparatus 1 (or when a predetermined condition is satisfied), the correction operation is executed. In the first embodiment, the correction operation includes steps S6 to S9. The correction instruction is given to the imaging device 1 by a predetermined operation on the operation unit 17 by the user, for example. However, the correction operation can be executed regardless of whether or not there is a correction instruction. The correction operation is executed in the reproduction mode. However, the correction operation can also be performed in the shooting mode.

Each process of steps S6 to S9 will be described. When a correction instruction is given to the imaging apparatus 1, in step S6, the correction processing unit 54 reads out the image data of the normal exposure image _{JL and} the image data of the small image _JSA extracted from the short exposure image from the recording medium 16. At the same time, the position data of the small image J _SA is also read from the recording medium 16. The correction processing unit 54 handles the normal exposure image _JL as a correction target image.

In subsequent step S7, the correction processing unit 54 calculates camera shake information representing a shake state included in the normal exposure image J _L based on the image data of the normal exposure image J _L and the small image J _SA . Shake information, for example, an image convolution function that normally represent the state of deterioration of the exposure image J _L, the point spread function as an image convolution function (Point Spread Function; hereinafter referred to as PSF) is obtained. Further, denoted by where resulting camera shake information symbols P _L.

In step S8, the correction processing unit 54, based on the camera shake information P _L obtained in step S7, the normal image restoration process for removing blur contained in the exposure image J _L normal exposure image J _L As a result, a blur correction image is generated from the normal exposure image J _L that is the correction target image. An example method for calculating camera shake information based on the image data of the normal exposure image J _L and the small image J _{SA and} an example method for image restoration processing using the camera shake information will be described later (see the seventh embodiment).

Thereafter, in step S9, the main control unit 13 newly generates an image file for recording the image data of the shake correction image generated in step S8, and records the image file on the recording medium 16. Alternatively without newly generating an image file, usually as image data of the exposure image J _L is overwritten, it corrected image blur in the body region of the normal image file which has been recorded image data of the exposure image J _L Image data may be recorded. The blur correction image is displayed on the display unit 15.

The short exposure image is usually less blurred than the normal exposure image, and if the normal exposure image is corrected with the edge state of the short exposure image as a target, the blur of the normal exposure image is reduced. For this reason, as described above, the normal exposure image and the short exposure image are taken in response to the pressing of the shutter button 17a. In this case, in step S5, the entire image data of an image obtained by JPEG compression of the entire image of the short-exposure image G _S, as correction information, the header area of an image file instead of the image data of the small image J _SA It is also possible to record. In this case, at the time of blur correction processing, a small image equivalent to the small image J _SA is extracted from the entire image of the recorded short exposure image using the processing in step S3, and the extracted small image is used as the small image J _SA. handle can create camera shake information P _L by executing the process of step S7. However, by doing so, the amount of increase in the size of the image file due to the recording of the correction information is considerably increased. Therefore, as described above, image data of a small image (that is, J _SA ) corresponding to a partial image of the short exposure image is recorded in the image file as correction information. Thereby, an increase in the size of the image file is suppressed.

However, the image data of a small image (that is, J _SA ) cannot be said to be sufficiently small. Increasing the size of the correction information decreases the number of recordable images on the recording medium 16 having a predetermined recording capacity. Therefore, it is desirable to provide a function for further suppressing an increase in the size of the image file. An embodiment for realizing this function will be described in the second and subsequent embodiments.

It is also conceivable to create camera shake information based on the image data of the normal exposure image and the short exposure image at the time of shooting. In this case, it is not necessary to record the correction information including the image data of the small image on the recording medium 16, and for example, the created camera shake information may be recorded in the header area of the image file. Since the data size of camera shake information is smaller than the data size of a small image (that is, J _SA ), the method of creating camera shake information at the time of shooting has a great effect of suppressing the increase in the size of the image file.

  However, a large amount of processing time is required to accurately calculate the camera shake information based on the image data of the normal exposure image and the short exposure image. If the camera shake information is created at the time of shooting, the creation process During the relatively long time until part or all of is completed, the next still image shooting cannot be executed. For blur detection and blur correction, it is not desirable to reduce the performance of continuous shooting, which is an important performance of the imaging apparatus. Therefore, the imaging apparatus 1 executes the camera shake information calculation in a period during which processing for a relatively long time is likely to be permitted (for example, during image reproduction).

[Second Embodiment]
A second embodiment will be described. FIG. 7 is a flowchart illustrating the flow of the shooting operation of the imaging apparatus 1 according to the second embodiment. Since the processes in steps S1 to S5 shown in FIG. 7 executed in the second embodiment are the same as those described in the first embodiment (see FIG. 5), step S1 is described in the description of the second embodiment. Explanation of each processing content of ~ S5 is simplified.

When the shutter button 17a is pressed in shooting mode, exposure time T _EP together with data representing the focal length f _D is obtained is calculated at step S1, then the process proceeds to step S11. As described in the first embodiment, the focal length f _D and the exposure period T _EP is generally to be the correction target image is the focal length and the exposure time at the time of shooting the exposure image G _L.

In step S11, the camera shake amount estimation unit 51 in FIG. 4 estimates the magnitude of the movement of the imaging device 1 during the exposure period of the normal exposure image. Although the movement of the imaging device 1 (that is, the movement of the housing of the imaging device 1 in real space) is mainly caused by camera shake, the housing of the imaging device 1 is not held by a hand and is fixed by a tripod or the like. Even when the image pickup apparatus 1 is, the image pickup apparatus 1 can move due to vibration of the ground. Although the movement of the imaging device 1 when the imaging device 1 is fixed on a tripod is not derived from camera shake, it is mainly assumed that the imaging device 1 moves due to camera shake, and during the exposure period of a normal exposure image The magnitude of the movement of the image pickup apparatus 1 in FIG. Therefore, the camera shake amount estimation unit 51 estimates the camera shake amount. The amount of camera shake is a one-dimensional amount, and the amount of camera shake increases as the magnitude of movement of the imaging apparatus 1 during the exposure period of the normal exposure image (for example, the amount of rotation of the imaging apparatus 1 in the yaw direction during the exposure period) increases. Shall increase. Shake amount estimated in step S11 represented by S _L. Shake amount S _L is supplied to the recording control unit 53 of FIG.

The amount of camera shake is generally proportional to the focal length and the exposure time. Therefore, we shake amount estimation unit 51, using the acquired focal length f _D and the exposure time T _EP in step S1, estimates the shake amount S _L according to the following formula (A-1). Here, k _EST is a conversion coefficient. Although the value of the conversion coefficient k _EST can be made variable, in the first embodiment, the value of the conversion coefficient k _EST is assumed to be a preset fixed value (for example, 0.7).
S _L = k _EST × T _EP × f _D (A-1)

After the camera shake amount S _L is obtained, in steps S12 and S13, the recording control unit 53 determines the magnitude relationship between the camera shake amount S _L and predetermined threshold values TH _{1 to} TH ₃ . Here, 0 <TH ₁ <TH ₂ <TH ₃ (see FIG. 8). First, in step S12, it is determined whether the first inequality “S _L <TH ₁ ” or the second inequality “S _L > TH ₃ ” is satisfied, and if the former inequality is satisfied, the image should be a correction target image. If the blur amount of the normal exposure image is so small as to be negligible, it is determined that the blur correction is unnecessary, and the process proceeds to step S17. If the latter inequality holds, the blur amount is too large and the blur correction is difficult. In this case as well, the process proceeds to step S17.

When the process proceeds to step S17, and executes only ordinary-exposure shooting in real shooting period, thereby to JPEG compression in the normal exposure image G _L a step S18 obtained to generate an image J _L, the image of the image J _L Data is recorded in an image file in step S19, and the image file is stored in the recording medium 16. That is, when the process proceeds to step S17, the correction information is not recorded on the recording medium 16.

In step S12, when both the first and second inequalities are not established, the process proceeds to step S13, and it is determined whether or not the third inequality “S _L <TH ₂ ” is established. If this third inequality is not satisfied, it is determined that the image restoration process based on the normal exposure image and the short exposure image should be performed because the amount of blurring of the normal exposure image to be the correction target image is large to some extent. From step S2 to steps S2 to S5. In other words, the ordinary-exposure shooting and short-exposure shooting acquires a normal exposure image G _L and the short-exposure image G _S (step S2), the normal exposure image obtained through a small region extraction process and the JPEG compression (Step S3 and S4) J _L and the small image J _SA in the short exposure image are recorded in an image file, and the image file is stored in the recording medium 16 (step S5).

On the other hand, when the third inequality is satisfied in step S13, the amount of blurring of the normal exposure image that should be the correction target image is slight. In such a case, it is possible not to perform image restoration processing using accurate camera shake information, but also to correct camera shake by other image processing. For example, even if edge enhancement processing is performed on a normal exposure image, it is possible to generate a blur correction image that has almost no visual difference compared to the case where image restoration processing based on a normal exposure image and a short exposure image is performed. is there. Therefore, when the third inequality “S _L <TH ₂ ” is satisfied in step S13, the process proceeds from step S13 to step S14, and the processes of steps S14, S15, and S16 are executed.

That is, in step S14, the imaging apparatus 1 executes only normal exposure shooting during the actual shooting period. Recording control unit 53 in step S15 to continue, the normal image data of the exposure image G _L obtained by ordinary-exposure shooting in step S14 and JPEG compression to generate an image J _L. Thereafter, in step S16, the recording control unit 53 creates an image file in the recording medium 16, and records the image data of the normal exposure image J _L in the main body area of the image file, while at the same time, the header area of the image file. The edge emphasis flag is recorded in The edge emphasis flag is correction information that is referred to when the blur correction process is executed, and its data size is several bytes at most.

As can be understood from the branch determination made in step S13, the blur correction image to be obtained by performing the edge enhancement process and the image restoration process based on the normal exposure image and the short exposure image. When contrasting with an image, if there is a certain visual difference between the two images, the process proceeds from step S13 to step S2, and if there is almost no visual difference, the process proceeds from step S13 to step S13. The threshold value TH ₂ may be set through experiments so that the process proceeds to S14.

When the recording process in step S5, S16, or S19 is completed, the operation for obtaining one normal exposure image is terminated. When the shutter button 17a is pressed again, the operations after step S1 described above are executed. In step S5, S16 and S19, the header area of the image file to the image data of the image J _L is recorded, the data representing the focal length f _D and the exposure time T _EP also recorded.

  Next, the correction operation will be described. FIG. 9 is a flowchart illustrating the flow of the correction operation of the imaging apparatus 1 according to the second embodiment. Since the processes of steps S7 to S9 shown in FIG. 9 executed in the second embodiment are the same as those described in the first embodiment (see FIG. 5), step S7 is described in the description of the second embodiment. The description of each processing content of S9 is simplified.

After the shooting operation shown in FIG. 7, when a correction instruction is given to the imaging apparatus 1 (or when a predetermined condition is satisfied), the correction operation after step S21 is executed. In step S21, the correction processing unit 54 reads the image data of the normal exposure image J _L from the recording medium 16. The correction processing unit 54 handles the normal exposure image _JL as a correction target image.

In step S22, the correction processing unit 54 determines whether image data based on the short exposure image, that is, image data of the small image _JSA exists in the header area of the image file in which the normal exposure image J _L is recorded. Confirm whether or not. If the image data of the small image J _SA exists, the process proceeds from step S22 to step S23, and the image data of the small image J _SA is read from the image file in the recording medium 16, and then the steps S7 to S7 are performed. The process of S9 is executed. That is, based on the image data of the image J _L and J _SA read in step S21 and S23 calculates the hand shake information P _L of the image J _L (step S7), and corrects the image restoration process based on the camera shake information P _L target to generate a corrected image blur by subjecting the image J _L is an image (step S8), and records it in the image file (step S9).

On the other hand, when the image data of the small image _JSA does not exist in the header area of the image file in which the normal exposure image J _L is recorded, the process proceeds from step S22 to step S24, and the correction processing unit 54 It is confirmed whether or not an edge emphasis flag exists in the region. If the edge enhancement flag does not exist, the correction operation shown in FIG. 9 is terminated without executing the blur correction process for the normal exposure image J _L. On the other hand, if the edge enhancement flag exists, steps S24 to S24 are performed. The process proceeds to S25, and each process of steps S25 and S26 is executed.

In step S25, correction processor 54 is normally exposed image J _L is a correction target image, usually performing the edge enhancement processing for enhancing edges in the exposure image J _L, ordinary-exposure image J after the edge emphasis processing _L is generated as a blur correction image.

For example, usually by spatial filtering the entire exposure image J _L, generates a normal exposure image J _L after the edge emphasis process using a sharpening filter that is set in advance with a predetermined filter size. Alternatively, the edge enhancement processing in step S25 may be realized by filtering using an unsharp mask filter. Filtering using an unsharp mask filter is also called unsharp masking. In unsharp masking, an input image (normal exposure image J _{L in} this example) is smoothed to generate a smoothed image, and then a difference image between the smoothed image and the input image is generated. Then, the difference image and the input image are synthesized so that the pixel values of the difference image and the pixel values of the input image are added together, thereby generating an output image (in this example, a shake correction image).

  The processing in step S26 following step S25 is the same as that in step S9. However, the shake correction image to be recorded in step S26 is the shake correction image generated in step S25.

  As described above, even if the edge enhancement processing is performed on the normal exposure image, it is possible to generate a blur-corrected image that is almost visually different from the case of performing the image restoration processing based on the normal exposure image and the short exposure image. If it is determined that the correction can be made, the correction information is simplified and information indicating a method of correcting the blur of the normal exposure image (correction target image) (in this embodiment, the edge enhancement flag) is recorded as the correction information. It is recorded on the medium 16. As a result, the data size of the correction information can be significantly reduced as compared with the first embodiment, without substantially reducing the visual effect of the blur correction process.

In addition, since the amount of camera shake is estimated by an extremely simple calculation using shooting conditions (specifically, f _D and T _EP ) obtained before shooting of a normal exposure image, it is a continuous performance that is an important performance of the imaging apparatus. It does not degrade the shooting performance.

In step S25 in the above-described example, a fixed edge enhancement process is performed on the normal exposure image J _L by using a fixedly set sharpening filter. However, the edge enhancement process in this edge enhancement process is performed. The degree may be changed according to the amount of camera shake S _L. In this case, for example, in step S16 of FIG. 7, data representing the camera shake amount S _L is recorded together with the edge enhancement flag in the header area of the image file in which the normal exposure image J _L is to be recorded. Then, the edge enhancement degree of the edge enhancement processing in step S25 in FIG. 9, may be changed in accordance with data representing the recorded image blur amount S _L.

In addition, assuming that image restoration processing based on a normal exposure image and a short exposure image is performed, if the amount of camera shake is relatively small, the PSF as camera shake information is accurately detected in order to detect the small camera shake well. Need to ask. On the other hand, when the amount of camera shake is relatively large, the normal exposure image includes a considerable amount of blur. Therefore, there is little practical problem even if the PSF calculation accuracy is not so high. Considering this, in step S4 in FIG. 7, the compression ratio of JPEG compression for the small image G _SA, may be increased in accordance with the camera shake amount S _L increases. As a result, the data size of the correction information can be further reduced.

Specifically, for example, in step S4, it is determined whether or not the inequality “TH ₁ ≦ S _L <TH ₄ ” is satisfied, and if it is satisfied, the JPEG when generating the image J _SA from the image G _SA is determined. While the compression rate of compression is the first compression rate, if it is not satisfied, the compression rate is set to a second compression rate that is greater than the first compression rate. Here, TH ₄ is a predetermined threshold value that satisfies “TH ₁ <TH ₄ <TH ₂ ” (see FIG. 8). In this example, the compression rate is variably set in two steps, but the compression rate may be variably set in three or more steps.

[Third embodiment]
A third embodiment will be described. Since the shape of the PSF approaches a point when the amount of camera shake is small, the PSF can be approximated to an image deterioration function that represents non-directional Gaussian blur (that is, blur due to smoothing using a Gaussian filter). In fact, when the amount of camera shake is small, even with the image restoration process using the approximate PSF due to Gaussian blur, the blur-corrected image that has almost no visual difference compared to the case of performing the image restoration process based on the normal exposure image and the short exposure image Can be generated. Therefore, in the third embodiment, when the estimated camera shake amount S _L is greater than or equal to the threshold value TH ₁ and less than the threshold value TH ₂ , an image restoration process using an approximate PSF due to Gaussian blur instead of the edge enhancement process. Is used as blur correction processing.

  The third embodiment corresponds to an embodiment obtained by modifying a part of the operation shown in the second embodiment, and the contents not particularly described in the description of the third embodiment are the same as those shown in the second embodiment. It is. Hereinafter, differences from the second embodiment will be described. Please refer to FIG. 7 and FIG.

When the inequality “TH ₁ ≦ S _L <TH ₂ ” is satisfied during the shooting operation shown in FIG. 7, the process proceeds from step S13 to step S14, and when step S16 is reached via step S15, in step S16, The recording control unit 53 creates an image file in the recording medium 16 and records the image data of the normal exposure image J _L in the main body area of the image file, while the Gaussian approximation flag and the step are recorded in the header area of the image file. recording the estimated shake amount S _L in S11. The Gaussian approximation flag and the camera shake amount S _L are correction information that is referred to when the camera shake correction process is executed, and the data size is at most several bytes.

Thereafter, in the correction operation shown in FIG. 9, when the process proceeds from step S22 to step S24, the correction processing unit 54 has a Gaussian approximation flag in the header area of the image file in which the normal exposure image _JL is recorded. Check if it exists. If the Gaussian approximation flag does not exist, the correction operation shown in FIG. 9 is terminated without executing the blur correction processing for the normal exposure image J _L. On the other hand, if the Gaussian approximation flag exists, Steps S24 to S24 are performed. The process proceeds to S <b> 25, and an image restoration process using an approximate PSF due to Gaussian blur is performed on the normal exposure image J _L to generate a shake correction image, and the generated shake correction image is recorded on the recording medium 16.

A point that is stationary in real space is called a “real stationary point”. The amount of camera shake S _L is the size of a trajectory where an ideal point image (that is, an image of an actual still point) moves on the imaging surface of the image sensor 33 during the exposure period of a normal exposure image as a correction target image. An estimated amount is represented, and the estimated amount is obtained in units of the adjacent pixel interval of the normal exposure image. In step S25 according to the third embodiment, the correction processing unit 54 reads the camera shake amount S _L from the header area of the image file in which the normal exposure image J _L is recorded, and the number of taps corresponding to the read camera shake amount S _L. Is used as an approximate PSF with Gaussian blur. For example, when the amount of camera shake S _L is three pixels of the normal exposure image, a Gaussian filter having a 3 × 3 filter size is adopted as an approximate PSF by Gaussian blur, and the size of camera shake S _L is In the case of 5 pixels of the normal exposure image, a Gaussian filter having a filter size of 5 × 5 is adopted as an approximate PSF by Gaussian blur. The dispersion in the Gaussian filter may be determined in advance.

Thereafter, the correction processing unit 54 obtains an image restoration function corresponding to the inverse function (general inverse matrix) of the approximate PSF. The image restoration function is expressed as a single image restoration filter, and each element of the general inverse matrix of the matrix representing the approximate PSF is used as each filter coefficient of the image restoration filter. The correction processing unit 54 applies the spatial filtering using the image restoration filter to the entire correction target image (that is, the normal exposure image J _L ), thereby reducing the blur included in the correction target image. Generate an image.

As will be understood from the branch determination made in step S13, the image restoration process based on the blur correction image to be obtained by performing the image restoration process using the approximate PSF due to Gaussian blur, and the normal exposure image and the short exposure image When the image is compared with the shake-corrected image to be obtained by performing the above, when there is some visual difference between the two images, there is almost no visual difference so that the process proceeds from step S13 to step S2. In such a case, the threshold value TH ₂ may be set through experiments so that the process proceeds from step S13 to step S14.

  Further, as described above, even if image restoration processing using an approximate PSF due to Gaussian blur is performed, there is no visual difference compared with the case where image restoration processing based on a normal exposure image and a short exposure image is performed. If it is determined that an image can be generated, the correction information is simplified and information indicating a method of correcting the blur of the normal exposure image (correction target image) (in this embodiment, a Gaussian approximation flag) Is recorded on the recording medium 16 as correction information. As a result, the data size of the correction information can be significantly reduced as compared with the first embodiment, without substantially reducing the visual effect of the blur correction process.

[Fourth embodiment]
A fourth embodiment will be described. FIG. 10 is a flowchart illustrating the flow of the shooting operation of the imaging apparatus 1 according to the fourth embodiment. Since the processes of steps S1 to S5, S12, S13, S15, S16, S18, and S19 shown in FIG. 10 executed in the fourth embodiment are the same as those in the second embodiment (see FIG. 7). In the description of the fourth embodiment, the description of each processing content is simplified. The correction operation of the image pickup apparatus 1 according to the fourth example is the same as that in the second example.

Estimation method of shake amount S _L in the fourth embodiment is different from that in the second embodiment. Focusing on this difference, the imaging operation of the imaging apparatus 1 will be described with reference to FIG.

When the shutter button 17a is pressed in shooting mode, exposure time T _EP is calculated together with data representing the focal length f _D is obtained at step S1. Thereafter, the flow proceeds to step S2, the image pickup apparatus 1 acquires the normal exposure image G _L and the short-exposure image G _S by executing the ordinary-exposure shooting and short-exposure shooting continuously. After the normal exposure image G _L and the short-exposure image G _S is acquired, the process proceeds to step S30. In step S30, shake amount estimation unit 51, typically estimates the shake amount S _L based on the image data of the exposure image G _L and the short-exposure image G _S.

FIG. 11 shows a detailed flowchart of the process in step S30. In step S30, the camera shake amount estimation unit 51 of FIG. 4 sequentially executes the processes of steps S31 to S35. First, shake amount estimation unit 51, an image of the small area as a small image G _LB usually by extracting the small area located in the center of the exposure image G _L, and, located in the center of the short-exposure image G _S the small image G _SB image of the small area by extracting a small area. The small images G _LB and G _SB have, for example, an image size of 128 × 128. Although here is to extract a central small region from each of the normal exposure image G _L and the short-exposure image G _S for the purpose of simplifying the calculation, using a corner detector or the like Harris, it features from them A typical small area may be extracted.

Next, in step S32, the luminance level of the small image _GSB is increased with the small image _GLB as a reference. That is, (as the average luminance of the average luminance and the small image G _LB of the small image G _SB equals) luminance level is small image G _LB of to equal the brightness level of the small image G _SB, the small image G _SB A luminance normalization process is performed in which the luminance value of each pixel is multiplied by a certain value. The small image G _SB after this luminance normalization processing is called a small image G _SB ′.

In subsequent step S33, 'by performing an edge extraction process for each of the small image G _LB edge extraction image E _LB and small image G _SB' small images G _LB and G _SB edge extraction image E _SB 'of obtain. For example, if an arbitrary edge detection filter (Laplacian filter or the like) is applied to each pixel of the small image G _LB, the edge extraction image E _LB is obtained from the output value of the edge detection filter. The same applies to the small image G _SB ′.

Thereafter, in step S34, the sum V _L of the pixel values of the edge extracted image E _LB and the sum V _S of the pixel values of the edge extracted image E _SB ′ are calculated. In step S35, according to the following equation (A-2), The amount of camera shake S _L to be estimated in step S30 is calculated. The conversion coefficient k _EST is the same as that described in the description of the second embodiment.
S _L = k _EST × (V _S / V _L ) (A-2)

As is well known, each pixel value in the edge extraction image represents the edge strength of the corresponding part in the edge extraction source image (in this example, the small image G _LB or G _SB ′). When it is considered that the edge in the short exposure image is sufficiently clear, the ratio of the value V _S corresponding to the edge intensity of the short exposure image and the value V _L corresponding to the edge intensity of the normal exposure image is used. The amount of camera shake S _L can be estimated.

After shake amount S _L is estimated in step S30, step S12 and S13, the recording control unit 53 determines the magnitude relation between the image blur amount S _L and a predetermined threshold value TH ₁ to TH _3. If the inequality “S _L <TH ₁ ” or “S _L > TH ₃ ” is satisfied, the processes of steps S18 and S19 are performed, and the photographing operation is terminated. If the inequality “TH ₁ ≦ S _L <TH ₂ ” is satisfied, the processes of steps S15 and S16 are performed, and the photographing operation is terminated. If the inequality “TH ₂ ≦ S _L ≦ TH ₃ ” is satisfied, the processing of steps S3 to S5 is performed and the photographing operation is terminated.

In the second embodiment corresponding to FIG. 7, if the inequality “S _L <TH ₁ ” or the inequality “S _L > TH ₃ ” is satisfied after the camera shake amount S _L is estimated, the processing in steps S 17 to S 19 is performed. When the inequality “TH ₁ ≦ S _L <TH ₂ ” is satisfied, the processes of steps S14 to S16 are performed. When the inequality “TH ₂ ≦ S _L ≦ TH ₃ ” is satisfied, the processes of steps S2 to S5 are performed. Although has been performed, in the fourth embodiment, since the amount of camera shake S _L normal exposure image G _L and the short-exposure image G _S before estimation is acquired, step S17, S14 after the estimation of the shake amount S _L And S2 need not be executed.

  According to the present embodiment, the data size of the correction information can be greatly reduced as compared with the first embodiment without substantially reducing the visual effect of the blur correction process. Further, since the amount of camera shake is estimated by a simple calculation based on the image data of the normal exposure image and the short exposure image, there is no (or minor) deterioration in continuous shooting performance due to this estimation.

  It should be noted that the contents shown in the present embodiment can be combined with the contents shown in the third embodiment.

[Fifth embodiment]
A fifth embodiment will be described. In the second to fourth embodiment is of a fixed value conversion factor k _EST for estimating a shake amount S _L. However, the actual amount of camera shake differs slightly if the user holding the imaging device 1 is different. Therefore, compared with the estimation method based on the focal length and exposure time, the amount of camera shake is estimated separately using an estimation method that seems to have higher estimation accuracy, and the conversion coefficient k _EST is updated using the amount of camera shake estimated separately. (In other words, correction) is desirable.

The method for updating the conversion coefficient k _EST will be described by applying it to the operation shown in the second embodiment. Of course, the conversion coefficient k _EST updating method described in the fifth embodiment can be applied to the third or fourth embodiment.

Shake amount estimation unit 51 of FIG. 4, the amount of camera shake S _L2 different from the amount of camera shake S _L, separately estimated from camera shake information P _L based on the normal-exposure image and the short-exposure image. The camera shake information P _L is PSF. In the matrix representing the PSF as the camera shake information P _L , an ideal point image (that is, an image of the actual still point) is captured during the exposure period of the normal exposure image by connecting the elements having values other than 0. The locus | trajectory which moves on the imaging surface of the element 33 is represented. The curve denoted by reference numeral 200 shown in FIG. 12 represents the locus on the normal exposure image. FIG. 13 shows a PSF expressed in a matrix corresponding to the locus 200. In the PSF of FIG. 13, the part filled with diagonal lines represents a matrix element having a value other than zero.

The camera shake amount estimation unit 51 estimates the camera shake amount S _L2 from the locus 200. Specifically, the distance between the start point 201 and the end point 202 of the trajectory 200 is obtained as the camera shake amount S _L2 . This distance is a distance on the normal exposure image.

The normal exposure image as the correction target image is obtained each time the shutter button 17a is pressed, and is arranged in time series as shown in FIG. 14 by pressing the shutter button 17a a plurality of times. (P-2) The (p-1) th and pth normal exposure images are obtained (p is an integer of 3 or more). The camera shake amount estimation unit 51 updates the conversion coefficient k _EST using the camera shake amount S _L2 obtained for the (p−1) th normal exposure image and uses the updated conversion coefficient k _EST. The camera shake amount S _L for the p th normal exposure image is estimated.

As a typical example, an example in which update processing of the conversion coefficient k _EST is executed during the correction operation will be described. In this case, after the camera shake information P _L for the (p−1) th normal exposure image is obtained in step S7 of FIG. 9, the camera shake amount estimation unit 51 performs the camera shake for the (p−1) th normal exposure image. Based on the camera shake amount S _L2 obtained from the information P _L, the exposure time T _EP and the focal length f _D at the time of shooting the (p−1) th normal exposure image, the update coefficient k according to the following equation (A-3): Calculate _tmp .
k _tmp = S _L2 / (T _EP × f _D ) (A-3)

Then, update the transform coefficients according to the following formula (A-4), using the transformation coefficient after updating as the conversion coefficient at the time of estimating the p-th shake amount S _L of the normal exposure image. In equation (A-4), k _EST is the conversion coefficient before update, that is, the conversion coefficient k _EST when estimating the camera shake amount S _L of the (p−1) -th normal exposure image, and k _EST ′ is updated. This is the subsequent conversion coefficient, that is, the conversion coefficient k _EST when estimating the camera shake amount S _L of the p-th normal exposure image. K _O is a predetermined attenuation coefficient (where 0 <k _O <1).
k _EST '= k _O × k _EST + (1−k _O ) × k _tmp (A-4)

In this way, by updating the conversion coefficient k _EST , the value of the conversion coefficient is gradually updated to a value suitable for the shake of the specific user every time it is used.

If it is determined that the update of the conversion coefficient k _EST has converged, the update may not be performed thereafter. For example, after _{obtaining the} updated conversion coefficient k _EST ′ from the conversion coefficient k _EST before update using the formula (A-4), the difference between the two is obtained, and the conversion is performed when the difference is smaller than the reference difference. It is determined that the update of the coefficient k _EST has converged.

Further, when the update processing of the conversion coefficient k _EST is simply applied to the operation shown in the second embodiment, a short exposure image is captured unless the process proceeds from step S11 in FIG. 7 to step S2 through steps S12 and S13. Conversion coefficient k _EST cannot be updated, and even if a short-exposure image is captured, conversion coefficient k _EST is not executed unless a correction operation for generating a shake correction image is performed. _{The EST} will not be updated.

Considering this, regardless of the estimated shake amount S _L in step S11 if the shutter button 17a is pressed in shooting mode, the images by performing a normal photographing exposure image G _L and the short-exposure image G _S Data may be temporarily recorded in the internal memory 14 (or the recording medium 16). Then, when the processing load in the imaging device 1 is light, the camera shake amount S _L2 is obtained from the temporarily recorded image data of the images G _L and G _S , and is according to the equations (A-3) and (A-4). The conversion coefficient may be updated. If an operation for stopping the driving of the imaging apparatus 1 is performed during the calculation process for updating the conversion coefficient, or if other tasks require memory, the calculation process is immediately stopped. , The memory used for the arithmetic processing is released.

In addition, it is possible to switch between a mode in which blur correction processing is automatically performed during a shooting operation (hereinafter referred to as a first mode) and a mode in which blur correction processing is performed during image reproduction (hereinafter referred to as a second mode). The imaging device 1 is formed in advance, the conversion coefficient k _EST is updated using the camera shake amount S _L2 obtained during the operation in the first mode, and the updated conversion coefficient k _EST is used in the second mode. It may be. The operations shown in FIGS. 7 and 9 are typical examples of operations in the second mode.

That is, during the operation in the first mode, the normal exposure image and the short exposure image are always continuously photographed as the shutter button 17a is pressed, and the camera shake information P _L is automatically based on the normal exposure image and the short exposure image. Is calculated. Thereafter, a shake-corrected image is generated by image restoration processing based on the camera shake information P _L , while a camera shake amount S _L2 is obtained every time based on the camera shake information P _L. Shake amount estimation unit 51 uses the exposure time T _EP and focal length f _D which is calculated each time the depression of the each sought shake amount S _L2 and the shutter button 17a, formula (A-3) and (A -4), the conversion coefficient k _EST is updated. After this update, when the operation in the second mode is executed, the camera shake amount S _L is estimated using the conversion coefficient k _EST updated in the first mode.

[Sixth embodiment]
A sixth embodiment will be described. Although the first to fifth embodiments have been described mainly assuming that the housing of the imaging device 1 is held by hand, even if the housing of the imaging device 1 is fixed to the ground using a tripod or the like. A small movement may occur in the imaging apparatus 1 due to the vibration of the ground. The magnitude of this movement can be estimated from the preview image sequence acquired in the preview period before the actual shooting period. In the sixth embodiment, the camera shake amount _SL is estimated using this preview image sequence.

This estimation method will be described more specifically. As shown in FIG. 15, preview images PI ₁ , PI ₂ , PI ₃ ,..., PI _q-1 and PI _q are sequentially photographed and previewed in the preview period immediately before a certain actual photographing period. Assume a case where a shift is made to an actual shooting period of interest immediately after shooting of the image PI _q (q is an integer of 3 or more).

  The camera shake amount estimation unit 51 in FIG. 4 pays attention to the preview image sequence acquired during the preview period, and compares the preview images of the adjacent frames forming the preview image sequence, so that the preview image sequence between adjacent frames is compared. Find the optical flow. The preview image of the adjacent frame refers to two preview images taken adjacent in time, and the time difference between the intermediate time of the exposure period in one preview image and that of the other is the frame period (for example, 1 / 30 seconds).

  The optical flow is obtained based on the image data of the contrasted preview image using a representative point matching method, a block matching method, or a gradient method, and is formed from a plurality of motion vectors representing the motion of the object between the two images. . The camera shake amount estimation unit 51 obtains an average vector of a plurality of motion vectors obtained for two preview images, and when the magnitude of the average vector continues to be a predetermined value or less for a predetermined time, It is determined that one housing is fixed with a tripod or the like.

Consider a case where the predetermined time corresponds to the photographing period of the preview images PI _{1 to} PI _q . In this case, the camera shake amount estimation unit 51 includes an average vector m _{[1: 2]} of a plurality of motion vectors between the preview images PI ₁ and PI ₂ and a plurality of motion vectors between the preview images PI ₂ and PI _3. mean vector _{m [2: 3], ···} , the average of a plurality of motion vectors between the preview image PI _q-1 and PI _q vector m _{[q-1: q]} obtained. Then, their mean vector m _{[1: 2] ~m:} the magnitude of _{[q-1 q]} is compared with a predetermined constant value, the mean vector _{m [1: 2] ~m [} q-1: q] If the size of the image pickup device 1 is less than or equal to the predetermined value, it is determined that the housing of the imaging device 1 is fixed by a tripod or the like. Otherwise, the housing of the imaging device 1 is gripped by a hand. Judge that

If the housing of the imaging apparatus 1 is determined to be manually grasped, usually it estimates the shake amount S _L for exposing the image G _L by the method described in the second to fifth embodiment, the imaging device 1 If the housing is determined to be fixed by a tripod or the like, according to the following formula (a-5), estimates the shake amount S _L for the normal exposure image G _L. Here, FR is a frame rate (reciprocal of the frame period) during the preview period, and | m _AVE | is a total of (q−1) average vectors m _{[1: 2] to} m _{[q-1: q].} Is the size of the vector. _TEP is the length of the exposure period of the normal exposure image _GL as described above.
S _L = FR × | m _AVE | × T _EP (A-5)

Photographing operation and correction operation other than estimation operation shake amount S _L are the same as those explained in any one of the second to fifth embodiment. The camera shake amount S _L estimated using the preview image sequence is used as, for example, the camera shake amount S _L to be estimated in step S11 of FIG.

If the camera shake amount for the normal exposure image is estimated from the preview image sequence obtained before shooting the normal exposure image as in this embodiment, it is estimated using the shooting conditions (f _D and T _EP ). As in the case, the deterioration of the continuous shooting performance due to the estimation of the amount of camera shake is completely or substantially avoided.

[Seventh embodiment]
A seventh embodiment will be described. In the seventh embodiment, an example of a method for calculating the camera shake information P _L based on the image data of the normal exposure image J _L and the small image J _{SA and} an example of an image restoration process using the camera shake information P _L will be described. The contents described in the seventh embodiment are applied to the first to sixth embodiments (specifically, they are used for the processes in steps S7 and S8 shown in FIG. 5 or FIG. 9). In the description of the seventh embodiment, when “memory” is simply used, it means the internal memory 14.

FIG. 16 is a flowchart showing the flow of blur correction processing according to the seventh embodiment. Correction processing unit 54 of FIG. 4, on the basis of the position data normally read out from the recording medium 16 representing the position of the image data and small image J _SA exposure image J _L and the small image J _SA, the process of step S71~S77 A blur correction image is generated by sequentially executing.

As described above, the small image J _SA is an image in a characteristic small region extracted from the short exposure image G _S (more precisely, it is an image obtained by JPEG compression of the image; step of FIG. 7). (See S3 and S4). First, in step S71, a small area corresponding to a characteristic small area extracted from the short exposure image G _{S is} extracted from the normal exposure image J _L as the correction target image, and the small area extracted from the normal exposure image J _{L is} extracted. The image in the area is stored on the memory as a small image _JLA . When the small image _JSA has an image size of 128 × 128 pixels, the small image _JLA also has an image size of 128 × 128 pixels.

When ignoring the misalignment between the normal exposure image and the short exposure image, it is extracted from the center coordinates of the small image _JLA extracted from the normal exposure image (center coordinates in the normal exposure image) and the short exposure image. that the center coordinates of the small image J _SA as the (center coordinates in the short-exposure image) are equal, and extracts a small region based on the positional data of the small image J _SA. If the positional deviation cannot be ignored, a corresponding small region may be searched using a template matching method or the like. That is, for example, a small area having the highest similarity to the template is searched from the normal exposure image J _L using the small image _JSA as a template, and the image in the searched small area is searched. _{Is a} small image _JLA . The search range of this search is set based on the position data of the small image _JSA .

Since the exposure time of the short-exposure image is relatively short, the signal-to-noise ratio of the small image J _SA, along with relatively low small image J _SA luminance is relatively low. Therefore, in step S72, the subjected to noise removal processing using a median filter or the like to the small image J _SA, further increases the brightness level of the small image J _SA after noise removal processing. Thus, for example, (as the average luminance of the average luminance and the small image J _LA of the small image J _SA equals) luminance level is small image J to equal the luminance level of the _LA of the small image J _SA, small image J _SA A luminance normalization process is performed in which the luminance value of each pixel is multiplied by a certain value. The small image J _SA after the noise removal processing and the luminance normalization processing is stored on the memory as a small image J _SA ′.

The small image _JLA obtained as described above is _treated as a deteriorated image and the small image J _SA ′ is treated as an initial restored image (step S73). obtaining an image degradation function representing the state of degradation due to blurring of the image J _LA. The image degradation function obtained here is camera shake information P _L to be calculated in step S7 (see FIG. 9 and the like).

PSF is obtained as an image degradation function. Since motion blur uniformly deteriorated on the entire image, PSF found for the small image J _LA can be utilized as a PSF for the entire normal exposure image J _L.

  The Fourier iteration method is a method for obtaining a restored image from which deterioration is removed or reduced from a deteriorated image including deterioration (see Non-Patent Document 1 above). When performing the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image). This initial restored image is called an initial restored image. This Fourier iteration method will be described in detail with reference to FIGS. FIG. 17 is a detailed flowchart of the process in step S74 of FIG. FIG. 18 is a block diagram of a part of the correction processing unit 54 of FIG.

First, in step S101, a restored image is set as f ′, and an initial restored image is set in the restored image f ′. That is, the small image J _SA ′ is used as the initial restored image f ′. Next, in step S102, the deteriorated image (that is, the small image J _LA ) is set as g. Then, the result of Fourier transform of the deteriorated image g is stored in the memory as G (step S103). For example, when the image sizes of the initial restored image and the degraded image are 128 × 128 pixels, f ′ and g can be expressed as a matrix having a matrix size of 128 × 128.

Next, in step S110, F ′ obtained by Fourier transform of the restored image f ′ is obtained, and in step S111, H is calculated by the following equation (B-1). H corresponds to the result of Fourier transform of PSF. In Formula (B-1), F ′ ^* is a conjugate complex matrix of F ′, and α is a constant.

  Next, in step S112, PSF is obtained by performing inverse Fourier transform on H. The PSF obtained here is assumed to be h. Next, in step S113, after the PSF h is corrected under the constraint condition of the following formula (B-2a), it is further corrected under the constraint condition of the formula (B-2b).

  Since PSF h is expressed as a two-dimensional matrix, each element of this matrix is represented by h (x, y). Each element of the PSF should originally take a value of 0 or more and 1 or less. Therefore, in step S113, it is determined whether each element of the PSF is 0 or more and 1 or less, and the value of the element that is 0 or more and 1 or less is left as it is. The value of the element is corrected to 1, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (B-2a). Then, the PSF is normalized so that the sum of the elements of the PSF after correction is 1. This normalization is correction by the constraint condition of the formula (B-2b).

  The PSF corrected by the constraints of the equations (B-2a) and (B-2b) is set as h ′.

Next, in step S114, H ′ obtained by Fourier transforming PSF h ′ is obtained, and in step S115, F is calculated by the following equation (B-3). F corresponds to a Fourier transform of the restored image f. In the formula (B-3), H ′ ^* is a conjugate complex matrix of H ′, and β is a constant.

  Next, in step S116, a restored image is obtained by performing an inverse Fourier transform on F. Let f be the restored image obtained here. Next, in step S117, the restored image f is corrected under the constraint condition of the following formula (B-4), and the corrected restored image is newly set as f '.

  Since the restored image f is represented as a two-dimensional matrix, each element of the matrix is represented by f (x, y). Now, it is assumed that the pixel value of each pixel of the degraded image and the restored image is represented by a digital value from 0 to 255. Then, each element (that is, each pixel value) of the matrix representing the restored image f should originally have a value of 0 or more and 255 or less. Accordingly, in step S117, it is determined whether each element of the matrix representing the restored image f is 0 or more and 255 or less, and the value of the element that is 0 or more and 255 or less is left as it is. If there is an element, the value of the element is corrected to 255, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (B-4).

  Next, in step S118, it is determined whether or not the convergence condition is satisfied, thereby determining whether or not the iterative process is converged.

  For example, the absolute value of the difference between the latest F ′ and the previous F ′ is used as an index for convergence determination. If this index is less than or equal to a predetermined threshold value, it is determined that the convergence condition is satisfied. Otherwise, it is determined that the convergence condition is not satisfied.

  When the convergence condition is satisfied, a final PSF is obtained by performing inverse Fourier transform on the latest H ′. That is, the latest H ′ obtained by inverse Fourier transform is the PSF to be obtained in step S74 of FIG. When the convergence condition is not satisfied, the process returns to step S110, and the processes of steps S110 to S118 are repeated. In the repetition of the processes of steps S110 to S118, f ′, F ′, H, h, h ′, H ′, F, and f (see FIG. 18) are sequentially updated to the latest ones.

  Another index can be used as an index for determining convergence. For example, the absolute value of the difference between the latest H ′ and the previous H ′ obtained may be used as the convergence determination index to determine whether or not the convergence condition is satisfied. Further, for example, the correction amount in step S113 using the above formulas (B-2a) and (B-2b) or the correction amount in step S117 using formula (B-4) is used as an index for convergence determination. Whether or not the convergence condition is satisfied may be determined. This is because if the iterative process is toward convergence, the amount of correction becomes small.

Further, when the number of repetitions of the loop process consisting of steps S110 to S118 reaches a predetermined number, it may be determined that convergence is impossible and the process may be terminated without calculating a final PSF. This case, normal correction of the exposure image J _L is a correction target image is not performed.

Returning to the description of each step in FIG. After the PSF is calculated in step S74, the process proceeds to step S75. In step S75, each element of the inverse matrix of PSF obtained in step S74 is obtained as each filter coefficient of the image restoration filter. This image restoration filter is a filter for obtaining a restored image from a deteriorated image. Actually, each element of the matrix represented by the following equation (B-5) corresponding to a part of the right side of the equation (B-3) corresponds to each filter coefficient of the image restoration filter. The intermediate calculation result of the Fourier iteration method in S74 can be used as it is. However, the formula (B-5) H 'in ^* and H' is, H obtained satisfied immediately before the convergence condition in step S118 ^'* and H' (i.e., the finally obtained H ^'* and H' ).

After each filter coefficient of the image restoration filter is obtained in step S75, the process proceeds to step S76, and the entire normal exposure image J _L as the correction target image is filtered (spatial filtering) using this image restoration filter. . That is, the image deconvolution filter having the filter coefficients calculated usually applied to each pixel of the exposure image J _L filtering the normal exposure image J _L. Accordingly, the filtered image with reduced blur contained in the ordinary-exposure image J _L is generated. Although the size of the image restoration filter is smaller than the image size of the normal exposure image J _L , camera shake is considered to give uniform degradation to the entire image, so this image restoration filter is applied to the entire normal exposure image J _L. by application to the overall blurring of the normal exposure image J _L is reduced.

  The filtering image may include ringing accompanying filtering. For this reason, in step S77, a final blurring correction image is generated by applying a ringing removal process for removing this to the filtered image. Since the method for removing ringing is known, a detailed description is omitted. For example, a technique described in JP 2006-129236 A may be used.

  The blur correction image is an image in which blur included in the correction target image is reduced and ringing due to filtering is reduced. However, since the filtered image is also an image with reduced blur, the filtered image can be regarded as a shake-corrected image.

  Since the amount of blur included in the short-exposure image is small, its edge component is close to the edge component of an ideal image without camera shake. Therefore, as described above, an image obtained from this short-exposure image is used as an initial restored image in the Fourier iteration method.

  By repetitive loop processing by the Fourier iteration method, the restored image (f ′) gradually approaches an image from which camera shake has been removed as much as possible. However, since the initial restored image itself is already close to an image without camera shake, Convergence is faster than a random image or a deteriorated image as an initial restored image (in the shortest, convergence is achieved by a single loop process). As a result, the processing time for creating camera shake information necessary for the camera shake correction process is shortened. If the initial restored image is far from the image to be converged, the probability of convergence to a local solution (an image different from the image to be truly converged) increases. However, by setting the initial restored image as described above, , The probability of convergence to a local solution is low (that is, the probability that camera shake correction fails) is low.

  In addition, since camera shake is considered to cause uniform degradation on the entire image, a small area is extracted from each image, and camera shake information is created from the image data of each small area, and is applied to the entire image. As a result, the amount of calculation required is reduced, and the processing time for creating camera shake information and the processing time for correcting camera shake are shortened. Of course, the required circuit scale can be reduced and the cost reduction effect can be expected.

  At this time, as described above, camera shake information is created from a characteristic small region containing a lot of edge components. An increase in the edge component in the PSF calculation source image means an increase in the ratio of the signal component to the noise component. Therefore, the influence of noise is reduced by extracting a characteristic small region, and the PSF can be detected more accurately. become able to.

  In the process of FIG. 17, the degraded image g and the restored image f ′ on the spatial domain are transformed to the frequency domain by Fourier transform, so that the function G representing the degraded image g on the frequency domain and the frequency domain A function F ′ representing the restored image f ′ is obtained (note that the frequency domain is, of course, a two-dimensional frequency domain). A function H representing PSF in the frequency domain is obtained from the obtained functions G and F ′, and this function H is converted into a function in the spatial domain, that is, PSF h by inverse Fourier transform. The PSF h is corrected using a predetermined constraint condition, and a corrected PSF h ′ is obtained. The process of correcting the PSF is hereinafter referred to as “first correction process”.

  PSF h ′ is again transformed onto the frequency domain by Fourier transform to obtain a function H ′, and a function F representing a restored image on the frequency domain is obtained from the functions H ′ and G. By performing inverse Fourier transform on this function F, a restored image f in the spatial domain is obtained, and this restored image f is corrected using a predetermined constraint condition, and a corrected restored image f ′ is obtained. The processing for correcting the restored image is hereinafter referred to as “second correction processing”.

  In the above-described example, it has been described that, after that, the above-described processing is repeated using the corrected restored image f ′ until the convergence condition is satisfied in step S <b> 118 of FIG. 17. Further, considering the characteristic that the correction amount becomes smaller when the iterative process is toward convergence, the establishment / non-establishment of the convergence condition is determined by the correction amount in step S113 corresponding to the first correction process or the second correction. It has also been stated that the determination may be made based on the correction amount in step S117 corresponding to the processing. When this determination is made based on the correction amount, a reference correction amount is set in advance, and the correction amount in step S113 or the correction amount in step S117 is compared with the reference correction amount, and the former is smaller than the latter. In this case, it is determined that the convergence condition is satisfied. However, if the reference correction amount is set sufficiently large, the processes in steps S110 to S117 are not repeatedly executed. That is, in this case, PSF h ′ obtained by performing the first correction process only once is the final PSF to be derived in step S74 in FIG. Thus, even if the process of FIG. 17 is used, the first and second correction processes are not always repeatedly executed.

  The increase in the number of repeated executions of the first and second correction processes contributes to improving the accuracy of the finally obtained PSF. However, in this example, since the initial restoration image itself is close to an image without camera shake, the first correction is performed. The accuracy of PSF h ′ obtained by performing the treatment only once is also high enough to cause no problem. In consideration of this, the determination process itself in step S118 can be omitted. In this case, the PSF h ′ obtained by executing the process of step S113 only once becomes the final PSF to be derived in step S74 of FIG. 16, and the function obtained by executing the process of step S114 only once. From H ′, each filter coefficient of the image restoration filter to be derived in step S75 in FIG. 16 is obtained. Therefore, when the process of step S118 is omitted, the processes of steps S115 to S117 are also omitted.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. FIG. 19 is an overall block diagram of an imaging apparatus 1a according to the second embodiment of the present invention. The imaging device 1a is a digital still camera capable of capturing and recording still images, or a digital video camera capable of capturing and recording still images and moving images.

  The imaging device 1a includes an imaging unit 11, an AFE (Analog Front End) 12, an internal memory 14, a display unit 15, a recording medium 16, and an operation unit 17, which are the same as those in the imaging device 1 of FIG. Is. The imaging apparatus 1a further includes a main control unit 13a and a sensor unit 18. The main control unit 13a has a function equivalent to the function of the main control unit 13 of FIG. The imaging device 1a is different from the imaging device 1 in that it performs shake correction processing based on the angular velocity data given from the sensor unit 18, but in other points, the configuration and function of both imaging devices are common. ing. Therefore, description of the common points between the two image pickup devices will be omitted, and the second embodiment will be described by paying particular attention to the difference between the two image pickup devices.

  The matters described in the first embodiment are applied to the second embodiment as long as there is no contradiction. However, in this application, the difference in reference numerals between parts referred to by the same name is ignored (for example, the main control unit 13 and the main control unit 13a are different in sign (difference between 13 and 13a is appropriately ignored).

  FIG. 20 shows an external perspective view of the imaging apparatus 1a. An alternate long and short dash line 300 in FIG. 20 represents the optical axis of the imaging unit 11. Now assume that the optical axis is parallel to the horizontal plane.

  The sensor unit 18 includes two angular velocity sensors 18Y and 18P, and detects so-called camera shake that acts on the imaging device 1a. The angular velocity sensor 18Y periodically measures the angular velocity in the yaw direction (horizontal direction) of the imaging device 1a at a constant sampling interval, and outputs a signal representing the angular velocity in the yaw direction of the imaging device 1a in each sampling period. Similarly, the angular velocity sensor 18P periodically measures the angular velocity in the pitch direction (vertical direction) of the imaging device 1a at the sampling interval, and outputs a signal representing the angular velocity in the pitch direction of the imaging device 1a in each sampling period. .

  The yaw direction and the pitch direction are orthogonal to each other and also orthogonal to the optical axis. If the image pickup device 1a is shaken in the yaw direction, the optical axis of the image pickup device 1a rotates on a plane parallel to the horizontal plane. If the image pickup device 1a is shaken in the pitch direction, the optical axis of the image pickup device 1a is the vertical plane. Rotate on a plane parallel to. An image formed on the image pickup surface of the image pickup device 33 due to camera shake in the yaw direction moves in the left-right direction (horizontal direction of the image pickup surface) on the image pickup surface, and is formed on the image pickup surface of the image pickup device 33 by camera shake in the pitch direction. The image to be moved moves in the vertical direction (perpendicular to the imaging surface) on the imaging surface. Angular velocity data generated from the output signal of each angular velocity sensor is transmitted to the main controller 13a. The angular velocity data is information representing the movement of the imaging apparatus 1a (that is, the movement of the imaging apparatus 1a in the yaw direction and the pitch direction).

  The photographed image can include a camera shake caused by camera shake. The main control unit 13a includes a blur correction unit (blur correction device) that performs a blur correction process, and has a function of correcting blur of a still image by image processing. As in the first embodiment, a captured image as a still image whose blur is to be corrected by the blur correction unit is particularly referred to as a “correction target image”, and an image after the blur correction is referred to as a “blur correction image”.

  FIG. 21 shows a partial block diagram of the imaging apparatus 1a. The main control unit 13a determines, based on the angular velocity data, whether the blur of the correction target image resulting from the movement of the imaging device 1a during the exposure period of the correction target image belongs to a predetermined type of blur. The data to be recorded on the recording medium 16 is generated based on the determination result of the determination unit 61, the imaging control unit 62 that performs imaging control including control over the exposure time of the image sensor 33, and the data to be recorded on the recording medium 16. Are recorded on the recording medium 16, and a correction processing unit 64 that receives the recording data of the recording medium 16 and executes shake correction processing on the correction target image. The type determination unit 61 is inherent in the recording control unit 63. The angular velocity data from the sensor unit 18 is given to the recording control unit 63 and the type determination unit 61 included therein.

  FIG. 22 is a flowchart showing the flow of the shooting operation of the imaging apparatus 1a according to the second embodiment. With reference to this flowchart, the photographing operation of the imaging apparatus 1a will be described. Each process of steps S201 to S211 shown in FIG. 22 is executed in the photographing mode.

Acquired when the shutter release button 17a is pressed in shooting mode, in step S201, the main control unit 13a, the data representing the focal length f _D of the imaging unit 11 based on the position and the known information of the lens in the optical system 35 while, the exposure amount of the image sensor 33 to calculate the exposure time T _EP to be appropriate based on the brightness information obtained from the photometric circuit for measuring the brightness of an object (not shown).

Thereafter, in step S202, the imaging apparatus 1a acquires the normal exposure image _GL by executing normal exposure shooting. _TEP is used as the exposure time for normal exposure photography. The exposure control unit 62 generates an exposure time control signal for normal exposure shooting based on the exposure time _TEP and supplies it to the image sensor 33, whereby normal exposure shooting is executed. Further, the focal length of the imaging unit 11 at the time of executing the normal exposure shooting is f _D. The image data of the normal exposure image _GL is given to the recording control unit 63.

On the other hand, in step S203, the recording control unit 63 acquires angular velocity data during the exposure period of the normal exposure image _GL . Now, as shown in FIG. 23, the start time and end time of the exposure period of the normal exposure image _GL are t ₀ and t _n , respectively, and every time the sampling interval of the angular velocity sensors 18P and 18Y elapses starting from the time t _0. time _{t 1, t 2, ···,} t n-1 and t _n is assumed to come sequentially (n is 2 or more an integer example 33) in. Each of a period between time t ₀ and time t ₁ , a period between time t ₁ and time t ₂ ,..., A period between time t _n−1 and time t _n is a sampling period. It is. The length of the sampling period is, for example, 1/1000 second.

The sensor unit 18 outputs one angular velocity data for each sampling period. The angular velocity data for one particular sampling period represents the angular velocities in the yaw direction and the pitch direction of the imaging apparatus 1a during the noticed sampling period. All angular velocity data (that is, n angular velocity data) during the exposure period of the normal exposure image _GL is referred to as an angular velocity data group D _ALL .

The angular velocity data group D _ALL is given to the type determination unit 61. Type determination unit 61, based on such focal length f _D and normal number of pixels in the horizontal and vertical directions of the exposure image G _L, respectively of the angular velocity data forming the angular velocity data groups D _ALL, on normal exposure image G _L Convert to motion vector. Since this conversion method is well known, a detailed description thereof will be omitted (for example, see JP-A-2006-129236). The motion vectors obtained by converting the angular velocity data during the sampling phase between times t _i-1 -t _i expressed by m _i (where, i is an integer from 1 to n). The type determining unit 61 calculates _n motion vectors m _{1 to} m _n for the normal exposure image _GL (see FIG. 23).

For example, an image of a real still point (a point stationary in real space) is formed at the center of the imaging surface of the imaging unit 11 at time t _i−1 , and between time t _{i−1 and} t _i . When the imaging device 1a rotates in the yaw direction or the pitch direction, the image of the actual still point moves from the center on the imaging surface between times t _{i-1 and} t _i . The direction and magnitude of this movement, a normal exposure image G _L on a representation of the vector in the motion vector m _i.

Moreover, the combined vector obtained by combining the motion vector m ₁ ~m _i that integral vector S _i. The type determination unit 61 obtains an integration vector S _i (that is, integration vectors S _{1 to} S _n ) when the variable i is an integer of _{1 to n} . S ₁ is the motion vector m ₁ itself, but for convenience, S ₁ is also called an integral vector. As shown in FIG. 24, for example, integral vector S ₂ is generated in accordance sequentially connecting the motion vector m ₁ and m _2, a composite vector of the motion vectors m ₁ and m _2, integral vector S _3, the motion generated combined sequentially connecting the vector m ₁ ~m _3, a synthetic vector of the motion vector m ₁ ~m _3.

For convenience, a two-dimensional coordinate plane XY with the x axis and the y axis orthogonal to each other as shown in FIG. 25 is defined, and the normal exposure image _GL is projected on the coordinate plane XY. To do. The x-axis and y-axis directions are assumed to match the horizontal and vertical directions of the normal exposure image _GL , respectively. All motion vectors m ₁ ~m _n and integral vector S ₁ ~S _n is a vector on the coordinate plane XY, all motion vectors m ₁ of the start point and the integral vector S ₁ ~S _n start point on the coordinate plane XY of It shall be placed at the origin.

The type determination unit 61, after the motion vector m ₁ ~m _n and integral vector S ₁ to S _n is determined from the angular velocity data group D _ALL, the program proceeds from step S203 in FIG. 22 to step S204.

In step S204, the type determining unit 61 determines whether or not the shape of the blurring trajectory can be mathematically expressed. The blur locus means a blur locus of the normal exposure image _GL caused by the movement of the imaging device 1a during the exposure period of the normal exposure image _GL . Locus obtained when the end point went sequentially connecting the integral vector S ₁ to S _n are blurring corresponds to the trajectory (see Figure 24 also).

  The determination as to whether the expression can be made is performed by determining whether the shape of the blur locus can be approximated to the shape of a point, a straight line, or a quadratic curve. If it is determined that the expression can be expressed, the process proceeds from step S204 to step S205, and the shape determination unit 61 generates the shape expression data DF (hereinafter sometimes abbreviated as expression data DF). FIG. 26 shows the processing contents of steps S204 and S205 including the determination method.

More specifically, in step S204, compares the magnitude of the integral vector S _n with a predetermined threshold value TH _A, if the former is smaller than the latter, it determines that it approximates the shape of the trajectory of blur in the shape of a point, If the former is greater than or equal to the latter, it is determined that the approximation cannot be performed.

When it is determined that the shape of the blur locus can be approximated to the shape of a point, the approximate equation of the blur locus is regarded as “x ² + y ² <r ² ”, and data including the value of r and the point flag in the approximate equation is used. Generated as mathematical formula data DF. X and y in the approximate expression of the blur locus represent the x-axis component and the y-axis component of the blur locus on the coordinate plane XY. The value of r is set in advance. Alternatively, the magnitude of the integral vector S _n may be substituted for the value of r.

Further, in step S204, we obtain an approximate straight line for the point group consisting of the end point of the integral vector S ₁ to S _n (total of n end points) by the least square method. FIG. 27 shows the points 311, 312, 313 and 314 as the end points of the integral vectors S _n−3 , S _n−2 , S _n−1 and S _n and the obtained approximate straight line 320. The distance between the end point (for example, the point 311) of an integration vector and the approximate line 320 represents the error of the approximate line with respect to the end point. Type determination unit 61, the error of the approximation straight line calculated for each of the integral vector S ₁ to S _n, obtains the value of the sum of the error divided by n for the integral vector S ₁ to S _n as the evaluation value E _V1 . Then, the evaluation value E _V1 is compared with a predetermined threshold value TH _B , and when the former is smaller than the latter, it is determined that the shape of the blurring locus can be approximated to the shape of a straight line. Determine that approximation is not possible.

  If it is determined that the shape of the blur locus can be approximated to a straight line shape, the approximate equation of the blur locus is regarded as “y = ax + b”, and data including the values of a and b and the straight line flag in the approximate equation is expressed as mathematical data. Generate as DF. a and b are the slope and y-intercept of the approximate line 320, respectively.

Further, in step S204, it obtains the approximate quadratic curve for the integral vector S ₁ to S _n point group consisting of the end point (the sum of n end points) of the least squares method. The distance between the end point of an integral vector and the approximate quadratic curve represents the error of the approximate quadratic curve with respect to the end point. Type determination unit 61, the error of the approximate quadratic curve determined for each of the integral vector S ₁ to S _n, the value obtained by dividing the error summation for the integral vector S ₁ to S _n in n evaluation value E _V2 Asking. Then, by comparing the evaluation value E _V2 with a predetermined threshold value TH _C, if the former is smaller than the latter, it determines that the shape of the trajectory of blur can be approximated to the shape of the quadratic curve, if the former is the latter more It is determined that the approximation cannot be performed.

When it is determined that the shape of the blur locus can be approximated to the shape of a quadratic curve, the approximate equation of the blur locus is regarded as “y = ax ² + bx + c”, and the values of a, b, and c in the approximate equation and the quadratic curve Data including a flag is generated as mathematical formula data DF. The values of a, b, and c are obtained from an approximate quadratic curve.

  If it is determined in step S204 that the shape of the blur locus can be approximated to the shape of a point, straight line, or quadratic curve, the mathematical formula data DF is generated in step S205, and then the process proceeds to step S206. When it is determined that the shape cannot be approximated to any of a point, a straight line, and a quadratic curve, the process proceeds from step S204 to step S210.

  When the process proceeds to step S206, in principle, the determination process of step S206 is executed. However, when it is determined that the shape of the blurring locus can be approximated to the shape of a point, the processes of steps S206 and S207 are exceptionally performed. Without performing, the process proceeds to step S208.

  In step S206, the type determination unit 61 determines whether or not the change state of the shake speed can be expressed as a formula. The blur speed means the speed at which the blur trajectory is drawn. The determination as to whether or not the expression can be expressed is performed by determining whether or not the change state of the shake speed can be approximated to a constant speed or a constant acceleration. Here, equal acceleration is a concept including negative acceleration (that is, equal deceleration). If it is determined that the expression can be expressed, the process proceeds from step S206 to step S207, and the type determination unit 61 generates speed change expression data DS (hereinafter sometimes abbreviated as expression data DS). FIG. 28 shows the processing contents of steps S206 and S207 including the determination method.

Specifically, in step S206, variance of the norm of the motion vector m ₁ ~m _n (i.e., the variance of the magnitude of the motion vector m ₁ ~m _n) is compared with a predetermined threshold value TH _D, the former than the latter Is smaller, it is determined that there is no change in the shake speed in the process of drawing the blur locus, that is, it is determined that the change state of the shake speed can be approximated at a constant speed.

When it is determined that the change state of the shake speed can be approximated at a constant speed, the approximate function representing the shake speed V (t) is regarded as “V (t) = V _CONST ” and is equal to the value of V _{CONST in} the approximate function. Data including a speed flag is generated as mathematical formula data DS. The value of V _CONST is an average value of the magnitudes of the motion vectors m _{1 to} m _n . Further, t in this approximate function is a variable that increases by 1 every time one sampling period elapses.

Further, as shown in FIG. 29, the variable i and assume the two-dimensional coordinate plane taking the size respectively the horizontal and vertical axes of motion vector m _i, vector m ₁ ~m _n motion to the two-dimensional coordinate plane Plot the size of. Then, an approximate straight line for a point group consisting of a total of n points plotted is obtained by the method of least squares. In FIG. 29, points 331 to 334 are plotted with respect to the magnitudes of the motion vectors m _{1 to} m ₄ , and a broken line 340 indicates the obtained approximate line.

A distance between a plotted point (for example, the point 331) and the approximate line 340 represents an error of the approximate line with respect to the point. Type determination unit 61, the error of the approximation straight line, asking for each point plotted versus the motion vector m ₁ ~m _n, evaluation value E a value the sum of the error divided by n for each point _V3 Asking. Then, the evaluation value E _V3 is compared with a predetermined threshold value TH _E , and when the former is smaller than the latter, the blurring speed changes at a constant rate in the process of drawing the blurring trajectory. It is determined that the acceleration can be approximated, and when the former is greater than or equal to the latter, it is determined that the approximation cannot be performed.

When it is determined that the change state of the shake speed can be approximated to equal acceleration, the approximate function representing the shake speed V (t) is regarded as “V (t) = V ₀ + dt”, and V ₀ and d of the approximate function are Data including a value and an equal acceleration flag is generated as mathematical formula data DS. In this approximate function, V ₀ is the magnitude of the motion vector m ₁ , d is the slope of the approximate line 340, and t is a variable that increases by 1 every time one sampling period elapses.

  If it is determined in step S206 that the change state of the shake speed can be approximated to constant speed or constant acceleration, the mathematical formula data DS is generated in step S207, and then the process proceeds to step S208, where the change state of the shake speed is changed to constant speed. If it is determined that the acceleration cannot be approximated to any of the constant accelerations, the process proceeds from step S206 to step S210.

When the process proceeds to step S208, the recording control unit 63 performs the processes of steps S208 and S209, and then the photographing operation ends. In steps S208 and S209, the normal exposure image _GL is JPEG-compressed to generate an image J _L and an image file is created in the recording medium 16, and the image data of the normal exposure image J _L is stored in the main body area of the image file. On the other hand, the mathematical formula data DF and DS are recorded as camera shake data (correction information) in the header area of the image file. At this time, the recording control unit 63, data representing the focal length f _D and the exposure time T _EP also recorded in the header area. If it is determined that the shape of the blur locus can be approximated to the shape of a point, the process of step S207 is not executed as described above, and the mathematical formula data DS is not recorded.

When the process proceeds to step S210, the recording control unit 63 performs the processes of steps S210 and S211 and thereafter ends the photographing operation. In steps S210 and 211, the normal exposure image _GL is JPEG compressed to generate an image J _L and an image file is created in the recording medium 16, and the image data of the normal exposure image J _L is stored in the main body area of the image file. On the other hand, the angular velocity data group D _ALL is recorded as camera shake data (correction information) in the header area of the image file. At this time, the recording control unit 63, data representing the focal length f _D and the exposure time T _EP also recorded in the header area.

  Next, the correction operation of the imaging apparatus 1a will be described with reference to FIG. FIG. 30 is a flowchart showing the flow of this correction operation. After the above-described shooting operation, when a correction instruction is given to the imaging apparatus 1a (or when a predetermined condition is satisfied), the correction operation is executed. The correction operation includes steps S221 to S224. The correction instruction is given to the imaging device 1a by a predetermined operation on the operation unit 17 by the user, for example. However, the correction operation can be executed regardless of whether or not there is a correction instruction. The correction operation is executed in the reproduction mode. However, the correction operation can also be performed in the shooting mode.

When a correction instruction is given to the imaging apparatus 1a, in step S221, the correction processing unit 64 in FIG. 21 reads the image data of the normal exposure image J _L from the image file in the recording medium 16, and the normal exposure image J _L is recorded. The camera shake data is read from the header area of the image file. The correction processing unit 64 handles the normal exposure image _JL as a correction target image.

Shake data read here is if I in the above header area is the angular velocity data groups D _ALL is recorded is the angular velocity data group D _ALL, if my above header area formulas data DF and DS is recorded The mathematical formula data DF and DS.

In subsequent step S222, the correction processing unit 64 obtains a PSF which normally represents the state of blurring of the exposed image J _L on the basis of the read-out camera shake data.

If the read hand movement data is angular velocity data group D _ALL, known methods (e.g., methods described in JP 2006-279807 and JP 2006-129236 JP) by, normal exposure from the angular velocity data group D _ALL The PSF of the image J _L can be obtained. That is, determine the motion vector m ₁ ~m _n from the angular velocity data group D _ALL, obtains a spatial filter weighting filter coefficients is made in accordance with the direction and magnitude of the motion vector m ₁ ~m _n as PSF.

When the read camera shake data is the mathematical formula data DF and DS, the correction processing unit 64 calculates a PSF similar to the PSF obtained based on the angular velocity data group D _ALL using the mathematical formula data DF and DS.

A method for calculating the PSF from the mathematical formula data DF and DS will be described with reference to FIG. The upper part of FIG. 31 shows an example of the angular velocity data group D _ALL generated during the photographing operation. Now, mathematical formula data DF and DS are generated from the angular velocity data group D _ALL and recorded on the recording medium 16, and the shape mathematical formula data DF includes the straight line flag and the values of a and b, and the velocity change mathematical formula data DS is equal to Consider the case where the speed flag and the value of V _CONST are included. In this case, the correction processing unit 64 considers that the blur locus is represented by the near-order expression “y = ax + b” and the blur speed V (t) is represented by the approximate function “V (t) = V _CONST ”. An angular velocity data group generated for the photographing operation is estimated from the data DF and DS. The angular velocity data group estimated here is referred to as an estimated angular velocity data group D _{ALL_E} . In the lower part of FIG. 31, an estimated angular velocity data group D _{ALL_E} is shown.

For example, when a = _4/3 , b = 0 and V _CONST = 5, the magnitude of the motion vector in the entire sampling period is 5, and the horizontal component (x-axis component) of the motion vector in the entire sampling period Estimated angular velocity data group D _{ALL_E} is obtained so that the ratio of the vertical component (y-axis component) is 3: 4.

Thereafter, motion vectors m _{1 to mn} determined by the angular velocity data group D _ALL are estimated based on the estimated angular velocity data group D _{ALL — E} , and a PSF is obtained from the estimated motion vector. The PSF calculation method based on the estimated angular velocity data group D _{ALL_E} is the same as the PSF calculation method based on the angular velocity data group D _ALL . Since it is possible to directly estimate the motion vectors m _{1 to} m _n from the mathematical formula data DF and DS without _{obtaining the} estimated angular velocity data group D _{ALL_E} , in _actuality , when _{obtaining the} PSF from the mathematical formula data DF and DS, the estimated angular velocity data It is not necessary to find the group D _{ALL_E} .

  When the shape formula data DF includes a point flag and a value of r, a Gaussian filter having the number of taps corresponding to the value of r is obtained as PSF. For example, when the value of r is three pixels of the normal exposure image, a Gaussian filter having a 3 × 3 filter size is obtained as the PSF. The dispersion in the Gaussian filter may be determined in advance.

Thereafter, in step S223, the correction processing unit 64 obtains an image restoration function corresponding to the inverse function (general inverse matrix) of the PSF obtained in step S222. The image restoration function is expressed as a single image restoration filter, and each element of the general inverse matrix of the matrix representing the PSF is used as each filter coefficient of the image restoration filter. The correction processing unit 64 performs the spatial filtering using the image restoration filter on the entire correction target image (that is, the normal exposure image J _L ), thereby reducing the blur included in the correction target image. Generate an image. You may make it obtain a final blurring correction image by performing a ringing removal process further with respect to the image obtained by performing the spatial filtering which used the image restoration filter for the correction object image.

In subsequent step S224, the main control unit 13a newly generates an image file for recording the image data of the shake correction image generated in step S223, and stores the image file in the recording medium 16. Alternatively, without newly generating an image file, usually as image data of the exposure image J _L is overwritten, corrected image blur in the body region of the normal image file which has been recorded image data of the exposure image J _L Image data may be recorded. The blur correction image is displayed on the display unit 15.

  When camera shake can be approximated to a simple motion (for example, constant velocity linear motion), a shake correction image obtained by image restoration processing using accurate angular velocity data, and a blur correction image obtained by image restoration processing using mathematical approximation (In other words, the determination conditions in steps S204 and S206 in FIG. 22 are determined so that the difference hardly occurs). Considering this, in the present embodiment, whether the blur of the correction target image belongs to a predefined type of blur (linear and constant velocity blur, quadratic curve and equal acceleration blur, or the like). Is determined by the determination processing in steps S204 and S206, and if it belongs to a typical blur, a mathematical expression representing a typical blur trajectory shape and a speed state for drawing the trajectory instead of actual angular velocity data Data is recorded as correction information. As a result, the data size of the correction information can be made smaller than when actual angular velocity data is recorded.

In the above-described example, the switching determination as to which of the angular velocity data group D _ALL and the mathematical formula data is recorded is made based on the motion vectors m _{1 to mn} obtained by combining the angular velocities in the yaw direction and the pitch direction. This determination may be made individually for the angular velocity in the yaw direction and the angular velocity in the pitch direction.

For example, as shown in FIG. 32, the angular velocity in the yaw direction during the entire sampling period during the exposure period of the normal exposure image _GL is constant or substantially constant, and the angular velocity in the pitch direction during the exposure period of the normal exposure image _GL. When increasing at a constant or substantially constant rate, the type determination unit 61 uses an approximate function representing the angular velocity V _X (t) in the yaw direction as “V _X (t) = V _XCONST ” and the angular velocity V _{Y in the} pitch direction. The approximate function representing (t) is regarded as “V _Y (t) = V _Y0 + d _Y t”, and the values of V _XCONST , V _Y0 and d _Y in these approximate functions and the angular velocity in the yaw direction are constant. Data including a yaw direction constant speed flag indicating that this is the case and a pitch direction constant acceleration flag indicating that the change state of the angular velocity in the pitch direction is equal acceleration is generated as mathematical formula data. The mathematical formula data is recorded in the header area of the image file where the image data of the normal exposure image is to be recorded, and then read out from the image file by the correction processing unit 64 during the correction operation. The angular velocity data group D _ALL during the exposure period of the exposure image _GL is estimated.

The value of V _XCONST is the average value of the angular velocities in the yaw direction during the exposure period of the normal exposure image _GL , and the value of V _{Y0 is} the value of the first sampling period during the exposure period of the normal exposure image _GL . It is an angular velocity in the pitch direction, and the value of d _Y is an average increase rate of the angular velocity in the pitch direction during the exposure period of the normal exposure image _GL . Further, t in the approximate function is a variable that increases by 1 every time one sampling period elapses.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the second embodiment, the example in which the sensor unit 18 is configured by the angular velocity sensor that detects the angular velocity has been described above (see FIG. 19), but the sensor that detects the physical quantity other than the angular velocity that represents the movement of the imaging device 1a. The sensor unit 18 may be configured. For example, the sensor unit 18 may be formed by an acceleration sensor that detects the acceleration of the imaging device 1a or an angular acceleration sensor that detects the angular acceleration of the imaging device 1a. It is also possible to obtain the PSF of the correction target image based on the output signal of the acceleration sensor or the angular acceleration sensor that represents the acceleration or the angular acceleration of the imaging device 1a.

[Note 2]
The imaging device (1 or 1a) can be realized by hardware or a combination of hardware and software. In particular, all or part of the processing executed in the main control unit (13 or 13a) can be realized by hardware, software, or a combination of hardware and software. When the imaging apparatus (1 or 1a) is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

  Further, all or part of the arithmetic processing executed in the main control unit (13 or 13a) is described as a program, and the program is executed on a program execution device (for example, a computer), so that the arithmetic processing is executed. You may make it implement | achieve all or one part.

[Note 3]
In each of the above-described embodiments, the correction processing unit (54 or 64) that executes the shake correction process is provided in the imaging apparatus (1 or 1a), and the shake correction process is executed in the imaging apparatus (1 or 1a). However (see FIG. 4 or FIG. 21), the blur correction process may be executed by an external device (not shown) different from the imaging device (1 or 1a). In this case, a correction processing unit equivalent to the correction processing unit 54 or 64 described above is provided in the external device, and the blur correction image is extracted from the correction target image by supplying the recording data of the recording medium 16 to the external device. It only has to be generated.

[Note 4]
For example, it can be considered as follows. In the first embodiment, the shake detection unit (blur detection device) included in the main control unit 13 includes the shake amount estimation unit (device motion detection means) 51 and the recording control unit 53 of FIG. Can be included. A portion obtained by adding the correction processing unit 54 to the shake detection unit functions as a shake correction unit (blur correction device). In the second embodiment, the shake detection unit (blur detection device) included in the main control unit 13a includes the recording control unit 63 of FIG. 21 and may further include an imaging control unit 62. A portion obtained by adding the correction processing unit 64 to the shake detection unit functions as a shake correction unit (blur correction device).

1 is an overall block diagram of an imaging apparatus according to a first embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure which shows the time relationship between a preview period and an actual imaging | photography period. It is a partial block diagram of the imaging device of FIG. 3 is a flowchart of a shooting operation and a correction operation of the imaging apparatus according to the first embodiment of the present invention. It is a figure which shows the structure of the image file which should be preserve | saved in the recording medium of FIG. It is a flowchart of imaging | photography operation | movement of an imaging device based on 2nd Example of this invention. It is a figure which concerns on 2nd Example of this invention and shows the relationship between the estimated amount of camera shake and a predetermined threshold value. It is a flowchart of the correction | amendment operation | movement of an imaging device based on 2nd Example of this invention. It is a flowchart of imaging | photography operation | movement of an imaging device based on 4th Example of this invention. It is a flowchart of the estimation operation | movement of the amount of camera shake based on 4th Example of this invention. FIG. 16 is a diagram illustrating a locus of an ideal point image moving on an imaging surface of an imaging element during an exposure period of a normal exposure image according to the fifth example of the present invention. FIG. 14 is a diagram illustrating a matrix-represented PSF corresponding to the locus in FIG. 12 according to the fifth embodiment of the present invention. It is a figure which shows the normal exposure image sequence arranged in a time series concerning 5th Example of this invention. FIG. 20 is a diagram illustrating a preview image sequence acquired during a preview period before a certain actual shooting period according to the sixth embodiment of the present invention. It is a flowchart of the blurring correction process which concerns on 7th Example of this invention. FIG. 16 is a detailed flowchart of a Fourier iteration method used in shake correction processing according to the seventh embodiment of the present invention. FIG. It is a block diagram of the site | part which implement | achieves the Fourier iteration method utilized in the 7th Example of this invention, and used in a blurring correction process. It is a whole block diagram of the imaging device concerning a 2nd embodiment of the present invention. It is an external appearance perspective view of the imaging device of FIG. It is a partial block diagram of the imaging device of FIG. It is a flowchart of imaging operation | movement of an imaging device based on 2nd Embodiment of this invention. It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship between several time in the exposure period of a normal exposure image, and the relationship between those time, angular velocity data, and a motion vector. It is a figure which shows the motion vector and integral vector calculated | required from angular velocity data in connection with 2nd Embodiment of this invention. It is a figure which shows the two-dimensional coordinate surface in which the motion vector and integration vector calculated | required from angular velocity data are arrange | positioned regarding 2nd Embodiment of this invention. It is a figure for demonstrating the determination process performed in the 2nd Embodiment of this invention at the time of imaging | photography operation | movement of an imaging device. It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship of the approximate curve with respect to a point group and the point group. It is a figure for demonstrating the determination process performed in the 2nd Embodiment of this invention at the time of imaging | photography operation | movement of an imaging device. It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship of the approximate curve with respect to a point group and the point group. It is a flowchart of correction | amendment operation | movement of an imaging device based on 2nd Embodiment of this invention. The figure which concerns on 2nd Embodiment of this invention, and shows the angular velocity data group obtained by the actual measurement, the numerical formula data derived | led-out from the angular velocity data group, and the estimated angular velocity data group calculated | required from the numerical formula data It is. It is a figure which concerns on 2nd Embodiment of this invention and shows the angular velocity data group obtained by actual measurement, and the numerical formula data derived | led-out from the angular velocity data group. It is a figure which shows a mode that a decompression | restoration image is produced | generated from a camera shake image while showing the locus | trajectory of camera shake, PSF, and an image restoration filter according to a prior art. It is a block diagram of the site | part which implement | achieves the conventional Fourier iteration method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1a Image pick-up device 11 Image pick-up part 13, 13a Main control part 14 Internal memory 16 Recording medium 17a Shutter button 18 Sensor part 33 Image pick-up element 35 Optical system 51 Camera shake amount estimation part 52, 62 Shooting control part 53, 63 Recording control part 54 64 Correction processing unit

Claims (12)

In a shake detection apparatus including a recording control unit that records correction information for correcting blur of a correction target image, which is a still image obtained by an imaging device by shooting, by image processing.
The recording control unit can generate a plurality of types of correction information, and selects the type of the correction information to be recorded in the recording unit according to the movement of the imaging apparatus during the exposure period of the correction target image. A shake detection device characterized by: A device motion estimation means for estimating the motion magnitude of the imaging device during the exposure period of the correction target image;
The shake detection apparatus according to claim 1, wherein the recording control unit selects a type of the correction information to be recorded in the recording unit based on the estimated size. The recording control means determines whether or not the estimated size belongs to either the first range to which a relatively large size should belong or the second range to which a relatively small size should belong. When the estimated size belongs to the first range and the second range, different types of correction information are generated, and when the estimated size belongs to the second range, the correction is performed. The blur detection apparatus according to claim 2, wherein information indicating a correction method of the target image is recorded on the recording medium as the correction information. When the estimated size belongs to the first range, the recording control unit is an image shot before or after shooting the correction target image and is longer than an exposure period of the correction target image. The blur detection apparatus according to claim 3, wherein part or all of image data of a short exposure image photographed in a short exposure period is recorded on the recording medium as the correction information. The shake detection apparatus according to claim 2, wherein the apparatus motion estimation unit estimates the size based on a shooting condition of the correction target image. The apparatus motion estimation means is an edge of a short-exposure image that is captured before or after the correction target image and is captured in an exposure period that is shorter than the exposure period of the correction target image. The blur detection apparatus according to claim 2, wherein the magnitude is estimated based on an intensity and an edge intensity of the correction target image. The apparatus motion estimation means includes
Treating each of a plurality of still images arranged in time series photographed by the imaging device as a correction target image, estimating the magnitude of movement of the imaging device during the exposure period of each correction target image,
The size to be estimated for the target correction target image,
Image data of the past correction target image of the target correction target image, and an image shot before or after shooting the past correction target image, and longer than an exposure period of the past correction target image The blur detection apparatus according to claim 5 or 6, wherein estimation is performed also using image data of a short exposure image taken in a short exposure period. The apparatus motion estimation means pays attention to an image sequence captured before capturing the correction target image, and a state in which the size of a motion vector between temporally adjacent images forming the image sequence is a predetermined value or less. Is continuously observed for a predetermined time, the imaging during the exposure period of the correction target image based on the motion vector, the frame rate at the time of shooting the image sequence, and the length of the exposure period of the correction target image The shake detection apparatus according to claim 2, wherein the magnitude of movement of the apparatus is estimated. The recording control unit determines whether the blur of the correction target image belongs to a predefined type of blur based on an output signal of a sensor that measures the movement of the imaging device, and based on the determination result, The blur detection apparatus according to claim 1, wherein a type of the correction information recorded in the recording unit is selected. The recording control means includes
When it is determined that the blur of the correction target image belongs to the type blur, the mathematical data representing the trajectory shape of the type blur and the speed state of drawing the trajectory are recorded as the correction information in the recording unit,
When it is determined that the blur of the correction target image does not belong to the type blur, data representing the output signal of the sensor during the exposure period of the correction target image is recorded in the recording unit as the correction information. The shake detection device according to claim 9. The shake detection device according to any one of claims 1 to 10,
A shake correction apparatus comprising: correction processing means for correcting shake of the correction target image based on the correction information recorded in the recording means. An imaging apparatus comprising the shake detection apparatus according to claim 1, wherein an image is acquired by photographing.