JP2009118434A - Blurring correction device and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve blurring correction, at low cost and with high precision, for continuously captured images. <P>SOLUTION: A burring correction device includes: an image capturing unit for capturing a plurality of source images (element images) by consecutive photographing or bracket photographing; a correction target extracting unit for extracting, as a correction target image, one of the plurality of source images; a reference image extracting unit which defines source images, other than the correction target image, as candidate reference images and extracts, as a reference image, a candidate reference image of which a blurring amount included is smaller than that of the correction target image, from among the plurality of candidate reference images; and a correction processing unit for correcting blurring of the correction target image on the basis of the correction target image and the reference image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a camera shake correction apparatus and an imaging apparatus that correct camera shake caused by camera shake or the like.

  The camera shake correction technique is a technique for reducing camera shake at the time of photographing, and is regarded as important as a differentiation technique in an imaging apparatus such as a digital still camera. As one type of camera shake correction, there is optical camera shake correction in which camera shake is detected by a camera shake detection sensor and a correction lens and an image sensor are driven based on the detection result.

  On the other hand, as special shooting for still images, continuous shooting that continuously acquires multiple still images without changing various settings related to shooting and various settings related to shooting (exposure, white balance, etc.) are changed. However, there is bracket shooting that acquires multiple still images continuously. It is also conceivable to apply optical image stabilization to such continuous shooting and bracket shooting. However, if the number of continuous shooting is large, the total amount of camera shake from the start to the end of continuous shooting becomes very large, and the required drive amount of the correction lens and image sensor may exceed their movable range. In the second half of continuous shooting, camera shake correction does not work at all. Further, the use of the camera shake detection sensor itself increases the cost.

  On the other hand, electronic image stabilization that does not use an image stabilization sensor has also been developed. In this electronic camera shake correction, an image obtained by photographing is analyzed to detect camera shake, and camera shake correction is performed based on the detection result.

  As conventional methods for electronic image stabilization, there are the following methods. In one conventional method, a plurality of sub-images are captured before and after the main image to be corrected, the blur information at the time of main image shooting is estimated from the sub-images, and the blur of the main image is estimated from the blur information. It correct | amends (for example, refer patent document 1). However, in this method, since the blur of the main image is estimated from the amount of motion between the sub-images obtained before and after the main image (the amount of motion including the exposure interval), the detection accuracy of the blur is lowered.

  In another conventional method, camera shake is detected from an image obtained by converting a camera shake image into a two-dimensional frequency space (see, for example, Patent Document 2). Specifically, the image obtained by the conversion is projected onto a circle centered on the origin of the frequency coordinate, and the magnitude and direction of the shake are obtained from the projection data. However, with this method, only linear and constant velocity blur can be estimated, and when the frequency component in a specific direction of the subject is small, detection of the blur direction may fail.

  A method for estimating camera shake information (a point spread function or an image restoration filter) representing a camera shake during shooting from a single camera shake image, and generating a restored image free from the camera shake information and the camera shake image by digital signal processing. Has also been proposed. Among these types of methods, a method using the Fourier iteration method is disclosed in Non-Patent Document 1 below. FIG. 30 shows a block diagram of a configuration for realizing the Fourier iteration method. In the Fourier iteration method, the final restored image is estimated from the degraded image by repeatedly executing Fourier transform and inverse Fourier transform while correcting the restored image and the point spread function (PSF). In order to execute the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image). Generally, the initial restored image is a random image or a degraded image as a camera shake image. Used.

JP 2001-346093 A JP-A-11-27574 G. R. Ayers and J. C. Dainty, "Iterative blind deconvolution method and its applications", OPTICS LETTERS, 1988, Vol. 13, No. 7, p.547-549

  Electronic camera shake correction that does not require a camera shake detection sensor can reduce costs compared to optical camera shake correction, but the actual situation is that no suitable camera for continuous shooting or bracket shooting has been proposed. . In addition, the problem of the prior art has been described with particular attention to still image continuous shooting or bracket shooting, but the same problem occurs when shooting a moving image that is a bundle of continuously shot still images. There is an urgent need for a suitable image stabilization technology.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a shake correction apparatus and an imaging apparatus that can realize shake correction with respect to continuously shot images at low cost and high accuracy.

  A first shake correction apparatus according to the present invention includes an image acquisition unit that acquires a plurality of original images by continuous shooting or bracket shooting using an imaging unit, and a single original image from the plurality of original images. Correction target image extraction means for extracting as a correction target image, and each of the plurality of original images excluding the correction target image is a candidate reference image, and the amount of blur included in the plurality of candidate reference images is Reference image extraction means for extracting a candidate reference image smaller than that of the correction target image as a reference image; and correction processing means for correcting blur of the correction target image based on the correction target image and the reference image. It is characterized by that.

  Thereby, blur correction is performed on an image obtained by continuous shooting or bracket shooting. Since a camera shake detection sensor is not required, a camera shake correction device can be formed at a low cost. Further, not only the reference image but also the image to be corrected is corrected based on the image to be corrected. I can expect. Furthermore, since the plurality of original images is an image group that should be originally acquired by continuous shooting or bracket shooting, it is not necessary to acquire a separate dedicated image for blur correction. Therefore, there is no extra load on the shooting time and memory usage.

  Specifically, for example, the correction target image extracting means includes the face orientation of the person in each original image, the eye open state of the person in each original image, the facial expression of the person in each original image, and the composition of each original image. The correction target image is extracted based on at least one of the above.

  If the correction target image is automatically extracted from the face orientation or the like, it is possible to automatically correct an image that is highly likely to be corrected by the user.

  A second shake correction apparatus according to the present invention includes an image acquisition unit that acquires a moving image including a plurality of frame images by continuous shooting using an imaging unit, and a frame in a frame of interest among the plurality of frame images. A correction processing unit that corrects a shake included in the correction target image and an image as a correction target image, and each frame image in a frame near the frame of interest is a candidate reference image, and is included among a plurality of candidate reference images Reference image extraction means for extracting, as a reference image, a candidate reference image whose blur amount is smaller than that of the correction target image, the correction processing means based on the correction target image and the reference image. The blur of the image is corrected.

  Thereby, it is possible to provide a moving image with clear image quality to the user. Since a camera shake detection sensor is not required, a camera shake correction device can be formed at a low cost. Further, not only the reference image but also the image to be corrected is corrected based on the image to be corrected. I can expect. Furthermore, since the correction target image and the frame image serving as the reference image are images that should be acquired originally, it is not necessary to acquire a separate dedicated image in order to perform blur correction. Therefore, there is no extra load on the shooting time and memory usage.

  Specifically, for example, in the first or second shake correction apparatus, the reference image extraction unit includes an edge intensity between the correction target image and each candidate reference image, and a contrast between the correction target image and each candidate reference image. The reference image is extracted by estimating the amount of blur between the correction target image and each candidate reference image based on at least one of the above.

  As a result, it is possible to estimate the amount of shake with less processing time and power consumption.

  Further, for example, in the first or second shake correction apparatus, the reference image extraction unit includes a rotation angle of each candidate reference image with respect to the correction target image, and a photographing time difference between each candidate reference image and the correction target image. The reference image is extracted by further using at least one of.

  This contributes to the improvement of the blur correction effect.

  More specifically, for example, in the first or second shake correction apparatus, the correction processing unit creates shake information corresponding to the shake of the correction target image based on the correction target image and the reference image. The blur of the correction target image is corrected based on the blur information.

  More specifically, for example, in the first or second shake correction apparatus, the correction processing unit synthesizes a luminance signal of the reference image with a color signal of the correction target image, thereby the correction target image. Correct blur.

  More specifically, for example, in the first or second shake correction apparatus, the correction processing unit corrects the shake of the correction target image by sharpening the correction target image using the reference image. To do.

  In addition, for example, in the first shake correction apparatus, after the continuous shooting or the bracket shooting is finished, the correction image obtained by automatically correcting the shake of the correction target image is output to the display unit.

  Thus, after continuous shooting or bracket shooting, the user can quickly confirm the corrected image without taking time and effort.

  Further, for example, in the second shake correction apparatus, the correction processing unit corrects the shake included in each frame image by sequentially treating each frame image forming the moving image as the correction target image, The blur correction apparatus cuts out a part of each frame image after the blur correction, outputs a clipped image to a display unit or a recording medium, and a positional deviation between adjacent frames with respect to the adjacent frame. Misalignment detection means for detecting based on the frame image before or after blur correction, and the clipped image output means determines the clipped position of the clipped image based on the detected shift. Set.

  Thereby, the correction effect with respect to camera shake between frames is also obtained.

  An imaging apparatus according to the present invention includes an imaging unit and the shake correction apparatus according to any one of the above.

  Further, another imaging apparatus according to the present invention is an imaging apparatus including an imaging unit and the first blur correction device, and corrects the blur of the plurality of original images and the correction target image. An association means for associating the corrected images obtained in this way with each other and an operation accepting means for accepting an operation on the associated image, and collectively applying the processing based on the operation to all of the associated images. It is characterized by.

  Thereby, the convenience for the user is enhanced.

  According to the present invention, it is possible to provide a camera shake correction apparatus and an imaging apparatus that can realize camera shake correction for continuously photographed images at low cost and with high accuracy.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to sixth embodiments will be described later. First, matters common to each embodiment or items referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. An imaging apparatus 1 in FIG. 1 is a digital video camera that can capture and record still images (still images) and moving images (moving images). However, the imaging apparatus 1 can also be a digital still camera that can capture and record only still images.

  The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, an operation unit 17, and a camera shake correction unit 19. It is equipped with. The operation unit 17 includes a shutter button 17a and a recording button 17b.

  The imaging unit 11 includes an optical system, an aperture, an imaging device such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a driver (all not shown) for controlling the optical system and the aperture. And have. The driver controls the zoom magnification and focal length of the optical system and the aperture of the diaphragm based on the AF / AE control signal from the main control unit 13. The image sensor photoelectrically converts an optical image representing a subject incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (imaging device), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and also functions as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image” or “frame image”). The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device composed of a liquid crystal display panel or the like, and displays an image taken in the immediately previous frame or an image recorded on the recording medium 16 under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and stores captured images and the like under the control of the main control unit 13.

  The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing to capture and record a still image. The recording button 17b is a button for instructing the start or end of moving image shooting and recording.

  The operation mode of the imaging device 1 includes a shooting mode capable of shooting and recording a still image or a moving image, and a reproduction mode for reproducing and displaying the still image or moving image recorded on the recording medium 16 on the display unit 15. . Transition between the modes is performed according to the operation on the operation unit 17.

  In the shooting mode, the imaging unit 11 sequentially performs shooting at a predetermined frame period (for example, 1/60 seconds). The main control unit 13 generates a through display image from the output of the imaging unit 11 in each frame, and sequentially displays the through display images obtained sequentially on the display unit 15.

  As still image shooting, single shooting for acquiring and recording one still image according to an instruction by pressing the shutter button 17a, and continuous shooting for continuously acquiring and recording a plurality of still images according to an instruction by pressing the shutter button 17a There is shooting. When an instruction for single shooting is given, one shot image is acquired, and the image data of the one shot image is recorded in the recording medium 16 as image data of one still image. When an instruction for continuous shooting is given, continuous shooting is performed, and the image data of a plurality of shot images obtained by the continuous shooting is recorded as image data of a plurality of still images shot continuously. Recorded on the medium 16. Hereinafter, a plurality of captured images obtained by continuous shooting are collectively referred to as a “continuous shooting image group”. Each forming element of the continuous shot image group, that is, each captured image forming the continuous shot image group is also referred to as an “element image”.

  When the recording button 17b is pressed in the shooting mode, the image data of the frame image obtained in each frame after the pressing is sequentially recorded on the recording medium 16, and when the recording button 17b is pressed again, The recording ends. Regardless of whether the subject to be photographed is a still image or a moving image, the image data recorded on the recording medium 16 is also recorded in the internal memory 14 for camera shake detection and camera shake correction, which will be described later, as necessary. Can be done.

  An image obtained by photographing by the imaging unit 11 is an image that may include a shake due to camera shake, and is corrected by the camera shake correction unit 19 in accordance with a correction instruction given via the operation unit 17 or the like. . An image whose blurring is to be corrected is particularly referred to as a “correction target image”. An image obtained by correcting blur included in the correction target image is referred to as a “corrected image”. The camera shake correction unit 19 refers to not only the correction target image but also other images, and corrects the shake included in the correction target image based on the correction target image and other images. This other image to be referred to is referred to as a “reference image”. Blur included in a certain image corresponds to deterioration of the image (image deterioration) caused by camera shake or the like.

  The camera shake correction unit 19 corrects the shake included in the correction target image based on the image data obtained from the output signal of the imaging unit 11 without using a camera shake detection sensor such as an angular velocity sensor. That is, the blur included in the correction target image is removed. In the present embodiment, removal related to blur means complete removal or partial removal, which can be read as reduction.

  The camera shake correction (camera shake correction) performed by the camera shake correction unit 19 includes still image camera shake correction for still images and moving image camera shake correction for moving images.

  By using continuous shooting, it is possible to select and use an image based on shutter timing and shooting conditions preferable for the user from a group of continuous shot images. However, since camera shake during continuous shooting is not uniform, there is a possibility that an image due to the preferable shutter timing and shooting conditions described above includes large camera shake. Therefore, in still image camera shake correction according to the present embodiment, an element image based on the preferred shutter timing and shooting conditions is selected from the group of continuous shot images as a best shot image (correction target image) and the best shot image is used. Is selected from the group of continuously shot images as a reference image, and the blur of the best shot image is corrected using both selected images. FIG. 2 shows a conceptual diagram of this correction method.

  As a result, it is possible to provide the user with the best shot image with less blur, that is, the image obtained with less shutter shake and preferable shutter timing and shooting conditions for the user. Further, since the continuous image group is an image group that should be originally acquired by continuous shooting, it is not necessary to acquire a separate dedicated image for camera shake correction. Therefore, there is no extra load on the shooting time and memory usage.

  In addition, since the camera shake at the time of moving image shooting is irregular, the frame image forming the moving image includes a frame image having a large shake and a frame image having a small shake. Therefore, in the video image stabilization according to the present embodiment, paying attention to an arbitrary frame, if the blur of the frame image of the target frame is large, refer to a frame image with a small blur in a frame near the target frame, The blur of the frame image of the target frame is corrected based on the frame image with the small blur and the frame image of the target frame.

  Accordingly, it is possible to provide a user with a clear moving image with reduced blurring in each frame image.

  Hereinafter, first to sixth examples will be described as examples relating to camera shake correction. FIG. 3 shows a summary of the first to sixth embodiments in a table. In the first embodiment, the overall movement of still image stabilization will be described. In the second embodiment, a method for selecting a correction target image as a best shot image will be described. In the third embodiment, a reference image selection method will be described. In the fourth embodiment, a correction method for a correction target image based on the correction target image and the reference image will be described. In the fifth embodiment, a motion blur correction operation will be described. In the sixth embodiment, application examples and modified sequences for the items described in the first to fifth embodiments will be described. In the description of each embodiment, the “memory” in which an image or the like is stored means a memory (not shown) provided in the internal memory 14 or the camera shake correction unit 19 unless otherwise specified.

<< First Example >>
First, the first embodiment will be described. With reference to FIG. 4, the flow of the still image camera shake correction operation will be described. FIG. 4 is a flowchart showing the flow of the operation.

In the shooting mode, when an instruction for continuous shooting is given, the exposure amount of the image sensor of the imaging unit 11 is optimized based on brightness information obtained from a photometry circuit (not shown) that measures the brightness of the subject. so that, the main control unit 13 calculates an exposure time T _EP of the image pickup device (steps S11 and S12). Next, in step S13, acquires a continuous image group consisting of N photographed images by N times continuous shooting at an exposure time T _EP, the N photographed images as an element image Cw ₁ ~Cw _N Store on memory. The number of element images (that is, the value of N) coincides with the number of images to be acquired by continuous shooting, and this number is designated by the user through an operation on the operation unit 17, for example. Here, N is an integer of 3 or more. However, it is possible to set N = 2. The image sizes of the element images Cw _{1 to} Cw _N are the same.

In the following step S14, the main control unit 13 or the camera shake correction unit 19 compares the exposure time T _EP and the threshold T _TH, if the exposure time T _EP is smaller than the threshold value T _TH, shake the element image Cw ₁ ~Cw _N 4 is regarded as being not included (or very slight), and the process of FIG. 4 is terminated without performing camera shake correction. As the threshold value T _TH , for example, a camera shake limit exposure time is used. The camera shake limit exposure time is a limit exposure time at which camera shake is determined to be negligible, and is calculated from the reciprocal of the focal length f _D of the imaging unit 11.

If the exposure time T _EP is larger than the threshold value T _TH, the process proceeds to step S15. In step S15, the camera shake correction unit 19 selects the best shot image from the element images Cw _{1 to} Cw _N and stores the selected best shot image in the memory as the correction target image Lw (the selection method is the second embodiment). Later in the example). In subsequent step S16, the camera shake correction unit 19 selects one element image with a small shake from the element images Cw _{1 to} Cw _N and stores the selected element image in the memory as a reference image Rw (selection method). Is described later in the third embodiment). The element image with the smallest blur to be selected here is basically the element image with the smallest blur among the element images Cw _{1 to} Cw _N. As will be apparent from the camera shake correction method described later, the reference image Rw is used as an image from which deterioration due to camera shake has been removed in the camera shake correction processing. For this reason, the smaller the amount of blur in the element image to be selected as the reference image, the better.

  In subsequent step S17, the camera shake correction unit 19 determines whether or not the correction target image Lw and the reference image Rw are the same, and if both are the same, the process of FIG. 4 is terminated without performing the camera shake correction. To do.

  When the correction target image Lw and the reference image Rw are not the same, the process proceeds from step S17 to step S18. When the correction target image Lw and the reference image Rw are not the same, the blur included in the reference image Rw is smaller than that in the correction target image Lw. That is, the blur amount of the reference image Rw is smaller than the blur amount of the correction target image Lw. Therefore, in step S18, the camera shake correction unit 19 performs camera shake correction on the correction target image Lw based on the correction target image Lw and the reference image Rw, and generates a correction image in which the blur in the correction target image Lw is reduced. This is stored in the memory as a corrected image Qw (a correction method will be described later in the fourth embodiment).

In step S19, the image data representing the corrected image Qw is output to the display unit 15 via the main control unit 13, and the corrected image Qw is automatically displayed on the display screen of the display unit 15. Further, the main control unit 13 or the camera shake correction unit 19 records association information for associating the corrected image Qw with the continuous shot image group (element images Cw _{1 to} Cw _N ) in the internal memory 14 or the recording medium 16.

Each image data of the element images Cw _{1 to} Cw _N and the corrected image Qw is recorded on the recording medium 16. At this time, the element image selected as the correction target image Lw may be overwritten and recorded with the correction image Qw. That is, one element image is selected as the correction target image Lw from the element images Cw _{1 to} Cw _N , but the selected element image is not recorded on the recording medium 16, and instead the corrected image Qw is selected. You may make it record on the recording medium 16. FIG.

  By performing the processing as described above, it is possible to provide the user with the best shot image that has been subjected to camera shake correction. Further, since the continuous image group is an image group that should be originally acquired by continuous shooting, it is not necessary to acquire a separate dedicated image for camera shake correction. Therefore, there is no extra load on the shooting time and memory usage.

  In addition, after continuous shooting, the last captured image is usually displayed on the display screen. However, when camera shake correction is performed, the corrected image Qw is automatically displayed on the display unit 15 immediately after the correction. Is done. This makes it possible to check the most desirable image immediately after continuous shooting, which is convenient for the user.

Further, as described above, the corrected image Qw and the continuous shot image group are associated with each other by the association information, and the operation unit 17 (operation reception unit) in FIG. 1 receives an image file operation on the associated image. When an image file operation is performed, the main control unit 13 collectively applies the process according to the image file operation to all the associated images. The image file operation is an operation for instructing deletion, duplication, image rotation, image size change, image cutout, printing by a printer, file name change, folder classification, and the like recorded on the recording medium 16. For example, when an image file operation for instructing deletion is performed, all the image data for the corrected image Qw and the continuous shot image group are deleted from the recording medium 16 at once. By making such an image file operation possible, user convenience is enhanced. Note that the correction image Qw and the element images Cw _{1 to} Cw _N can be individually deleted by other operations on the operation unit 17.

<< Second Example >>
Next, a second embodiment will be described. In the second embodiment, a method for selecting the correction target image Lw as the best shot image from among the element images Cw _{1 to} Cw _N forming the continuous shot image group will be described. This selection process is executed in step S15 of FIG. Selection processing can be performed from various viewpoints so that an element image based on shutter timing and photographing conditions preferable for the user is selected as the best shot image. Hereinafter, a plurality of selection methods including the first to fourth best shot image selection methods will be exemplified.

[First best shot image selection method]
First, the first best shot image selection method will be described. In many cases, it is preferable that a person's face as a subject is facing the front of the imaging apparatus 1 as a photograph. Therefore, in the first best shot image selection method, the face orientation of a person appearing in the element image is detected for each element image, and the most face of all the element images faces the front with respect to the imaging device 1. Is selected as the best shot image. The degree to which the face is facing the front with respect to the imaging device 1 is referred to as the face-facing front degree for convenience. It is assumed that the greater the degree of front facing the face, the more the face is facing the imaging device 1.

  A method for detecting the face of a person in the image and a method for detecting the face direction of the person are known, and any known method can be applied. For example, a method described in JP-A-10-307923 may be used. A specific method is illustrated with reference to FIG. 5 showing one element image 200. For example, using a known method, a face 201 of a person in the element image 200 is detected and parts of the face 201 (eyebrows, eyes, nose, mouth, etc.) are detected. Next, on the element image of the midpoint 233 of the line segment connecting each position on the element image of the two end points 211 and 221 located outside both eyes of the face 201 and both end points 231 and 232 of the mouth of the face 201 The end points 211 and 221 are located at the left end of the left eye and the right end of the right eye, and the end points 231 and 232 are located at both ends in the longitudinal direction of the mouth. Subsequently, the area on the element image of a triangle having three points 211, 221 and 233 as vertices is obtained. Then, the larger the obtained area, the higher the degree of front facing the face.

[Second best shot image selection method]
Next, a second best shot image selection method will be described. It is often preferable for a photograph that the degree of eye opening of a person as a subject is large. Therefore, in the second best shot image selection method, the eye opening degree of the person appearing in the element image is detected for each element image, and the element image having the highest eye opening degree is selected as the best shot image among all the element images. select. The degree of eye opening means the degree of eye opening of a person in the element image, and the degree of eye opening is larger as the eye is opened more widely.

  The degree of eye opening can be estimated from the curvature in the convex direction of the upper eyelid, for example. A specific method is illustrated with reference to FIG. FIG. 6 is an enlarged view near the eyes of the face 201 in the element image 200 shown in FIG. First, the eye which is a part of the face 201 is detected from the element image 200 using the above-described known method, and then the left and right end points of the eyes are detected for each of both eyes. The left and right end points in one eye are referred to by reference numerals 211 and 212, and the left and right end points in the other eye are referred to by reference numerals 221 and 222. Next, edge detection processing is performed on a partial image near the eyes of the face 201 in the element image 200. Then, an edge point group having strong edge strength is detected along the edge of each upper eyelid of both eyes. Therefore, paying attention to one eye, a parabola fitting is performed on three points of the two end points 211 and 212 of the eye and an arbitrary point 213 in the edge point group located near the center of the two end points 211 and 212. The curvature for the three points is obtained. This is repeated for a plurality of points in the edge point group. Further, the curvature is similarly obtained for the other eye. And the average curvature of the some curvature calculated | required in this way is calculated, and it is estimated that the degree of eye opening is so high that this average curvature is large in the convex direction of the upper eyelid.

  Instead of this, for example, it is also possible to estimate the degree of eye opening from the black eye area (area on the element image) of the face 201 appearing in the element image 200.

[Third best shot image selection method]
Next, a third best shot image selection method will be described. In many cases, it is preferable for a photograph to have a better expression of a person as a subject. Therefore, in the third best shot image selection method, the facial expression of the person appearing in the element image is detected for each element image, and the element image having the best expression among all the element images is selected as the best shot image. . For example, the smile level of the face is detected as an index representing the quality of the person's facial expression, and the element image having the highest smile level is selected as the best shot image. In this case, as the face in the element image is determined to be a laughing face, the degree of smile increases and the expression is determined to be better.

  A method for detecting the smile level is known, and any known method can be applied. For example, a method described in JP-A-2005-56388 may be used. In this case, the feature amount is obtained for each of the predetermined portion group of the face included in the element image, and the feature amount is obtained for each of the predetermined portion group of the face in the image including the face having a predetermined expression (in this example, a smile). A face is included in the element image based on the distribution of the points calculated for each of the predetermined part groups. Find the degree of smile.

  Instead of this, for example, the degree of smile of the face can be estimated from the area of the teeth appearing in the element image (area on the element image). In this case, it is estimated that the smile degree is higher as the ratio of the tooth area to the entire face area in the element image is larger. Alternatively, it is also possible to estimate the degree of smile of the face from the degree of opening of the face (degree of opening of the mouth) in the element image. In this case, it is estimated that the larger the ratio of the mouth area to the entire face area in the element image, the higher the smile degree. Alternatively, it is possible to estimate the smile level of the face from the curvature of the cheek muscles of the face in the element image. In this case, it is estimated that the higher the curvature of the facial cheek muscles in the element image, the higher the smile degree.

[Fourth best shot image selection method]
Next, a fourth best shot image selection method will be described. In the fourth best shot image selection method, the quality of the photographic composition is evaluated for each element image, and the element image evaluated as having the best composition is selected as the best shot image. A good degree of composition is defined as an index representing the quality of composition. It is determined that the better the composition is, the better the composition is.

  When photographing a person, a composition in which the head is biased above and below the photographing range (that is, the upper end or the lower end in the entire region of the photographed image) is often not preferable. However, there may be such an unfavorable composition due to the movement of the subject and camera shake at the time of shooting. Therefore, first, the photographer sets the preferred center position and size of the head in advance by operating the operation unit 17. The center position and size here are the center position and size of the head on the captured image. Next, the center position and size of the person's head in the element image are detected by the known face detection process as described above. Then, it is determined that the better the composition is, the closer the detected center position and size are to the set center position and size.

As described above, the setting operation for the center position and size can be omitted. For example, as shown in FIG. 7, a distance D ₂₅₃ between the upper end 251 of the head in the element image 250 and the upper end 252 of the element image 250 itself is detected. A distance D ₂₅₃ is a distance on the element image 250. On the other hand, the optimum distance of this distance may be stored in advance in the memory, and it may be determined that the better the composition is, the closer the detected distance D ₂₅₃ is to the optimum distance.

  By automatically selecting the best shot image using the first to fourth best shot image selection methods described above, it is possible to save the user from having to select a preferable image from the group of continuously shot images.

However, it is also possible to select the best shot image from the element images Cw _{1 to} Cw _N based on the user's selection instruction without depending on the first to fourth best shot image selection methods. In this case, after the acquisition of the element image Cw ₁ ~Cw _N, to display the elemental images Cw ₁ ~Cw _N on the display unit 15, seek user selection instruction. The user can instruct the selection of the best shot image to the operation unit 17 in FIG.

Alternatively, M candidates of best shot images are selected from the element images Cw _{1 to} Cw _N using any one of the first to fourth best shot image selection methods, and the selected M candidates One best shot image may be selected according to the user's selection instruction (M is an integer equal to or greater than 2 and N> M). That is, the user is finally selected after narrowing down the best shot image candidates to M. In this case, the narrowed down M element images as candidates for the best shot image are displayed on the display unit 15, and selection by the user for selecting one best shot image from the M element images Ask for instructions. For example, when the first best shot image selection method is used, M element images having a face-facing front degree of 1st to Mth may be used as best shot image candidates. Similarly, when the second, third, or fourth best shot image selection method is used, the degree of eye opening, the degree of smile, or the degree of composition goodness is 1 to Mth, not the degree of front facing the face. Large M element images may be used as best shot image candidates.

  Also, two, three, or four of the first to fourth best shot image selection methods described above can be used in combination. In this case, for example, for each element image, a weighted addition value of two or more values is calculated among a value representing the degree of front facing the face, a value representing the degree of eye opening, a value representing the degree of smile, and a value representing the degree of good composition. . Then, the element image corresponding to the maximum weighted addition value is selected as the best shot image. It is also possible to combine this with a selection instruction by the user. That is, M element images having the first to Mth weighted addition values are selected and displayed as best shot image candidates, and one best shot image is selected from the selected M candidates according to a user's selection instruction. You may make it select.

<< Third Example >>
Next, a third embodiment will be described. In the third embodiment, a method of selecting the reference image Rw from the element images Cw _{1 to} Cw _N forming the continuous shot image group will be described. This selection process is executed in step S16 of FIG. Hereinafter, a plurality of selection methods including the first to fourth reference image selection methods will be exemplified.

[First Reference Image Selection Method]
First, the first reference image selection method will be described. In the first reference image selection method, the reference image Rw is selected based on the edge strength of the image. A specific flow of the selection operation will be described with reference to FIGS. FIG. 8 is a flowchart showing the flow of operations related to the first reference image selection method. FIG. 9 is a diagram showing the relationship of images used for this operation.

First, in step S31, a variable i that can take each integer value between 1 and N is introduced, and 1 is substituted into the variable i as an initial value. In subsequent step S32, it is determined whether or not the variable i is currently 1. If i = 1, the process proceeds to step S33, whereas if i ≠ 1, the process proceeds to step S34. In step S33, it extracts the central or small region located near the center of the element images Cw _i, the image of the small area as the small image Cs _i. The extracted small area is, for example, a small area of 128 × 128 pixels. i = only since it is the processing leads to the step S33 if 1 is, in step S33, so that the small image Cs ₁ is extracted from the first element image Cw ₁ to form a continuous image group.

After step S33, the process proceeds to step S35. In step S35, subjected to edge extraction processing for the small image Cs _i, obtaining the small image Es _i. For example, to generate an edge extracted image of the small image Cs _i by applying any edge detection operator to each pixel of the small image Cs _i, to the edge extraction image and the small image Es _i. Thereafter, in step S36, the sum of the pixel values of the small image Es _i is calculated, and the sum is taken as an evaluation value Ka _i . After step S36, it is determined in step S37 whether the variable i is equal to N or not.

If i ≠ N in step S37, 1 is added to variable i in step S38, and the process returns to step S32. In this case, since i ≠ 1 in step S32, the process proceeds from step S32 to step S34. In step S34, a small area corresponding to the small area extracted from the element image Cw _{1 is} extracted from the element image Cw _i (≠ Cw ₁ ), and an image in the small area extracted from the element image Cw _{i is} extracted from the small image Cs. _i . The search for the corresponding small area is performed by image processing using a template matching method or the like. Specifically, for example, the small image Cs ₁ extracted from the element image Cw ₁ as a template, using a known template matching method, and searches for a small area most similar to the template from the element image Cw _i, the search the image in the small area and the small image Cs _i. After the small image Cs _i is extracted in step S34, the processes of steps S35 and S36 are performed on the small image Cs _i .

When the process of step S36 is performed for each of the variables i = 1, 2,... N and N evaluation values Ka _{1 to} Ka _N are calculated, i = N is established in step S37. The process of FIG. 8 ends. As is clear from the above processing contents, the evaluation value Ka _i increases as the edge strength of the small image Cs _i increases.

If the images have the same composition, the smaller the camera shake during the exposure period, the clearer the edge and the higher the edge strength in the image. Moreover, since motion blur uniformly deteriorated on the entire image, the edge intensity in the entire element image Cw _i is a one corresponding to the edge intensity in the small image Cs _i. Therefore, it can be estimated that the larger the evaluation value Ka _i is, the smaller the amount of shake of the corresponding small image Cs _i and element image Cw _i is. Since the amount of blurring of the element image to be selected as the reference image is preferably as small as possible, the image corresponding to the maximum evaluation value is selected as the reference image Rw among the element images Cw _{1 to} Cw _N. For example, among the evaluation values Ka ₁ ~Ka _N, evaluation value Ka _N-1 is equal to the maximum, selects the element image Cw _N-1 as a reference image Rw.

  In general, in order to obtain the blur amount of an image from one image, as described in Japanese Patent Application Laid-Open No. 11-27574, a Fourier transform is performed on the image to generate a converted image on the frequency space, and a camera shake is caused. Therefore, processing with a large calculation cost is required, such as measuring the frequency interval of attenuation. The calculation cost means a load due to calculation, and an increase in calculation cost increases processing time and power consumption. On the other hand, in this selection method, the blur amount is estimated from the edge strength using the relationship between the edge strength and the blur amount. For this reason, compared with the conventional method using a Fourier transform etc., the calculation cost for estimating the amount of blurring can be made small. Further, by calculating the evaluation value by focusing on the extracted small image instead of the entire image, an effect of further reducing the calculation cost can be obtained. In addition, since evaluation values are compared between corresponding small regions using template matching or the like, even if there is a change in composition during continuous shooting, the effect of the change is negligible.

[Second Reference Image Selection Method]
Next, the second reference image selection method will be described. In the second reference image selection method, the reference image Rw is selected based on the contrast of the image. A specific flow of the selection operation will be described with reference to FIG. FIG. 10 is a flowchart showing an operation flow related to the second reference image selection method.

  Since the processing contents of steps S41 to S44, S47, and S48 in FIG. 10 are the same as the processing contents of steps S31 to S34, S37, and S38 in FIG. 8, respectively, their overlapping descriptions are omitted. However, in the second reference image selection method, after step 43 or S44, the process proceeds to step S45, and after steps S45 and S46, the process proceeds to step S47.

In step S45, it extracts a luminance signal of each pixel of the small image Cs _i. Of course, for example, when i = 1, the luminance signal of each pixel of the small image Cs ₁ is extracted, and when i = 2, the luminance signal of each pixel of the small image Cs ₂ is extracted. In step S46, a histogram of the luminance value (that is, the value of the luminance signal) of the small image Cs _i is generated, the variance of the histogram is calculated, and the variance is set as the evaluation value Kb _i . Thereafter, the process proceeds to step S47. When the process of step S46 is executed for each of the variables i = 1, 2,... N and N evaluation values Kb _{1 to} Kb _N are calculated, i = N is established in step S47. The process of FIG. 10 is finished.

For images with the same composition, the greater the camera shake that occurs during the exposure period, the smoother the luminance between adjacent pixels, and the proportion of pixels in the intermediate gradation increases, and the distribution of luminance values in the histogram becomes intermediate gradation. Centralize. The greater the degree of smoothing, the smaller the variance in the histogram and the smaller the evaluation value Kb _i , so the larger the evaluation value Kb _i , the smaller the amount of blurring of the corresponding small image Cs _i and element image Cw _i. Can be estimated. Since the amount of blurring of the element image to be selected as the reference image is preferably as small as possible, the element image corresponding to the maximum evaluation value is selected as the reference image Rw among the element images Cw _{1 to} Cw _N. For example, among the evaluation values Kb ₁ ~Kb _N, evaluation value Kb _N-1 is equal to the maximum, selects the element image Cw _N-1 as a reference image Rw.

  As an example of an element image that forms a continuous image group, an element image 261 is shown in FIG. 11A and an element image 262 is shown in FIG. While the element image 261 is a clear image, the element image 262 includes a large amount of shake due to a large amount of camera shake during the exposure period of the element image 262. FIGS. 12A and 12B show the histograms generated in step S46 for the element images 261 and 262, respectively. In contrast to the histogram of the element image 261 (see FIG. 12A), the histogram of the element image 262 (see FIG. 12B) shows a concentration of distribution to the intermediate gradation. This concentration reduces the variance (and standard deviation).

For a certain image of interest, a small variance in the histogram corresponds to a low contrast of the image of interest, and a large variance in the histogram corresponds to a high contrast of the image of interest. Therefore, in the above-described method, the contrast of the target image is estimated by calculating the variance in the histogram, and the blur amount of the target image is estimated based on the estimated contrast. Then, the estimated contrast value is derived as the evaluation value Kb _i .

  In this selection method, the blur amount is estimated from the contrast using the relationship between the contrast and the blur amount. For this reason, compared with the conventional method using a Fourier transform etc., the calculation cost for estimating the amount of blurring can be made small. Further, by calculating the evaluation value by focusing on the extracted small image instead of the entire image, an effect of further reducing the calculation cost can be obtained. In addition, since evaluation values are compared between corresponding small regions using template matching or the like, even if there is a change in composition during continuous shooting, the effect of the change is negligible.

[Third Reference Image Selection Method]
Next, a third reference image selection method will be described. In the third reference image selection method, (N−1) element images excluding the element image selected as the correction target image Lw from the element images Cw _{1 to} Cw _N forming the continuous shot image group are used as reference images. Treat as a candidate for Rw. Element images that are handled as candidates for the reference image Rw are also referred to as “candidate images”. Then, each rotation angle of each candidate image with respect to the correction target image Lw is obtained, and the reference image Rw is selected using each obtained rotation angle.

This selection method will be described more specifically with reference to FIG. Now, it is assumed that the element image Cw _N are selected as the correction target image Lw. In this case, each of the element images Cw _{1 to} Cw _N−1 is set as a candidate image.

First, a plurality of characteristic small regions are extracted from the correction target image Lw. A characteristic small area refers to a rectangular area that has a relatively large number of edge components (in other words, a relatively strong contrast) in the source image. For example, a Harris corner detector is used. Extracted. A characteristic small area refers to a rectangular area having a relatively large number of edge components (in other words, a relatively strong contrast) in the extraction source image, and is an area including a characteristic pattern, for example. A characteristic pattern has, for example, a luminance change in two or more directions such as a corner of an object, and the position of the pattern (position on the image) can be easily detected by image processing based on the luminance change. Meaning a simple pattern. Here, a plurality of small areas of 32 × 32 pixels are extracted as characteristic small areas. As shown in FIG. 13, it is assumed that two small regions 281 and 282 are extracted from the correction target image Lw (the element image Cw _N under the current assumption). The center points of the small areas 281 and 282 are referred to by reference numerals 291 and 292, respectively. In the example shown in FIG. 13, the direction of the line segment connecting the center points 291 and 292 matches the horizontal direction of the correction target image Lw.

Next, two small regions corresponding to the two small regions 281 and 282 extracted from the correction target image Lw are extracted from each of the candidate images. The search for the corresponding small region may be performed using the above-described method using a template matching method or the like (see the description of the first reference image selection method). FIG. 13 shows two small areas 281 a and 282 a extracted from the element image Cw ₁ as the candidate image, and two small areas 281 b and 282 b extracted from the element image Cw ₂ as the candidate image. Yes. The small areas 281 a and 281 b correspond to the small area 281, and the small areas 282 a and 282 b correspond to the small area 282. Further, the center points of the small regions 281a, 282a, 281b, and 282b are referred to by reference numerals 291a, 292a, 291b, and 292b, respectively.

Thereafter, paying attention to the element image Cw ₁ , the rotation angle (ie, inclination) θ ₁ of the line connecting the center points 291a and 292a with respect to the line connecting the center points 291 and 292 is obtained. Similarly, paying attention to the element image Cw ₂ , the rotation angle (ie, inclination) θ ₂ of the line connecting the center points 291 b and 292 b with respect to the line connecting the center points 291 and 292 is obtained. Rotation angle θ ₃ ~θ _N-1 is also determined in the same manner for the element image Cw ₃ ~ element images Cw _N-1, obtains the reciprocal of the rotation angle theta _i as the evaluation value Kc _i.

As described later, it is desirable to perform the selection of the reference image Rw using both the evaluation value Kc _i and other evaluation values, also select candidate images corresponding to the minimum evaluation value Kc _i as a reference image Rw Is possible. In this case, for example, among the evaluation values Kc ₁ ~Kc _N-1 obtained, the evaluation value Kc _N-1 is equal to the maximum, selects the element image Cw _N-1 as a reference image Rw.

  Since the imaging time (imaging time) of the correction target image Lw and the reference image Rw are different, a compositional deviation may occur between the two images. In order to perform high-accuracy camera shake correction, it is necessary to perform alignment for canceling the positional shift between the correction target image Lw and the reference image Rw resulting from this shift. This alignment is realized by coordinate transformation (affine transformation or the like), but if this alignment includes image rotation processing, the circuit scale and calculation cost increase. According to this selection method, an element image having a small rotation angle with respect to the correction target image Lw is preferentially selected. Therefore, a relatively good camera shake correction effect can be obtained only by alignment by parallel movement, and the circuit scale can be increased. Contributes to reduction.

  In addition, when camera shake correction is performed using the Fourier iteration method described later, linear calculation is performed between frequency space images obtained by Fourier transform of the correction target image Lw and the reference image Rw (in the fourth embodiment). Detailed explanation). In this case, if there is a shift in the rotation direction between the correction target image Lw and the reference image Rw, camera shake detection and camera shake correction accuracy are significantly reduced due to the characteristics of Fourier transform. For this reason, when camera shake correction is performed using the Fourier iteration method, camera shake detection and camera shake correction accuracy are greatly improved by selecting the reference image Rw using this selection method.

[Fourth Reference Image Selection Method]
Next, a fourth reference image selection method will be described. In the fourth reference image selection method, similarly to the third reference image selection method, the element image selected as the correction target image Lw is excluded from the element images Cw _{1 to} Cw _N forming the continuous shot image group (N -1) Treat element images as candidates for the reference image Rw. Element images that are handled as candidates for the reference image Rw are also referred to as “candidate images”. Then, a photographing time difference (photographing time difference) between each candidate image and the correction target image Lw is obtained. Then, the reference image Rw is selected using each obtained photographing time difference.

This will be described more specifically. Now, it is assumed that the element image Cw _N are selected as the correction target image Lw. In this case, each of the element images Cw _{1 to} Cw _N−1 is set as a candidate image. In addition, it is assumed that continuous photographing is performed in the order of element images Cw ₁ , Cw ₂ ,..., Cw _N−1 and Cw _N. The difference in shooting time between the focused candidate image and the correction target image Lw is the time difference between the intermediate point in the exposure period when shooting the focused candidate image and the intermediate point in the exposure period when shooting the correction target image Lw. The inverse of the photographing time difference determined for the element image Cw _i as a candidate image as the evaluation value Kd _i. For this example, a total of (N-1) evaluation value Kd ₁ ~Kd _N-1 is calculated, as a matter of course, that the _{Kd 1 <Kd 2 <··· <} Kd N-1 is established Become.

As will be described later, it is desirable to select the reference image Rw using both the evaluation value Kd _i and another evaluation value, but it is also possible to select a candidate image corresponding to the minimum evaluation value Kd _i as the reference image Rw. Is possible. In this case, in the present example, the evaluation value Kd _N-1 is maximized, and thus the element image Cw _N-1 is selected as the reference image Rw.

  The larger the shooting time difference between the correction target image Lw and the reference image Rw, the higher the probability that the subject will move during that time, and the higher the probability that shooting conditions such as illuminance will change. The movement of the subject and the change in shooting conditions act in the direction of reducing the camera shake detection and camera shake correction accuracy. According to this selection method, since an element image having a small photographing time difference from the correction target image Lw is preferentially selected, the influence of these changes is reduced, and more accurate camera shake detection and camera shake correction are performed. Is possible.

Note that two, three, or four of the first to fourth reference image selection methods described above may be used in combination. For example, when combining four, a total evaluation value K _i for the element image Cw _i is calculated by the following formula (A-1), may be selected element image corresponding to the maximum total evaluation value as the reference image Rw . Here, ka, kb, kc, and kd are predetermined weighting factors having zero or positive values. When combining two or three selection methods, the necessary weighting factor may be zero. For example, when combining the three selection methods of the first to third reference picture selection method may be calculated overall evaluation value K _i on which the "kd = 0".
K _i = ka × Ka _i + kb × Kb _i + kc × Kc _i + kd × Kd _i (A-1)

In order to realize the processing described in the first embodiment, the requirement that “the amount of blur in the element image to be selected as the reference image Rw is smaller than that of the best shot image (correction target image Lw)” is satisfied. There is a need. On the other hand, the shake amount is reflected only on the evaluation values Ka _i and Kb _i among the first to fourth evaluation values Ka _i , Kb _i , Kc _i and Kd _i . Therefore, in calculating the overall evaluation value K _i based on the above formula (A-1), at least one of the weighting coefficients ka and kb it should not be zero.

Furthermore, two of the first through fourth reference picture selection method, see when using three or four in combination, based on the overall evaluation value K _i as described above requirements are met The image Rw should be selected. Thus, for example, if the element image Cw _N are selected as the correction target image Lw, obtaining the total evaluation value K ₁ ~K _N-1 for elemental image Cw ₁ ~Cw _N-1. At this stage, the shake amount of each of the element images Cw _{1 to} Cw _N is estimated by the shake amount estimation method described in the description of the first or second reference image selection method. In this case, _{"K 1 <K 2 <··· <} K N-2 <K N-1 " if a blur estimated for blurring amount and the element image Cw _N-1 which is estimated for an element image Cw _N If the former is large, the element image Cw _N-1 is selected as the reference image Rw. On the other hand, if the latter is larger, to meet the above requirements, the shake amount estimated for an element image Cw _N focusing on an element image Cw _N-2 corresponding to the large total evaluation value K _N-2 in the second And the shake amount estimated for the element image Cw _N-2 are compared. Here, if the former is large, the element image Cw _N-2 is selected as the reference image Rw. If the latter is large, the element image Cw _N-3 corresponding to the third largest comprehensive evaluation value K _N-3 is selected. The same processing is performed focusing on the above. By processing in this way, the most desirable element image that satisfies the above-described requirements is selected as the reference image Rw.

<< 4th Example >>
Next, a fourth embodiment will be described. In the fourth embodiment, a correction method for the correction target image Lw based on the correction target image Lw and the reference image Rw will be described. The process for performing this correction is executed in step S18 of FIG. As a correction method for the correction target image Lw, first to third correction methods will be exemplified below. The first, second, and third correction methods are correction methods using an image restoration method, an image synthesis method, and an image sharpening method, respectively (see FIG. 3).

[First correction method]
The first correction method will be described with reference to FIG. FIG. 14 is a flowchart showing the flow of correction processing based on the first correction method. When the first correction method is adopted, the process in step S18 in FIG. 4 is formed from the processes in steps S71 to S77 in FIG.

  First, in step S71, a characteristic small area is extracted from the correction target image Lw, and an image in the extracted small area is stored in the memory as a small image Ls. For example, a small area of 128 × 128 pixels is extracted as a characteristic small area by using a Harris corner detector. The significance of the characteristic small area is as described above.

  Next, in step S72, a small region having the same coordinates as the small region extracted from the correction target image Lw is extracted from the reference image Rw, and an image in the small region extracted from the reference image Rw is stored in the memory as a small image Rs. To remember. The center coordinates of the small area extracted from the correction target image Lw (center coordinates in the correction target image Lw) and the center coordinates of the small area extracted from the reference image Rw (center coordinates in the reference image Rw) are equal, and the correction target Since the image sizes of the image Lw and the reference image Rw are equal, the image sizes of both small regions are also made equal.

  The small image Ls obtained as described above is treated as a deteriorated image and the small image Rs is treated as an initial restored image (step S73), and the Fourier iteration method is performed in step S74, whereby the small image Ls is obtained. An image deterioration function representing a state of deterioration due to blur is obtained.

  When performing the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image). This initial restored image is called an initial restored image.

  A point spread function (hereinafter referred to as PSF) is obtained as an image degradation function. An operator or a spatial filter in which an ideal point image is weighted in accordance with a trajectory drawn on the image by a shake of the image pickup apparatus 1 is called a PSF, and is generally used as a mathematical model of camera shake. Since camera shake uniformly degrades the entire image, the PSF obtained for the small image Ls can be used as the PSF for the entire correction target image Lw.

  The Fourier iteration method is a method for obtaining a restored image from which deterioration is removed or reduced from a deteriorated image including deterioration (see Non-Patent Document 1 above). This Fourier iteration method will be described in detail with reference to FIGS. FIG. 15 is a detailed flowchart of the process in step S74 of FIG. FIG. 16 is a block diagram of a part of the image stabilization unit 19 shown in FIG.

  First, in step S101, a restored image is set as f ', and an initial restored image is set in the restored image f'. That is, the small image Rs is used as the initial restored image f ′. Next, in step S102, the deteriorated image (that is, the small image Ls) is set to g. Then, the result of Fourier transform of the deteriorated image g is stored in the memory as G (step S103). For example, when the image sizes of the initial restored image and the degraded image are 128 × 128 pixels, f ′ and g can be expressed as a matrix having a matrix size of 128 × 128.

Next, in step S110, F ′ obtained by Fourier transform of the restored image f ′ is obtained, and in step S111, H is calculated by the following equation (B-1). H corresponds to the result of Fourier transform of PSF. In Formula (B-1), F ′ ^* is a conjugate complex matrix of F ′, and α is a constant.

  Next, in step S112, PSF is obtained by performing inverse Fourier transform on H. The PSF obtained here is assumed to be h. Next, in step S113, after the PSF h is corrected under the constraint condition of the following formula (B-2a), it is further corrected under the constraint condition of the formula (B-2b).

  Since PSF h is expressed as a two-dimensional matrix, each element of this matrix is represented by h (x, y). Each element of the PSF should originally take a value of 0 or more and 1 or less. Therefore, in step S113, it is determined whether each element of the PSF is 0 or more and 1 or less, and the value of the element that is 0 or more and 1 or less is left as it is. The value of the element is corrected to 1, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (B-2a). Then, the PSF is normalized so that the sum of the elements of the PSF after correction is 1. This normalization is correction by the constraint condition of the formula (B-2b).

  The PSF corrected by the constraints of the equations (B-2a) and (B-2b) is set as h ′.

Next, in step S114, H ′ obtained by Fourier transforming PSF h ′ is obtained, and in step S115, F is calculated by the following equation (B-3). F corresponds to a Fourier transform of the restored image f. In the formula (B-3), H ′ ^* is a conjugate complex matrix of H ′, and β is a constant.

  Next, in step S116, a restored image is obtained by performing an inverse Fourier transform on F. Let f be the restored image obtained here. Next, in step S117, the restored image f is corrected under the constraint condition of the following formula (B-4), and the corrected restored image is newly set as f '.

  Since the restored image f is represented as a two-dimensional matrix, each element of the matrix is represented by f (x, y). Now, it is assumed that the pixel value of each pixel of the degraded image and the restored image is represented by a digital value from 0 to 255. Then, each element (that is, each pixel value) of the matrix representing the restored image f should originally have a value of 0 or more and 255 or less. Accordingly, in step S117, it is determined whether each element of the matrix representing the restored image f is 0 or more and 255 or less, and the value of the element that is 0 or more and 255 or less is left as it is. If there is an element, the value of the element is corrected to 255, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (B-4).

  Next, in step S118, it is determined whether or not the convergence condition is satisfied, thereby determining whether or not the iterative process is converged.

  For example, the absolute value of the difference between the latest F ′ and the previous F ′ is used as an index for convergence determination. If this index is less than or equal to a predetermined threshold value, it is determined that the convergence condition is satisfied. Otherwise, it is determined that the convergence condition is not satisfied.

  When the convergence condition is satisfied, a final PSF is obtained by performing inverse Fourier transform on the latest H ′. That is, the latest H ′ obtained by inverse Fourier transform is the PSF to be obtained in step S74 of FIG. When the convergence condition is not satisfied, the process returns to step S110, and the processes of steps S110 to S118 are repeated. In the repetition of each process of steps S110 to S118, f ′, F ′, H, h, h ′, H ′, F, and f (see FIG. 16) are sequentially updated to the latest ones.

  Another index can be used as an index for determining convergence. For example, the absolute value of the difference between the latest H ′ and the previous H ′ obtained may be used as the convergence determination index to determine whether or not the convergence condition is satisfied. Further, for example, the correction amount in step S113 using the above formulas (B-2a) and (B-2b) or the correction amount in step S117 using formula (B-4) is used as an index for convergence determination. Whether or not the convergence condition is satisfied may be determined. This is because if the iterative process is toward convergence, the amount of correction becomes small.

  Further, when the number of repetitions of the loop process consisting of steps S110 to S118 reaches a predetermined number, it may be determined that convergence is impossible and the process may be terminated without calculating a final PSF. In this case, the correction target image Lw is not corrected.

Returning to the description of each step in FIG. After the PSF is calculated in step S74, the process proceeds to step S75. In step S75, each element of the inverse matrix of PSF obtained in step S74 is obtained as each filter coefficient of the image restoration filter. This image restoration filter is a filter for obtaining a restored image from a deteriorated image. Actually, each element of the matrix represented by the following equation (B-5) corresponding to a part of the right side of the equation (B-3) corresponds to each filter coefficient of the image restoration filter. The intermediate calculation result of the Fourier iteration method in S74 can be used as it is. However, the formula (B-5) H 'in ^* and H' is, H obtained satisfied immediately before the convergence condition in step S118 ^'* and H' (i.e., the finally obtained H ^'* and H' ).

  After each filter coefficient of the image restoration filter is obtained in step S75, the process proceeds to step S76, and the correction target image Lw is filtered (spatial filtering) using this image restoration filter. In other words, the correction target image Lw is filtered by applying an image restoration filter having the obtained filter coefficients to each pixel of the correction target image Lw. As a result, a filtered image in which the blur included in the correction target image Lw is removed or reduced is generated. Although the size of the image restoration filter is smaller than the image size of the correction target image Lw, it is considered that camera shake causes uniform degradation on the entire image. Therefore, this image restoration filter is applied to the entire correction target image Lw. As a result, the entire blur of the correction target image Lw is removed.

  The filtering image may include ringing accompanying filtering. For this reason, in step S77, a final correction image Qw is generated by performing ringing removal processing for removing this on the filtered image. Since the method for removing ringing is known, a detailed description is omitted. For example, a technique described in JP 2006-129236 A may be used.

  The corrected image Qw is an image in which blur included in the correction target image Lw is removed or reduced and ringing due to filtering is removed or reduced. However, since the filtering image is also an image in which the blur is removed or reduced, the filtering image can be regarded as the corrected image Qw.

  Since the amount of blur included in the reference image Rw is small, its edge component is close to the edge component of an ideal image without camera shake. Therefore, as described above, an image obtained from this reference image Rw is used as an initial restored image in the Fourier iteration method.

  By repetitive loop processing by the Fourier iteration method, the restored image (f ′) gradually approaches an image from which camera shake has been removed as much as possible. However, since the initial restored image itself is already close to an image without camera shake, Convergence is faster than a random image or a deteriorated image as an initial restored image (in the shortest, convergence is achieved by a single loop process). As a result, the processing time for generating camera shake information (PSF or filter coefficient of the image restoration filter) and the processing time for camera shake correction are shortened. If the initial restored image is far from the image to be converged, the probability of convergence to a local solution (an image different from the image to be truly converged) increases. However, by setting the initial restored image as described above, , The probability of convergence to a local solution is low (that is, the probability that camera shake correction fails) is low.

  In addition, camera shake is thought to give uniform degradation to the entire image, so a small area is extracted from each image to create camera shake information (PSF or filter coefficient of the image restoration filter) from the image data of each small area, Apply it to the entire image. As a result, the amount of calculation required is reduced, and the processing time for creating camera shake information and the processing time for correcting camera shake are shortened. Of course, the required circuit scale can be reduced and the cost reduction effect can be expected.

  At this time, as described above, a characteristic small region containing a lot of edge components is automatically extracted. An increase in the edge component in the PSF calculation source image means an increase in the ratio of the signal component to the noise component. Therefore, the influence of noise is reduced by extracting a characteristic small region, and camera shake information is detected more accurately. Will be able to.

[Second correction method]
Next, the second correction method will be described with reference to FIGS. 17 and 18. FIG. 17 is a flowchart showing the flow of correction processing based on the second correction method. FIG. 18 is a conceptual diagram showing the flow of this correction process. When the second correction method is adopted, the process in step S18 in FIG. 4 is formed from the processes in steps S151 to S154 in FIG.

  1 is a color image including information on luminance and information on color. Accordingly, the pixel signal of each pixel that forms the correction target image Lw is formed from a luminance signal that represents the luminance of the pixel and a color signal that represents the color of the pixel. Now, it is assumed that the pixel signal of each pixel is expressed in YUV format. In this case, the color signal is formed from the two color difference signals U and V. The pixel signal of each pixel forming the correction target image Lw is formed from a luminance signal Y that represents the luminance of the pixel and two color difference signals U and V that represent the color of the pixel.

Then, as shown in FIG. 18, the correction target image Lw includes an image Lw _Y including only the luminance signal Y as a pixel signal, an image Lw _U including only the color difference signal U as a pixel signal, and only the color difference signal V as a pixel signal. an image Lw _V containing, can be decomposed into. Similarly, the reference image Rw includes an image Rw _Y including only the luminance signal Y as a pixel signal, an image Rw _U including only the color difference signal U as a pixel signal, an image Rw _V including only the color difference signal V as a pixel signal, (In FIG. 18, only the image Rw _Y is shown).

In step S151 in FIG. 17, first, images Lw _Y , Lw _U and Lw _V are generated by extracting the luminance signal and color difference signal of the correction target image Lw. In step S152, by extracting a brightness signal of the reference image Rw, and generates an image Rw _Y.

Thereafter, in step S153, the pixel signal of the image Lw _{Y and} the pixel signal of the image Rw _Y are compared, thereby calculating a positional deviation amount ΔD between the image Lw _Y and the image Rw _Y. The positional deviation amount ΔD is a two-dimensional amount including a horizontal component and a vertical component, and is expressed as a so-called motion vector. The positional deviation amount ΔD can be calculated using a well-known representative point matching method or a block matching method using template matching. For example, an image in a small region extracted from the image Lw _Y is used as a template, and a small region having the highest similarity with the template is searched from the image Rw _Y by template matching. Then, to calculate the amount of deviation between the searched position of the small area (position on the image Rw _Y) and the position of the small area extracted from the image Lw _Y (position on the image Lw _Y) as the positional deviation amount [Delta] D. Note that the small area to be extracted from the image Lw _Y is desirably a characteristic small area as described above.

Considering the image Lw _Y as a reference, it is assumed that the positional deviation amount ΔD is the positional deviation amount of the image Rw _Y with respect to the image Lw _Y. The image Rw _Y can be regarded as an image in which a positional deviation has occurred by an amount corresponding to the positional deviation amount ΔD with respect to the image Lw _Y. Therefore, in step S154, positional deviation correcting the image Rw _Y by performing coordinate conversion (such as affine transformation) in the image Rw _Y as the positional deviation amount ΔD is canceled. A pixel located at the coordinates (x + ΔDx, y + ΔDy) in the image Rw _Y before the positional deviation correction is converted into a pixel located at the coordinates (x, y) by the positional deviation correction. ΔDx and ΔDy are a horizontal component and a vertical component of ΔD, respectively.

In step S154, the images Lw _U and Lw _V and the image Rw _Y after the positional deviation correction are further combined, and the image obtained by this combination is output as the corrected image Qw. The pixel signal of the pixel located at the coordinates (x, y) in the corrected image Qw is the pixel signal of the pixel in the image Lw _U located at the coordinates (x, y) and the image located at the coordinates (x, y). It is formed from the pixel signal of the pixel in Lw _V and the pixel signal of the pixel in the image Rw _Y after the positional deviation correction located at the coordinates (x, y).

  In color images, visual blur is mainly caused by luminance blur, and if the edge component of luminance is close to that of an ideal image without blur, the viewer feels that there is little blur. Therefore, in this correction method, a pseudo camera shake correction effect is obtained by combining the luminance signal of the reference image Rw with a relatively small amount of blur with the color signal of the correction target image Lw. According to this method, although color misregistration occurs in the vicinity of the edge, an image with less blurring in appearance can be generated at a low calculation cost.

[Third correction method]
Next, the third correction method will be described with reference to FIGS. 19 and 20. FIG. 19 is a flowchart showing the flow of correction processing based on the third correction method. FIG. 20 is a conceptual diagram showing the flow of this correction process. When the third correction method is employed, the process of step S18 in FIG. 4 is formed from the processes of steps S201 to S206 in FIG.

  First, in step S201, a small image Ls is generated by extracting a characteristic small region from the correction target image Lw. In step S202, a small region corresponding to the small image Ls is extracted from the reference image Rw. An image Rs is generated. The processing in steps S201 and S202 is the same as the processing in steps S71 and S72 in FIG.

Next, in step S203, eight smoothed small images Rs _G1 , Rs _G2 ,..., Rs _{G8 having} different smoothing levels are obtained by filtering the small image Rs using eight different smoothing filters. Is generated. Now, eight different Gaussian filters are used as the eight smoothing filters, and the variance of the Gaussian distribution represented by each Gaussian filter is represented by σ ² .

When attention is paid to a one-dimensional image and the pixel position in the one-dimensional image is represented by x, a Gaussian distribution having an average of 0 and a variance of σ ² is generally represented by the following formula (C-1). (See FIG. 21). When this Gaussian distribution is applied to a Gaussian filter, each filter coefficient of the Gaussian filter is represented by h _g (x). That is, when the Gaussian filter is applied to the pixel at position 0, the filter coefficient at position x is represented by h _g (x). In other words, the contribution ratio of the pixel value at the position x before filtering to the pixel value at the position 0 after filtering by the Gaussian filter is represented by h _g (x).

When this concept is extended to two dimensions and the pixel position in the two-dimensional image is represented by (x, y), the two-dimensional Gaussian distribution is represented by the following equation (C-2). Note that x and y represent a horizontal position and a vertical position, respectively. When this two-dimensional Gaussian distribution is applied to the Gaussian filter, each filter coefficient of the Gaussian filter is represented by h _g (x, y). When the Gaussian filter is applied to the pixel at the position (0, 0), the position (x, The filter coefficient in y) is represented by h _g (x, y). That is, the contribution ratio of the pixel value at the position (x, y) before filtering to the pixel value at the position (0, 0) after filtering by the Gaussian filter is represented by h _g (x, y).

In step S203, Gaussian filters with σ = 1, 3, 5, 7, 9, 11, 13, and 15 are used as the eight Gaussian filters. In subsequent step S204, image matching is performed between the small image Ls and each of the smoothed small images Rs _{G1 to} Rs _G8 , and among the smoothed small images Rs _{G1 to} Rs _G8 , the smoothing small that has the smallest matching error. An image (that is, a smoothed small image having the highest correlation with the small image Ls) is specified.

Focusing on the smoothed small image Rs _G1, briefly explaining the calculation method of the small image Ls and the smoothed small image Rs _G1 matching error when the comparison (matching residuals). The image sizes of the small image Ls and the smoothed small image Rs _G1 are the same, and the number of pixels in the horizontal direction and the number of pixels in the vertical direction are respectively M _N and N _N (M _N and N _N are 2 or more) Integer). The pixel value of the pixel at the position (x, y) in the small image Ls is represented by V _Ls (x, y), and the pixel value of the pixel at the position (x, y) in the smoothed small image Rs _G1 is represented by V _Rs. It is represented by (x, y). (Where x and y are integers satisfying 0 ≦ x ≦ M _N −1 and 0 ≦ y ≦ N _N −1). Then, R _SAD representing SAD (Sum of Absolute Difference) between contrast images is calculated according to the following equation (C-3), and R _SSD representing SSD (Sum of Square Difference) between contrast images is represented by the following equation (C− Calculated according to 4).

Let R _SAD or R _SSD be a matching error between the small image Ls and the smoothed small image Rs _G1 . Similarly, a matching error between the small image Ls and each of the smoothed small images Rs _{G2 to} Rs _G8 is also obtained, and the smoothed small image with the smallest matching error is specified. Now, it is assumed that the smoothed small image Rs _G3 corresponding to σ = 5 is specified. In step S204, σ corresponding to the smoothed small image Rs _G3 is set to σ ′. That is, the value of σ ′ is 5.

  In the subsequent step S205, the Gaussian blur represented by σ 'is treated as an image deterioration function representing the deterioration state of the correction target image Lw, thereby deteriorating the correction target image Lw.

Specifically, in step S205, blurring of the correction target image Lw is removed by applying an unsharp mask filter to the entire correction target image Lw based on σ ′. Processing contents of the unsharp mask filter will be described with an image before application of the unsharp mask filter as an input image I _INPUT and an image after application of the unsharp mask filter as an output image I _OUTPUT . First, a σ ′ Gaussian filter (that is, a σ = 5 Gaussian filter) is used as an unsharp filter, and the input image I _INPUT is filtered using this σ ′ Gaussian filter to generate a blurred image I _BLUR . To do. Then, by subtracting each pixel value of the blurred image I _BLUR from each pixel value of the input image I _INPUT, and generates a difference image I _DELTA between the input image I _INPUT and the blurred image I _BLUR. Finally, an image obtained by adding each pixel value of the difference image I _DELTA to each pixel value of the input image I _INPUT is defined as an output image I _OUTPUT . Expression (C-5) shows a relational expression between the input image I _INPUT and the output image I _OUTPUT . In Expression (C-5), (I _INPUT · Gauss) represents the result of filtering the input image I _INPUT using a Gaussian filter of σ ′.

I _OUTPUT = I _INPUT + I _DELTA
= I _INPUT + (I _INPUT -I _BLUR )
= I _INPUT + (I _INPUT- (I _INPUT · Gauss) (C-5)
In step S205, the filtering target image Lw is handled as the input image I _INPUT to obtain a filtered image as the output image I _OUTPUT . In step S206, the ringing of the filtered image is removed to generate a corrected image Qw (the process in step S206 is the same as the process in step S77 in FIG. 14).

By using an unsharp mask filter, the edge of the input image (I _INPUT ) is enhanced, and an image sharpening effect is obtained. However, if the blurring degree at the time of generating the blurred image (I _BLUR ) is significantly different from the actual blur amount included in the input image, an appropriate blur correction effect cannot be obtained. For example, if the degree of blur at the time of blur image generation is larger than the actual blur amount, the output image (I _OUTPUT ) becomes an extremely sharp and unnatural image. On the other hand, if the blurring degree at the time of generating the blurred image is smaller than the actual blur amount, the sharpening effect is too weak. In this correction method, a Gaussian filter whose degree of blurring is defined by σ is used as the unsharp filter, and σ ′ corresponding to the image degradation function is used as σ of the Gaussian filter. For this reason, an optimal sharpening effect is obtained, and a corrected image from which blur is well removed is obtained. That is, it is possible to generate an image with less blur in appearance at a low calculation cost.

  It is also conceivable to apply so-called optical camera shake correction by driving a correction lens or an image sensor during continuous shooting. However, if the number of continuous shots is large, the total amount of camera shake from the start to the end of continuous shooting becomes very large, and the required drive amount of the correction lens and image sensor often exceeds their movable range. In the second half of continuous shooting, camera shake correction does not work at all. On the other hand, according to the above-described first to third correction methods, it is possible to perform appropriate camera shake correction on an arbitrary element image in the continuous-shot image group without such limitation.

<< 5th Example >>
Next, a fifth embodiment will be described. In the fifth embodiment, the operation of the imaging apparatus 1 during moving image shooting will be described.

  The above-described camera shake correction technology can also be used during movie shooting. On the other hand, at the time of moving image shooting, a part of each frame image acquired from the imaging unit 11 is cut out and the cut-out image is output to the display unit 15 or the like. A method of controlling the cutout position so as to cancel the positional deviation between cutout images is known. This method is also a type of camera shake correction. In the following description, in order to distinguish the camera shake correction by the cutout position control from the camera shake correction described in each of the above-described embodiments, the former is sometimes referred to as out-of-frame camera shake correction, and the latter is referred to as in-frame camera shake correction.

  22 and 23 are flowcharts showing the flow of operations of the imaging apparatus 1 during moving image shooting. The flow of this operation will be described with reference to these drawings. The processing of each step shown in FIGS. 22 and 23 is basically executed by the camera shake correction unit 19 under the control of the main control unit 13. However, some processes (display and the like) are executed mainly by other parts in the imaging apparatus 1.

  First, when a moving image shooting instruction is given by an operation on the recording button 17b in FIG. 1, the process proceeds to step S301. Therefore, the variable i is introduced, and 1 is assigned to the variable i, and then the process proceeds to step S302. The value of variable i represents the current frame number. The first, second, third,... Frames are sequentially visited in a predetermined frame period (for example, 1/60 seconds), and one frame image is acquired in each frame.

In step S302, a frame image in the current frame is extracted. The extracted frame image and Fw _i. For example, when the current frame is the first frame (ie, when i = 1), the frame image Fw ₁ is extracted, and when the current frame is the second frame (ie, when i = 2). A frame image Fw ₂ is extracted.

In subsequent step S303, it is confirmed whether or not i = 1. If the current frame is a moving image shooting start frame, that is, if i = 1, the process proceeds to step S304. On the other hand, if i ≠ 1, the process proceeds to step S307. In step 304, the extracted characteristic small areas from the frame image Fw _i, stored in the memory an image of the small area as a small image Fs _i. Since step S304 is reached only when i = 1, the small image Fs ₁ is extracted from the frame image Fw ₁ . For example, the central region when the respectively divided into three frame images Fw _i in the horizontal and vertical directions to extract a characteristic small area of 32 × 32 pixels. The significance of the characteristic small area is as described above. By using a characteristic small area instead of the entire area of the frame image, it is possible to reduce the memory usage while maintaining good alignment accuracy between subsequent frame images and blur information detection accuracy. .

In order to realize the above-described out-of-frame camera shake correction, a motion vector between frame images of adjacent frames is calculated during moving image shooting. The motion vectors calculated for the frame image Fw _i represented by VEC _i. The motion vector VEC _i is also a motion vector for an image Qw _i described later.

When i = 1, only one frame image has been acquired yet. Therefore, in step S305 following step S304, the horizontal and vertical components of the motion vector VEC ₁ for the frame image Fw ₁ and the image Qw ₁ are set to zero (the magnitude of VEC ₁ is set to zero). Further, in the subsequent step S306, without performing image stabilization in a frame, and stores in the memory the current frame image Fw _i as an image Qw _i. Accordingly, no intra-frame camera shake correction is performed on the frame image of the start frame. After step S306, the process proceeds to step S321 (see FIG. 23).

Prior to the description of the processing after step S321, the processing content after step S307 performed when i ≠ 1 will be described. In step S307, it is confirmed whether the current value of the variable i is smaller than the specified value N. Now, N is an integer of 3 or more, but it is possible to set N = 2. If i <N, the process proceeds from step S307 to step S308, and 1 is substituted for the variable b. On the other hand, if i ≧ N, the process proceeds from step S307 to step S309 to set the variable b to (i−N + 1). Is assigned. Step S308 or S309 after determining the value of the variable b in, in step S310, the extracted small area corresponding to the small image Fs _b extracted from a past frame image Fw _b from the current frame image Fw _i, the current frame , stored in the memory and the image in the small area extracted from the image Fw _i as a small image Fs _i. The search for the corresponding small area can be performed by the above-described method using a template matching method or the like. Thereafter, in step S311, from among the small image Fs _b ~Fs _i, selects the most blurring amount of small one small image is stored in the memory of the small image selected as a small image Rs. Thereafter, if the small image Fs _i corresponding to the current frame is different from the small image Rs, intra-frame camera shake correction for the current frame image Fw _i is performed (steps S312 and S313; details will be described later).

Now, considering the case where N = 5, a supplementary explanation will be given to the processing contents of steps S307 to S311. As will be apparent from the description below, when N = 5, camera shake correction of the current frame image is performed using frame images for five frames. In the case of N = 5, when 2 ≦ i ≦ 4, images for 5 frames have not yet been acquired, so 1 is substituted into the variable b (step S308), and four or less small images Fs _{b to} Fs _A small image with the smallest blur amount is selected from _i (= Fs _{1 to} Fs _i ) to be a small image Rs. The process of selecting the small image Rs corresponds to the process of selecting the reference image Rw described above (see step S16 in FIG. 4). On the other hand, for example, i = 6 in the case of, b = i-N + 1 = 6-5 + 1 = ( see step S309) from 2 second to sixth frame is focused in the small image Fs from the current frame image Fw ₆ In step S310 _The small image Fs ₆ corresponding to ₂ is extracted, and in step S311, the small image with the smallest blur amount is selected as the small image Rs from the five small images Fs _{2 to} Fs ₆ . Since camera shake is considered to cause uniform degradation on the entire image, the amount of blur of the frame image including the small image Rs is the smallest among the amounts of blur of the frame images Fw _{2 to} Fw ₆ .

As a method for selecting one small image as the small image Rs from the small images Fs _{2 to} Fs ₆ in step S 311, the reference image selection method described in the third embodiment can be used.
That is, for example, when adopting the first reference picture selection method corresponding to FIG. 8, handling the small image Fs ₂ ~Fs ₆ as a small image Cs ₂ to CS _6, the evaluation value Ka for small images Fs ₂ ~Fs _{6 2 to} Ka ₆ are calculated to estimate the shake amount of the small images Fs _{2 to} Fs ₆ (and the shake amount of the frame images Fw _{2 to} Fw ₆ ). For example, if the evaluation value Ka ₄ is the maximum among the evaluation values Ka _{2 to} Ka ₆ , the small image Fs ₄ is selected as the small image Rs.
Alternatively, in the case of employing the second reference picture selection method corresponding to FIG. 10, handling the small image Fs ₂ ~Fs ₆ as a small image Cs ₂ to CS _6, the evaluation value Kb for small images Fs ₂ ~Fs _{6 2} calculates the ～Kb ₆ estimates the blur amount of the small image Fs ₂ ~Fs ₆ (and blurring amount of the frame image Fw ₂ ~Fw _6). For example, if the evaluation value Kb ₄ is the maximum among the evaluation values Kb _{2 to} Kb ₆ , the small image Fs ₄ is selected as the small image Rs.
In addition, it is possible to select the small image Rs in consideration of both the evaluation values Ka _{2 to} Ka ₆ and the evaluation values Kb _{2 to} Kb ₆ .

After obtaining the small image Rs by the selection process of step S311, at step S312, the a small image Fs _i extracted from the small image Rs and the current frame image Fw _i to determine whether it is the same. If they are the same, that is, if the current frame image has the smallest amount of shake among the most recent N frame images including the current frame image, the process proceeds to step S306 without performing intra-frame image stabilization. migrating the current frame image Fw _i after stored in the memory as an image Qw _i to step S321 of FIG. 23. On the other hand, if it is determined in step S312 that the small image Rs and the small image Fs _i are not the same, the process proceeds to step S313 and the current frame image Fw _i and the past frame image including the small image Rs are displayed based on the current frame image Fs _i. It performs frame-in image stabilization with respect to the frame image Fw _i, moves from the stored in the memory in step S321 the current frame image Fw _i after camera shake compensation in the frame as an image Qw _i. For example, when the small image Rs is the small image Fs ₄ , the past frame image including the small image Rs indicates the frame image Fw ₄ .

The method described in the fourth embodiment can be used as the intra-frame camera shake correction method in step S313. That is, using any of the first to third correction method described in the fourth embodiment deals with a past frame image and including the small image Rs the current frame image Fw _i as a correction target image Lw as the reference image Rw by, you get an image Qw _i as corrected image Qw. When the first correction method corresponding to FIG. 14 is adopted, the small image Fs _i extracted from the current frame image Fw _i is used as the small image Ls to be extracted in step S71 of FIG. In addition, the small image Rs selected in step S311 may be used as the small image Rs to be extracted in step S72 of FIG.

Processing after step S321 in FIG. 23 will be described. In step S321, using a representative point matching method, a block matching method, or the like, based on the image data of the images Qw _i-1 and Qw _i , a motion vector representing the amount of positional deviation of the image Qw _i viewed from the image Qw _i-1. VEC _i is calculated. This motion vector VEC _i is a two-dimensional quantity including a horizontal component and a vertical component. As a method for calculating the positional deviation amount (motion vector) between two images, the method already described in other embodiments can be used. In particular, for example, this calculation may be performed using characteristic small regions corresponding to each other extracted from the images Qw _i-1 and Qw _i .

In subsequent step S322, a cumulative vector W _i obtained by accumulating the motion vectors VEC _{1 to} VEC _i calculated so far is calculated. As shown in FIG. 24, the accumulated vector W _i sequentially connects the motion vectors VEC _{1 to} VEC _i while matching the end point of the focused motion vector with the start point of the motion vector obtained next. Is generated.

After the accumulated vector W _i is obtained, in step S323, as shown in FIG. 25, a fixed size cut-out centered on a point 351 moved by the amount represented by the accumulated vector W _i from the center 350 of the image Qw _i. set the frame 352 is extracted from the image Qw _i as an image Qc _i cut an image of the trimming frame 352. For example, horizontal and vertical size of the clipping frame 352 is 80% of their image Qw _i.

Then, in step S324, the motion blur correction portion 19, as cut image Qc _i is recorded on and the recording medium 16 is displayed on the display unit 15, the image data of the clipped image Qc _i via the main control unit 13 displays Output to the unit 15 and the recording medium 16.

  Thereafter, in step S325, it is determined whether a scene change has occurred, that is, whether the subject to be photographed has changed significantly. For example, when a change in the optical zoom magnification occurs or when a panning or tilting operation is applied to the image pickup apparatus 1, it is determined that a scene change has occurred, and step S326 is performed after substituting zero for the variable i. The process proceeds to S327. If it is determined that no scene change has occurred, the process proceeds from step 325 to step S327 as it is. In step S327, the main control unit 13 confirms whether or not the end of the moving image shooting is instructed by an operation on the operation unit 17, and when the instruction is given, the moving image shooting operation is ended while the instruction is made. If not, 1 is added to the variable i in step S328, and the process returns to step S302 (FIG. 22) to repeat the above-described processing. However, the loop process composed of steps S302 to S313 and S321 to S328 is executed only once in one frame.

  As described above, by performing in-frame image stabilization, even if camera shake occurs during the exposure period of each frame image, the user can be provided with a clear video with reduced image blur in each frame image. Is possible. Further, since the correction target image and the frame image serving as the reference image are images that should be acquired originally, it is not necessary to acquire a separate dedicated image in order to perform blur correction. Therefore, there is no extra load on the shooting time and memory usage. In addition, by performing the out-of-frame camera shake correction by the cutout position control, the composition of the cutout image of the start frame (or the reference frame) and the current frame becomes the same. That is, camera shake between frames is also corrected.

According to the above-described method, there is an advantage that camera shake correction can be performed in real time during moving image shooting. However, each frame image before the intra-frame camera shake correction is temporarily recorded in the recording medium 16, and after moving image shooting is finished, it is also possible to generate an image Qw _i of number of sheets of frame images forming the moving image by performing the image stabilization in the frame. For this, it is sufficient to simply execute the above-described processing shown in FIGS. 22 and 23 using each frame image recorded in the recording medium 16 before the intra-frame camera shake correction. In this case, in-frame image stabilization for a frame image of a certain frame of interest is executed using the frame image of the past frame of the frame of interest as shown in FIG. The same applies when camera shake correction is performed).

  However, after the moving image shooting is completed, the intra-frame camera shake correction for the frame image of a certain frame of interest can be performed using the frame images of the frames before and after the frame of interest. That is, frame images for a total of N frames including the target frame and frames before and after the target frame are referenced, and a frame image with the smallest blur amount is extracted from the frame images for the N frames. Then, the extracted frame image with the smallest blur amount is treated as the reference image Rw and the frame image of the target frame is treated as the correction target image Lw, and the intra-frame camera shake correction may be performed on the frame image of the target frame (however, refer to This is not possible if the image Rw and the correction target image Lw are the same). By using the frames before and after the frame of interest in this way, more accurate intra-frame camera shake correction can be performed. This is because intra-frame camera shake correction is performed using a reference image having a relatively short photographing time difference from the correction target image.

Although an example in which the motion vector VEC _i is calculated using the image data of the images Qw _i-1 and Qw _i after the intra-frame camera shake correction (see step S321 in FIG. 23), the frame before the intra-frame camera shake correction is shown. using the image data of the image Fw _i-1 and Fw _i calculates a motion vector between frame images Fw _i-1 and the frame image Fw _i, which as the motion vector VEC _i to be calculated in step S321 You may make it handle. For example, the motion vector VEC _i may be calculated based on the extraction result in step S310 (or S304 and S310). That is, the position of the small image Fs _i-1 in the frame image Fw in _i-1, from the amount of positional deviation between the position of the small image Fs _i in the frame image Fw _i to calculate the motion vector VEC _i Also good.

In the selection process in step S311, the third or fourth reference image selection method described in the third embodiment can be used. This will be described taking the case where N = 5 and i = 6 as an example. When using the third reference picture selection method, and calculates a rotation angle theta ₂ through? ₅ each of the past frame image Fw ₂ ~Fw ₅ for the current frame image Fw ₆ to be a correction target image, the rotation angle Reciprocals of θ _{2 to} θ ₅ are obtained as evaluation values Kc _{2 to} Kc ₅ . Then, among the frame images Fw _{2 to} Fw ₅ , the small image in the frame image corresponding to the large evaluation value is preferentially selected as the small image Rs. When using a fourth reference picture selection method, for each frame image Fw ₂ ~Fw _5, obtains a photographing time difference for the current frame image Fw _6, the photographing time difference obtained for each frame image Fw ₂ ~Fw ₅ The reciprocal is obtained as evaluation values Kd _{2 to} Kd ₅ . Then, among the frame images Fw _{2 to} Fw ₅ , the small image in the frame image corresponding to the large evaluation value is preferentially selected as the small image Rs.

If the selection processing in step S311 using the third or fourth reference picture selection method, at step S311, from among the small image Fs ₂ ~Fs _6, most blurring amount of small small image is selected Not exclusively. However, to perform the frame-in image stabilization in step S313 is "shake amount of the small image Rs is smaller than the shift amount of the small image Fs _i extracted from the current frame image Fw _i" must satisfy the requirements that is there. Accordingly, even when the third or fourth reference image selection method is used, the evaluation value (Ka ₂ or the like) and / or the second reference image selection method is used so that this necessary condition is satisfied. The evaluation value (Kb ₂ etc.) according to the reference image selection method should be preferentially considered. As described in the third embodiment with the expression (A-1), the evaluation value (Kc) by the third reference image selection method is supplementarily used after mainly using them (Ka ₂ , Kb _2, etc.). ₂ ) and / or an evaluation value (Kd ₂ or the like) obtained by the fourth reference image selection method, and an appropriate small image Rs may be selected after satisfying the above-mentioned necessary conditions.

In addition, since the third and fourth reference image selection methods are based on the premise that the current frame image and the frame image including the small image Rs are different, in step S311, the third and / or fourth reference image selection method is selected. In step S312, the following determination is made (N = 5 and i = 6 are taken as an example). That is, it is determined whether a frame image having a smaller blur amount than the current frame image Fw ₆ is present in the frame images Fw _{2 to} Fw ₅ . If it does not exist, the process proceeds to step S306. If it exists, the process proceeds to step S313. The amount of blur included in the frame image can be estimated using the method described in the description of the first or second reference image selection method.

<< Sixth Example >>
Next, a sixth embodiment will be described. In the sixth embodiment, application examples, modifications, and annotations for the above-described contents will be described.

First, technical matters useful for estimating the amount of image blur will be described. In a third embodiment, it extracts the small image Cs _i from the element image Cw _i, said to be able to estimate the blur amount for the entire element image Cw _i based on the edge intensity or contrast of the small image Cs _i (FIG. 8 and (See FIG. 10). In this case, an example of extracting the small image Cs _i from the vicinity of the center of the element images Cw _i, it is not always necessary to the small image Cs _i is extracted from the vicinity of the center of the element image Cw _i. For example, the following may be performed. For the sake of concrete explanation, consider the case where N = 5, that is, the case where the continuous shot image group is formed from _five element images Cw _{1 to} Cw ₅ .

First, by using a block matching method or the like, an optical flow between two element images Cw _i−1 and element images Cw _i photographed continuously in time is obtained. The optical flow here includes a bundle of motion vectors between contrast images. FIG. 27 shows an example of the obtained optical flow. Next, a small image extraction region in the series of element images Cw _{1 to} Cw ₅ is detected based on the obtained optical flow. The small image extraction area is defined in each element image. Then, so as to extract the small image Cs _i from the small-image-extraction area of the element image Cw _i.

For example, during continuous shooting, when the imaging apparatus 1 is substantially fixed and a person located near the center of the shooting area is moving in real space, a significant motion vector is detected at a position corresponding to the person. On the other hand, such a motion vector is not detected in the peripheral portion occupying most of the element image. A significant motion vector is a motion vector having a magnitude greater than or equal to a predetermined magnitude, and simply means a motion vector whose magnitude is not zero. FIG. 27 shows an optical flow in such a case. In this case, the region where no significant motion vector is detected is a still subject region where a stationary subject is depicted in real space, and this still subject region is detected as a small image extraction region. The inside of the area surrounded by the broken line in the element images Cw _{1 to} Cw ₅ in FIG. 27 corresponds to the detected small image extraction area.

  Also, for example, during continuous shooting, a person located near the center of the shooting area is moving in the right direction in real space, and the housing (not shown) of the imaging device 1 follows the person in the right direction. 28, as shown in FIG. 28, no significant motion vector is detected at a position corresponding to the person, while a significant motion vector is detected in the peripheral portion (background portion) that occupies most of the element image. Detected. Moreover, the magnitude and direction of the detected significant motion vector are uniform. In this case, a region where the significant motion vector is detected, that is, a dominant motion region in the image is detected as a small image extraction region (resulting in the same small image extraction region as that shown in FIG. 27). Area is detected).

  Also, for example, when all the subjects and the imaging device 1 are stationary in real space during continuous shooting, no significant motion vector is detected in the entire region of the element image. In this case, the entire region of each element image is a still subject region, and this still subject region is detected as a small image extraction region. Further, for example, during continuous shooting, when all subjects are stationary in real space but the casing of the imaging device 1 is swung to the right, or the imaging device 1 is stationary in real space. When all the subjects are moving in the left direction evenly, as shown in FIG. 29, a significant motion vector having a uniform size and orientation is detected over the entire element image. In this case, it is determined that the entire region of the element image is a dominant motion region, and the dominant motion region is detected as a small image extraction region.

  As described above, the small image extraction region can be specified by statistically processing a plurality of motion vectors forming the optical flow.

  Alternatively, a moving subject (such as a person) moving in real space may be detected, and an area where the moving subject is not located may be detected as a small image extraction region. If a known moving subject tracking technique based on image processing is used, it is possible to detect and track the moving subject from the output of the imaging unit 11 including the image data of the element image.

A small image Cs _i is extracted from a region where a moving subject that moves irregularly in the imaging region is drawn, and the evaluation value (Ka _i or Kb _i ) described in the third embodiment is extracted based on the small image Cs _i. If the calculation is performed (see FIGS. 8 and 10), the evaluation value is affected by the motion of the moving subject, and the blur amount estimation accuracy for the small image Cs _i and the element image Cw _i deteriorates. As a result, there is an increased possibility that selection of an element image with a small blur amount, that is, selection of the reference image Rw will fail. However, as described above, if to extract small image Cs _i from the detection to the small-image-extraction area of small-image-extraction areas (static subject area or dominant motion areas), in the element image Cw _i Even if an irregularly moving moving subject is included, an element image with a small amount of blur can be accurately extracted as the reference image Rw.

Further, (see FIG. 13) even when calculating the evaluation value Kc _i based on the rotation angle of the element images Cw _i, the plurality of small areas are extracted from the correction target object Lw, the evaluation value Kc _i Gado subject In order to avoid the influence of movement, the plurality of small areas may be extracted from the small image extraction area.

Although the small image extraction method has been described focusing on continuous shooting, the same concept can be applied to moving image shooting corresponding to the fifth embodiment. That is, using the detection method of the small-image-extraction area as described above, by extracting the small image Fs _b ~Fs _i which is referred to in step S311 of FIG. 22 from the small-image-extraction area of the frame image Fw _b ~Fw _i It is good to leave.

  Other application examples, modified examples, and annotations for the above-described contents are listed below as annotations 1 to 4. The matters described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
Although the camera shake correction method in continuous shooting was performed, this method can be applied to bracket shooting as it is. In this case, the continuous shooting described above may be read as bracket shooting, and N captured images continuously acquired by bracket shooting may be regarded as _N element images Cw _{1 to} Cw _N. In this case, the number of element images (that is, the value of N) also coincides with the number of images to be acquired by bracket shooting, as in continuous shooting.

  In continuous shooting, a plurality of shot images are continuously acquired without changing various settings relating to shooting. The various settings relating to shooting are the focal length of the image pickup unit 11, the exposure amount of the image pickup unit 11, white balance, and the like. On the other hand, as is generally known, in bracket shooting, a plurality of shot images are continuously acquired while changing these settings. For example, when performing bracket photography with respect to the exposure amount, continuously changing the aperture of the imaging unit 11 (not shown) sequentially so that a plurality of photographed images are acquired with different exposure amounts. Take a picture.

[Note 2]
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values.

[Note 3]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, the function of the camera shake correction unit 19 can be realized by hardware, software, or a combination of hardware and software, and the function can also be realized by an external device (such as a computer) of the imaging device 1. It is. All or part of the functions realized by the camera shake correction unit 19 are described as a program, and the program is executed on a program execution device (for example, a computer) to realize all or part of the functions. It may be. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 4]
Regarding continuous shooting or bracket shooting, for example, the following can be considered. An image acquisition unit that acquires N element images by continuous shooting or bracket shooting, and a correction target image extraction unit (correction target image selection unit) that executes the process of step S15 in FIG. 4 to extract a correction target image. 4. Reference image extraction means (reference image selection means) for executing the process of step S16 in FIG. 4 to extract a reference image; Correction process means for generating a corrected image by executing the process in step 18 of FIG. Is included in the imaging apparatus 1 of FIG. This camera shake correction device is mainly formed by the camera shake correction unit 19, or is formed by the camera shake correction unit 19 and the main control unit 13. Further, the imaging apparatus 1 includes an association unit that associates the corrected image with the group of continuously shot images by executing the process of step S19 in FIG.

Regarding moving image shooting, for example, the following can be considered. Image acquisition means for acquiring a plurality of frame images forming a moving image by moving image shooting, and correction processing means for generating a correction image (image Qw _i ) from the correction target image and the reference image by performing the processing of step S313 in FIG. And a reference image extraction unit (reference image selection unit) that performs the process of step S311 in FIG. 22 and extracts a reference image based on the processing result. Yes. This camera shake correction device is mainly formed by the camera shake correction unit 19, or is formed by the camera shake correction unit 19 and the main control unit 13. The shake correction apparatus further includes a misalignment detection unit that performs the processes of steps S321 and S322 in FIG. 23 and a cut-out image output unit that performs the processes of steps S323 and S324 in FIG.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. FIG. 6 is a conceptual diagram of a camera shake correction method for a continuous shot image group according to the embodiment of the present invention. It is the table | surface which showed the outline of the 1st-6th Example regarding this invention. 4 is a flowchart showing a flow of still image camera shake correction operation according to the first embodiment of the present invention. It is a figure for demonstrating the best shot image selection method which concerns on 2nd Example of this invention, and is a figure showing one element image which forms a continuous-shot image group. FIG. 6 is an enlarged view near the eyes of a person in the element image of FIG. 5. It is a figure for demonstrating the best shot image selection method which concerns on 2nd Example of this invention, and is a figure showing one element image which forms a continuous-shot image group. It is a flowchart showing the flow of operation | movement regarding the 3rd Example of this invention regarding the 1st reference image selection method. It is a conceptual diagram which assists operation | movement description corresponding to FIG. It is a flowchart showing the flow of operation | movement regarding 2nd Example of this invention regarding the 2nd reference image selection method. It is a figure for demonstrating the significance of the selection operation | movement corresponding to FIG. 10, and is a figure which shows a clear element image and a blurred unclear element image with a big blur. It is a figure which shows the brightness | luminance histogram corresponding to each element image of FIG. It is a figure for demonstrating the selection operation | movement of the reference image by the 3rd reference image selection method concerning 3rd Example of this invention. FIG. 14 is a flowchart illustrating an operation flow of camera shake correction processing according to the first correction method according to the fourth embodiment of the present invention. FIG. 15 is a detailed flowchart of a Fourier iteration method, which is a part of the camera shake correction processing of FIG. 14. It is a block diagram of the site | part which implements the Fourier iteration method of FIG. FIG. 14 is a flowchart illustrating an operation flow of camera shake correction processing according to a second correction method according to the fourth embodiment of the present invention. FIG. 18 is a conceptual diagram of camera shake correction processing corresponding to FIG. 17. FIG. 14 is a flowchart illustrating an operation flow of camera shake correction processing according to a third correction method according to the fourth embodiment of the present invention. FIG. 20 is a conceptual diagram of camera shake correction processing corresponding to FIG. 19. It is a figure which concerns on 4th Example of this invention and shows a one-dimensional Gaussian distribution. It is a flowchart showing the flow of the operation | movement of the moving image camera shake correction which concerns on 5th Example of this invention. It is a flowchart showing the flow of the operation | movement of the moving image camera shake correction which concerns on 5th Example of this invention. FIG. 10 is a diagram illustrating a relationship between a plurality of motion vectors and their accumulated vectors according to the fifth embodiment of the present invention. It is a figure for demonstrating cutout position control which concerns on 5th Example of this invention, and implement | achieves out-of-frame camera shake correction. In the fifth embodiment of the present invention, a diagram showing that the frame image of the target frame is corrected using the past frame image, and the frame image of the target frame is corrected using the frame images of the previous and subsequent frames. It is the figure which showed that. FIG. 20 is a diagram illustrating an example of an optical flow between each element image forming a continuous-shot image group and adjacent element images according to the sixth embodiment of the present invention. It is a figure which concerns on 6th Example of this invention and shows the other example of the optical flow between adjacent element images. It is a figure which concerns on 6th Example of this invention and shows the further another example of the optical flow between adjacent element images. It is a block diagram of the structure which implement | achieves the conventional Fourier iteration method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Main control part 19 Camera shake correction part

Claims (12)

Image acquisition means for acquiring a plurality of original images by continuous shooting or bracket shooting using an imaging means;
Correction target image extraction means for extracting one original image as a correction target image from the plurality of original images;
Among the plurality of original images, each original image excluding the correction target image is set as a candidate reference image, and among the plurality of candidate reference images, a candidate reference image whose amount of blur is smaller than that of the correction target image is selected. Reference image extraction means for extracting as a reference image;
A blur correction apparatus comprising: correction processing means for correcting blur of the correction target image based on the correction target image and the reference image. The correction target image extracting means includes
The face orientation of the person in each original image,
The person's open eye state in each original image,
The facial expression of the person in each original image, and
Composition of each original image,
The shake correction apparatus according to claim 1, wherein the correction target image is extracted based on at least one of the two. Image acquisition means for acquiring a moving image consisting of a plurality of frame images by continuous shooting using an imaging means;
A correction processing means for correcting a shake included in the correction target image, with the frame image in the frame of interest as a correction target image among the plurality of frame images;
Each frame image in a frame in the vicinity of the target frame is used as a candidate reference image, and a reference image that extracts a candidate reference image having a smaller blur amount than that of the correction target image as a reference image from among a plurality of candidate reference images. Extracting means,
The blur correction apparatus, wherein the correction processing unit corrects the blur of the correction target image based on the correction target image and the reference image. The reference image extraction means includes
Edge strength of the correction target image and each candidate reference image, and
The contrast between the correction target image and each candidate reference image,
4. The reference image is extracted by estimating a blur amount between the correction target image and each candidate reference image based on at least one of the reference image. Blur correction device. The reference image extraction means includes
The rotation angle of each candidate reference image with respect to the correction target image, and
Difference in shooting time between each candidate reference image and the correction target image,
The blur correction device according to claim 4, wherein the reference image is extracted by further using at least one of the reference image. The correction processing unit creates blur information corresponding to the blur of the correction target image based on the correction target image and the reference image, and corrects the blur of the correction target image based on the blur information. The shake correction apparatus according to claim 1, wherein: The correction processing unit corrects the blur of the correction target image by synthesizing a luminance signal of the reference image with a color signal of the correction target image. The camera shake correction device according to claim 1. The correction processing unit corrects the blur of the correction target image by sharpening the correction target image using the reference image. Blur correction device. The correction image obtained by correcting the blur of the correction target image is automatically output to the display unit after the continuous shooting or the bracket shooting is finished. The described blur correction device. The correction processing means corrects blur included in each frame image by sequentially handling each frame image forming the moving image as the correction target image,
The blur correction device is
Cut out a part of each frame image after the blur correction, and a cut image output unit that outputs each cut image to a display unit or a recording medium;
A positional deviation detecting means for detecting a positional deviation between adjacent frames based on a frame image of the adjacent frame before or after blur correction, and
The blur correction device according to claim 3, wherein the cut-out image output unit sets a cut-out position of the cut-out image based on the detected positional deviation. Imaging means;
An image pickup apparatus comprising: the shake correction apparatus according to claim 1. Imaging means;
An image pickup apparatus comprising the shake correction apparatus according to claim 1 or 2,
Association means for associating the corrected images obtained by correcting the blur of the plurality of original images and the correction target image with each other;
An operation receiving means for receiving an operation on the associated image;
An image pickup apparatus that applies the process based on the operation to all of the associated images at once.