JP2009153046A - Blur correcting device and method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To apply a blur correction to consecutively taken images inexpensively and highly precisely, and to reduce the increase of required memory capacity caused when correcting blurs of consecutively taken normal-exposure images by using a short-exposure image. <P>SOLUTION: An image sequence to be recorded constituted of (m) normal-exposure images and (n) short-exposure images is consecutively taken in accordance with consecutive photographing instructions and image data thereof are recorded on a recording medium ((m) and (n) are integers). In such a case, a ratio between the number of normal-exposure images and the number of short-exposure images to be taken is set so as to establish m<n. Furthermore, based on a displacement amount (D<SB>A</SB>) between preview images taken before consecutive photographing, the ratio of (n) in (m) is variably set. When the displacement amount is great, the ratio is increased but when the displacement amount is small, the ratio is decreased. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a camera shake correction apparatus, a camera shake correction method, and an imaging apparatus that correct camera shake caused by camera shake or the like.

  The camera shake correction technique is a technique for reducing camera shake at the time of photographing, and is regarded as important as a differentiation technique in an imaging apparatus such as a digital still camera. As one type of camera shake correction, there is optical camera shake correction in which camera shake is detected by a camera shake detection sensor and a correction lens and an image sensor are driven based on the detection result.

  On the other hand, as special shooting for still images, there is continuous shooting (including so-called bracket shooting) in which an imaging unit composed of a CCD or the like continuously performs shooting. It is also conceivable to apply optical camera shake correction to such continuous shooting. However, if the number of continuous shootings is large, the total amount of camera shake from the start to the end of continuous shooting will be very large, and the required drive amount of the correction lens and image sensor may exceed their movable range. In this case, camera shake correction is not effective at the second half of continuous shooting. Further, the use of the camera shake detection sensor itself increases the cost.

  On the other hand, electronic image stabilization that does not use an image stabilization sensor has also been developed. In this electronic camera shake correction, an image obtained by photographing is analyzed to detect camera shake, and camera shake correction is performed based on the detection result.

  As conventional methods for electronic image stabilization, there are the following methods. In one conventional method, a plurality of sub-images are captured before and after the main image to be corrected, the blur information at the time of main image shooting is estimated from the sub-images, and the blur of the main image is estimated from the blur information. It correct | amends (for example, refer patent document 1). However, in this method, since the blur of the main image is estimated from the amount of motion between the sub-images obtained before and after the main image (the amount of motion including the exposure interval), the detection accuracy of the blur is lowered.

  In another conventional method, camera shake is detected from an image obtained by converting a camera shake image into a two-dimensional frequency space (see, for example, Patent Document 2). Specifically, the image obtained by the conversion is projected onto a circle centered on the origin of the frequency coordinate, and the magnitude and direction of the shake are obtained from the projection data. However, with this method, only linear and constant velocity blur can be estimated, and when the frequency component in a specific direction of the subject is small, detection of the blur direction may fail.

  A method for estimating camera shake information (a point spread function or an image restoration filter) representing a camera shake during shooting from a single camera shake image, and generating a restored image free from the camera shake information and the camera shake image by digital signal processing. Has also been proposed. Among these types of methods, a method using the Fourier iteration method is disclosed in Non-Patent Document 1 below. FIG. 26 shows a block diagram of a configuration for realizing the Fourier iteration method. In the Fourier iteration method, the final restored image is estimated from the degraded image by repeatedly executing Fourier transform and inverse Fourier transform while correcting the restored image and the point spread function (PSF). In order to execute the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image). Generally, the initial restored image is a random image or a degraded image as a camera shake image. Used.

JP 2001-346093 A JP-A-11-27574 G. R. Ayers and J. C. Dainty, "Iterative blind deconvolution method and its applications", OPTICS LETTERS, 1988, Vol. 13, No. 7, p.547-549

  Electronic camera shake correction that does not require a camera shake detection sensor can reduce the cost compared to optical camera shake correction, but it has not been proposed yet that is suitable for continuous shooting.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a shake correction apparatus and an imaging apparatus that can realize shake correction with respect to continuously shot images at low cost and high accuracy.

  The blur correction apparatus according to the present invention includes m first images and an exposure time of the first image in an image sequence arranged in time series obtained by continuous shooting by an imaging unit according to a predetermined continuous shooting instruction. Photographing control means for controlling the imaging means so that n second images obtained by photographing with an exposure time shorter than that are included (where m and n are integers and m> n ≧ 1), and m Correction target image extraction means for extracting one first image as a correction target image from among the first images, and one second image from the n second images as a reference image. Reference image extracting means, and correction processing means for correcting blur of the correction target image based on the correction target image and the reference image are provided.

  Thereby, blur correction is performed on an image obtained by continuous shooting. Since a camera shake detection sensor is unnecessary, a camera shake correction device can be formed at low cost. In addition, it is possible to use a reference image that is assumed to have less blur due to the short exposure time, and to correct the correction target image with the edge state of the reference image as a target. .

  On the other hand, in order to reduce the effects of camera shake and subject movement, it is desirable to photograph the second image that is the target of camera shake correction immediately before or after each first image. However, if one second image is taken each time one first image is taken, twice the memory capacity is required as compared to the case where the second image is not taken. In addition, the photographing time required to obtain m first images is increased. Considering this, the number of second images (ie, n) is made smaller than the number of first images (ie, m). Thereby, an increase in required memory capacity is suppressed, and an increase in photographing time for obtaining m first images is suppressed.

  Specifically, for example, it further includes first misalignment detection means for detecting misalignment amounts between a plurality of images obtained from the imaging means before photographing the image sequence, and the photographing control means includes a first position Based on the detection result of the deviation detection means, the ratio of n to m is set.

Alternatively, for example, the m first images are formed from m _A first images and m _B first images taken after the m _A first images (provided that, m _A and m _B are integers, and m _A ≧ 2, m _B ≧ 1, m = m _A + m _B ), and the blur correction device detects the amount of misalignment between the first images of m _A sheets. A first misregistration detecting means that sets the ratio of n to m _B based on the detection result of the first misregistration detection means.

If the ratio of n to m or the ratio of n to m _B is set using the first misregistration detection means, an increase in necessary memory capacity and the like can be suppressed as much as possible while maintaining high-precision blur correction. Is possible.

  Alternatively, for example, the shooting control unit may set n = 1 so that only one second image is acquired before or after shooting the m first images, or n = 2 and the m images are acquired. The imaging unit is controlled so that the second image is acquired one by one before and after the first image is captured.

  According to this, it is possible to avoid an increase in the shooting interval of the first image resulting from the shooting of the second image.

  More specifically, for example, the image processing apparatus further includes a second misregistration detection unit that detects a misregistration amount between the correction target image and the reference image, and the correction processing unit detects the second misregistration detection unit. Using the result, the blur of the correction target image is corrected through the correction of the misalignment between the correction target image and the reference image, and the second misalignment detection unit is configured to detect the m first images. And when the n second images are arranged in time series, when another image exists between the first image as the correction target image and the second image as the reference image, the other The accumulation target image sequence composed of the image, the correction target image and the reference image is specified, and the amount of positional deviation between two temporally adjacent images forming the accumulation target image sequence is obtained and accumulated. The correction target image and the reference Detecting the positional deviation amount between the images.

  As a result, it is possible to obtain a positional deviation amount between the correction target image and the reference image with high accuracy, and as a result, high-accuracy blur correction is expected.

  Further, for example, it further comprises recording control means for storing in the recording means m image files each recording the image data of the m first images, wherein the recording control means adds the n second images to the n second images. The image data of one included second image is recorded in only one of the m image files.

  As a result, the image data of the second image is not redundantly recorded, and an increase in the required recording capacity resulting from the recording of the second image is suppressed.

  According to another aspect of the present invention, there is provided an image pickup apparatus including an image pickup unit that outputs a signal representing an image by shooting, and the shake correction device that receives an output signal of the image pickup unit.

  Another imaging apparatus according to the present invention includes an imaging unit that outputs a signal representing an image by shooting, an instruction receiving unit that receives a predetermined continuous shooting instruction, and image data of an image based on an output signal of the imaging unit And recording means for recording m first images according to the continuous shooting instruction, and recording the image data of the m first images in the recording means. In the imaging device (where m is an integer), the exposure time of the m first images and the first image in an image sequence arranged in time series obtained by continuous shooting according to the continuous shooting instruction. An imaging control means for controlling the imaging means so that n second images obtained by imaging with a short exposure time are included (where n is an integer and m> n ≧ 1), and the image of the image sequence Record data in the recording means Recording control means for performing correction, image correction means for extracting one first image from the m first images as a correction target image, and one second image from the n second images. Reference image extraction means for extracting two images as reference images, and correction processing means for correcting blur of the correction target image based on the correction target image and the reference image are provided.

  Further, the blur correction method according to the present invention includes the m first images and the first images in the time series arranged in a time series obtained by continuous shooting by the imaging unit according to a predetermined continuous shooting instruction. The imaging means is controlled so that n second images obtained by photographing with an exposure time shorter than the exposure time are included (where m and n are integers and m> n ≧ 1), While extracting one first image from the first image as a correction target image, extracting one second image from the n second images as a reference image, The blur of the correction target image is corrected based on the reference image.

  According to the present invention, it is possible to realize blur correction on continuously captured images with low cost and high accuracy.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to fourth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital still camera that can capture and record a still image, or a digital video camera that can capture and record a still image and a moving image.

  The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, and an operation unit 17. The operation unit 17 is provided with a shutter button 17a.

  FIG. 2 shows an internal configuration diagram of the imaging unit 11. The imaging unit 11 drives and controls the optical system 35, the diaphragm 32, the imaging element 33 including a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like, and the optical system 35 and the diaphragm 32. And a driver 34. The optical system 35 is formed from a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 drives and controls the positions of the zoom lens 30 and the focus lens 31 and the opening degree of the diaphragm 32 based on the drive control signal from the main control unit 13, so that the focal length (angle of view) and the imaging unit 11 are controlled. The focal position and the amount of light incident on the image sensor 33 are controlled.

  The image sensor 33 photoelectrically converts an optical image representing a subject incident through the optical system 35 and the diaphragm 32 and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and in each photographing, each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time. An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1. In the following description, “exposure” means exposure of the image sensor 33.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and functions as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image”). The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15. The main control unit 13 outputs an exposure time control signal for controlling the exposure time of the image sensor 33 to the imaging unit 11.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device including a liquid crystal display panel and the like, and displays a photographed image, an image recorded on the recording medium 16, and the like under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and stores captured images and the like under the control of the main control unit 13.

  The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing photographing and recording of an image.

  The shutter button 17a is formed so that it can be pressed in two stages. When the photographer lightly presses the shutter button 17a, the shutter button 17a is half pressed, and when the shutter button 17a is further pressed from this state, the shutter button 17a is fully pressed.

  The photographed image can include a camera shake caused by camera shake. The main control unit 13 includes a shake correction unit (blur correction device) that performs a shake correction process. The shake correction unit outputs an output signal of the imaging unit 11 without using a camera shake detection sensor such as an angular velocity sensor. Provided with a function of correcting blurring of a captured image based on image data obtained from. Hereinafter, a captured image whose shake is to be corrected by the blur correction unit is particularly referred to as a “correction target image”, and an image after the blur correction is referred to as a “blur correction image”.

  The operation mode of the imaging device 1 includes a shooting mode capable of shooting and recording a still image or a moving image, and a reproduction mode for reproducing and displaying the still image or moving image recorded on the recording medium 16 on the display unit 15. . Transition between the modes is performed according to the operation on the operation unit 17.

  When the user gives a continuous shooting instruction to the imaging apparatus 1 in the shooting mode, a sequence of captured images arranged in time series to be recorded on the recording medium 16 is obtained by continuous shooting. This captured image sequence is hereinafter referred to as a “recorded image sequence”. The continuous shooting instruction is given to the imaging apparatus 1 by a predetermined operation on the shutter button 17a. The recorded image sequence is composed of a plurality of photographed images arranged in time series, and a photographed image to be a correction target image is included in the plurality of photographed images. Of course, it is also possible to give a single shooting instruction to the imaging apparatus 1 by a predetermined operation on the operation unit 17, and when a single shooting instruction is given, one shot image is acquired and recorded in the recording medium 16. Is done. In the following description, it is assumed that a continuous shooting instruction is given to the imaging apparatus 1 by an operation of pressing the shutter button 17a.

  Regardless of whether there is a continuous shooting instruction or a single shooting instruction, in the shooting mode, the imaging unit 11 sequentially performs shooting at a predetermined frame period (for example, 1/60 seconds), and the shooting is obtained. Each captured image other than the recorded image sequence is referred to as a preview image. The preview images obtained one after another are sequentially updated and displayed on the display unit 15.

  FIG. 3 shows a temporal relationship between the shooting period of the preview image and the shooting period of the recorded image sequence in the shooting mode. In a state where the preview image sequence is acquired by sequential shooting, when a continuous shooting instruction is given, the recorded image sequence is captured, and when the recorded image sequence is captured, the preview image sequence is captured again. Transition to the state.

  Hereinafter, first to fourth examples will be described as examples for explaining the operation of the imaging apparatus 1 in detail. The matters described in one embodiment can be applied to other embodiments as long as no contradiction arises.

<< First Example >>
First, the first embodiment will be described. Please refer to FIG. 4 and FIG. FIG. 4 is a partial block diagram of the imaging apparatus 1 according to the first embodiment. 4 are provided in a shake correction unit (see FIG. 1) included in the main control unit 13.

[Operation when shooting]
First, the operation during shooting will be described. FIG. 5 is a flowchart illustrating the flow of the shooting operation of the image pickup apparatus 1 in the shooting mode according to the first embodiment. During the shooting operation, the first misalignment detection unit 51, the shooting control unit 52, and the recording control unit 53 in FIG. 4 function significantly.

First, in step S11, the main control unit 13 confirms whether or not the shutter button 17a is in a half-pressed state. If it is in the half-pressed state, the process proceeds from step S11 to step S12, and then the positions between the preview images are referred to with reference to a plurality of preview images obtained until the shutter button 17a is fully pressed. _A deviation amount D _A is calculated. A period from when the shutter button 17a is half-pressed to when it is fully pressed is referred to as a “shooting preparation period”. In this example, the operation of fully pressing the shutter button 17a corresponds to a continuous shooting instruction. Note that known autofocus control is performed during the shooting preparation period, and the position of the focus lens 31 is driven and controlled so as to form an optical image of the subject on the image sensor 33 (see FIG. 2).

The first misalignment detection unit 51 is provided with image data of each captured image that is sequentially acquired. In a first embodiment, the first displacement detection unit 51 is obtained during the shooting preparation time, by obtaining the positional displacement amount between the preview images that are temporally adjacent to calculate a position deviation amount D _A. The displacement amount is a two-dimensional vector amount including a horizontal component and a vertical component, and is also referred to as a motion amount, a motion vector, or an optical flow (the same applies to each displacement amount described later other than the displacement amount D _A ).

For example, as shown in FIG. 6, ten preview images P ₁ , P ₂ ,... P ₈ , P ₉ and P ₁₀ were taken in this order during the shooting preparation period, and a preview image P ₁₀ was obtained. immediately after becoming shutter button 17a is fully pressed state, the next preview image P ₁₀ consider the case where exposure for the first captured image to form a recording image sequence is performed.

In this case, the first misregistration detection unit 51 calculates the misregistration amount between the preview image P _i and the preview image P _{i + 1} that are temporally adjacent by using the misregistration detection method (where, i is an integer from 1 to 9. The displacement detection method includes a known representative point matching method, block matching method, and gradient method. For example, the amount of positional deviation between two preview images obtained immediately before the recording period of the recorded image sequence, that is, the amount of positional deviation between the preview image P ₉ and the preview image P ₁₀ is calculated as the amount of positional deviation. Let D _A. Alternatively, the displacement amount D _A may be obtained from a plurality of displacement amounts based on three or more preview images. For example, the average of the positional deviation amount between the preview image P ₈ and the preview image P ₉ and the positional deviation amount between the preview image P ₉ and the preview image P ₁₀ may be obtained as the positional deviation amount D _A. .

In parallel with the calculation of the amount of positional deviation between the preview images, in step S13, the main control unit 13 checks whether or not the shutter button 17a has transitioned from the half-pressed state to the fully-pressed state. When the transition to the fully pressed state is confirmed, the process proceeds to step S14. In FIG. 5, the positional deviation amount D _A is shown as if it was calculated before confirmation of this transition, but actually, the positional deviation amount D _A is determined after this transition is confirmed. The The misregistration amount D _A is calculated in units of pixels.

In step S <b> 14, the shooting control unit 52 determines the shooting pattern of the recorded image sequence based on the positional deviation amount D _A. The recorded image sequence is formed from m normal exposure images and n short exposure images. The shooting pattern of the recorded image sequence is also referred to as a shooting pattern of a normal exposure image and a short exposure image. Here, m is an integer of 2 or more, n is an integer of 1 or more, and the inequality “m> n” holds. The value of m is determined according to a user operation on the operation unit 17, for example. The normal exposure image means a photographed image obtained by exposure with appropriate exposure time T _EP later. On the other hand, a short exposure image means a captured image obtained with an exposure time shorter than the exposure time of a normal exposure image.

  Hereinafter, photographing for obtaining a normal exposure image is sometimes referred to as “normal exposure photographing”, and photographing for obtaining a short exposure image is sometimes referred to as “short exposure photographing”. In addition, image data representing a normal exposure image may be referred to as “normal exposure image data”, and image data representing a short exposure image may be referred to as “short exposure image data”. In this specification, description of image data may be omitted for simplification of description in a sentence explaining that some processing (recording, saving, reading, etc.) is performed on image data of a certain image. is there. For example, recording of image data of a normal exposure image is synonymous with recording of a normal exposure image.

Specifically, in step S14, the shooting control unit 52 determines the ratio of the number of shots of the normal exposure image and the short exposure image, that is, the ratio of m and n, based on the positional deviation amount D _A.

  A specific example will be given assuming that m = 6. In the following description, a case where m = 6 is assumed unless otherwise specified.

When the amount of displacement D _A is 20 pixels or more, one normal exposure image and one short exposure image are alternately photographed. However, it is assumed that the last captured image in the recorded image sequence is a normal exposure image. Therefore, the recording image sequence consisting of six normal exposure image A ₁ to A ₆ and five short-exposure image B ₁ .about.B ₅ Metropolitan is obtained. More specifically, as shown in FIG. 7A, the order of images A ₁ , B ₁ , A ₂ , B ₂ , A ₃ , B ₃ , A ₄ , B ₄ , A ₅ , B ₅ and A ₆ . The continuous shooting is done. The shooting pattern corresponding to this continuous shooting is hereinafter referred to as a “first shooting pattern”.

If the magnitude of the positional deviation amount D _A is less than and 20 pixels was 10 pixels or more repeats the imaging operation of taking a single short-exposure image after the two normal exposure image capturing. However, it is assumed that the last captured image in the recorded image sequence is a normal exposure image. For this reason, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and two short exposure images B ₁ and B ₂ is obtained. More specifically, as shown in FIG. 7B, continuous shooting is performed in the order of images A ₁ , A ₂ , B ₁ , A ₃ , A ₄ , B ₂ , A ₅ and A ₆ . The shooting pattern corresponding to the continuous shooting is hereinafter referred to as a “second shooting pattern”.

When the amount of misregistration D _A is less than 10 pixels, the photographing operation of photographing one short exposure image after photographing three normal exposure images is repeated. However, it is assumed that the last captured image in the recorded image sequence is a normal exposure image. Therefore, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and one short exposure image B ₁ is obtained. More specifically, as shown in FIG. 7C, continuous shooting is performed in the order of images A ₁ , A ₂ , A ₃ , B ₁ , A ₄ , A ₅ and A ₆ . The shooting pattern corresponding to this continuous shooting is hereinafter referred to as a “third shooting pattern”.

  In FIGS. 7A to 7C, the brightness of the image is expressed by shading. Since the exposure time of the short exposure image is short, the brightness (luminance level) of the short exposure image is lower than that of the normal exposure image. It is assumed that the aperture of the diaphragm 32 and the signal amplification degree in the AFE 12 are the same between the normal exposure photography and the short exposure photography (however, they can be different).

In this way, the imaging control unit 52 determines the ratio between the number of normal exposure images and the number of short exposure images to be included in the recorded image sequence based on the positional deviation amount D _A , so that the normal exposure image and the short exposure are determined. The image capturing order is determined as described above.

In the above specific example, since m is 6, when the displacement amount D _A is 20 pixels or more, m: n = 6: 5, and the displacement amount D _A is 10 pixels. When the above is less than 20 pixels, m: n = 3: 1, and when the displacement D _A is less than 10 pixels, m: n = 6: 1. When m is sufficiently large, the ratio m: n converges to 1: 1 when the displacement amount D _A is 20 pixels or more, and the displacement amount D _A is 10 pixels or more. The ratio m: n converges to 2: 1 in the case of less than 20 pixels, and the ratio m: n converges to 3: 1 in the case where the amount of displacement D _A is less than 10 pixels. I will do it.

In step S15 following step S14, the imaging control unit 52 performs an exposure time T at which the exposure amount of the image sensor 33 is appropriate based on brightness information obtained from a photometric circuit (not shown) that measures the brightness of the subject. Calculate _EP .

In the following step S16, is compared with the exposure time T _EP and the threshold time T _TH, if the exposure time T _EP is shorter than the threshold time T _TH, blur from camera shake Outpatient exposure image is not included (or an extremely small Therefore, normal continuous shooting is executed without proceeding to step S17 to be described later. In normal continuous shooting, the imaging unit 11 is controlled so that only m normal exposure images are included in the recorded image sequence. As the threshold time T _TH , for example, a camera shake limit exposure time is used. The camera shake limit exposure time is a limit exposure time at which camera shake is determined to be negligible, and is calculated from the reciprocal of the focal length f _D of the imaging unit 11.

On the other hand, when the exposure time T _EP is longer than a threshold time T _TH proceeds to step S17. In step S17, the shooting control unit 52 generates an exposure time control signal so that m normal exposure images and n short exposure images are continuously shot with the shooting pattern determined in step S14. The imaging operation by the imaging unit 11 is controlled. The exposure time of each normal exposure image is the exposure time _TEP, and the exposure time of each short exposure image is, for example, _TEP / 4.

  Since each normal exposure image has a relatively long exposure time, each normal exposure image is likely to contain camera shake. However, each short exposure image has a relatively short exposure time. The image contains almost no blurring caused by camera shake or the like.

  Each normal exposure image data and each short exposure image data obtained by the continuous shooting in step S17 is sent to the recording control unit 53 in FIG. In step S <b> 18, the recording control unit 53 performs recording control for recording these image data on the recording medium 16. The recording control can be performed after the continuous shooting of the recorded image sequence is completed, or the continuous shooting in step S17 and the recording control in step S18 can be performed in parallel.

  In the recording control in step S18, the recording control unit 53 creates an image file for recording the normal exposure image data. For example, an image file is created so as to comply with the file format of Exif (Exchangeable image file format). FIG. 8 shows the structure of one image file. The image file is formed of a main body area and a header area. When conforming to the Exif file format, the header area is also referred to as an Exif tag or an Exif area.

For the sake of specific description, an example of a case in which continuous shooting is performed with the second shooting pattern corresponding to FIG. 7B will be described, and an image file group created by the recording control in step S18 will be described. FIG. 9 is a schematic diagram of the data structure of the image file group. When continuous shooting is performed according to the second shooting pattern, six image files FI _{1 to} FI ₆ are created and stored in the recording medium 16. Image data of normal exposure images A _{1 to} A ₆ are recorded in the main body regions of the image files FI _{1 to} FI ₆ , respectively.

On the other hand, a dedicated image file for recording image data of a short exposure image is not created. Image data of a certain short exposure image is recorded in a header area of an image file in which a normal exposure image obtained by photographing immediately before the short exposure image is to be recorded. Therefore, as shown in FIG. 9, the image data of the short exposure image B ₁ is recorded in the header area of the image file FI ₂ , and the image data of the short exposure image B ₂ is recorded in the header area of the image file FI ₄ . In addition, in the header area of an image file in which short-exposure image data is not recorded, a short-exposure image photographed closest in time with respect to the photographing point of the normal exposure image recorded in the main body area of the image file. Records only the name of the recorded image file. Accordingly, only the image file name of the image file FI ₂ is recorded in the header areas of the image files FI ₁ and FI ₃ , and only the image file name of the image file FI ₄ is recorded in the header areas of the image files FI ₅ and FI _6. Accordingly, the short exposure image data is not recorded in the header areas of the image files FI ₁ , FI ₃ , FI ₅ and FI ₆ . However, in the header area of each image file, additional data other than the image data or image file name (shooting date and time, focal length at the time of shooting, etc.) is also recorded. Further, when recording image data of a normal exposure image or a short exposure image in an image file, the image data is compressed using a compression method such as JPEG (Joint Photographic Experts Group).

  FIG. 10 shows a flowchart of processing for realizing the recording control described above. FIG. 10 corresponds to the detailed flowchart of step S18 of FIG. 5, and the process of step S18 is formed from the processes of steps S21 to S28 of FIG.

In the recording control in step S18, first, the variable i is introduced, 1 is substituted into the variable i (step S21), and the process proceeds to step S22. In step S22, the i-th acquired captured image is selected as the image I _i out of the captured images forming the recorded image sequence, and the process proceeds to step S23. When continuous shooting is performed in the second shooting pattern corresponding to FIG. 7B, the images I ₁ , I ₂ , I ₃ , I ₄ , I ₅ , I ₆ , I ₇ and I ₈ are respectively Corresponds to images A ₁ , A ₂ , B ₁ , A ₃ , A ₄ , B ₂ , A ₅ and A ₆ . If the image I _i is a normal exposure image, the process proceeds from step S23 to step S24, where an image file is created and the image data of the image I _i after JPEG compression is output to the main body area of the image file. Let me record. In step S25, the name of the image file in which the short exposure image data corresponding to the image I _i is to be recorded is output and recorded in the header area of the image file. After the process of step S25, the process proceeds to step S27. On the other hand, when the image I _i is a short exposure image, the process proceeds from step S23 to step S26. In step S26, the image data of the image I _i after JPEG compression is output and recorded in the header area of the image file in which the image data of the image I _i-1 is recorded, and then the process proceeds to step S27. In step S27, it is confirmed whether or not the recording of all the image data has been completed. If not, 1 is added to the variable i in step S28, and the process returns to step S22. Execute. On the other hand, when the recording of all the image data is completed, the recording control is terminated.

  Actually, the image data is compressed when the image file is recorded as described above. However, in the following description, the existence of the compression process for the image data is ignored for the sake of simplicity.

[Operation during image stabilization]
Next, the operation at the time of blur correction will be described. FIG. 11 is a flowchart illustrating the flow of the shake correction operation of the imaging apparatus 1 according to the first embodiment. The shake correction operation is executed in the playback mode, for example. During the shake correction operation, the correction target image extraction unit 54, the reference image extraction unit 55, the second misregistration detection unit 56, and the correction processing unit 57 in FIG. 4 function significantly.

  When the shake correction operation is performed, the processes of steps S31 to S36 are sequentially performed. First, in step S31, the main control unit 13 indicates an instruction (hereinafter referred to as a correction target instruction) indicating which of the m normal exposure images stored in the recording medium 16 should be corrected. Called). The correction target instruction is given by a predetermined operation on the operation unit 17 by the user, for example. According to the content of the correction target instruction, the correction target image extraction unit 54 selects one of the m normal exposure images, and the image data of the selected one normal exposure image is stored in the recording medium 16. Read from the main body area of the image file onto the internal memory 14. The single normal exposure image selected by the correction target image extraction unit 54 is subsequently handled as a correction target image.

  In subsequent step S32, the reference image extraction unit 55 specifies one short exposure image to be referred to for blur correction of the normal exposure image selected as the correction target image as the reference image, and the correction target image is recorded. Based on the recorded contents of the header area of the image file, the image data of the reference image is read from the recording medium 16. When image data of a short exposure image is recorded in the header area of the image file in which the correction target image is recorded, the short exposure image is specified as a reference image. When the image data of the short exposure image is not recorded in the header area of the image file in which the correction target image is recorded, the image file name recorded in the header area is referred to and indicated by the image file name. A short-exposure image recorded in the header area of the image file to be recorded is specified as a reference image.

A specific example will be given assuming that the _six image files FI _{1 to} FI 6 shown in FIG. 9 are created and stored in the recording medium 16. In this case, when the correction target image is the normal exposure image A ₂ , the image data of the short exposure image B ₁ is recorded in the header area of the image file FI ₂ storing the normal exposure image A _2. The exposure image B ₁ is selected as a reference image, and the image data of the short exposure image B ₁ is read onto the internal memory 14 from the header area of the image file FI ₂ . Similarly, when the correction target image is the normal exposure image A ₄ , the short exposure image B ₂ is selected as the reference image, and the image data of the short exposure image B ₂ from the header area of the image file FI ₄ is stored in the internal memory 14. Reads up.

On the other hand, when the correction target image is the normal exposure image A ₁ , the image file FI ₂ indicated by the image file name stored in the header area of the image file FI ₁ storing the normal exposure image A ₁ is focused. The Then, the short exposure image B ₁ in the header area of the image file FI ₂ is selected as a reference image, and the image data of the short exposure image B ₁ is read from the header area of the image file FI ₂ onto the internal memory 14. Similarly, the correction when the target image is the normal exposure image A _3, A ₅ and A _6, respectively, the image file FI _3, FI ₅ and the image file name short-exposure images based on the header region of the FI ₆ B ₁ , B ₂ , and B ₂ are selected as reference images, and the image data of the reference image is read onto the internal memory 14.

After the image data of the correction target image and the reference image are read in steps S31 and S32, the process proceeds to step S33. In step S33, the second displacement detection unit 56, using the position deviation detecting method such as the representative point matching method, calculates a position deviation amount D _B between the reference image and the correction target image.

  This calculation method will be described in more detail with reference to FIG. FIG. 12 is a detailed flowchart of step S33. The process of step S33 is formed from the processes of steps S41 to S43. First, in step S41, it is determined whether or not another image exists between the correction target image and the reference image in the recorded image sequence arranged in time series. That is, it is determined whether or not another normal exposure image is shot between the correction target image and the reference image. If no other image exists, the process proceeds to step S42. If another image exists, the process proceeds to step S43.

For example, in the case where continuous shooting is performed with the third shooting pattern corresponding to FIG. 7C, if the correction target image is the normal exposure image A ₃ or A ₄ , the reference image is the short exposure image B ₁ . Therefore, the process proceeds from step S41 to step S42. If the correction target image is the normal exposure image A ₁ , A ₂ , A ₅ or A ₆ , the reference image is the short exposure image B ₁ and therefore the process proceeds from step S41 to step S43. To do.

In step S42, the correction target image and the reference image are directly compared based on the image data of the correction target image and the reference image, and a positional shift amount D _B1 between the two images is detected using a positional shift detection method such as a representative point matching method. Ask for. Then, handling of the positional deviation amount D _B1 as the positional deviation amount D _B.

On the other hand, when the process proceeds to step S43, an image sequence including the correction target image and the reference image and another normal exposure image existing between the two images is specified as the accumulation target image sequence. Then, by using a positional deviation detection method such as a representative point matching method, the positional deviation amount between two temporally adjacent images included in the accumulation target image sequence is sequentially obtained, and the obtained positional deviation amount is accumulated. A deviation amount D _B2 is obtained. Then, handling of the positional deviation amount D _B2 as the positional deviation amount D _B.

For example, when continuous shooting is performed with the third shooting pattern corresponding to FIG. 7C, when the normal exposure image A ₁ is selected as the correction target image, the accumulation target image sequence is the normal exposure image A _1. It will be formed from a to a ₃ and the short-exposure image B _1. In this case, the positional deviation amount D _Ba between the image A ₁ and the image A ₂ with the image A ₁ as a reference, and the positional deviation amount between the image A ₂ and the image A ₃ with the image A ₂ as a reference. D _Bb, or, positions calculated displacement amount D _Bc, positional deviation amount D _Ba, the positional displacement amount obtained by accumulating the D _Bb and D _Bc position between the image a ₃ and the image B ₁ relative to the image a _{3 Obtained} as the deviation amount _DB2 . As described above, since the displacement amount is a two-dimensional vector amount, the displacement amounts D _B2 , D _Ba , D _Bb, and D _Bc are read as vectors D _B2 , D _Ba , D _Bb, and D _Bc , respectively. Then, as shown in FIG. 13, when joining the starting point of the end point and a vector D _Bc Vector D _Ba endpoint and vector D _Bb starting the splicing and vector D _Bb of the vector D _Ba of the start point and the vector D _Bc The end point coincides with the start point and end point of the vector _DB2 .

If the shooting interval between the correction target image and the reference image is long, a change in the subject and a change in the ambient illuminance of the imaging apparatus 1 tend to occur between the shootings. When the subject changes or the like, the image matching degree decreases between the correction target image and the reference image, and in principle, the positional deviation detection accuracy between the two images decreases. Also, if the shooting interval between the correction target image and the reference image is long, camera shake accumulates and the total amount of camera shake between the shooting of both images increases. The increase in the amount of camera shake occurring between the contrast images acts in the direction of reducing the positional deviation detection accuracy between the contrast images. For example, when there is a large rotational displacement between contrast images, the normal displacement detection method cannot detect the correct displacement amount. Since blur correction is performed through misalignment correction between the correction target image and the reference image based on the detected misalignment amount, the accuracy of misalignment correction deteriorates when the misalignment detection accuracy decreases. Eventually, the accuracy of blur correction is degraded. Considering this, as processing of step S43, by accumulating the positional displacement amount between images temporally adjacent, to determine a position deviation amount D _B between the reference image and the correction target image. For temporally positional shift amount between adjacent images is less small and changes in the subject, it is possible to determine the positional deviation amount with high accuracy, a result, the position deviation amount D _B corresponding to their cumulative amount The accuracy of is also increased.

Although different from the above-described method, even when step S43 is reached, the positional deviation amount _DB1 between the two images is calculated by direct comparison between the correction target image and the reference image, and the positional deviation based on the direct comparison is calculated. You may make it obtain | require the average amount or one of the amount D _B1 and the amount of displacement D _B2 based on the cumulative calculation as the amount of displacement D _B. Alternatively, even when step S43 is reached, the positional deviation amount _DB1 based on the direct comparison is calculated, and the positional deviation amount is obtained only when the minimum value of the matching error at the time of calculating the positional deviation amount _DB1 is large. D _B2 may be obtained as the positional deviation amount D _B. The minimum value of the matching error is an index indicating the reliability of the calculated misregistration amount, and if it is large, the reliability becomes low. The matching error is SAD (Sum of Absolute Difference) or SSD (Sum of Square Difference) between contrast images, which is obtained when the representative point matching method or the block matching method is used.

Further, in order to obtain a large amount of positional deviation, it is necessary to increase the search range for the matching position. Expansion of the search range increases the processing burden. Also, if the search range is insufficient, the amount of positional deviation cannot be detected accurately. Considering this, when the process reaches step S43, first, the above-described positional deviation amount _DB2 is calculated, and the position between the two images is directly compared with the reference image based on the positional deviation amount _DB2. obtains a shift amount, may be handled as the position displacement amount D _B should seek it eventually. That is, after the search range of the matching position is shifted so that the positional shift amount _DB2 is canceled, the correction target image and the reference image are directly compared, and the positional shift amount between the two images obtained by the direct comparison is finally determined. it may be obtained as a position deviation amount D _B to.

Returning to the flowchart of FIG. When the positional deviation amount D _B obtained, the process proceeds to step S34. Each image data and the positional deviation amount D _B of the correction target image and the reference image are given to the correction processing unit 57 of FIG. In step S34, the correction processing unit 57 corrects the positional deviation between the reference image and the correction target image based on the positional deviation amount D _B.

Now, positional deviation amount D _B is the corrected target image is obtained is obtained as a reference. Then, the reference image is divided by positional deviation corresponding to the displacement amount D _B the correction target image as a reference can be regarded as an image generated. Therefore, the reference image to the positional deviation correction by performing coordinate conversion on the reference image (such as affine conversion) as the positional deviation amount D _B is canceled. For example, geometric displacement parameters for performing this coordinate conversion are obtained, and positional deviation correction is performed by converting the reference image on the coordinate plane on which the correction target image is defined. A pixel located at the coordinates (x + D _Bx , y + D _By ) in the reference image before the positional deviation correction is converted into a pixel located at the coordinates (x, y) by the positional deviation correction. D _Bx and D _{By A,} respectively, the horizontal and vertical components of D _B. Note that the same misalignment correction may be realized by performing coordinate conversion on the correction target image or by performing coordinate conversion on both the correction target image and the reference image.

  The correction target image and the reference image after the positional deviation correction in step S34 are referred to as a correction target image Lw and a reference image Rw, respectively. The correction target image Lw represents the whole image of the correction target image after the positional deviation correction, and the reference image Rw represents the whole reference image after the positional deviation correction. In step S <b> 35 subsequent to step S <b> 34, the correction processing unit 57 performs shake correction processing on the correction target image Lw based on the correction target image Rw and the reference image Lw, so that the blur correction in which the blur of the correction target image Lw is reduced is performed. An image Qw is generated. A method for generating the shake correction image Qw will be described later. Thereafter, in step S <b> 36, the main control unit 13 newly generates an image file for recording the image data of the shake correction image Qw, and stores the image file in the recording medium 16. Alternatively, the image data of the shake correction image Qw is added to the main body area of the image file in which the image data of the correction target image is recorded so that the image data of the correction target image is overwritten without generating a new image file. You may remember. Further, the shake correction image Qw is displayed on the display unit 15.

[Action and effect]
Consider the action or effect of processing as described above. Basically, the short exposure image has less blur than the normal exposure image, and if the normal exposure image is corrected with the edge state of the short exposure image as a target, the blur of the normal exposure image is reduced. Since camera shake and subject movement also occur during continuous shooting, it is desirable that the short-exposure image that is the target of camera shake correction be captured immediately before or after the normal exposure image. However, if one short exposure image is taken every time one normal exposure image is taken, twice the memory capacity is required as compared with normal continuous shooting. In addition, the shooting time required to obtain m normal exposure images becomes longer and the shooting interval between adjacent normal exposure images becomes longer.

  In the first embodiment, considering this, the number of short exposure images (that is, n) to be photographed in association with the photographing of m normal exposure images is set to be larger than the number of normal exposure images (that is, m). It is small. As a result, an increase in the required memory capacity resulting from the shooting of the short exposure image is suppressed, and an increase in the shooting time for obtaining m normal exposure images is also suppressed.

However, when the number of short-exposure images is small, a combination in which the photographing interval between the normal exposure image and the short-exposure image becomes large occurs. At this time, if the subject moves further or a large amount of camera shake occurs, the amount of misalignment between the normal exposure image as the correction target image and the short exposure image as the reference image increases, and the misalignment amount The detection accuracy of the image quality deteriorates, and the accuracy of blur correction eventually deteriorates. Therefore, in this embodiment, the ratio of the number of short-exposure images to the number of normal-exposure images is set based on the positional deviation amount D _A between preview images. That is, when the displacement amount D _A is large, it is estimated that the displacement amount between the correction target image and the reference image also increases. Therefore, the ratio is increased to ensure highly accurate displacement detection. On the other hand, if the positional displacement amount D _A is small, the correction for the positional deviation amount between the target image and the reference image is estimated to be small, by decreasing the ratio, the required memory capacity increase suppression, m Like The increase in shooting time for obtaining the normal exposure image and the increase in the shooting interval between adjacent normal exposure images are suppressed.

  In this embodiment, as shown in FIG. 9, the image data of one short exposure image is recorded only in one image file, and the image file in which the image data of the short exposure image is not recorded is referred to. Only the link information (image file name) of the power image file is recorded in the header area. This eliminates redundant recording of short-exposure image data, thereby reducing an increase in file size.

Furthermore, by recording the image data of the short exposure image in the image file of the normal exposure image taken immediately before it, the number of file accesses related to the opening and closing of the file can be reduced. For example, when continuous shooting is performed with the second shooting pattern corresponding to FIG. 7B, the image files FI _{1 to} FI ₆ shown in FIG. 9 are sequentially generated and stored in the recording medium 16. According to the embodiment, a recording operation as shown in FIG. That is, after the image file FI ₁ is opened and the image data of the image A ₁ is recorded, the image file FI ₁ is closed. Then, after the image file FI ₂ is opened image data of the image A ₂ and B ₁ is recorded, the image file FI ₂ is closed. If the image data of the image B ₁ is recorded not in the image file FI ₂ but in the header area of the image file FI ₁ , as shown in FIG. 14B, once the image data of the image A ₁ is recorded. The closed image file FI ₁ must be opened and closed again when the image data of the image B ₁ is recorded. Since opening and closing a file takes a finite time, reducing the number of file accesses contributes to shortening the recording operation time.

  In the present embodiment, the entire image data of the short-exposure image is recorded on the recording medium 16. However, only the image data of the portion necessary for the blur correction process in step S35 in FIG. It may be recorded. Thereby, the increase in file size can be further reduced. For example, a characteristic small area in the short exposure image corresponding to a part of the short exposure image is extracted, and the image data of the image in the small exposure image is used instead of the entire image data of the short exposure image. May be recorded. However, this cannot be applied to a second correction method (see the fourth embodiment) described later. A characteristic small region refers to a region having a relatively large number of edge components (in other words, a relatively strong contrast) in the extraction source image, and is, for example, a region including a characteristic pattern. A characteristic pattern has, for example, a luminance change in two or more directions such as a corner of an object, and the position of the pattern (position on the image) can be easily detected by image processing based on the luminance change. Meaning a simple pattern.

<< Second Example >>
In the first embodiment, the shooting pattern of the recorded image sequence is determined based on the positional deviation amount D _A between the preview images, but the position between the normal exposure images obtained in the vicinity of the recording start of the recorded image sequence. A subsequent photographing pattern may be determined based on the amount of deviation. As an example for explaining this, a second example will be described. Since the second embodiment corresponds to an embodiment obtained by modifying a part of the first embodiment, only portions different from the first embodiment will be described. The operation and configuration other than the method of determining the shooting pattern are the same as those of the first embodiment, and the description is applicable to the second embodiment as long as the description of the first embodiment is consistent.

In the shooting mode, after the shutter button 17a is half-pressed and then the shutter button 17a is fully pressed via the drive control of the focus lens 31, continuous shooting for acquiring a recorded image sequence is performed. Be started. Photographing control unit 52, based on the brightness information obtained from the photometric circuit for measuring the brightness of an object (not shown), the exposure amount of the image sensor 33 to calculate the exposure time T _EP to be appropriate. Then, compared with the exposure time T _EP and the threshold time T _TH, the exposure time T _EP is shorter than the threshold time T _TH, executes the normal continuous shooting. In normal continuous shooting, the imaging unit 11 is controlled so that only m normal exposure images are included in the recorded image sequence.

On the other hand, when the exposure time T _EP is longer than a threshold time T _TH, the imaging control unit 52, first, continuous shooting the m _A single normal exposure image by ordinary-exposure shooting of the exposure time T _EP. Here, m _A is an integer of 2 or more. Eventually, the m normal exposure images are included in the recorded image sequence, but the m _A normal exposure images are part of the m normal exposure images. That is, the inequality “m _A <m” is established. Further, it is assumed that m = m _A + m _B. m _B is an integer of 1 or more.

Now, suppose that m _A is 2, and that two normal exposure images A ₁ and A ₂ are taken in this order. The first displacement detection unit 51, using the position deviation detecting method such as the representative point matching method, positional deviation amount positional deviation amount between the normal exposure image A ₁ and a normal exposure image A ₂ temporally adjacent Calculated as D _A2 . The positional deviation amount D _A2 is calculated in units of pixels.

Thereafter, the shooting control unit 52 determines the shooting pattern of the recorded image sequence based on the positional deviation amount D _A2 . However, as a matter of course, the photographing pattern determined here is the normal exposure image A.
A shooting pattern except ₁ and A _2. As in the first embodiment, the recorded image sequence is formed from m normal exposure images and n short exposure images, and m is an integer of 2 or more and n is an integer of 1 or more. Furthermore, the inequality “m> n” holds. Specifically, the shooting control unit 52 determines the ratio of the number of shots of the normal exposure image and the short exposure image excluding the normal exposure images A ₁ and A ₂ based on the positional deviation amount D _A2 . That is, the ratio of m _B and n is determined based on the positional deviation amount D _A2 . Since m = m _A + m _B is established, if the ratio of m _B and n is determined, the ratio of m and n is automatically determined. Therefore, it can be said that the imaging control unit 52 determines the ratio of m and n based on the positional deviation amount D _A2 .

A specific example will be given assuming that m = 6.
When the size of the displacement amount D _A2 is 20 pixels or more, one normal exposure image and one short exposure image are alternately captured after the normal exposure images A ₁ and A ₂ are captured. However, it is assumed that the last captured image in the recorded image sequence is a normal exposure image, and short exposure shooting is performed immediately after shooting of the normal exposure images A ₁ and A ₂ . For this reason, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and four short exposure images B _{1 to} B ₄ is obtained. More specifically, as shown in FIG. 15A, continuous shooting is performed in the order of images A ₁ , A ₂ , B ₁ , A ₃ , B ₂ , A ₄ , B ₃ , A ₅ , B ₄ and A _6. Shooting is done.

When the size of the misregistration amount D _A2 is 10 pixels or more and less than 20 pixels, after the normal exposure images A ₁ and A ₂ are captured, one short image is captured after the two normal exposure images are captured. The photographing operation of photographing the exposure image is repeated. However, it is assumed that the last captured image in the recorded image sequence is a normal exposure image, and short exposure shooting is performed immediately after shooting of the normal exposure images A ₁ and A ₂ . For this reason, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and two short exposure images B ₁ and B ₂ is obtained. More specifically, as shown in FIG. 15B, continuous shooting is performed in the order of images A ₁ , A ₂ , B ₁ , A ₃ , A ₄ , B ₂ , A ₅ and A ₆ . As a result, the shooting pattern by the continuous shooting is the same as that of the second shooting pattern corresponding to FIG.

When the amount of displacement D _A2 is less than 10 pixels, after taking the normal exposure images A ₁ and A ₂ , taking one short exposure image after taking three normal exposure images. Repeat the operation. However, the last captured image in the recorded image sequence is a normal exposure image, and the _first short exposure shooting is performed immediately after the three normal exposure images including the normal exposure images A ₁ and A ₂ are captured. Assumed to be performed. Therefore, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and one short exposure image B ₁ is obtained. More specifically, as shown in FIG. 15C, continuous shooting is performed in the order of images A ₁ , A ₂ , A ₃ , B ₁ , A ₄ , A ₅ and A ₆ . As a result, the shooting pattern by the continuous shooting is the same as that of the third shooting pattern corresponding to FIG.

As described above, the shooting control unit 52 determines the ratio of m _B and n (and the ratio of m and n) based on the positional deviation amount D _A2 , and sets the shooting order of the normal exposure image and the short exposure image as described above. Determine.

  The obtained image data of each normal exposure image and image data of each short exposure image are recorded on the recording medium 16 by the same recording control as in the first embodiment (step S18 in FIG. 5; see FIG. 10). The operation of the image pickup apparatus 1 at the time of blur correction is the same as that in the first embodiment.

According to this embodiment, the same effect as that of the first embodiment can be obtained. However, in the above-described example, when the size of the displacement amount D _A2 is 20 pixels or more, another normal exposure is performed between the normal exposure image A ₁ and the short exposure image B ₁ closest in time to the normal exposure image A _1. An image is inserted. Accordingly, the blur correction accuracy is deteriorated as compared with the first embodiment only when the amount of the displacement D _A2 is 20 pixels or more and the normal exposure image A ₁ is the correction target image.

In addition, although the example in which m _A is 2 has been described above, m _A may be an arbitrary integer of 3 or more. For example, when m _A is 3, the imaging control unit 52 sets the first, second, and third captured images forming the recorded image sequence as normal exposure images A ₁ , A _2, and A ₃ , respectively. In this case, the first misregistration detection unit 51 obtains the misregistration amount between the image A ₁ and the image A ₂ and the misregistration amount between the image A ₂ and the image A _3. was calculated that the average as positional deviation amount D _A2, based on the position deviation amount D _A2, the imaging control unit 52 typically determines the shooting pattern of the recorded image columns except exposure image a ₁ to a ₃ .

<< Third Example >>
Next, a third embodiment will be described. In the third embodiment, a modified example of the shooting pattern will be described. Since the third embodiment corresponds to an embodiment obtained by modifying a part of the first embodiment, only the portions different from the first embodiment will be described. The operation, configuration, and the like other than the shooting pattern setting method are the same as those of the first embodiment, and the description is applicable to the third embodiment as long as the description of the first embodiment is consistent.

In the shooting mode, when the shutter button 17a is fully pressed, continuous shooting for acquiring a recorded image sequence is started. Photographing control unit 52, based on the brightness information obtained from the photometric circuit for measuring the brightness of an object (not shown), the exposure amount of the image sensor 33 to calculate the exposure time T _EP to be appropriate. Then, compared with the exposure time T _EP and the threshold time T _TH, the exposure time T _EP is shorter than the threshold time T _TH, executes the normal continuous shooting. In normal continuous shooting, the imaging unit 11 is controlled so that only m normal exposure images are included in the recorded image sequence.

On the other hand, when the exposure time T _EP is longer than a threshold time T _TH, the imaging control unit 52, using any one of the fourth to sixth shooting pattern shown below, m pieces of normal exposure images and n pieces of short-exposure The image-recording unit 11 causes the imaging unit 11 to continuously record a recorded image sequence including images. In the fourth shooting pattern, only one short exposure image is shot, and then m normal exposure images are shot continuously. In the fifth shooting pattern, after the m normal exposure images are continuously shot, only one short exposure image is shot. In the sixth shooting pattern, only one short exposure image is shot, then m normal exposure images are shot continuously, and then only one short exposure image is shot.

A specific example will be given assuming that m = 6.
When the fourth photographing pattern is adopted, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and one short exposure image B ₁ is obtained. More specifically, as shown in FIG. 16 (a), the image _{B 1, A 1, A 2} , A 3, A 4, continuous shooting in the order of A ₅ and A ₆ is made.
When the fifth shooting pattern is adopted, a recorded image sequence composed of _six normal exposure images A _{1 to} A ₆ and one short exposure image B ₁ is obtained. More specifically, as shown in FIG. 16B, continuous shooting is performed in the order of images A ₁ , A ₂ , A ₃ , A ₄ , A ₅ , A ₆ and B ₁ .
When the sixth shooting pattern is adopted, a recorded image sequence including _six normal exposure images A _{1 to} A ₆ and two short exposure images B ₁ and B ₂ is obtained. More specifically, as shown in FIG. 16C, continuous shooting is performed in the order of images B ₁ , A ₁ , A ₂ , A ₃ , A ₄ , A ₅ , A ₆ and B ₂ .

The obtained image data of each normal exposure image and image data of each short exposure image are recorded on the recording medium 16 by the same recording control as in the first embodiment (step S18 in FIG. 5; see FIG. 10). However, if the normal performed taking short-exposure image B ₁ before taking the exposure image A _1, the image data of short-exposure image B ₁ represents, normal exposure image A ₁ is recorded in the header area of an image file recorded The On the other hand, as described in the recording control of the first embodiment, the image data of short-exposure image B ₁ in the case of adopting the fifth shot pattern, the header area of an image file is normally exposed image A ₆ is recorded The recorded image data of the short exposure image B ₂ in the case of adopting the sixth shooting pattern is recorded in the header area of the image file in which the normal exposure image A ₆ is recorded. Then, in the header area of the image file in which no short exposure image is recorded, the short exposure image photographed closest in time with respect to the photographing time point of the normal exposure image recorded in the main body area of the image file is recorded. Only the name of the image file to be recorded is recorded. The operation of the imaging apparatus 1 during blur correction is the same as that in the first embodiment.

  In order to satisfy m> n, the imaging control unit 52 adopts the sixth imaging pattern only when m is 3 or more. You may make it perform switching control of the imaging | photography pattern employ | adopted according to the value of m. For example, the sixth shooting pattern is adopted when m is a predetermined value of 3 or more, and the fourth or fifth shooting pattern is adopted when m is less than the predetermined value.

  Also in the third embodiment, the number of short-exposure images (that is, n) to be photographed in association with the photographing of m normal-exposure images is made smaller than the number of normal-exposure images (that is, m). . As a result, an increase in the required memory capacity resulting from the shooting of the short exposure image is suppressed, and an increase in the shooting time for obtaining m normal exposure images is also suppressed. Further, if a short exposure image is shot during shooting of m normal exposure images, the shooting interval between adjacent normal exposure images is widened. In the third embodiment, continuous shooting of m normal exposure images is performed. Since the short-exposure image is captured before or after this, the interval between adjacent normal-exposure images is not increased by the short-exposure shooting. Further, as in the first embodiment, since the short exposure image data is not recorded redundantly, an increase in file size can be reduced.

  As described in the first embodiment, among the entire image data of the short-exposure image, only the image data of the portion necessary for the blur correction process in step S35 in FIG. 11 is recorded on the recording medium 16. May be. Thereby, the increase in file size can be further reduced.

<< 4th Example >>
Next, a fourth embodiment will be described. In the fourth embodiment, a shake correction method for the correction target image Lw based on the correction target image Lw and the reference image Rw will be described. The shake correction process for performing this correction is executed in step S35 of FIG. As the blur correction methods for the correction target image Lw, first to third correction methods will be exemplified below. The first, second, and third correction methods are correction methods using an image restoration method, an image composition method, and an image sharpening method, respectively.

[First correction method]
The first correction method will be described with reference to FIG. FIG. 17 is a flowchart showing the flow of blur correction processing based on the first correction method. When the first correction method is employed, the process of step S35 in FIG. 11 is formed from the processes of steps S71 to S78 in FIG. In the description of the first correction method, “memory” in which an image or the like is stored means the internal memory 14.

  First, in step S71, a characteristic small area is extracted from the correction target image Lw, and an image in the extracted small area is stored in the memory as a small image Ls. For example, a small area of 128 × 128 pixels is extracted as a characteristic small area by using a Harris corner detector. The significance of the characteristic small area is as described above.

  Next, in step S72, a small region having the same coordinates as the small region extracted from the correction target image Lw is extracted from the reference image Rw, and an image in the small region extracted from the reference image Rw is stored in the memory as a small image Rs. To remember. Now, it is assumed that the image size of the correction target image Lw and the reference image Rw are equal. Then, the center coordinates of the small area extracted from the correction target image Lw (center coordinates in the correction target image Lw) and the center coordinates of the small area extracted from the reference image Rw (center coordinates in the reference image Rw) are equal, and The image sizes of both small areas are also equal.

  Since the exposure time of the reference image Rw is relatively short, the signal-to-noise ratio (hereinafter referred to as S / N ratio) of the small image Rs is relatively low and the luminance of the small image Rs is relatively low. Therefore, in step S73, noise removal processing using a median filter or the like is performed on the small image Rs, and the luminance level of the small image Rs after noise removal processing is further increased. That is, for example, the luminance level of each pixel of the small image Rs is set so that the luminance level of the small image Rs is equal to the luminance level of the small image Ls (the average luminance of the small image Rs is equal to the average luminance of the small image Ls). Luminance normalization is performed by multiplying the value by a fixed value. The small image Rs after the noise removal process and the luminance normalization process is stored in the memory as a small image Rs ′.

  The small image Ls obtained as described above is treated as a deteriorated image and the small image Rs ′ is treated as an initial restored image (step S74), and then the Fourier iteration method is performed in step S75, whereby the small image Ls is obtained. An image deterioration function representing a state of deterioration due to blurring is obtained.

  When performing the Fourier iteration method, it is necessary to give an initial restored image (initial value of the restored image). This initial restored image is called an initial restored image.

  A point spread function (hereinafter referred to as PSF) is obtained as an image degradation function. An operator or a spatial filter in which an ideal point image is weighted in accordance with a trajectory drawn on the image by a shake of the image pickup apparatus 1 is called a PSF, and is generally used as a mathematical model of camera shake. Since camera shake uniformly degrades the entire image, the PSF obtained for the small image Ls can be used as the PSF for the entire correction target image Lw.

  The Fourier iteration method is a method for obtaining a restored image from which deterioration is removed or reduced from a deteriorated image including deterioration (see Non-Patent Document 1 above). This Fourier iteration method will be described in detail with reference to FIGS. FIG. 18 is a detailed flowchart of the process in step S75 of FIG. FIG. 19 is a block diagram of a part in the correction processing unit 57 of FIG.

  First, in step S101, a restored image is set as f ', and an initial restored image is set in the restored image f'. That is, the small image Rs ′ is used as the initial restored image f ′. Next, in step S102, the deteriorated image (that is, the small image Ls) is set to g. Then, the result of Fourier transform of the deteriorated image g is stored in the memory as G (step S103). For example, when the image sizes of the initial restored image and the degraded image are 128 × 128 pixels, f ′ and g can be expressed as a matrix having a matrix size of 128 × 128.

Next, in step S110, F ′ obtained by Fourier transform of the restored image f ′ is obtained, and in step S111, H is calculated by the following equation (A-1). H corresponds to the result of Fourier transform of PSF. In Formula (A-1), F ′ ^* is a conjugate complex matrix of F ′, and α is a constant.

  Next, in step S112, PSF is obtained by performing inverse Fourier transform on H. The PSF obtained here is assumed to be h. Next, in step S113, PSF h is corrected under the constraint condition of the following formula (A-2a), and further corrected under the constraint condition of formula (A-2b).

  Since PSF h is expressed as a two-dimensional matrix, each element of this matrix is represented by h (x, y). Each element of the PSF should originally take a value of 0 or more and 1 or less. Therefore, in step S113, it is determined whether each element of the PSF is 0 or more and 1 or less, and the value of the element that is 0 or more and 1 or less is left as it is. The value of the element is corrected to 1, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (A-2a). Then, the PSF is normalized so that the sum of the elements of the PSF after correction is 1. This normalization is a correction by the constraint condition of the formula (A-2b).

  The PSF corrected by the constraints of the expressions (A-2a) and (A-2b) is set as h ′.

Next, in step S114, H ′ obtained by Fourier transforming PSF h ′ is obtained, and in step S115, F is calculated by the following equation (A-3). F corresponds to a Fourier transform of the restored image f. In Formula (A-3), H ′ ^* is a conjugate complex matrix of H ′, and β is a constant.

  Next, in step S116, a restored image is obtained by performing an inverse Fourier transform on F. Let f be the restored image obtained here. Next, in step S117, the restored image f is corrected under the constraint condition of the following formula (A-4), and the corrected restored image is newly set as f '.

  Since the restored image f is represented as a two-dimensional matrix, each element of the matrix is represented by f (x, y). Now, it is assumed that the pixel value of each pixel of the degraded image and the restored image is represented by a digital value from 0 to 255. Then, each element (that is, each pixel value) of the matrix representing the restored image f should originally have a value of 0 or more and 255 or less. Accordingly, in step S117, it is determined whether each element of the matrix representing the restored image f is 0 or more and 255 or less, and the value of the element that is 0 or more and 255 or less is left as it is. If there is an element, the value of the element is corrected to 255, and if there is an element smaller than 0, the value of the element is corrected to 0. This is correction by the constraint condition of Formula (A-4).

  Next, in step S118, it is determined whether or not the convergence condition is satisfied, thereby determining whether or not the iterative process is converged.

  For example, the absolute value of the difference between the latest F ′ and the previous F ′ is used as an index for convergence determination. If this index is less than or equal to a predetermined threshold value, it is determined that the convergence condition is satisfied. Otherwise, it is determined that the convergence condition is not satisfied.

  When the convergence condition is satisfied, a final PSF is obtained by performing inverse Fourier transform on the latest H ′. That is, the latest H ′ obtained by inverse Fourier transform is the PSF to be obtained in step S75 of FIG. When the convergence condition is not satisfied, the process returns to step S110, and the processes of steps S110 to S118 are repeated. In the repetition of each process of steps S110 to S118, f ′, F ′, H, h, h ′, H ′, F, and f (see FIG. 19) are sequentially updated to the latest ones.

  Another index can be used as an index for determining convergence. For example, the absolute value of the difference between the latest H ′ and the previous H ′ obtained may be used as the convergence determination index to determine whether or not the convergence condition is satisfied. Further, for example, the correction amount in step S113 using the above formulas (A-2a) and (A-2b) or the correction amount in step S117 using formula (A-4) is used as an index for convergence determination. Whether or not the convergence condition is satisfied may be determined. This is because if the iterative process is toward convergence, the amount of correction becomes small.

  Further, when the number of repetitions of the loop process consisting of steps S110 to S118 reaches a predetermined number, it may be determined that convergence is impossible and the process may be terminated without calculating a final PSF. In this case, the correction target image Lw is not corrected.

Returning to the description of each step in FIG. After the PSF is calculated in step S75, the process proceeds to step S76. In step S76, each element of the inverse matrix of the PSF obtained in step S75 is obtained as each filter coefficient of the image restoration filter. This image restoration filter is a filter for obtaining a restored image from a deteriorated image. Actually, each element of the matrix represented by the following equation (A-5) corresponding to a part of the right side of the above equation (A-3) corresponds to each filter coefficient of the image restoration filter. The intermediate calculation result of the Fourier iteration method in S75 can be used as it is. However, the formula H ^'* and H' is in the (A-5), H obtained satisfied immediately before the convergence condition in step S118 ^'* and H' (i.e., the finally obtained H ^'* and H' ).

  After each filter coefficient of the image restoration filter is obtained in step S76, the process proceeds to step S77, and the entire correction target image Lw is filtered (spatial filtering) using the image restoration filter. In other words, the correction target image Lw is filtered by applying an image restoration filter having the obtained filter coefficients to each pixel of the correction target image Lw. Thereby, a filtered image with reduced blur included in the correction target image Lw is generated. Although the size of the image restoration filter is smaller than the image size of the correction target image Lw, it is considered that camera shake causes uniform degradation on the entire image. Therefore, this image restoration filter is applied to the entire correction target image Lw. Thus, the overall blurring of the correction target image Lw is reduced.

  The filtering image may include ringing accompanying filtering. For this reason, in step S78, a final blurring correction image Qw is generated by applying a ringing removal process for removing this to the filtered image. Since the method for removing ringing is known, a detailed description is omitted. For example, a technique described in JP 2006-129236 A may be used.

  The blur correction image Qw is an image in which blur included in the correction target image Lw is reduced and ringing due to filtering is reduced. However, since the filtered image is also an image with reduced blur, the filtered image can also be regarded as the shake-corrected image Qw.

  Since the amount of blur included in the reference image Rw is small, its edge component is close to the edge component of an ideal image without camera shake. Therefore, as described above, an image obtained from this reference image Rw is used as an initial restored image in the Fourier iteration method.

  By repetitive loop processing by the Fourier iteration method, the restored image (f ′) gradually approaches an image from which camera shake has been removed as much as possible. However, since the initial restored image itself is already close to an image without camera shake, Convergence is faster than a random image or a deteriorated image as an initial restored image (in the shortest, convergence is achieved by a single loop process). As a result, the processing time for generating camera shake information (PSF or filter coefficient of the image restoration filter) and the processing time for camera shake correction are shortened. If the initial restored image is far from the image to be converged, the probability of convergence to a local solution (an image different from the image to be truly converged) increases. However, by setting the initial restored image as described above, , The probability of convergence to a local solution is low (that is, the probability that camera shake correction fails) is low.

  In addition, camera shake is thought to give uniform degradation to the entire image, so a small area is extracted from each image to create camera shake information (PSF or filter coefficient of the image restoration filter) from the image data of each small area, Apply it to the entire image. As a result, the amount of calculation required is reduced, and the processing time for creating camera shake information and the processing time for correcting camera shake are shortened. Of course, the required circuit scale can be reduced and the cost reduction effect can be expected.

  At this time, as described above, a characteristic small region containing a lot of edge components is automatically extracted. An increase in the edge component in the PSF calculation source image means an increase in the ratio of the signal component to the noise component. Therefore, the influence of noise is reduced by extracting a characteristic small region, and camera shake information is detected more accurately. Will be able to.

  In the process of FIG. 18, the degraded image g and the restored image f ′ on the spatial domain are transformed to the frequency domain by Fourier transform, so that the function G representing the degraded image g on the frequency domain and the frequency domain A function F ′ representing the restored image f ′ is obtained (note that the frequency domain is, of course, a two-dimensional frequency domain). A function H representing PSF in the frequency domain is obtained from the obtained functions G and F ′, and this function H is converted into a function in the spatial domain, that is, PSF h by inverse Fourier transform. The PSF h is corrected using a predetermined constraint condition, and a corrected PSF h ′ is obtained. The process of correcting the PSF is hereinafter referred to as “first correction process”.

  PSF h ′ is again transformed onto the frequency domain by Fourier transform to obtain a function H ′, and a function F representing a restored image on the frequency domain is obtained from the functions H ′ and G. By performing inverse Fourier transform on this function F, a restored image f in the spatial domain is obtained, and this restored image f is corrected using a predetermined constraint condition, and a corrected restored image f ′ is obtained. The processing for correcting the restored image is hereinafter referred to as “second correction processing”.

  In the above-described example, it has been described that, after that, the above-described processing is repeated using the corrected restored image f ′ until the convergence condition is satisfied in step S <b> 118 of FIG. 18. Further, considering the characteristic that the correction amount becomes smaller when the iterative process is toward convergence, the establishment / non-establishment of the convergence condition is determined by the correction amount in step S113 corresponding to the first correction process or the second correction. It has also been stated that the determination may be made based on the correction amount in step S117 corresponding to the processing. When this determination is made based on the correction amount, a reference correction amount is set in advance, and the correction amount in step S113 or the correction amount in step S117 is compared with the reference correction amount, and the former is smaller than the latter. In this case, it is determined that the convergence condition is satisfied. However, if the reference correction amount is set sufficiently large, the processes in steps S110 to S117 are not repeatedly executed. That is, in this case, PSF h ′ obtained by performing the first correction process only once is the final PSF to be derived in step S75 of FIG. As described above, even if the process of FIG. 18 is used, the first and second correction processes are not always repeatedly executed.

  The increase in the number of repeated executions of the first and second correction processes contributes to improving the accuracy of the finally obtained PSF. However, in this example, since the initial restoration image itself is close to an image without camera shake, the first correction is performed. The accuracy of PSF h ′ obtained by performing the treatment only once is also high enough to cause no problem. In consideration of this, the determination process itself in step S118 can be omitted. In this case, PSF h ′ obtained by executing the process of step S113 only once becomes the final PSF to be derived in step S75 of FIG. 17, and the function obtained by executing the process of step S114 only once. From H ′, each filter coefficient of the image restoration filter to be derived in step S76 in FIG. 17 is obtained. Therefore, when the process of step S118 is omitted, the processes of steps S115 to S117 are also omitted.

Further, when the first correction method is adopted, it is not always necessary to record all image data of the short-exposure image from which the reference image Rw is generated on the recording medium 16, and among all the image data of the short-exposure image, If only the image data of the small image Rs is recorded in the recording medium 16, it is possible to generate the shake correction image Qw. A processing example for realizing this will be described below. This processing example can also be applied to a third correction method described later. For the sake of specific description, a recorded image sequence is acquired using the second shooting pattern corresponding to FIG. 7B, the short exposure image that is the basis of the reference image Rw is the short exposure image B ₁ , and Consider a case where the normal exposure image that is the basis of the correction target image Lw is the normal exposure image A ₁ .

In this case, during the photographing operation, the recording control unit 53 in FIG. 4 extracts a characteristic small area from the short-exposure image B ₁ by using a Harris corner detector or the like, and the extracted small area The image data of the inside image is recorded in the header area of the image file FI ₂ together with the center coordinates of the small area (center coordinates in the short exposure image B ₁ ). Then, an image obtained by subjecting the extracted image to edge normalization processing and luminance normalization processing for canceling a luminance level difference resulting from a difference in exposure time between normal exposure photography and short exposure photography. Are handled as a small image Rs ′. Note that the image data of the small image Rs ′ may be recorded in the header area of the image file FI ₂ instead of the image data of the small image Rs. At the time of the shake correction operation corresponding to FIG. 11, the correction processing unit 57 of FIG. 4 uses the small image Rs ′ as a template and uses a known template matching method to correct a small region having the highest similarity with the template. Search is performed from the normal exposure image A ₁ as the image Lw. The search range is set based on the center coordinates (center coordinates for the small image Rs ′) recorded in the header area of the image file FI ₂ . Then, the searched image in the small area is handled as the small image Ls. The operation after the small image Rs ′ and the small image Ls are obtained is as described above.

[Second correction method]
Next, the second correction method will be described with reference to FIGS. FIG. 20 is a flowchart showing the flow of blur correction processing based on the second correction method. FIG. 21 is a conceptual diagram showing the flow of this blur correction process. When the second correction method is employed, the process in step S35 in FIG. 11 is formed from the processes in steps S151 to S154 in FIG.

  1 is a color image including information on luminance and information on color. Accordingly, the pixel signal of each pixel that forms the correction target image Lw is formed from a luminance signal that represents the luminance of the pixel and a color signal that represents the color of the pixel. Now, it is assumed that the pixel signal of each pixel is expressed in YUV format. In this case, the color signal is formed from the two color difference signals U and V. The pixel signal of each pixel forming the correction target image Lw is formed from a luminance signal Y that represents the luminance of the pixel and two color difference signals U and V that represent the color of the pixel.

Then, the correction target image Lw includes, as shown in FIG. 21, an image Lw _Y including only the luminance signal Y as the pixel signal, an image Lw _U including only the color difference signal U as the pixel signal, and only the color difference signal V as the pixel signal. an image Lw _V containing, can be decomposed into. Similarly, the reference image Rw includes an image Rw _Y including only the luminance signal Y as a pixel signal, an image Rw _U including only the color difference signal U as a pixel signal, an image Rw _V including only the color difference signal V as a pixel signal, (In FIG. 21, only the image Rw _Y is shown).

In step S151 of FIG. 20, first, images Lw _Y , Lw _U and Lw _V are generated by extracting the luminance signal and color difference signal of the correction target image Lw. In step S152, by extracting a brightness signal of the reference image Rw, and generates an image Rw _Y.

Since the image Rw _Y has low luminance, the luminance level of the image Rw _Y is increased in step S153. That is, for example, the luminance level of each pixel of the image Rw _Y is set so that the luminance level of the image Rw _Y is equal to the luminance level of the image Lw _Y (so that the average luminance of the image Rw _Y is equal to the average luminance of the image Lw _Y ). Luminance normalization is performed by multiplying the value by a fixed value. Furthermore, noise removal processing using a median filter or the like is performed on the image Rw _Y after the luminance normalization processing. The image Rw _Y after the luminance normalization processing and noise removal processing is stored on the memory as an image Rw _Y ′.

Thereafter, in step S154, the images Lw _U and Lw _V and the image Rw _Y 'are combined, and the image obtained by this combination is output as a blurring corrected image Qw. The pixel signal of the pixel located at the coordinate (x, y) in the shake correction image Qw is located at the coordinate (x, y) with the pixel signal of the pixel in the image Lw _U located at the coordinate (x, y). It is formed from the pixel signal of the pixel in the image Lw _V and the pixel signal of the pixel in the image Rw _Y 'located at the coordinates (x, y).

  In color images, visual blur is mainly caused by luminance blur, and if the edge component of luminance is close to that of an ideal image without blur, the viewer feels that there is little blur. Therefore, in this correction method, a pseudo camera shake correction effect is obtained by combining the luminance signal of the reference image Rw with a relatively small amount of blur with the color signal of the correction target image Lw. According to this method, although color misregistration occurs in the vicinity of the edge, an image with less blurring in appearance can be generated at a low calculation cost.

[Third correction method]
Next, the third correction method will be described with reference to FIGS. FIG. 22 is a flowchart showing the flow of blur correction processing based on the third correction method. FIG. 23 is a conceptual diagram showing the flow of this blur correction process. When the third correction method is employed, the process of step S35 in FIG. 11 is formed from the processes of steps S201 to S207 in FIG.

  First, in step S201, a small image Ls is generated by extracting a characteristic small region from the correction target image Lw. In step S202, a small region corresponding to the small image Ls is extracted from the reference image Rw. An image Rs is generated. The processing in steps S201 and S202 is the same as the processing in steps S71 and S72 in FIG. Subsequently, in step S203, noise removal processing using a median filter or the like is performed on the small image Rs, and the luminance level of the small image Rs after noise removal processing is further increased. That is, for example, the luminance level of each pixel of the small image Rs is set so that the luminance level of the small image Rs is equal to the luminance level of the small image Ls (the average luminance of the small image Rs is equal to the average luminance of the small image Ls). Luminance normalization is performed by multiplying the value by a fixed value. The small image Rs after the noise removal process and the luminance normalization process is stored in the memory as a small image Rs ′.

Next, in step S204, by filtering the small image Rs ′ using eight different smoothing filters, eight smoothed small images Rs _G1 , Rs _G2 ,. Generate _G8 . Now, eight different Gaussian filters are used as the eight smoothing filters, and the variance of the Gaussian distribution represented by each Gaussian filter is represented by σ ² .

When attention is paid to a one-dimensional image and the pixel position in the one-dimensional image is represented by x, a Gaussian distribution having an average of 0 and a variance of σ ² is generally represented by the following formula (B-1). (See FIG. 24). When this Gaussian distribution is applied to a Gaussian filter, each filter coefficient of the Gaussian filter is represented by h _g (x). That is, when the Gaussian filter is applied to the pixel at position 0, the filter coefficient at position x is represented by h _g (x). In other words, the contribution ratio of the pixel value at the position x before filtering to the pixel value at the position 0 after filtering by the Gaussian filter is represented by h _g (x).

When this concept is extended to two dimensions and the pixel position in the two-dimensional image is represented by (x, y), the two-dimensional Gaussian distribution is represented by the following formula (B-2). Note that x and y represent a horizontal position and a vertical position, respectively. When this two-dimensional Gaussian distribution is applied to the Gaussian filter, each filter coefficient of the Gaussian filter is represented by h _g (x, y). When the Gaussian filter is applied to the pixel at the position (0, 0), the position (x, The filter coefficient in y) is represented by h _g (x, y). That is, the contribution ratio of the pixel value at the position (x, y) before filtering to the pixel value at the position (0, 0) after filtering by the Gaussian filter is represented by h _g (x, y).

In step S204, Gaussian filters with σ = 1, 3, 5, 7, 9, 11, 13, and 15 are used as the eight Gaussian filters. In the subsequent step S205, image matching is performed between the small image Ls and each of the smoothed small images Rs _{G1 to} Rs _G8 , and among the smoothed small images Rs _{G1 to} Rs _G8 , the smoothing small that has the smallest matching error. An image (that is, a smoothed small image having the highest correlation with the small image Ls) is specified.

Focusing on the smoothed small image Rs _G1, briefly explaining the calculation method of the small image Ls and the smoothed small image Rs _G1 matching error when the comparison (matching residuals). The image sizes of the small image Ls and the smoothed small image Rs _G1 are the same, and the number of pixels in the horizontal direction and the number of pixels in the vertical direction are respectively M _N and N _N (M _N and N _N are 2 or more) Integer). The pixel value of the pixel at the position (x, y) in the small image Ls is represented by V _Ls (x, y), and the pixel value of the pixel at the position (x, y) in the smoothed small image Rs _G1 is represented by V _Rs. It is represented by (x, y). (Where x and y are integers satisfying 0 ≦ x ≦ M _N −1 and 0 ≦ y ≦ N _N −1). Then, R _SAD representing SAD (Sum of Absolute Difference) between contrast images is calculated according to the following formula (B-3), and R _SSD representing SSD (Sum of Square Difference) between contrast images is represented by the following formula (B− Calculated according to 4).

Let R _SAD or R _SSD be a matching error between the small image Ls and the smoothed small image Rs _G1 . Similarly, a matching error between the small image Ls and each of the smoothed small images Rs _{G2 to} Rs _G8 is also obtained, and the smoothed small image with the smallest matching error is specified. Now, it is assumed that the smoothed small image Rs _G3 corresponding to σ = 5 is specified. In step S205, σ corresponding to the smoothed small image Rs _G3 is set to σ ′. That is, the value of σ ′ is 5.

  In the subsequent step S206, the Gaussian blur represented by σ 'is treated as an image degradation function representing the degradation state of the correction target image Lw, thereby deteriorating the correction target image Lw.

Specifically, in step S206, blurring of the correction target image Lw is removed by applying an unsharp mask filter to the entire correction target image Lw based on σ ′. Processing contents of the unsharp mask filter will be described with an image before application of the unsharp mask filter as an input image I _INPUT and an image after application of the unsharp mask filter as an output image I _OUTPUT . First, a σ ′ Gaussian filter (that is, a σ = 5 Gaussian filter) is used as an unsharp filter, and the input image I _INPUT is filtered using this σ ′ Gaussian filter to generate a blurred image I _BLUR . To do. Then, by subtracting each pixel value of the blurred image I _BLUR from each pixel value of the input image I _INPUT, and generates a difference image I _DELTA between the input image I _INPUT and the blurred image I _BLUR. Finally, an image obtained by adding each pixel value of the difference image I _DELTA to each pixel value of the input image I _INPUT is defined as an output image I _OUTPUT . A relational expression between the input image I _INPUT and the output image I _OUTPUT is shown in Expression (B-5). In Expression (B-5), (I _INPUT · Gauss) represents the result of filtering the input image I _INPUT using a Gaussian filter of σ ′.
I _OUTPUT = I _INPUT + I _DELTA
= I _INPUT + (I _INPUT -I _BLUR )
= I _INPUT + (I _INPUT- (I _INPUT · Gauss) (B-5)

In step S206, the correction target image Lw is handled as the input image I _INPUT to obtain a filtered image as the output image I _OUTPUT . In step S207, the ringing correction image Qw is generated by removing the ringing of the filtered image (the process in step S207 is the same as the process in step S78 in FIG. 17).

By using an unsharp mask filter, the edge of the input image (I _INPUT ) is enhanced, and an image sharpening effect is obtained. However, if the blurring degree at the time of generating the blurred image (I _BLUR ) is significantly different from the actual blur amount included in the input image, an appropriate blur correction effect cannot be obtained. For example, if the degree of blur at the time of blur image generation is larger than the actual blur amount, the output image (I _OUTPUT ) becomes an extremely sharp and unnatural image. On the other hand, if the blurring degree at the time of generating the blurred image is smaller than the actual blur amount, the sharpening effect is too weak. In this correction method, a Gaussian filter whose degree of blurring is defined by σ is used as the unsharp filter, and σ ′ corresponding to the image degradation function is used as σ of the Gaussian filter. For this reason, an optimal sharpening effect is obtained, and a corrected image from which blur is well removed is obtained. That is, it is possible to generate an image with less blur in appearance at a low calculation cost.

FIG. 25 shows an image (that is, an original corrected image) 302 obtained when an optimal σ Gaussian filter is used together with a camera shake image 300 as an input image I _INPUT and a too small σ Gaussian filter. The image 301 obtained in FIG. 5 and the image 303 obtained when using a Gaussian filter with too large σ are shown. It can be seen that if σ is too small, the sharpening effect is weak, and if σ is too large, an extremely sharp unnatural image is generated.

  Further, when the third correction method is adopted, it is not always necessary to record all the image data of the short-exposure image from which the reference image Rw is generated on the recording medium 16 as in the case where the first correction method is adopted. However, if only the image data of the small image Rs or Rs ′ is recorded in the recording medium 16 among all the image data of the short exposure image, the blur correction image Qw can be generated.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 and 2 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The continuous shooting described above is synonymous with continuous shooting (that is, continuous shooting), but various conditions relating to shooting may be fixed or changed during the continuous shooting period. The various conditions related to shooting are the focal position of the image pickup unit 11, the exposure amount of the image pickup unit 11, white balance, and the like. Continuous shooting performed while changing various conditions related to shooting is generally called bracket shooting. For example, when performing bracket shooting with respect to the focal position, the position of the focus lens 31 in FIG. 2 is sequentially changed so that a plurality of captured images are acquired at different focal positions (focus positions). Take a picture.

[Note 2]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, the function of the main control unit 13 can be realized by hardware, software, or a combination of hardware and software. All or part of the functions realized by the main control unit 13 are described as a program, and the program is executed on a program execution device (for example, a computer) to realize all or part of the functions. It may be. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure which shows the time relationship between the imaging | photography period of a preview image, and the imaging | photography period of a to-be-recorded image sequence. It is a partial block diagram of the imaging device of FIG. 6 is a flowchart illustrating a flow of a shooting operation of the imaging apparatus according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating a preview image sequence obtained during a shooting preparation period according to the first embodiment of the present invention. It is a figure which illustrates the imaging | photography pattern of the to-be-recorded image sequence which concerns on 1st Example of this invention. FIG. 2 is a diagram illustrating a structure of one image file recorded on the recording medium of FIG. 1 according to the first embodiment of the present invention. FIG. 2 is a schematic diagram of a data structure of an image file group recorded on the recording medium of FIG. 1 according to the first embodiment of the present invention. 6 is a detailed flowchart of recording control in step S18 of FIG. 7 is a flowchart illustrating a flow of a shake correction operation of the imaging apparatus according to the first embodiment of the present invention. 12 is a detailed flowchart of a positional deviation amount calculation process in step S33 of FIG. It is a figure which shows a mode that a positional offset amount (vector) is accumulated according to 1st Example of this invention. It is a figure which shows the mode of the access with respect to the 1st Example of this invention with respect to an image file. It is a figure which illustrates the imaging | photography pattern of the to-be-recorded image sequence which concerns on 2nd Example of this invention. It is a figure which illustrates the imaging | photography pattern of the to-be-recorded image sequence which concerns on 3rd Example of this invention. FIG. 14 is a flowchart illustrating an operation flow of a shake correction process according to a first correction method according to the fourth embodiment of the present invention. 18 is a detailed flowchart of a Fourier iteration method, which is a part of the blur correction process of FIG. 17. It is a block diagram of the site | part which implements the Fourier iteration method of FIG. It is a flowchart concerning the 4th Example of this invention and represents the flow of operation | movement of the blurring correction process by a 2nd correction method. FIG. 21 is a conceptual diagram of shake correction processing corresponding to FIG. 20. FIG. 14 is a flowchart illustrating an operation flow of shake correction processing according to a third correction method according to the fourth embodiment of the present invention. FIG. 23 is a conceptual diagram of shake correction processing corresponding to FIG. 22. It is a figure which concerns on 4th Example of this invention and shows a one-dimensional Gaussian distribution. It is a figure for demonstrating the effect of the blurring correction process corresponding to FIG. It is a block diagram of the structure which implement | achieves the conventional Fourier iteration method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 12 AFE
DESCRIPTION OF SYMBOLS 13 Main control part 16 Recording medium 33 Image pick-up element 51 1st position shift detection part 52 Shooting control part 53 Recording control part 54 Correction object image extraction part 55 Reference image extraction part 56 2nd position shift detection part 57 Correction process part

Claims (9)

In an image sequence arranged in a time series obtained by continuous shooting by an imaging unit in accordance with a predetermined continuous shooting instruction, n by shooting with the first image of m sheets and an exposure time shorter than the exposure time of the first image. Photographing control means for controlling the imaging means so that the second image of the sheet is included (where m and n are integers and m> n ≧ 1);
Correction target image extraction means for extracting one first image as a correction target image from the m first images;
Reference image extraction means for extracting one second image as a reference image from the n second images;
A blur correction apparatus comprising: correction processing means for correcting blur of the correction target image based on the correction target image and the reference image. A first misregistration detecting means for detecting a misregistration amount between a plurality of images obtained from the imaging means before photographing the image sequence;
The blur correction apparatus according to claim 1, wherein the imaging control unit sets a ratio of n to m based on a detection result of the first misregistration detection unit. The m first images are formed from m _A first images and m _B first images taken after the m _A first images, where m _A and m _B are Integer, m _A ≧ 2, m _B ≧ 1, m = m _A + m _B ),
The blur correction apparatus further comprises a first displacement detection means for detecting a positional deviation amount between the m _A piece of first image,
It said imaging control means, based on a detection result of the first displacement detection means, blur correction device according to claim 1, characterized in that setting the ratio of n to m _B. The shooting control means sets n = 1 so that only one second image is acquired before or after shooting the m first images, or n = 2, and the m first images. The blur correction apparatus according to claim 1, wherein the imaging unit is controlled so that the second image is acquired one by one before and after the image is captured. A second misregistration detection unit for detecting a misregistration amount between the correction target image and the reference image;
The correction processing unit corrects a blur of the correction target image through a positional shift correction between the correction target image and the reference image using a detection result of the second positional shift detection unit,
The second misregistration detection means, when the m first images and the n second images are arranged in time series, the first image as the correction target image and the first image as the reference image. When another image exists between two images, the accumulation target image sequence including the other image, the correction target image, and the reference image is specified, and the accumulation target image sequence is temporally adjacent. 5. The positional shift amount between the correction target image and the reference image is detected by respectively obtaining and accumulating the positional shift amount between two images. The shake correction apparatus according to any one of the above. A recording control means for storing m image files respectively recording the image data of the m first images in a recording means;
The recording control means records image data of one second image included in the n second images in only one of the m image files. The blur correction device according to any one of claims 1 to 5. Imaging means for outputting a signal representing an image by photographing;
An image pickup apparatus comprising: the shake correction apparatus according to claim 1 that receives an output signal of the image pickup means. Imaging means for outputting a signal representing an image by photographing;
Instruction receiving means for receiving a predetermined continuous shooting instruction;
Recording means for recording image data of an image based on the output signal of the imaging means,
In an imaging apparatus that causes the imaging unit to continuously capture m first images in accordance with the continuous shooting instruction and records the image data of the m first images on the recording unit (where m is an integer) ),
In the image sequence arranged in a time series obtained by continuous shooting according to the continuous shooting instruction, the m first images and n second images obtained by shooting with an exposure time shorter than the exposure time of the first image. Photographing control means for controlling the imaging means so that an image is included (where n is an integer and m> n ≧ 1);
Recording control means for causing the recording means to record image data of the image sequence;
Correction target image extraction means for extracting one first image as a correction target image from the m first images;
Reference image extraction means for extracting one second image as a reference image from the n second images;
An imaging apparatus comprising: correction processing means for correcting blur of the correction target image based on the correction target image and the reference image. In an image sequence arranged in a time series obtained by continuous shooting by an imaging unit in accordance with a predetermined continuous shooting instruction, n by shooting with the first image of m sheets and an exposure time shorter than the exposure time of the first image. Controlling the imaging means so that the second image of the sheet is included (where m and n are integers and m> n ≧ 1);
While extracting one first image from the m first images as a correction target image,
Extracting one second image from the n second images as a reference image;
A blur correction method, wherein blur correction of the correction target image is corrected based on the correction target image and the reference image.