JP2006033760A - Imaging apparatus and image correction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To restore a high quality image by preventing a deterioration caused by hand-shaking in a frame, at photographing of a dynamic image. <P>SOLUTION: The deterioration of image data due to camera shake in the frame is restored by a time series shaking detecting signal corresponding to the frame stored, by detecting the shaking at photographing to the image data obtained sequentially by making an imaging element 114 operate continuously, and the deterioration in the image data by the shaking generated among the frames is restored, by correcting the relative position of the image data according to an image shift generated among the frames. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus and an image correction method for detecting camera shake and restoring a blurred image to an image without blur.

  When camera shake is detected during moving image shooting, image blurring between frames causes an image to be shaken and gives an unpleasant feeling. Therefore, a configuration that corrects camera shake during shooting to prevent image shake is known. For example, in Japanese Patent Application Laid-Open No. 11-266392, as camera shake correction of a video camera, exposure is performed only when the detected camera shake amount is within a predetermined range, so that only a screen with a stable subject position is photographed and camera shake correction is performed. Is getting a picture. However, this configuration is effective for still images, but when applied to moving images, the exposure interval between frames is different, so the captured moving image becomes jerky, and a sense of incongruity cannot be avoided. Is difficult.

In Japanese Patent Laid-Open No. 2003-234946, camera shake signals detected from an angular velocity sensor or the like are sampled at a predetermined interval, and a correction amount for reducing camera shake, for example, a correction amount for the center position of the exposure time is calculated from the sampled value. Output images without image shaking. However, according to this configuration, although moving images that are not jerky can be recorded, blurring during the exposure period is unavoidable, and blurred images are recorded in the frame, and high-quality images cannot be obtained.
JP-A-11-266392 JP 2003-234946 A

  An object of the present invention is to provide an imaging apparatus and an image correction method for restoring a high-quality image by preventing deterioration due to blurring in a frame during moving image shooting.

  In order to achieve the above object, in the present invention according to claim 1, an optical system for imaging a subject; an imaging means for obtaining image data from a subject image formed by the optical system; Exposure control means for controlling exposure to the element; camera shake detection means for detecting camera shake; and storing time-series camera shake detection signals output from the camera shake detection means during the exposure period of the image sensor Camera shake detection signal storage means for performing continuous operation means for continuously operating the image pickup means; image data sequentially obtained from the image pickup means by the continuous operation means for deterioration caused by camera shake generated in the frame Operates an intra-frame camera shake correction unit that restores using a time-series camera shake detection signal corresponding to the corresponding frame stored in the camera shake detection signal storage unit. And a camera shake correction controller that operates an inter-frame camera shake correction unit that performs correction by correcting the relative position of image data in accordance with an image shift that occurs between frames. It is comprised.

  According to a second aspect of the present invention, a monitor for displaying image data from the image pickup means; and image data that has undergone camera shake correction by the intra-frame camera shake correction means and the inter-frame camera shake correction means are displayed on the monitor. Display control means for doing so.

  The present invention according to claim 3 further comprises recording means for recording on the recording medium to which image data subjected to camera shake correction by the intra-frame camera shake correction means and the inter-frame camera shake correction means is applied. Has been.

  According to a fourth aspect of the present invention, there is provided an image compression means for compressing image data that has been subjected to camera shake correction by the intra-frame camera shake correction means and the inter-frame camera shake correction means; and the compressed data compressed by the image compression means. And a recording means for recording on the applied recording medium.

  In the present invention according to claim 5, the image stabilization controller operates the in-frame image stabilization means, and then operates the image shift generated between frames from the image data in which the deterioration due to the image stabilization occurring in the frame is corrected. The inter-frame camera shake correction means is operated based on the image shift occurring between the frames.

  In the present invention according to claim 6, the camera shake correction controller operates the intra-frame camera shake correcting means after operating the inter-frame camera shake correcting means.

  In the present invention according to claim 7, the camera shake correction controller obtains an image shift generated between frames from the image data, and operates an inter-frame camera shake correction unit based on the image shift generated between the frames.

  In the present invention according to claim 8, the camera shake correction controller obtains an image shift generated between frames according to the camera shake detection signal, and operates an inter-frame camera shake correction unit based on the image shift generated between the frames. ing.

  In the present invention according to claim 9, the camera shake correction controller can selectively operate both or one of the intra-frame camera shake correcting means and the inter-frame camera shake correcting means.

  In the present invention according to claim 10, image data deterioration due to camera shake in a frame is detected and stored for image data sequentially obtained by continuously operating the imaging means. The image data is restored by the time-series camera shake detection signal corresponding to the frame, and the image data deterioration due to camera shake occurring between frames is corrected by correcting the relative position of the image data according to the image shift occurring between frames. Restoring.

  According to the configuration of the present invention according to claim 1, in addition to the camera shake correction in the frame in addition to the camera shake correction between the frames in moving image shooting, in addition to performing only the camera shake correction between the frames, a high-quality image is obtained. Is obtained.

  According to the configuration of the present invention according to claim 2, in the image monitoring the subject, in addition to the camera shake correction between the frames, the camera shake correction in the frame is performed. A high-quality image can be observed.

  According to the configuration of the present invention according to claim 3, in addition to the camera shake correction in the frame in addition to the camera shake correction between frames in moving image shooting, in addition to performing only the camera shake correction between the frames, Can be recorded.

  According to the configuration of the present invention according to claim 4, in order to compress and record the image after performing the camera shake restoration calculation, it can be performed without deterioration before compression during the image restoration calculation, and an accurate camera shake restoration calculation can be performed. . Further, since it can be recorded after being compressed, a large number of images can be stored even in a small and inexpensive storage medium with a small capacity.

  According to the configuration of the present invention according to claim 5, since the camera shake correction in the frame is performed in moving image shooting, and the image shift between frames is obtained from the image data based on the data, the camera shake in the frame Compared with the case of obtaining an image shift between frames using an image that is not corrected, an accurate shift amount between frames can be calculated, and more accurate camera shake correction can be performed.

  According to the configuration of the present invention according to claim 6, since the inter-frame correction is performed first and the intra-frame correction can be performed after the area that is actually displayed as an image is determined, the useless portion that is not used for display However, the amount of processing is smaller than when correction is applied.

  According to the configuration of the present invention of claim 7, regarding the shift of the frame, since the output of the camera shake detection means is generally longer between frames than within the frame, the shift of the frame is caused by integration of noise components. There is no inaccuracy and an accurate shift is possible.

  According to the configuration of the present invention according to claim 8, since both the frames and within the frame are based on the output of the camera shake detection means, even if there is something moving on the subject side in the screen, it is influenced by it and the frame The shift does not become inaccurate, and it is possible to reliably correct deterioration due to camera shake of a non-moving subject.

  According to the configuration of the present invention according to claim 9, since the operation of the unnecessary portion is not performed, for example, when deterioration in the frame due to camera shake is small, power consumption can be suppressed.

  According to the configuration of the present invention according to claim 10, in addition to the camera shake correction in the frame in addition to the camera shake correction between the frames in the moving image shooting, in addition to performing only the camera shake correction between the frames, a high-quality image is obtained. Is obtained.

  According to the present invention, the degradation of image data in a frame is restored by a time-series camera shake detection signal corresponding to the frame, and the degradation of image data between frames depends on the image shift occurring between frames. By correcting and restoring the relative position, high-quality image restoration is realized by eliminating the influence of blurring within and between frames during moving image shooting.

  Embodiments of the present invention will be described below with reference to the drawings. 1A and 1B show a first embodiment of the present invention applied to a digital camera as an imaging device, and FIGS. 1A and 1B are a front perspective view and a rear perspective view of the digital camera.

  As can be seen from FIGS. 1A and 1B, the lens unit 2 is connected to the front surface of the camera body 1, and the finder 6 is integrally assembled to the back surface of the camera body 1. The lens unit 2 is composed of a plurality of photographing lenses and driving means thereof, the details of which will be described later with reference to FIG.

  Reference numeral 3 denotes a release switch, which is pressed (turned on) to start a photographing operation. A zoom switch 4 includes a T button 4-1 and a W button 4-2. When the T button is pressed, the zooming operation of the taking lens is performed to the telephoto side, and when the W button is pressed, the zooming to the wide side is performed. Operation is performed. Reference numeral 5 denotes a camera shake mode setting switch. When the camera shake mode switch 5 is pressed to set the camera shake mode, the mode lamp 5-1 is lit to show that the camera shake mode is set. Reference numeral 6 denotes a viewfinder, and the viewfinder 6 is an electronic viewfinder that magnifies a small LCD with a loupe, for example, and displays an image of an image pickup device (CCD) in real time, and a so-called through image can be displayed on the viewfinder. Yes. Reference numeral 7 denotes a mode key (slide key) for switching between a still image and a moving image. When the mode key 7 is set to the S side (STILL), the still image shooting mode is set, and when the mode key 7 is set to the M side (MOVIE), the moving image shooting mode is set. .

  A flash 8 is an auxiliary light that illuminates the subject by emitting light when the luminance is low. Reference numeral 9 denotes a mode operation key. Four buttons are arranged around a determination button at the center of the mode operation key 9, and the macro operation, self-timer, flash and the like are turned on by the mode operation key 9. Reference numeral 10 denotes a rear LCD panel that reproduces a photographed image and can display a through image. Reference numeral 11 denotes a power switch. By pressing the power switch 11 to turn it on, exposure and imaging can be performed. The rear LCD panel 10 is used as a monitor together with the viewfinder 6.

  FIG. 2 is a schematic diagram of a lens unit 2 that is an optical system, and the lens unit has, for example, three lenses 12, 13, and 14. Among the three lenses, the lenses 12 and 13 are so-called variable magnification lenses (zoom lenses) that change the focal length of the lenses by changing the positional relationship between them. The driving force is transmitted to the zoom lens driving cam mechanism 17 via the gears 18a and 18b, and the lenses 12 and 13 are moved along the optical axis by the lens driving cam mechanism.

  The lens 14 is a so-called focus lens that adjusts the focus shift by moving back and forth along the optical axis. During focus adjustment, the driving force of the focus motor 105 is applied to the focus lens via the gears 20a and 20b. This is transmitted to the drive cam mechanism 19 and the lens 14 is moved by the lens drive cam mechanism. An imaging device (imaging means) 114 made of, for example, a CCD is positioned behind the lens 14, and the light flux that has passed through the lenses 12, 13, and 14 is imaged on the imaging device and is photoelectrically converted by each pixel of the imaging device. Take an image. A diaphragm 15 and a shutter 16, and the light amount (exposure) to the image sensor 114 is controlled by the diaphragm 15 and the shutter 16. Instead of the mechanical shutter 16, an element shutter (electronic shutter) of the image sensor 114 may be used.

  FIG. 3 is a configuration diagram of a control circuit of the digital camera, and 101 is a battery made of a rechargeable battery such as a lithium ion rechargeable battery. Reference numeral 102 denotes a power supply circuit that generates a power supply of a necessary voltage from the battery 101 by a booster circuit or a step-down circuit and supplies it to each processing circuit. A motor driver circuit 103 includes an electric circuit including a switching transistor, and drives and controls the focus motor 104, the zoom motor 105, the shutter motor 106, and the aperture motor 107 in accordance with instructions from the sequence control circuit 119. Reference numerals 108 and 109 denote angular velocity sensors (camera shake detection means) that detect angular velocities around the X axis and Y axis that are orthogonal to each other, and are orthogonal to each other with the long side direction of the element as an axis, as shown in FIG. The angular velocity along the axis is detected.

  Reference numeral 110 denotes an analog processing circuit that cancels the offset of the outputs of the angular velocity sensors 108 and 109 and amplifies the output, and constitutes a camera shake detection unit together with the angular velocity sensors 108 and 109. The output of the analog processing circuit 110 is converted into a digital signal by the A / D conversion circuit 111 and input to the basic trajectory calculation circuit 112. The basic trajectory calculation circuit 112 integrates the input with time to calculate a displacement angle for each time, and outputs the displacement angle corresponding to the time, that is, outputs in time series, (CCD 114 on the imaging surface). (Upper) Shake locus in the vertical and horizontal directions due to camera shake near the optical axis of the image is detected. Here, the detection of camera shake is not limited to the angular velocity sensors 108 and 109. If the calculation process is changed instead of the angular velocity sensors 108 and 109, the camera shake is detected by an angular acceleration sensor or two pairs of acceleration sensors. A similar blur locus may be calculated. Reference numeral 113 denotes a trajectory memory circuit that stores the shake trajectory detected by the basic trajectory calculation circuit 112, and functions as a camera shake detection signal storage means.

  Reference numeral 114 denotes an image pickup device composed of a CCD located behind the lens unit 2 described in FIG. 2, 115 denotes a CCD output processing circuit that processes output from the image pickup device (CCD) 114, and 116 denotes output data of the image pickup device 114 and This is an image memory that temporarily holds image data being processed by the CCD output processing circuit 115. Data stored in the image memory 116 is subjected to basic processing such as RGB processing and shading correction processing by the image processing circuit 117. However, the image processing circuit 117 does not perform γ conversion or image compression that impedes the image restoration operation of the camera shake image (these are performed by the image compression / decompression circuit 151 described later), and the data is stored in the image restoration operation circuit 123 or It is sent to the image shift circuit 132. The image sensor 114 is driven and controlled by a control signal from the sequence control circuit 119 via a CCD driver (not shown).

Reference numeral 122 denotes a circuit that calculates an image restoration function f ⁻¹ for restoring image degradation due to camera shake, and the image restoration function f ⁻¹ indicates how the original image changes from the output of the basic trajectory calculation circuit 112. Will be output. The image restoration function f ⁻¹ is directly calculated from the output of the basic trajectory calculation circuit 112 at the center of the screen, and in other than the center of the screen, the digital camera lenses 12, 13, 14 cause image distortion depending on the zoom position and the focus position. Because it has (distortion), correction is necessary. For this reason, in the digital camera of the embodiment, image distortion (distortion) correction data corresponding to the zoom position and the focus position is stored in the correction value storage memory 118 (distortion information storage means) for each area of the screen.

For example, if the image around the screen is compressed with respect to the image at the center of the screen due to the distortion, the locus change is also compressed at the periphery of the screen with respect to the center of the screen. The trajectory data output from the basic trajectory calculation circuit 112 is corrected for each image area based on the value in the correction value storage memory 118, and the corrected trajectory data is output to the image restoration function calculation circuit 122. That is, the trajectory correction data stored in the correction value storage memory 118 is input to the trajectory correction circuit 121, and the image restoration function calculation circuit 122 performs image restoration function f ⁻ for each screen area based on the output of the trajectory correction circuit 121. ¹ is calculated. Here, the image restoration function f ⁻¹ is an inverse function of the image degradation function f generated by camera shake.

In the image restoration calculation circuit 123 to which data not subjected to γ conversion or image compression is sent from the image processing circuit 117, image conversion is performed by the image restoration function f ⁻¹ calculated by the image restoration function calculation circuit 122 for each area of the screen. It is. An image that has been restored from image degradation due to camera shake by eliminating the effects of image distortion (distortion) by the image restoration arithmetic circuit 123 is compressed by the image compression / decompression circuit 151 and is stored in an image storage medium such as a built-in flash memory. The data is written in the area 153 using the recording means 152. An external memory such as a loadable memory card may be used as the image storage medium 153 instead of the built-in flash memory. It should be noted that the electronic shake of a still image that electronically corrects the image distortion (distortion) of the lenses 12, 13, and 14 for each area of the screen from the trajectory correction circuit 121, the image restoration function calculation circuit 122, and the image restoration calculation circuit 123. The correction circuit 120 is formed, the trajectory correction circuit 121 is used as a camera shake detection signal correction unit, the image restoration function calculation circuit 122 is used as an image restoration function calculation unit, and the image restoration calculation circuit 123 is used as a camera shake restoration unit. Reference numerals 151 each function as compression means.

  Reference numeral 119 denotes a sequence control circuit composed of a CPU such as a microcomputer, which detects the on / off state of the release switch 3, zoom switch 4 (T, W), power switch 11, camera shake mode switch 5, mode key 7, etc. The movement of the component is controlled to control the entire digital camera. Specifically, the sequence control circuit 119 includes a (sequence) controller, continuous operation means for continuously operating the image sensor, display control means for controlling display on the monitor (viewfinder 6, rear LCD panel 10), first , A controller of the second camera shake correction means (image restoration arithmetic circuit 123, image shift circuit 132).

  Reference numeral 131 denotes a circuit that calculates an inter-frame shift amount during a period during which a through image is acquired. The inter-frame shift amount calculation circuit 131 receives a camera shake trajectory for each frame period from the basic trajectory calculation circuit 112 and responds. The amount of the image to be shifted is calculated. Reference numeral 132 denotes an image shift circuit which receives an image from the image sensor (CCD) 114 from the image memory 116, shifts by the amount of camera shake based on the output of the inter-frame shift amount calculation circuit 131, and produces a moving image (or a through image). Perform camera shake correction at. A moving image electronic image stabilization circuit 130 is formed from the inter-frame shift amount calculation circuit 131 and the image shift circuit 132. If the image restoration calculation means 123 in the still image is the first camera shake correction means, the image shift circuit 132 in the moving image can be said to be the second camera shake correction means.

  The motion-corrected moving image processed by the moving image electronic image stabilization circuit 130 is compressed by the image compression / expansion circuit 151 (compression unit) and recorded on the image recording medium 153 by the recording unit 152. Regardless of whether it is a still image or a moving image, it is sent as a monitor image to the rear LCD panel 10 or the viewfinder 6 on the back of the camera body and displayed. The image compression / decompression circuit 151 also has a decompression function for displaying the image data read from the image storage medium 153 by the recording unit 152 on the display 10 and the viewfinder 6. If the output from the image restoration arithmetic circuit 123 is recorded by the recording means 152 on an image storage medium 153 such as a built-in flash memory or an external memory (for example, a loadable memory card), a sharp image can be recorded on the entire screen.

  The electronic camera shake correction in the still image will be described. FIG. 4 is an image of the electronic camera shake correction in the still image, and FIGS. 4A and 4B are the camera shake (rotation angles) θx and θy on the X axis and the Y axis. FIG. 4C shows a blur locus on the image sensor (CCD) 114, and FIG. 4D shows the relationship between the original image and the captured image.

As described with reference to FIG. 3, the X-axis and Y-axis camera shakes detected by the angular velocity sensors 108 and 109 are sent to the basic trajectory calculation circuit 112 in response to time as shown in FIGS. In other words, time-series displacement angles θx and θy data are output. Next, since the focal length of the lens is known from the zoom position at that time, the displacement locus of the blur on the image sensor (CCD) 114 is calculated by paraxial calculation as shown in FIG. Then, the image degradation coefficient f due to camera shake is calculated from the blur locus on the image sensor 114, and the captured image (original image) i should be degraded to the blurred image j by the image degradation coefficient f. When the function f ^−1, that is, the image restoration function is calculated and obtained and inversely transformed using the image restoration function f ⁻¹ , the captured image i is restored.

As described above, in the still image, the image degradation coefficient f is calculated from the blur locus on the image sensor 114 due to the time-series camera shake due to the camera shake at the time of shooting, and the image blur is caused by the inverse function f ^{−1 of} f, that is, the inverse transformation by the image restoration function. The image is restored. At this time, the trajectory correction circuit 121 corrects the blur trajectory to eliminate the influence of the distortion of the optical system. Therefore, even if there is distortion in the optical system, the accurate correction is performed for each screen area from the center to the periphery. An image trajectory due to camera shake is output, and the camera shake can be accurately restored over the entire screen, and a sharp image can be obtained over the entire screen.

  5A and 5B are images of moving image electronic image stabilization. FIGS. 5A, 5B, and 5C show three fluctuating frames, and FIGS. 5D and 5E simply display the three frames sequentially. The image and the image which displayed the corrected image sequentially are shown. The image in FIG. 4D corresponds to an image not subjected to camera shake correction, and the image in FIG. 5E corresponds to an image subjected to camera shake correction.

  In a moving image, a shift between frames is recognized as camera shake, and is corrected by image shift. As shown in FIGS. 5A, 5B, and 5C, for example, when three images 1, 2, and 3 are considered, an image represented by (u →) is between 1 and 2 on the paper surface. The vector movement in the direction deviating to the lower left side is assumed to be the vector movement in the direction deviating to the lower right side on the paper between the images 2 and 3 (v →). In this case, when images 1, 2, and 3 are simply displayed sequentially, the images appear blurred as shown in FIG. However, if the inverse vectors of u → and v → are shifted and displayed sequentially (image 1 + image 2 * (− u →) + image 3 * (− u →) * (− v →))), FIG. As you can see, there is a clear image without camera shake. Here, “*” is an operator representing image shift.

  6A and 6B show the correction amount (maximum shift) and the image clipping range for moving images, a through image in the moving image mode, a through image in the still image mode, and an image stabilization in the still image. Show.

  Assuming that the effective range of the CCD image is 100% when the camera shake mode is not set, the image has a predetermined spread by the image restoration function in the still image camera shake mode, and there is no image data outside the imaging range. Therefore, for example, a 95% range of diagonal length ratio is set as an imaging range, and a captured image in this range is recorded with electronic camera shake correction. However, the amount of camera shake of a still image is small because it is within the shutter exposure time as compared to the case of sequentially shifting with a moving image, and the margin of the periphery may be smaller than that of a moving image.

  The size of the effective imaging range in the moving image blurring mode is smaller than that of a still image, and for example, a range of 70% in the diagonal length ratio is set as the imaging range. This is because an image is shifted in a moving image, and thus the shift amount is larger than a still image and takes a longer time.

  Next, an imaging range corresponding to an image displayed as a through image will be described. When both the still image and the moving image are not in the camera shake correction mode, the imaged and recorded range is a range equivalent to 100% in the diagonal ratio on the CCD, and the through image is also 100 in the diagonal ratio on the CCD. Displays an image in the% range.

  In the case of moving image shooting and in the camera shake correction mode, the same range as the captured and recorded range is displayed as a through image. This is a size of 70% in the diagonal ratio, and is sequentially shifted (moved) within the range of the effective pixels of the CCD (in the range of 100% in the diagonal ratio) in order to correct camera shake. On the other hand, in the case of still image shooting and in the camera shake correction mode, the range on the CCD that is captured and recorded is different from the range on the CCD that is displayed as a through image. This is because the camera shake correction method when imaged and recorded is different from the camera shake correction method when displaying a through image. However, even with different camera shake correction methods, it is necessary to substantially match the range that is captured and recorded with the range that is displayed as a through image. For this reason, in the case of the camera shake correction mode in the case of still image shooting, for example, the captured and recorded range is 95% of the CCD diagonal ratio, whereas the through image range is the size of the CCD diagonal ratio of 90%. is there. The range of the through image is sequentially shifted within the range of 95% of the CCD diagonal ratio for camera shake correction. This still image live image stabilization amount (shift amount) is in the range of 5%, and the maximum shift amount is small compared to the moving image live image, and it cannot cope with large camera shake. A range substantially the same as the range can be displayed on the viewfinder 6 and the rear LCD panel 10.

  7 and 8 show the main flow of the image restoration operation. First, the power switch 11 is pressed (S101), a retracted lens is set up (S102), and it is determined whether or not the camera shake correction mode is set (S103). Each time the camera shake mode switch 5 is pressed, it is repeatedly turned on and off. If it is on, the mode lamp 5-1 is turned on and the camera shake correction flag 1 is set (S104). If it is off, the mode lamp 5-1 is turned off. Then, the camera shake correction flag is set to 0 (S105).

  Next, a still image and a moving image are determined as shooting modes (S106), and if the mode key 7 is positioned on the M side and the moving image mode is selected, the process proceeds to S120 in FIG. If it is the still image mode, it is determined whether or not the camera shake flag is 1 (S107). If the flag is 1, a through image that has been subjected to camera shake correction using the CCD 90% screen is displayed (S108), and if the flag is 0, a through image is displayed, but the camera shake correction is not performed, and the image remains blurred. Through images are displayed (S109). Here, either the viewfinder 6 or the rear LCD panel 10 is selected by the photographer (user) as the LCD to be displayed, and a through image is displayed on the selected LCD. Alternatively, the image may be displayed on both the viewfinder 6 and the rear LCD panel 10 so that the photographer can see one of the displays.

  Then, it is confirmed that the release switch 3 is turned on (S110). If the release switch 3 is on (if the release switch is pressed), a still image is captured (S111), and the image processing circuit 117 performs image processing (S113). It is determined whether or not the correction flag is 1 (S114). If the release switch 3 is not pressed, it is determined whether or not another switch is operated (S112). If any switch is on, the corresponding processing is performed. If any switch is off, the process returns to S103. .

  If the flag is 1 in S114, the image restoration function calculation circuit 122 calculates an image restoration function that eliminates the influence of image distortion (distortion) corresponding to each area of the screen, and the image restoration calculation circuit 123 calculates the CCD 95. The camera shake correction using the% screen is performed (S115). If the flag is 0, the camera shake correction is not performed and the process proceeds to S116. In S116, image processing such as γ conversion and image compression is performed in the image compression / decompression circuit 151, and a captured image (still image) is displayed on the rear LCD panel 10 or the like (S117), and is recorded on the image recording medium 153 by the recording unit 152. Is written (S118), and after writing, the process returns to S103.

  Next, the main flow of moving images will be described with reference to FIG. If it is set to a moving image in S106 (if the mode key 7 is located on the M side), it is determined whether or not the camera shake flag is 1 (S120). A through image that has been shifted by the image shift circuit 132 by the amount of shift calculated in step S1 and corrected for camera shake using the CCD 70% screen is displayed (S121). If the flag is 0, the through image is displayed, but the camera shake correction is not performed, and the image that remains blurred is displayed on the LCD (S122). Note that the image in FIG. 5D corresponds to the through image blurred in S122, and the image in FIG. 5E corresponds to the through image subjected to shift correction in S121.

  Then, it is confirmed that the release switch 3 is turned on (S123). If the release switch 3 is turned on (if the release switch is pressed), moving image shooting is started (S124), and it is determined whether or not the camera shake flag is 1 (S124). S126). If the release switch 3 is not pressed, it is determined whether or not another switch is operated (S125). If any switch is on, the corresponding processing is performed. If any switch is off, the process returns to S103. It is.

  If the flag is 1 in S126, the captured image that has been shifted using the CCD 70% screen and corrected for camera shake is displayed on the LCD in real time (S127). If the flag is 0, camera shake correction is not performed and imaging is performed. The image is displayed on the LCD while being shaken in real time (S128). Similarly to the through image display in S121 and S122, the blurred captured image in S128 is displayed as an image in FIG. 5D, and the shifted corrected captured image in S127 is displayed as an image in FIG. 5E. Is done. Then, images are continuously captured until the release switch 3 is pressed again. When the release switch is pressed again (S129), the imaging stops (S130), and the moving image is written to the image recording medium 153 (S131). The process returns to S103.

  With such a configuration, it is possible not only to confirm that camera shake is corrected even when taking a still image or a moving image with the viewfinder 6 and the rear LCD panel 10, but also because the range of the through image substantially matches the range that can be actually captured. Easy and framing can be set quickly. In addition, since the trajectory due to image distortion such as distortion is corrected for each screen range, the influence of the distortion of the lens image is eliminated, and a correct change amount of the trajectory for each screen range can be obtained, which is good over the entire screen. Camera shake correction is possible.

A modification of the first embodiment will be described. In the first embodiment, the trajectory data output from the basic trajectory calculation circuit 112 is corrected for each image area by the trajectory correction circuit 121 based on the value in the correction value storage memory 118, and the corrected trajectory data is calculated as an image restoration function. Output to the circuit 122. Next, the image restoration function calculation circuit 122 calculates an image restoration function f ⁻¹ for each screen area based on the output of the trajectory correction circuit 121, and then the image restoration calculation circuit 123 performs an image restoration function calculation circuit. The image restoration calculation is performed by the image restoration function f ⁻¹ calculated in 122. On the other hand, you may make it as a modification as follows.

First, the locus correction circuit 121 is deleted, and the output line from the correction value storage memory 118 is modified to be connected to the image restoration function calculation circuit 122. Then, the trajectory data output from the basic trajectory calculation circuit 112 is directly processed by the image restoration function calculation circuit 122 to calculate and obtain only one type of image restoration function f ⁻¹ . Next, the image restoration function f ⁻¹ is corrected for each image area based on the value in the correction value storage memory 118 to obtain a different image restoration function f ⁻¹ for each image area. Next, the image restoration calculation circuit 123 performs an image restoration calculation using an image restoration function f ⁻¹ that differs for each image area. In this modification, the image restoration function calculation circuit 122 functions as an image restoration function calculation unit and also functions as an image restoration function correction unit.

According to the configuration of the modified example, even with the same camera shake, the trajectory of the image changes due to the distortion, and the area other than the center of the screen and the area other than the center of the screen changes due to compression, enlargement, and direction change, resulting in an image degradation function. Even when f ^-1 varies from area to area, the image degradation function f ^-1 is corrected for each area to obtain an optimal image restoration function f ^-1 , so that accurate camera shake restoration can be performed over the entire screen, and the entire screen is sharp. Images can be obtained.

  In a camera equipped with image stabilization means for performing restoration calculation from image data obtained after still image shooting, in the through image display for observing a subject in the preparation stage of still image shooting, a format for performing the restoration calculation described above The camera shake correction means cannot be applied. Or even if it is applied, the desired effect cannot be obtained. However, as shown in FIGS. 7 and 8, different camera shake correction is performed for still image capturing and live view display, and for live view display, moving image (through image) camera shake correction is activated to capture still images. At times, different image stabilization for still images is activated. In addition, the through image in still image mode and the through image in moving image mode have different image clipping ranges and maximum correction amounts for electronic image stabilization, and are set to be optimal for still images and moving images. A camera shake correction mode is set, camera shake correction for moving images is performed at the time of camera shake correction, and a restored image is displayed after performing camera shake correction correction based on the camera shake locus for still image capturing. This example is referred to as Example 2.

  According to the second embodiment, when the camera shake prevention mode is set, an image with less camera shake is displayed by camera shake correction having another format for the live view image and effective for the live view image. , The photographer can be notified that the camera shake mode is operating. For this reason, it is possible for the photographer to confirm that the camera shake mode is set during photographing while observing the subject. In addition, since camera shake during observation is reduced, it becomes easier to observe the subject. In addition, camera shake compensation for live view is also stopped when the camera is not in still image stabilization mode.If there is a large amount of camera shake, the camera operator should be aware of camera shake when observing the subject and set the camera shake compensation mode. There is an effect.

  In addition, since Example 1, 2 has shared FIG. 1-8, description of FIGS. 1-8 is abbreviate | omitted about Example 2. FIG.

  Another embodiment (embodiment 3) in which still image electronic camera shake correction and moving image electronic camera shake correction are performed on a captured image after correcting lens distortion will be described with reference to FIGS. FIG. 9 is a block diagram of the control circuit of the digital camera. The correction value storage memory 118 and the trajectory correction circuit 121 are omitted as constituent elements, the distortion correction value memory 171 (distortion information storage means, image deterioration information storage means), image 3 is different from FIG. 3 in that a distortion correction circuit 172 is added. In the third embodiment as well, the drawings of FIGS. 1 to 8 except for FIG. 3 are common. However, the third embodiment is different from the first embodiment in that correction in accordance with lens distortion (distortion) is added to the captured image by the image distortion correction circuit 172 in the image processing of S113.

  In the configuration diagram of the control circuit of the digital camera in FIG. 9, a distortion correction value corresponding to lens distortion (distortion) is stored in the distortion correction value memory 171. After applying distortion correction, still image electronic image stabilization and moving image electronic image stabilization are performed. Further, the distortion correction value memory 171 is simply a lens characteristic correction value memory, and correction data for aberrations caused by characteristics of the photographing lens other than the distortion correction value (distortion correction value) are also stored in the correction value memory. In addition, the image distortion correction circuit 172 may be used as a lens characteristic correction circuit to correct aberrations and the like due to the characteristics of other photographic lenses. According to this configuration, not only when there is an influence of distortion of the photographing lens, but also when there is image degradation caused by the optical system such as aberration, distortion of the optical system or aberration or the like before the camera shake restoration calculation Since the camera shake restoration calculation is performed after correcting the image deterioration due to the image and removing the influence of the image deterioration, the camera shake can be accurately restored over the entire screen by a simple calculation, and a sharp image can be obtained over the entire screen.

  FIG. 10 shows a processing flow of the sequence control circuit 119 in the third embodiment. First, the release switch 3 is pressed to capture an image (S201), and a distortion correction value for distortion from the zoom and subject distance is read from the distortion correction value memory 171 (S202), and the image distortion correction circuit 172 corrects the image distortion of the lens ( S203), a camera shake image restoration function is calculated by the image restoration function calculation circuit 122 from the time-series camera shake trajectory obtained from the camera shake detected by the angular velocity sensors 108 and 109 (S204), and according to the camera shake image restoration function. The image restoration calculation circuit 123 corrects camera shake (S205), compresses the image using the image compression / decompression circuit 151 (S206), and records it on the image recording medium 153 using the recording unit 152 (S207).

  FIG. 11 is a schematic diagram of image distortion when a building is photographed. FIG. 11A is an image of distortion zero, FIG. 11B is an image under barrel distortion, and FIG. An image under a pincushion type distortion, FIG. 11D shows a relation between image height and distortion correction, and FIG. 11E shows an explanatory diagram of the image height. As shown in FIG. 11E, the image height is zero at the center of the screen and 1 at the periphery (outermost circumference), and the same image height on the concentric rectangle.

  Even if lenses are molded under the same material and under the same conditions, it is difficult to avoid variations in lens characteristics. In order to accurately restore an image, it is necessary to consider differences in lens characteristics. Even if the image of the barrel distortion as shown in FIG. 11B or the image of the pincushion distortion as shown in FIG. 11C is brought close to the image of FIG. Variation may deviate from zero distortion. The human eye is familiar with the fisheye lens image and does not feel so uncomfortable with an image distorted into a barrel shape. However, the image distorted in a pincushion is uncomfortable and unnatural is noticeable. For this reason, when the distortion is corrected to zero, it is preferable that the restored image becomes an image distorted in a barrel shape rather than an image distorted in a pincushion shape if it deviates from zero due to variations in lens characteristics.

For this reason, as shown in FIG. 11D, the distortion of the lens image distortion (ball distortion) L ₁ is corrected to a level L ₀ of zero distortion (distortion correction 1), and then the image is corrected for camera shake correction. A restoration operation is performed, and then electronic correction is performed to reversely correct the image to level L ₂ and return to the mold direction (distortion correction 2). Here, the definition of words will be explained slightly. Distortion correction refers to eliminating or reducing the influence of distortion in image data affected by distortion, and reverse distortion correction refers to image data without distortion. On the other hand, it refers to a process of intentionally adding distortion or further increasing the influence of distortion on image data with distortion. Here, when the distortion amount of the distortion correction 2 which is the reverse correction with respect to the distortion correction 1 is reduced, the correction to the pincushion mold is set to + (plus), and the correction to the barrel mold is set to − (minus). The maximum distortion around the image height d = 1 is + 12% for distortion correction 1 and −4% for distortion correction 2. Similarly, with respect to the pincushion distortion, the lens image distortion (pincushion distortion) L ₃ is corrected for distortion zero level L ₀ (distortion correction 1), and then for camera shake correction. An image restoration operation is performed, and then the image is reverse-corrected to level L ₂ by electronic correction to obtain a mold image.

  In this way, by performing distortion correction aiming at zero distortion (distortion correction 1) and then applying reverse correction to the barrel (distortion correction 2), distortion correction 1 due to distortion correction variations caused by differences in lens characteristics. Thus, even if a pincushion image is generated, distortion correction 2 forcibly corrects the pincushion die to a barrel die. Therefore, generation of an image distorted in a pincushion is prevented, and an image without a sense of incongruity is restored. Even if the distortion varies from area to area due to the so-called Jinkasa distortion, which is a combination of pincushion and barrel distortions, the combination of zero distortion correction and reverse correction to the mold (distortion correction 2) makes it uncomfortable. No image is obtained. Distortion reverse correction (distortion correction 2) is performed by the image restoration calculation circuit 123. The image restoration calculation circuit can also be referred to as camera shake restoration means and distortion reverse correction means. Note that the distortion correction 2 in the pincushion distortion is also performed by the image restoration arithmetic circuit 123.

FIG. 12 shows a processing flow of the sequence control circuit 119 in the image restoration of FIG. 11 and is different from the processing flow of FIG. 10 in that distortion correction 2 is added. That is, the release switch 3 is pressed to capture an image (S301), and a distortion correction value for distortion is read from the zoom and subject distance from the distortion correction value memory 171 (S302), and a time series obtained from camera shake detected by the angular velocity sensors 108 and 109 is obtained. The image restoration function calculation circuit 122 calculates a camera shake image restoration function from the camera shake detection signal (camera shake trajectory) (S304), and the lens image distortion (the barrel distortion L ₁ and the pincushion distortion L ₃ ) is reduced to a distortion level L. Aiming at ₀ , the image distortion correction circuit 172 performs distortion correction (distortion correction 1) (S303). Then, and image stabilization is performed to restore operation by the image restoration calculation circuit 123 (S305), and the level L ₂ is inversely corrected to of occurrence of barrel distortion (S306), and image compression by the image compression and expansion circuit 151 From (S307), recording is performed on the image recording medium 153 using the recording unit 152 (S308).

  Another embodiment (embodiment 4) in consideration of image degradation due to camera shake between frames in a moving image will be described with reference to FIGS. Also in the fourth embodiment, the drawings of FIGS. 1 to 8 except for FIG. 3 are common, and the description of FIGS. 1 to 8 except for FIG. 3 is also applied to the fourth embodiment. FIGS. 13 and 14 are block diagrams of the control circuit of the digital camera, which differs from FIG. 3 in that the correction value storage memory 118 and the trajectory correction circuit 121 are omitted as components, and FIG. 15 shows the correction value storage. 3 is different from FIG. 3 in that an inter-frame shift amount calculation circuit 131 is omitted in addition to the memory 118 and the trajectory correction circuit 121, and an image shift calculation circuit 173 is added.

  FIG. 13 is intended for moving images and through-images, and after performing camera shake correction between frames, camera shake correction within the frame is performed. That is, the image shift circuit 132 performs shake correction for each frame in accordance with the camera shake detected by the angular velocity sensors 108 and 109, and the image restoration calculation circuit 123 performs image calculation based on the camera shake locus in the same manner as a still image. The image is displayed on the viewfinder 6 and the rear LCD panel 10 or recorded on the image recording medium 153. In this configuration, blurring in the frame is corrected, and clear through images and moving images are obtained. In addition, in moving image shooting, camera shake correction within a frame is performed in addition to camera shake correction between frames, so that a high-quality image can be obtained in addition to the case of performing only camera shake correction between frames. First, the inter-frame correction is performed, and the intra-frame correction can be performed after the area that is actually displayed as an image is determined. Therefore, the amount of processing is less than when correcting unnecessary parts that are not used for display. I'm sorry.

  The sequence control circuit 119 obtains an image shift generated between frames in accordance with a camera shake detection signal, and operates the image shift circuit 132 based on the image shift generated between the frames. Since the inter-frame and intra-frame outputs are based on the outputs of the angular velocity sensors 108 and 109, even if there is something moving on the subject side in the screen, it may be affected by this and the frame shift may become inaccurate. In addition, it is possible to reliably correct deterioration due to camera shake of a subject that is not moving.

  FIG. 14 also targets moving images and through-images. Contrary to FIG. 13, camera shake correction between frames is performed after camera shake correction within the frame. That is, as in the case of a still image, each frame is subjected to image restoration computation by the image restoration computation circuit 123 based on the camera shake trajectory, and then the image shift circuit 132 is adapted to the camera shake detected by the angular velocity sensors 108 and 109 for each frame. After blur correction, the image is displayed on the viewfinder 6 and the rear LCD panel 10 or recorded on the image recording medium 153. Even in this configuration, blurring in the frame is corrected, and a clear through image and moving image can be obtained.

  Also in the fourth embodiment, the camera shake correction between the frames and in the frame is performed and then compressed by the image compression / decompression circuit 151, and then recorded on the image recording medium 153 using the recording unit 152. In order to compress and record the image after performing the above, it can be performed without any deterioration before compression at the time of image restoration calculation, and an accurate camera shake restoration calculation can be performed. Since the camera shake correction between frames and frames is performed after compression, a large number of high-quality images can be stored in a small and inexpensive image recording medium 153 with a small capacity.

  FIG. 15 is the same as FIG. 14 except that an image shift calculation circuit 173 is arranged instead of the interframe shift amount calculation circuit 131. That is, in FIG. 15, the image shift is performed by calculating the shift amount between frames image-wise by the image shift amount calculation circuit 173, for example, by image correlation calculation or the like based on the change in the image between frames. In this configuration, if the image becomes unclear due to the blur between frames, the calculation of the shift amount between frames becomes inaccurate. Therefore, it is effective to perform the blur restoration calculation in the frame first.

  In addition, after moving image stabilization within a frame during movie shooting, the image shift between frames is obtained from the image data based on that data, so images between frames using images that are not subjected to camera shake correction within the frame are used. Compared with the case of obtaining the shift, the shift amount between frames can be calculated more accurately, and more accurate camera shake correction can be performed.

  The sequence control circuit 119 obtains an image shift generated between frames from the image data, and operates the image shift circuit 132 based on the image shift generated between the frames. Therefore, regarding the frame shift, the outputs of the angular velocity sensors 108 and 109 are generally longer between frames than within the frame, and the frame shift does not become inaccurate due to the accumulation of noise components. You can shift.

  Furthermore, it is preferable that the sequence control circuit 119 performs control so that either or both of the image shift calculation circuit 173 and the image shift circuit 132 are selectively operated. If so, deterioration in the frame due to camera shake is caused. Since it is not necessary to operate an unnecessary part such as when it is small, power consumption can be suppressed.

  The embodiments described above are for explaining the present invention, and the present invention is not limited to the embodiments. Modifications and applications other than the above can be made without departing from the gist of the present invention. Needless to say.

  According to the present invention, it is possible to restore high-quality images by eliminating the influence of blurring within and between frames at the time of moving image shooting. Therefore, the need for high-quality moving images is eliminated by eliminating the effects of camera shake. The present invention can be widely applied to various fields.

The digital camera in Example 1, 2 of this invention applied to the digital camera is shown, (A) (B) is the front perspective view of a digital camera, and a rear perspective view. 2 is a schematic diagram of a lens unit 2. FIG. 2 shows a configuration of a control circuit of a digital camera in Examples 1 and 2. (A) and (B) show changes in camera shake (rotation angles) θx and θy on the X-axis and Y-axis, and (C) shows an image on the image sensor (CCD) 114. The blur locus (D) shows the relationship between the original image and the captured image. (A), (B), and (C) show three fluctuating frames, and (D) and (E) show images that are simply displayed sequentially in three frames and corrected images. The images are sequentially displayed. The electronic image stabilization amount in each mode and a CCD image showing the image cutout range are shown. (A) is a moving image, a through image in the moving image mode, a through image in the still image mode, and an electronic image blur in the still image. A correction amount (maximum shift amount), (B) shows a CCD image in the image cutout range. It is a main flow in Example 1,2. It is the main flow in Example 1, 2 similarly to FIG. 4 shows a configuration of a control circuit of a digital camera in Embodiment 3 of the present invention corresponding to FIG. 10 is a processing flow of the sequence processing circuit in the third embodiment. FIG. 6 is a schematic diagram of image distortion in Example 3, (A) is an image diagram of distortion zero, (B) is an image under barrel-type distortion, (C) is an image under pincushion-type distortion, D) shows the relationship between image height and distortion correction, and (E) shows an explanatory diagram of image height. 12 is a processing flow of a sequence processing circuit in image distortion restoration in FIG. 4 shows a configuration of a control circuit of a digital camera in Embodiment 4 of the present invention corresponding to FIG. 10 shows a configuration of a control circuit of a digital camera in a first modification of the fourth embodiment of the present invention. 9 shows a configuration of a digital camera control circuit according to a second modification of the fourth embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera body 2 Lens unit 5 Camera shake mode switch 6 Viewfinder 7 Mode key 10 Rear LCD panel 11 Power switch 108, 109 X-axis, Y-axis angular velocity sensor 112 Basic trajectory calculation circuit 113 Trajectory memory circuit 114 Imaging device (CCD)
117 Image processing circuit (including RGB processing and shading correction)
118 Correction Value Storage Memory 119 Sequence Control Circuit 120 Still Image Camera Shake Processing Circuit 121 Trajectory Correction Circuit 122 Image Restoration Coefficient Calculation Circuit 123 Image Restoration Calculation Circuit 130 Moving Image Shake Processing Circuit 131 Interframe Shift Amount Calculation Circuit 132 Image Shift Circuit 151 Decompression circuit 152 Recording means 153 Image recording medium 171 Distortion correction value memory 172 Image distortion correction circuit

Claims (10)

An optical system for imaging a subject;
Imaging means for obtaining image data from a subject image formed by the optical system;
Exposure control means for controlling exposure to the image sensor;
Camera shake detection means for detecting camera shake;
A camera shake detection signal storage means for storing a time-series camera shake detection signal output from the camera shake detection means during an exposure period of the image sensor;
Continuous operation means for continuously operating the imaging means;
For image data obtained sequentially from the imaging means by the continuous operation means, deterioration due to camera shake occurring in a frame is caused by a time-series camera shake detection signal corresponding to the corresponding frame stored in the camera shake detection signal storage means. Inter-frame image stabilization is performed by operating the in-frame image stabilization unit to restore, and correcting the degradation caused by camera shake between frames by correcting the relative position of the image data according to the image shift occurring between frames. An image pickup apparatus comprising a camera shake correction controller that operates the means. A monitor for displaying image data from the imaging means;
2. The imaging apparatus according to claim 1, further comprising display control means for displaying image data on which camera shake correction has been performed by the intra-frame camera shake correction means and the inter-frame camera shake correction means. The imaging apparatus according to claim 1, further comprising a recording unit configured to record the image data on which the camera shake correction is performed by the intra-frame camera shake correcting unit and the inter-frame camera shake correcting unit on a recording medium to which the image data is applied. Image compression means for compressing image data that has been subjected to camera shake correction by the intra-frame camera shake correction means and the inter-frame camera shake correction means;
The imaging apparatus according to claim 1, further comprising recording means for recording the compressed data compressed by the image compression means on a recording medium to which the compressed data is applied. The image stabilization controller operates an in-frame image stabilization unit, obtains an image shift generated between frames from image data corrected for image stabilization generated in the frame, and generates an image generated between the frames. The imaging apparatus according to claim 1, wherein the inter-frame camera shake correction unit is operated by the shift. 2. The imaging apparatus according to claim 1, wherein the camera shake correction controller operates the intra-frame camera shake correction unit after operating the inter-frame camera shake correction unit. The imaging apparatus according to claim 6, wherein the camera shake correction controller obtains an image shift generated between frames from the image data, and operates an inter-frame camera shake correction unit based on the image shift generated between the frames. The imaging apparatus according to claim 6, wherein the camera shake correction controller obtains an image shift generated between frames in accordance with the camera shake detection signal, and operates an inter-frame camera shake correction unit based on the image shift generated between the frames. The image pickup apparatus according to claim 1, wherein the camera shake correction controller selectively operates both or one of the intra-frame camera shake correction unit and the inter-frame camera shake correction unit. For image data sequentially obtained by continuously operating the imaging means, image data deterioration due to camera shake in a frame is a time-series camera shake corresponding to the frame stored by detecting camera shake during shooting. An image correction method in which image data is restored by a detection signal, and image data deterioration due to camera shake occurring between frames corrects a relative position of the image data in accordance with an image shift generated between frames.

Images