JP2010233188A - Imaging apparatus, control method thereof and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an image with high vibration-proof accuracy by performing image blur correction of a main subject and image blur correction of a background in a well-balanced manner, even under a proximity photographing condition. <P>SOLUTION: An imaging apparatus includes: exposure control means 18, 113 for sequentially performing photographing to obtain a plurality of images; a weighting setting means 114 for setting weighting for feature points in different regions in an image on the basis of a photographic magnification; a deviation detecting means 115 for detecting the amount of deviation between the images from coordinates of the feature points weighted by the weighting setting means; a coordinate transforming means 116 for transforming the coordinates of the images on the basis of the deviation amount detected by the deviation detecting means; and an image composing means 117 for composing the images which are coordinate-transformed by the coordinate transforming means. The images are composed after corresponding image blurring in accordance with the deviation amount reflected with weighting based on the photographic magnification and image exposure is complemented. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus that improves the accuracy of a captured image by electronically correcting image blur.

  Recently, a system that corrects image blur due to camera shake applied to the camera has been studied, and there is almost no factor that causes a photographer to make a shooting mistake.

  In principle, the image blur is corrected by detecting the acceleration, angular acceleration, angular velocity, angular displacement, etc. with an accelerometer, vibration gyroscope, laser gyroscope, etc. It can be done by mounting on. Then, image blur correction is performed by driving a correction optical system that decenters the photographing optical axis based on camera shake detection information from the vibration detection unit.

  On the other hand, unlike the above-described optical image stabilization system, shooting is repeated a plurality of times with an exposure time that does not cause camera shake. Then, there is an electronic image stabilization system that obtains a photographed image (composite image) with a long exposure time by combining the images obtained by these photographing while correcting the image shift (for example, Patent Document 1).

JP 09-261526 A

  As described above, there are an optical image stabilization system and an electronic image stabilization system as a system that corrects image blur and obtains an image having no blur, but has a common drawback. This is a problem that during close-up shooting, an error in the correction amount due to the difference in shooting magnification becomes large, so that a good shake correction cannot be performed on the entire screen unless the shake correction amount is adjusted for each subject distance. Specifically, when the distance of the main subject is 10 cm and sufficient image blur correction is performed on the main subject, sufficient image blur correction cannot be performed in the background (for example, the shooting distance is 40 cm). On the contrary, there is a risk of increasing the swing.

  This will be explained in more detail. There are two types of shake applied to the image pickup apparatus: an angle shake that rotates around the center of rotation and a shift shake (also referred to as parallel shake) in which the entire image pickup apparatus moves in parallel. The degradation of the image due to the angular shake is constant regardless of the subject distance, and becomes larger as the focal length of the imaging apparatus is longer. On the contrary, image degradation due to shift shake is related to the subject distance and focal length (image magnification), and the larger the image magnification (closer subject distance and longer focal length), the larger the degradation amount. In the case of general shooting conditions (for example, a lens with a subject distance of 1 m and a focal length of 50 mm), the influence of image deterioration due to shift shake can be almost ignored.

  However, in close-up photography (for example, a macro lens with a subject distance of 10 cm and a focal length of 100 mm), image degradation due to shift shake cannot be ignored due to a large photographing magnification. In such a case, it is only necessary to detect shift shake (for example, using an accelerometer in an optical image stabilization system, or to detect image shift in electronic image stabilization), and correct the image shake according to the result. become.

  However, subjects with various shooting distances are mixed in the screen. Therefore, for example, when image shake correction is performed in accordance with the shooting distance of the main subject, image plane deterioration due to shift shake can be prevented for that subject. However, not only a sufficient shake correction cannot be performed as described above for a subject such as the background, but there may be a case where the image is deteriorated.

  In particular, in an electronic image stabilization system, shake is detected from an image deterioration component (image shift) that appears as a result of shift shake or angular shake. For this reason, it is difficult to determine whether the cause of the shake is due to angular shake or shift shake (in an optical image stabilization system, shift shake is detected by an accelerometer, and angular shake is detected by a gyro, so that each shake can be separated. ). Further, the actual shift shake amount varies depending on the shooting distance of each subject in the screen. For this reason, sufficient shake correction cannot be performed on images in which subject distances are mixed during close-up photography.

(Object of invention)
An object of the present invention is to provide an imaging apparatus that can perform image blur correction of a main subject and background image blur correction with good balance even under close-up shooting conditions and can obtain an image with high image stabilization accuracy.

  In order to achieve the above object, the present invention provides an exposure control unit that sequentially performs shooting to obtain a plurality of images, and a weight setting unit that sets weights of feature points of different regions in the image based on a shooting magnification. A deviation detecting means for detecting a deviation amount between the images from the coordinates of the feature points weighted by the weight setting means, and coordinates of the images based on the deviation amount detected by the deviation detecting means. A coordinate conversion unit that performs conversion, and an image combination unit that combines the images after the coordinate conversion by the coordinate conversion unit, and between the images according to the deviation amount in which the weighting based on the imaging magnification is reflected The image capturing apparatus compensates for image exposure by combining after correcting image blur.

  According to the present invention, it is possible to provide an imaging apparatus that can perform image blur correction of a main subject and image blur correction of a background in a well-balanced shooting condition and obtain an image with high image stabilization accuracy.

It is a block diagram which shows the whole structure of the camera which concerns on Example 1 thru | or 4 of this invention. It is a figure for demonstrating the image composition by Example 1 of this invention. It is a figure for demonstrating the lack of a screen by Example 1 of this invention. It is a figure for demonstrating the time of close-up photography by Example 1 of this invention. It is a figure for demonstrating the weighting which concerns on Example 1 of this invention. It is a flowchart which shows main operation | movement of the camera which concerns on Example 1 of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on Example 2 of this invention. It is a figure which shows the area division regarding the weighting of Example 2 of this invention. It is a flowchart which shows the detailed operation | movement in step # 2001 of FIG. It is a figure for demonstrating the weighting correction in Example 2 of this invention. It is a figure which shows the area | region division regarding the weighting of Example 3 of this invention. It is a figure for demonstrating the weighting of Example 3 of this invention. It is a flowchart which shows the main operation | movement of the camera of Example 4 of this invention.

  The best mode for carrying out the present invention is as shown in Examples 1 to 3 below.

  FIG. 1 is a diagram illustrating a configuration of a camera (photographing apparatus) including an electronic image stabilization system according to Embodiment 1 of the present invention. An incident light beam (photographing light) from the photographing lens 11 is limited in light quantity by the diaphragm 13a, and then passes through the shutter 12a and forms an image on the imaging unit 19 including a semiconductor imaging element such as a MOS or CCD.

  The photographic lens 11 is composed of a plurality of optical lens groups, and some or all of these lens groups move on the optical axis 10 in response to a driving force from an AF (autofocus) drive motor 14a, and have a predetermined alignment. The focus is adjusted by stopping at the focal position. The AF drive motor 14a is driven by receiving a drive signal from the focus drive unit 14b. Also, some optical lens groups of the photographic lens 11 move on the optical axis 10 in response to the driving force from the zoom drive motor 15a, and change the photographic angle of view by stopping at a predetermined zoom position. The zoom drive motor 15a is driven by receiving a drive signal from the zoom drive unit 15b.

  The diaphragm 13a has a plurality of diaphragm blades, and these diaphragm blades are actuated by receiving a driving force from the diaphragm driving unit 13b to change an aperture area (diaphragm aperture) serving as a light passage port. The shutter 12a has a plurality of shutter blades, and these shutter blades open and close an opening serving as a light passage port by receiving a driving force from the shutter driving unit 12b. Thereby, the light beam incident on the imaging unit 19 is controlled.

  The strobe 16a is driven (emits light) in response to a drive signal from the flash drive unit 16b in accordance with conditions at the time of photographing (subject brightness, etc.). The speaker 17a is for informing the photographer of the photographing operation, and is driven (sounded) in response to a driving signal from the sounding driving unit 17b.

  The driving of the focus driving unit 14b, the zoom driving unit 15b, the aperture driving unit 13b, the shutter driving unit 12b, the flash driving unit 16b, and the sound generation driving unit 17b is controlled by the photographing control unit 18.

  The photographing control unit 18 receives operation signals from a release operation unit 12c, an aperture operation unit 13c, a zoom operation unit 15c, a flash operation unit 16c, and an image stabilization operation unit 120 described later. Then, according to the shooting state of the camera, the operation signals are given to the focus driving unit 14b, the zoom driving unit 15b, the aperture driving unit 13b, the shutter driving unit 12b, and the flash driving unit 16b, and shooting conditions are set to perform shooting. I am doing so.

  The aperture diameter of the aperture 13a and the light emission of the strobe 16a are normally set automatically on the camera side during shooting. Therefore, the aperture operation unit 13c and the flash drive unit 16b are not necessary, but the photographer can arbitrarily shoot. It is provided for setting conditions.

  The photographing control unit 18 measures subject luminance (photometry) based on an image signal captured by a signal processing unit 111 described later, and based on the photometric result, the aperture diameter of the aperture 13a and the closing timing (exposure) of the shutter 12a. Time). Further, the photographing control unit 18 obtains the in-focus position of the photographing lens 11 based on the output from the signal processing unit 111 while driving the focus driving unit 14b.

The shooting control unit 18 includes a shooting condition detection unit 18a, which is shown separately in FIG. 1 for convenience of explanation. This shooting condition detection unit 18a detects the following conditions at the time of shooting: exposure time, shooting focal length, shooting subject distance, preset number of continuous shots taken, and power supply state. The output of the imaging condition detection unit 18a is input to a weight setting unit 114 described later.

  The video signal output from the imaging unit 19 is converted into a digital signal by the A / D conversion unit 110 and input to the signal processing unit 111. The signal processing unit 111 performs signal processing such as forming a luminance signal and a color signal on the input signal to form a color video signal. Then, the video signal signal-processed by the signal processing unit 111 is input to the second image correction unit 118b via the signal switching unit 112, and the input signal is subjected to gamma correction and compression processing. The signal of the second image correction unit 118b is input to the display unit 119 and the recording unit 121, and the captured image is displayed on the display unit 119 and recorded on the recording unit 121. The first image correction unit 118a will be described later.

  Here, when the photographer performs a half-press operation of the release button of the release operation unit 12c (switch sw1 is turned on), a shooting preparation operation (focus adjustment operation, photometry operation, etc.) is started. Then, the closing timing (exposure time) of the shutter 12a and the aperture diameter of the aperture 13a are set based on the photometric value obtained by the photometric operation. The aperture of the aperture 13a is changed to an appropriate aperture at this time.

  Next, when the photographer performs a push-off operation of the release button of the release operation unit 12c (switch SW2 is turned on), an exposure operation for image recording is started. Specifically, when the release button push-off operation (the switch sw2 is turned on) of the release operation unit 12c is performed, all the image data (charges) stored in the imaging unit 19 are once reset, and the charge is recharged at the set image size. Accumulation starts. The shutter 12a is closed when the exposure time determined by photometry when the release button is half-pressed (switch SW1 is turned on), and the charge is transferred while the shutter 12a is closed. After the transfer is completed, the shutter 12a is opened.

  Here, when the image stabilization operation unit 120 is on, the exposure time is limited to prevent image quality deterioration due to camera shake. However, if the exposure time is shortened, the exposure will be underexposed and susceptible to noise. In practice, multiple images are acquired by continuous shooting with a short exposure time, and then combined. To improve the exposure.

  In addition, since the composition of the respective images slightly changes due to camera shake between the plurality of captured images, the images are combined after the composition is adjusted. A detailed description of this function is given below.

  In the first embodiment, a plurality of image signals output from the imaging unit 19 for each shooting in accordance with continuous shooting are converted into digital signals by the A / D conversion unit 110 and subjected to signal processing by the signal processing unit 111. The

  On the other hand, when the image stabilization control unit 120 is operated to notify the imaging control unit 18 that the image stabilization system has been turned on, the image data from the signal processing unit 111 is transferred to the image storage unit 113 via the signal switching unit 112. Is input. That is, the input to the second image correction unit 118b is interrupted. The image storage unit 113 stores a plurality of captured images.

  Here, the image storage unit 113 does not store all of the obtained images, but a clearly defective image is determined by the signal processing unit 111 and the imaging control unit 18, and the image storage unit 113 also stores the second image. Switching is controlled by the signal switching unit 112 so as not to be transmitted to the image correction unit 118b.

  The weighting setting unit 114 sets the weighting of the feature points used for deviation detection by the deviation detection unit 115. This feature point is a point with high contrast with respect to the surroundings such as a bright spot or an edge in each image, and is easily discriminated in the image, and is related to a corresponding feature point in another image. A small area of an image in which a shift between images is easily detected is shown. The weight setting unit 114 is the most important component of the present invention, and details will be described later.

  The deviation detection unit 115 extracts the weighted feature points set by the weighting setting unit 114 in the image stored in the image storage unit 113, and determines the position coordinates of the feature points in the shooting screen.

  For example, as shown in FIG. 2, consider a case where a photograph is taken in a frame 122a where a person 122f stands against a building 123a. When shooting a plurality of images, an image having a composition shifted from the frame 122a due to camera shake may be shot like the frame 122b. The weight setting unit 114 takes out the edge 125a of the window 124a, which is a point with high luminance, from the buildings 123a located around the screen as a feature point by edge detection. Then, the deviation detection unit 115 compares the feature point 125a with the feature point 125b in the frame 122b, and corrects (coordinates conversion) this difference. In FIG. 2, the frame 122b is coordinate-transformed such that the feature point 125b of the frame 122b overlaps the feature point 125a of the frame 122a as indicated by an arrow 126.

  Here, the reason why the periphery of the shooting screen is selected as the feature point will be described below.

  In many cases, the main subject is located near the center of the screen, and the main subject is often a person. In such a case, when the main subject is selected as a feature point, inconvenience due to subject shake occurs. That is, when shooting a plurality of images, not only the camera shake of the photographer but also the subject shake is superimposed, so that coordinate conversion of the image is performed based on the subject shake. In this case, the coordinate transformation is performed so that the composition of the main subject is appropriate, so it seems that a preferable image can be produced, but generally the movement of the person is complicated and the deviation detection accuracy is large depending on the place where the feature point is selected. It depends.

  For example, if the eye of the main subject (person) is selected as a feature point, the effect of blinking will appear, and if the tip of the hand is selected as a feature point, the hand will move easily, which is different from the actual shake of the entire subject. End up. In this way, even if coordinate conversion of an image is performed using one point of a person as a feature point, not all of the persons are appropriately coordinate-converted. Even when a plurality of images are coordinate-converted and combined, The coordinate position varies every time, and a preferable image cannot be obtained.

  Therefore, it is possible to obtain an image that is preferable when the still subject such as the background as described above is selected as the feature point and the coordinate transformation of the image is performed. Note that the method for obtaining the image shift is not only the method of extracting the feature points such as the peripheral edges as described above and making the shift, but also detecting the motion vector between the images in each area in the entire screen, Alternatively, the most frequent motion vector may be shifted.

  The coordinate conversion unit 116 aligns a plurality of images based on the shift amount between the images obtained by the shift detection unit 115.

  Here, the coordinate conversion unit 116 also performs the following operations before synthesis. First, a reference image is selected when the coordinates of each image are converted and aligned. An image that is significantly deviated from the selected reference image is deleted so that it is not used for subsequent synthesis. Each image subjected to coordinate conversion is output to the image composition unit 117, and each image is composed into one image. In the case of a digital image, it is possible to correct the exposure by increasing the gain even for a single underexposed photo. However, if the gain is increased, the noise increases and the image becomes unsightly.

  However, when a large number of images are combined and the gain of the entire image is increased as in the first embodiment, since the noise of each image is averaged, an image with a large S / N ratio can be obtained. As a result, noise can be suppressed and exposure can be optimized. If another view is taken, it can be said that random noise included in the image is reduced by, for example, allowing noise and taking a plurality of images with the imaging unit 19 having high sensitivity and averaging these.

  The synthesized image data is input to the first image correction unit 118a and subjected to gamma correction and compression processing. Further, the edge missing portion of each image at the time of synthesis is cut out, and the image size is reduced. Complemented. In FIG. 3, the non-overlapping area 129 is cut, and only the area where the two images overlap is cut out and the diffusion complementing process is performed. Thereafter, it is recorded in the recording unit 121.

  Generally, in the case of a digital camera, a photographed image is displayed on the rear liquid crystal monitor (display unit 119) of the camera after photographing. However, when the images are combined and exposure complementation is performed as in the first embodiment, it takes time to create and display the combined image.

  Therefore, in the first embodiment, the pre-combination image is displayed on the display unit 119 of the camera. This is because the display unit 119 of the camera is small, so even if the gain of an underexposed image is increased and displayed, it does not matter at the time of viewing.

  A signal input from the recording unit 121 of FIG. 1 to the display unit 119 is required when an image recorded in the recording unit 121 is reproduced. As described above, when a plurality of images are photographed and combined, the specified frame is displayed immediately after photographing. However, the display prohibition unit 119a does not display the specific frame on the display unit 119 when reproducing the image after the synthesis process is completed, but can confirm the actual image by displaying the image after the synthesis process is completed. I am doing so.

  In the image reproduction mode, all the image data in the recording unit 121 can be generally viewed. However, when a plurality of images are photographed and combined, the display prohibition unit 119a pre-combines them. This image is not visible during playback. This is because if the plurality of composite images are displayed at the time of reproduction, many images not recognized by the photographer are displayed, which not only takes time to browse the images but also causes confusion. Because of that.

  In the above configuration, the details of the above-described weighting setting unit 114 will be described.

  The role of the weighting setting unit 114 is to set the weighting of the feature points in each image based on the shooting magnification. When the shooting magnification is low, the weighting of the feature points in the image peripheral area is increased, and when the shooting magnification is high, the weighting of the feature points in the image central area is increased. This indicates that when the shooting magnification is low, it is preferable to correct the background shake as described above, and when the shooting magnification is high, it is preferable to correct the main subject shake. It is because it came.

  Here, the photographing magnification is a magnification obtained from the photographing focal length and the photographing subject distance detected by the photographing condition detection unit 18a. For example, when the shooting focal length is long (telephoto) and the shooting subject distance is short, the shooting magnification is high, and when the shooting focal distance is short (wide) and the shooting subject distance is long, the shooting magnification is low. Even if the shooting focal distance is short, the shooting magnification increases if the shooting subject distance is short. Similarly, the shooting magnification may increase if the shooting focal distance is long even if the shooting subject distance is long.

  Note that the imaging magnification referred to here is the same size as the size of the subject to be photographed, and is the same size as that when the image is taken on the imaging surface of the imaging unit 19 (imaging magnification 1), with respect to the size of the subject to be photographed. A case where the image is captured on the imaging surface with a size of 0.25 times is referred to as an imaging magnification of 0.25 times.

  When shooting magnification is low, as mentioned above, most of the angle shake is not affected by shift shake, so the main subject that is often located in the center of the screen and the background that is often located in the vicinity are the same amount. It is swinging. Therefore, when a plurality of images are aligned and synthesized, as described with reference to FIG. 2, the background region (image peripheral region) is more than the main subject region (image central region) where there is a risk of subject shake. It is only necessary to detect misalignment between images and align and combine the images.

  However, since the influence of shift shake increases as the shooting magnification increases, the amount of shake differs when the shooting distance differs between the main subject (image central area) and the background (image peripheral area). For example, consider a case where a close-up photograph of a plant is taken.

  At this time, in the frame 41 of FIG. 4, the motion vectors of the feature points (for example, the edges of the flowers and leaves) of the central flower 42 of the image as the main subject and the leaves 43 around the image as the background are compared. Then, a flower with a close subject distance has a large motion vector 42a, and a leaf motion vector 43a with a far subject distance becomes small. In such a case, it is desirable to sufficiently correct the shake in the main subject region (image central region) that the photographer wants to photograph most.

  Therefore, the weight setting unit 114 increases the weighting of the deviation detection value of the feature point for detecting the deviation in the center area of the image with respect to the detection value of the deviation of the feature point for detecting the deviation around the image when the photographing magnification increases. For example, when the photographing magnification is extremely large, the weight of the deviation detection value of the feature point for detecting the deviation around the image is set to zero, and the alignment and synthesis of each image is performed only by the feature point deviation detection result in the image center region.

  However, the above processing is not necessarily performed for all cases where the photographing magnification is large. When the above processing is performed, on the contrary, the leaf 43 which is the background may be greatly shaken by overcorrection. This is because the actual shake is small as shown by the motion vector 43a, but the correction is performed by the motion vector 42a.

  When the subject magnification is extremely large and the main subject shake due to shift shake is very large, priority is given to the shake correction of the main subject, and the background shake can be allowed. This is because, under such shooting conditions, the main subject occupies most of the image and the background area is very small, so over-correction of the shake is not noticeable. However, at the shooting magnification as shown in FIG. 4 (conditions where the shooting magnification is larger than normal shooting but not the maximum shooting magnification), the image peripheral area is also subjected to predetermined shake correction, and the entire image (the center of the image that is the main subject) is corrected. I want to make a good image in the area and the background area around the image.

  Therefore, in such a case, the weight detection value of the feature point for detecting the deviation around the image is not set to zero, and the feature point deviation detection result in the center area of the image and the deviation around the image are detected. Each image is aligned and synthesized based on the detection result of the feature point deviation.

  Specifically, the feature point deviation detection value weighting for image periphery deviation detection is set to 0.4, and the feature point deviation detection value weighting in the center area of the image is set to 0.6. Each image is aligned and synthesized with a synthesis vector after weighting the result. In this case, the shake correction of the main subject in the center area of the image is insufficient (0.6 times, so correction is insufficient), but the overcorrection of shake is reduced for the background that is the peripheral area of the image.

  As the shooting magnification is smaller, the effect of shift shake is reduced, and the difference between the shake amount of the main subject and the background shake due to the different shooting distances disappears. As shown in FIG.

  In FIG. 5A, the weighting of feature points in the center area of the image is increased (solid line) as the shooting magnification on the horizontal axis is increased, and on the contrary, the weighting of feature points in the image peripheral area is decreased (dashed line). In the same magnification shooting, the weighting of the feature points in the image peripheral area is set to zero. Reference numeral 51 denotes a certain photographing magnification, which will be described later.

  The weighting method is not limited to FIG. 5 (a). As shown in FIG. 5 (b), when the photographing magnification is smaller than a predetermined value (for example, 0.25 times), the feature point weighting of the image peripheral area is set to 1. The feature point weighting in the center area of the image may be zero. Further, as shown in FIG. 5C, when the photographing magnification is larger than a predetermined photographing magnification (for example, 0.25 times), the image peripheral area is weighted from 1 to zero, The weight may be changed from zero to one.

  FIG. 6 is a flowchart summarizing the photographing operation of the main part according to the first embodiment of the present invention, and this flow starts when the power of the camera is turned on.

  In step # 1001, the photographer half-presses the release button, that is, the switch sw1 is turned on and waits in this step until the photographing preparation unit operation is performed. Thereafter, when the switch sw1 is turned on, the process proceeds to step # 1002. .

  In the next step # 1002, the subject is imaged by the imaging unit 19, and the AF motor 14a is driven by the imaging control unit 18 while the contrast of the image is detected by the signal processing unit 111, and the lens 11 starts to be extended. Then, the position with the highest contrast is detected, and the lens 11 is extended to that position to perform focusing. The above is a case where the TV-AF (so-called mountain climbing) method is taken as an example, but a phase difference AF method may be used. In this case, the focus state is first detected by an AF sensor (not shown), the lens extension amount is calculated from the result, the lens is extended, and the focus state is detected again. Then, when the focus is in focus, the lens driving is stopped, and when the focus is not in focus, the lens driving is repeated again to focus. In step # 1002, the brightness of the subject is obtained from the output simultaneously with the imaging unit 19 and a photometric sensor (not shown in FIG. 1).

  In the next step # 1003, it is determined whether or not the image stabilization operation unit 120 is turned on by the photographer. If it is turned on, the process proceeds to step # 1004, and if it remains off, the process proceeds to step # 1019. .

First, the case where the image stabilization operation unit 120 is turned on and the process proceeds to step # 1004 will be described. In step # 1004, the number of images to be photographed and the respective exposure times are obtained from the photographing conditions such as the brightness of the subject obtained in step # 1002. What are the shooting conditions here? • Subject brightness • Shooting focal length • Shooting subject distance • Shooting optical system brightness (aperture value)
-Five points of sensitivity of the imaging unit.

  In step # 1004, the photographing magnification is further obtained from the photographing focal length and the photographing subject distance. For example, assume that the sensitivity of the imaging unit is set to ISO200. At this time, in order to measure the brightness of the subject and appropriately expose it, it is assumed that the aperture 13a is fully opened (for example, f2.8) and the shutter 12a requires an exposure time 1/8. At this time, when the shooting focal length is 30 mm in terms of 35 mm film, there is a risk of camera shake in shooting with an exposure time of 1/8, so the exposure time is set to 1/32 with no risk of camera shake, and four shots are taken. Set as follows. Here, when the photographing magnification is larger than a predetermined value (for example, the photographing magnification is 0.25 times or more), the shake increases due to the influence of the shift shake, so the exposure time is further shortened. For example, when the shooting magnification is 0.25, the exposure time is 1/64, when the shooting magnification is 0.5, the exposure time is 1/128, and when the shooting magnification is equal, the exposure time is 1/256. Then, the number of shots is set accordingly (for example, 8 shots at a shooting magnification of 0.25, 16 shots at a shooting magnification of 0.5, and 32 shots at the same shooting magnification).

  As described above, the exposure time when shooting a plurality of images is determined according to the shooting conditions, and how many more images are to be shot is set according to the shooting conditions. Further, even when the same subject is photographed in a plurality of pieces, accurate information can be captured by the imaging unit 19 when the exposure condition for each photographing is as close to the appropriate exposure as possible. Therefore, in the case of a dark subject, when the aperture 13a is narrowed down, or when the sensitivity of the imaging unit 19 is set low, the exposure time of each shooting is effective as long as possible even when shooting multiple images. Use exposure conditions.

  However, if the exposure time is excessively long, the influence of deterioration due to camera shake appears on the image plane. As described above, when the shooting focal length is 30 mm in terms of 35 mm film, there is no risk of camera shake, which is about one focal length. The exposure time is set to 1/32. Then, the number of shots is supplemented for the lack of exposure time. If the focal length is long, image deterioration due to camera shake occurs unless the exposure time is further shortened. Therefore, the exposure time is further shortened, and the number of shots is increased accordingly to perform exposure complementation.

  As described above, the exposure time in the multiple-image shooting becomes longer as the shooting subject is darker and the shooting lens is darker, and becomes longer as the sensitivity of the imaging unit 19 is lower, and becomes shorter as the focal length of the lens is longer, and the shooting magnification is larger. It gets shorter. The number of shots in multiple shooting increases as the shooting subject becomes darker and the shooting lens darkens, increases as the sensitivity of the imaging unit 19 decreases, increases as the focal length of the lens increases, and increases as the shooting magnification increases. Become more.

  After the above calculation is completed, the camera viewfinder and rear LCD monitor display that the image stabilization mode (multiple shooting mode) has been set, and at the same time, display the number of shots taken and notify the photographer.

  In the next step # 1005, the weighting setting unit 114 sets weighting detection of feature point deviation in the center area of the image and weighting of feature points in the image peripheral area according to the photographing magnification. The weighting increases the weighting of the feature points in the center area of the image as the photographing magnification increases, and the change of the weighting depending on the photographing magnification is as described with reference to FIGS. 5 (a) to 5 (c).

  In the next step # 1006, it is determined whether or not the release button has been pressed and the switch sw2 is turned on. If not, the process returns to step # 1001 and the same operation is repeated. Thereafter, when it is determined that the switch sw2 is turned on and a photographing instruction is given, the process proceeds to step # 1007.

  When the process proceeds to step # 1007, the first image is taken. That is, first, the charge of the imaging unit 19 is reset and re-accumulation is started (in the case of a mechanical shutter, the shutter is opened after the accumulation is started). Next, the exposure time obtained in step # 1004 is waited, and then charge transfer is performed (in the case of a mechanical shutter, the shutter is closed prior to charge transfer). Then, the obtained image is temporarily stored in the image storage unit 113. At the same time, sounding at the start of photographing is performed by the speaker 17a via the sounding drive unit 17b. This sound is for example! An electronic sound such as a shutter opening sound in a film camera or a mirror-up sound may be used.

  Note that from Step # 1007 to Step # 1014, which will be described later, is an operation in a composite shooting mode in which shooting with a short exposure time is repeated a plurality of times and combined to make the apparent exposure appropriate.

  In the next step # 1008, it is determined whether or not all photographing has been completed. If not completed, steps # 1007 and # 1008 are circulated and waited. Then, when the photographing is completed, the process proceeds to step # 1009.

  In step # 1009, the sound of completion of shooting is generated by the speaker 17a via the sound generation drive unit 17b. This sound is similar to the pronunciation at the start of shooting, for example. An electronic sound such as a shutter closing sound, a mirror down sound or a film winding sound in a film camera or the like may be used. Even in the case of shooting a plurality of images in this way, the sound generation representing the operation is one set (once from the exposure start of the first shooting to the completion of the exposure of the last shooting), so that the photographer feels uncomfortable with the shooting of the plurality of images. There is nothing. In particular, when continuous shooting is performed using an electronic shutter or the like, the sound and vibration of the mechanical shutter operation do not occur at all, so that the photographer can perform smooth shooting without even recognizing that continuous shooting has been performed.

  In step # 1010, the shift detection unit 115 obtains the shift amount between the images for each region set by the weight setting unit 114. The amount of deviation is calculated by calculating the average of the feature points provided for each of the image central region and the image peripheral region, and the above steps are performed for each of the average of the feature point coordinates of the image central region and the average of the feature point coordinates of the image peripheral region Weighting obtained in # 1005 is performed to obtain final coordinates.

  In the next step # 1011, the coordinate conversion unit 116 performs coordinate conversion of each image. In subsequent step # 1012, it is determined whether or not coordinate conversion has been completed for all the images. If not completed, the process returns to step # 1010, and waits by circulating through steps # 1010 and # 1011 until completion. When the coordinate conversion of the image is completed, the process proceeds to step # 1013.

  In the next step # 1013, the image is synthesized. Here, image synthesis is performed by averaging the image signals of the corresponding coordinates of each image, and random noise in the image is reduced by averaging. Then, the gain of the image with reduced noise is increased to optimize the exposure. In the subsequent step # 1014, an area where the images do not overlap due to composition fluctuations such as the end of the synthesized image is cut, and the image is diffusion-supplemented so as to have the original frame size. In the next step # 1015, the composite image is subjected to signal gamma correction and compression processing.

  In the next step # 1016, the image obtained in step # 1015 is displayed on the rear liquid crystal monitor (display unit 119) arranged on the back of the camera. In the subsequent step # 1017, the image obtained in step # 1015 is recorded on a recording medium (recording unit 121) that can be attached to and detached from the camera, for example, with a semiconductor memory. In step # 1018, the process returns to the start.

  If the switch sw1 is still turned on at the step # 1018, the flow is advanced again from the step # 1001 → # 1002 → # 1003 → # 1004. However, when the switch sw2 continues to be on at the stage of step # 1018, the process does not return to the start and waits at step # 1018.

  Next, the case where the image stabilization operation unit 120 is turned off at step # 1003 and the process proceeds to step # 1019 will be described.

  When the process proceeds from step # 1003 to step # 1019, it is determined here whether or not the imaging condition is such that image deterioration due to camera shake occurs unless the image stabilization system is used. As described above, the shooting conditions are the brightness of the subject, the brightness of the lens, the imaging sensitivity, and the shooting focal length. In this step # 1019, the exposure time is obtained based on the brightness of the subject, the brightness of the lens, and the imaging sensitivity, and whether or not there is a possibility of image degradation due to camera shake at the current shooting focal length. Judgment. As a result, if it is determined that there is a possibility of image deterioration, the process proceeds to step # 1020. If not, the process immediately proceeds to step # 1021.

  In step # 1020, a display recommending that the image stabilization mode is set is performed on the camera finder or the rear liquid crystal monitor (display unit 119). Then, in the next step # 1021, the switch sw2 is turned on and the process waits while circulating from step # 1001 to step # 1021 until there is an instruction for photographing.

  Thereafter, when it is determined in step # 1021 that the switch sw2 is turned on, the process proceeds to step # 1022, and waits until normal photographing (normal photographing mode for forming an effective exposure condition by one exposure) is completed. Upon completion of exposure, the process proceeds to step # 1015 described above.

  Although omitted here, even in the case of normal shooting, the shooting operation sound is generated from the speaker 17a in accordance with the operation from the start to the end of shooting. In other words, the same type of shooting operation sound is generated in both the composite shooting mode (combination of a plurality of shots) and the normal shooting mode. When the electronic shutter is selected, the photographer can recognize whether or not the exposure time is long due to the difference in the length of the operation sound (the length from the shooting start sound to the shooting completion sound). The photographer does not know whether it is present or not. Therefore, the camera is easy to use without giving the photographer the recognition that special shooting is being performed even in the composite shooting mode.

  When the process proceeds from step # 1022 to step # 1015, signal gamma correction and compression processing are performed on the captured image. Then, in the next step # 1016, the image obtained in step # 1015 is displayed on the rear liquid crystal monitor (display unit 119) disposed on the back of the camera. In the subsequent step # 1017, the image obtained in the above step # 1015 is recorded on a recording medium (recording unit 121) that can be attached to and detached from the camera, for example, with a semiconductor memory, and the process returns to the start via step # 1018.

  As can be seen from the flow of FIG. 6, even when the image stabilization operation unit 120 is turned off, a display that prompts the photographer to use the image stabilization system (synthetic image capture mode) under image capturing conditions that cause image degradation due to camera shake. To prevent image degradation. Further, in the composite shooting mode, each exposure time is changed depending on the focal length, so that desirable shooting can be performed at any focal length.

  As described above, the present invention relates to a camera having an electronic image stabilization system that acquires a plurality of images, detects their mutual shift amounts, and aligns and combines the images based on the results. In this camera, the weighting of feature points of different regions in the image is set (changed) based on the shooting magnification. This enables well-balanced shake correction (image shake correction) even during close-up photography.

  Specifically, the camera according to the first embodiment includes the following components. The image capturing control unit 18 and the image storage unit 113 that sequentially perform image capturing to obtain a plurality of images, and the weight setting unit 114 that sets weights of feature points of different regions in the image based on the image capturing magnification. Further, a deviation detection unit 115 is provided that detects a deviation amount between the images from the coordinates of the feature points weighted for each region (image central region and image peripheral region) by the weight setting unit 114. Furthermore, a coordinate conversion unit 116 that performs coordinate conversion of each image based on the shift amount detected by the shift detection unit 115 is provided. Further, the image conversion unit 116 includes an image combining unit 117 that combines the images after the coordinate conversion by the coordinate conversion unit 116. Then, the image blur between the images is corrected by the shift amount reflecting the weighting based on the photographing magnification, and then the composition is performed to complement the image exposure.

  The weight setting unit 114 will be described in more detail. When the shooting magnification is low, the feature points in the image peripheral area are weighted higher than when the shooting magnification is high, and when the shooting magnification is high, the feature in the image center area is lower than when the shooting magnification is low. The point weighting is increased.

  Therefore, satisfactory image blur correction can be performed even in close-up photography. In other words, even under close-up shooting conditions, image blur correction of the main subject and image blur correction of the background can be performed in a balanced manner, and an image with high image stabilization accuracy can be obtained.

  In the first embodiment, the weighting of the image central region and the image peripheral region is changed according to the photographing magnification, and the amount of deviation is obtained by the weighting to perform image composition. In contrast, the second embodiment of the present invention differs in the following points. One is that the image is divided into a high-contrast area and a low-contrast area, and the weighting of feature points located in high-contrast areas and feature points located in low-contrast areas is variable depending on the shooting magnification. . The other is that the contrast of the high-contrast region and the low-contrast region is compared, and the weighting is changed according to the comparison result.

  Details of the weight setting unit 114 according to the second embodiment of the present invention will be described below. Others are almost the same as those of the first embodiment, and the description thereof is omitted.

  The role of the weighting setting unit 114 is to set the weighting of the feature points in each image based on the shooting magnification. Specifically, when the shooting magnification is low, the weighting of the feature points in the region where the image contrast is low is increased, and when the shooting magnification is high, the weighting of the feature points in the region where the image contrast is high is increased.

  It can be seen that when the shooting magnification is low, it is preferable to correct the background blur as described above, and when the shooting magnification is high, it is preferable to correct the main subject shake, but the image becomes better. It is because it came. In general, the background area is not in focus, so the contrast is low, and the main subject is in focus, so the contrast is high. That is, in the first embodiment, the main subject and the background are separated at the center and the periphery of the image, but in the second embodiment of the present invention, the main subject and the background are separated by contrast.

  Note that when the shooting magnification is low, as described above, the angular shake occupies most and there is no influence of the shift shake, so the main subject that is often located in a high-contrast area is also often located in a low-contrast area. Is also swinging by the same amount.

  Therefore, when aligning and synthesizing a plurality of images, as described with reference to FIG. 2, the background area (low contrast area) is higher than the main subject area (high contrast area) where there is a risk of subject shake. It is only necessary to detect misalignment between images and align and combine the images.

  However, since the effect of shift shake increases as the shooting magnification increases, the amount of shake differs when the shooting distance differs between the main subject (high contrast region) and the background (low contrast region).

  As shown in FIG. 4, in the frame 41, the motion vectors of the feature points (for example, the edges of flowers and leaves) of the central flower 42 of the image as the main subject and the leaves 43 around the image as the background are compared. Then, a flower with a close subject distance has a large motion vector 42a, and a leaf motion vector 43a with a far subject distance becomes small. In such a case, it is desirable to sufficiently correct the shake in the main subject region (region with high contrast) that the photographer wants to photograph most.

  Therefore, the weighting setting unit 114 increases the weighting of the feature point deviation detection value for detecting the deviation of the high-contrast region with respect to the feature point deviation detection value for detecting the deviation of the low-contrast region when the photographing magnification increases. To do. For example, when the shooting magnification is extremely high, the weight of the feature point deviation detection value for detecting the deviation of the low contrast region is set to zero, and the alignment and synthesis of each image is performed only by the feature point deviation detection result in the high contrast region. Do.

  However, the above processing is not necessarily performed for all cases where the photographing magnification is large. That is, when the above process is performed, the leaf 43 as a background may be greatly shaken by overcorrection. This is because the actual shake is small as shown by the motion vector 43a, but the correction is performed by the motion vector 42a.

  When the subject magnification is extremely large and the main subject shake due to shift shake is very large, priority is given to the shake correction of the main subject, and the background shake can be allowed. This is because, under such shooting conditions, the main subject occupies most of the image and the background area is extremely small, so over-correction of the shake is not noticeable.

  However, at a shooting magnification as shown in FIG. 4 (conditions where the shooting magnification is larger than normal shooting but not the maximum shooting magnification), the image peripheral area is also subjected to predetermined shake correction, and the entire image (the contrast of the main subject) is adjusted. I want a good image in a high area and a low contrast area.

  Therefore, in such a condition, the weight of the deviation detection value of the feature point for detecting the deviation in the low contrast region is not set to zero. Then, each image is aligned and synthesized based on the feature point shift detection result in the high contrast region and the feature point shift detection result for detecting the shift in the low contrast region. Specifically, the weight of the feature point deviation detection value for detecting the deviation in the low contrast region is 0.4, and the weight of the feature point deviation detection value in the high contrast region is 0.6. Then, each image is aligned and synthesized with a synthesized vector after weighting the detection result of each feature point.

  In this case, the shake correction of the main subject in the high-contrast area is insufficient (the correction is insufficient because it is 0.6 times), but the shake over-correction is reduced for the background that is the low-contrast area.

  The smaller the shooting magnification, the less the influence of shift shake, and the difference between the main subject shake amount and the background shake amount due to different shooting distances disappears. Therefore, the weighting of the feature points of each region determined by the weight setting unit may be set in the same manner as in FIGS. 5A to 5C described in the first embodiment.

  FIG. 7 is a flowchart summarizing the photographing operation of the main part according to the second embodiment of the present invention, and this flow starts when the power of the camera is turned on. The flowchart of FIG. 7 differs from the flowchart of FIG. 6 in that step # 2001 is performed instead of step # 1005.

  In step # 2001, the weight setting unit 114 sets weight detection for feature point shift in a region with high contrast and weighting for feature points in a region with low contrast according to the shooting magnification. Here, the contrast of the image is obtained by known image processing, but a high-contrast subject and a low-contrast subject are mixed in the image. Also, even if the subject is high in contrast, the contrast is low if the subject is not in focus.

  Therefore, as shown in FIG. 8, the image is roughly divided (region 101a as the second region and regions 101b to 101g as the first region). Then, an average contrast value is obtained for each divided area, and a high contrast area and a low contrast area are set based on the average value. This weighting increases the weighting of the feature points of the high contrast area (the second area indicated by area 101a in FIG. 8) as the shooting magnification increases. As described in FIG. 5C from a).

  As described above, in the second embodiment, an image is divided into a high-contrast region and a low-contrast region, and the feature points located in the high-contrast region and the feature points located in the low-contrast region are weighted according to the shooting magnification. It is made variable. This makes it possible to more accurately determine the main subject and the background and give appropriate weights to the feature points for deviation detection as compared with the first embodiment.

  In the second embodiment, the balance described below can be further improved by performing the processing described below.

  As described above, in close-up shooting, there is an image shift due to the effect of shift shake applied to the camera, and the shift amount changes depending on the distance of the shooting subject. When the subject to be photographed is close to the camera, the image shift due to the shift shake is larger than when the subject is far away.

  When the main subject and the background are at substantially the same distance from the camera, the image shift due to the shift shake is almost the same. Therefore, as described above, the shift between the images is detected and based on the shift. By aligning and synthesizing the images, the main subject and the background can be corrected well. However, when the shooting distance of the background is far from the shooting distance of the main subject, for example, when the main subject is at a position of 20 cm with respect to the camera and the background is at a position of 1 m with respect to the camera, The shift is large for the main subject and has little effect on the background. For this reason, when the shift detection is performed based on the shift amount of the main subject and the images are aligned and combined, a shift occurs in the background area. However, since the background is sufficiently distant from the main subject, the contrast of the background image is extremely low (because it is not in focus), and it does not matter much if the area is shifted.

  In the case of the above two conditions, it is only necessary to detect deviation using the feature points of the main subject region (region with high contrast). However, adjustment is necessary for deviation detection under a condition intermediate between the above two conditions. For example, the main subject is 20 cm from the camera and the background is 40 cm from the camera. In such a case, if the images are aligned and synthesized based on the shift detected in the main subject area (high contrast area), the background will be shifted (overcorrection). This is the same as when the background is 1 m. That is, since the shooting distance of the background is also close to the shooting distance of the main subject, there is little decrease in contrast with respect to the main subject, the overcorrection is conspicuous, and a good image is not obtained.

  Under such conditions, the main object shift correction amount (accuracy) is lowered, and the background overcorrection amount is reduced accordingly, so that the entire image is balanced. It is one feature.

  In the examples described so far, the higher the shooting magnification, the higher the main subject displacement correction accuracy with respect to the background, and this can be dealt with by increasing the weighting of the feature points in the center area of the image or in the high contrast area. I have done it. Here, in addition to this, even when the shooting magnification is large, the feature points of the main subject area (high contrast area) and the background (low contrast area) according to the background shooting magnification (shooting subject distance and contrast). The weight is changed.

  Specifically, the contrasts of the high contrast region and the low contrast region are compared, and when the difference or ratio is large, it is determined that the background is far from the main subject (the shooting distance is long). . Then, the weighting is corrected so as to increase the weighting of the feature points in the main subject (high contrast region). On the other hand, when the difference or ratio is small, it is determined that the background is close to the main subject, and correction is performed so as to reduce the weighting of the feature points in the main subject (region with high contrast).

  In this way, even when the background is far away, overcorrection of background deviation can be prevented and a good image can be obtained when the pattern has a high contrast.

  FIG. 9 is a flowchart for explaining the details of the processing contents in step # 2001 of FIG.

  When the process proceeds to step # 20012 via step # 20011, here, the contrast ratio of each of the high contrast region and the low contrast region is obtained. Note that the high contrast region is a main subject region such as the second region indicated by the region 101a in FIG. 8, and the low contrast region is a background region such as the first region indicated by the region 101f in FIG. .

  In the next step # 20013, the weighting of the feature points scattered in each region (the high contrast region 101a and the low contrast region 101f) is corrected based on the contrast ratio obtained in step # 20012. Then, in the next step # 20014, the process proceeds to step # 1006 in FIG.

  Here, correction of weighting of feature points will be described.

  FIG. 10 shows the weighting for each area under the photographing condition of a certain photographing magnification 51 in FIG. Here, the horizontal axis represents the ratio of contrast between a high contrast region (for example, the region 101a in FIG. 8) and a low contrast region (the region 101f in FIG. 8). The further to the right in the figure, the smaller the contrast ratio (the contrast is substantially the same, and the main subject and the background are in the vicinity of the same shooting distance). The vertical axis represents the weighting of each area (high contrast area and low contrast area), and corresponds to a high contrast area indicated by a broken line and a low contrast area indicated by a solid line.

  When the contrast ratio is large (the background is far away and the focus is blurred), each area is weighted at the position of the photographing magnification 51 shown in FIG. 5A. There is no difference in the weighting. Therefore, when the shooting distance between the main subject and the background is long, the image is synthesized based on the result of weight detection of the main subject, and when the main subject and the background are almost at the same shooting distance, the main subject and the background are combined. The images are synthesized based on the average deviation detection result of the deviations. Therefore, the main subject shift and the background shift can be corrected with good balance, and a good image can be obtained.

  As described above, as in the first embodiment, a well-balanced image blur correction can be performed even during close-up photography.

  The weighting setting unit 114 according to the second embodiment will be described in detail. When the shooting magnification is low, the weighting of the feature points in the first region where there are many images with low contrast is increased compared to when the shooting magnification is high. When the shooting magnification is high, the weighting of the feature points in the second region where there are many images with high contrast is increased compared to when the shooting magnification is low. Furthermore, the weighting setting unit 114 increases the weighting of the feature points in the second region as the contrast comparison result between the first region and the second region increases.

  Therefore, satisfactory image blur correction can be performed even in close-up photography.

  In the second embodiment, the weighting is set by detecting the contrast of the image. On the other hand, in Embodiment 3 of the present invention, the shooting distance of each part of the screen is obtained, and weighting is set according to the result.

  As described above, in one of the AF methods of the camera, the contrast of the image is evaluated while changing the focus by scanning the lens, and final focusing is performed based on the result (at the position with the highest contrast). There is a method of positioning the lens). As another method, there is a method in which a dedicated AF sensor (for example, a phase difference AF sensor) is provided and the lens is driven according to the detection result to perform focusing. In these methods, the focus state of each part of the image is detected, and the lens is adjusted so that the focus state of the region determined as the main subject (determined by the distance between the camera and the subject, the position on the image, etc.) is good. Driving.

  In both methods, since the focus state at each point on the screen can be detected, it is also possible to detect the focus state outside the region determined to be the main subject. For this reason, the main subject (in-focus area) and the background (in-focus area) can be separated from the focus state in the image, and the weighting of the feature points can be set for each area. In addition, weight correction can be performed in the same manner as in the second embodiment by comparing the focus state (shooting distance) between the main subject and the background.

  FIG. 11 shows the image to be picked up and the area (ranging frame) for detecting the focus state of each part of the image. The ranging frame 131a overlaps the main subject, and the ranging frame 131b overlaps the background. ing. Then, based on the focus state detection result of the distance measuring frame 131a, the lens is driven so as to focus on the main subject. The distance from the camera to the main subject is known from the lens driving amount, and the photographing magnification is known from the relationship with the focal length of the photographing lens at that time. When the photographing magnification is known in this way, as shown in FIG. 5, the weighting of the feature points is set.

  The deviation detection unit 115 detects a deviation between corresponding images of feature points in the area for each distance measurement frame. Then, the average value of the deviation detection results in the focused distance measurement frame (for example, the distance measurement frame 131a) and the average value of the deviation detection results in the distance measurement frame (for example, the distance measurement frame 131b) that is not in focus are obtained. Set to the shooting magnification. Then, the images are aligned based on the combined vector, and the images are combined.

  Further, the in-focus state measurement frame (for example, the distance measurement frame 131b) and the in-focus state measurement frame (for example, the distance measurement frame 131a) are compared, and weighting is performed in the same manner as in FIG. Perform the correction.

  Thereby, even when the shooting distance of the background is close to the main subject, a well-balanced image blur correction can be performed.

  As described above, as in the first embodiment, a well-balanced image blur correction can be performed even during close-up photography.

  In the third embodiment, the camera includes a focus state detection unit that detects the focus state of different areas in the shooting screen. The weighting setting unit 114 increases the weighting of the feature points in regions other than the vicinity of the focus of the image (such as the distance measurement frame 131b) when the shooting magnification is low compared to when the shooting magnification is high. When the photographing magnification is high, the weighting of the feature points (the distance measurement frame 131a and the like) in the in-focus vicinity region is increased compared to when the photographing magnification is low. Furthermore, as the comparison result of the in-focus state between the in-focus vicinity region of the image and the region other than the in-focus vicinity of the image is larger, the weighting of the feature points in the near-in-focus region is increased.

  Therefore, satisfactory image blur correction can be performed even in close-up photography.

  In the first to third embodiments, the weighting of the feature points is changed according to the photographing magnification. Thus, by aligning and synthesizing the images, it was possible to obtain a good image with little image blur and complemented exposure.

  As described in Embodiments 1 to 3 above, when images are shot in a plurality of pieces, each of them needs to be shot with a short exposure time so that image deterioration due to camera shake does not occur. The exposure time is adjusted according to the focal length of the camera and the photographing magnification. When the focal length is long or the photographing magnification is high, image degradation due to camera shake increases, so the exposure time of each image is short. The time is shorter than when the shooting magnification is low. Therefore, in order to complement the exposure, it is necessary to acquire more images and align and combine them.

  When the number of combined images is large, the relationship between the main subject and the background shake correction amount described in the first to third embodiments changes. As described above, image blur correction is possible in close-up shooting only for subjects with the same shooting distance, and over-correction may occur for subjects with different shooting distances such as the background. In order to avoid this, the main subject shake correction accuracy is adjusted (decrease accuracy), and the background overcorrection is weakened to obtain a well-balanced image as a whole. However, as the number of shots increases, the time difference between the first shot image and the last shot image becomes too long, and the amount of shake during that time also increases. Therefore, if the shake correction accuracy of the main subject is lowered in order to obtain a well-balanced image, the remaining shake may not be allowed.

  Therefore, in Embodiment 4 of the present invention, when the number of shots is large, the weighting of the feature points of the main subject area is increased to increase the shake correction accuracy of the main subject. On the other hand, when the number of shots is small, the weighting of the feature points in the background area is increased to perform shake correction that is balanced over the entire image.

  The configuration of the camera is the same as that in FIG. 1, and the description thereof is omitted. Only the processing unit for a plurality of images will be described.

  In the fourth embodiment, a plurality of image signals output from the imaging unit 19 for each shooting in response to continuous shooting are converted into digital signals by the A / D conversion unit 110 and then subjected to signal processing by the signal processing unit 111. Is done.

  On the other hand, when the image stabilization control unit 120 is operated and the imaging control unit 18 is informed that the image stabilization system is turned on, the image data from the signal processing unit 111 is transferred to the image storage unit 113 via the signal switching unit 112. Entered. The image storage unit 113 stores a plurality of captured images.

  The weighting setting unit 114 sets the weighting of the feature points used for the deviation detection of the next stage deviation detection unit 115, and details of the weighting setting unit 114 in the fourth embodiment will be described later. The deviation detection unit 115 extracts the weighting feature points set by the weighting setting unit 114 in the image stored in the image storage unit 113, and determines the position coordinates of the feature points in the shooting screen. The coordinate conversion unit 116 aligns a plurality of images based on the shift amount between the images obtained by the shift detection unit 115.

  Here, the coordinate conversion unit 116 also performs the following operations before synthesis.

  First, a reference image is selected when the coordinates of each image are converted and aligned. An image that is significantly deviated from the selected reference image is deleted so that it is not used for subsequent synthesis. Each image subjected to coordinate conversion is output to the image composition unit 117, and each image is composed into one image. The synthesized image data is input to the first image correction unit 118a and subjected to gamma correction and compression processing. Further, the edge missing portion of each image at the time of synthesis is cut out, and the image size is reduced. Complemented. That is, the region 129 in FIG. 3 that does not overlap each other is cut, and only the region where the two images overlap is cut out and the diffusion complement processing is performed. Thereafter, it is recorded in the recording unit 121.

  In the above configuration, details of the weighting setting unit 114 according to the fourth embodiment will be described.

  The role of the weight setting unit 114 is to set the weighting of the feature points in each image based on the number of shots. When the number of shots is small, the weighting of the feature points in the image peripheral area is increased. On the other hand, when the number of shots is large, the weighting of the feature points in the center area of the image is increased.

  This is because, as described above, the shake amount to be corrected becomes large when the number of shots is large, so that it is necessary to increase the weighting of the feature points in the main subject area in order to maintain the correction accuracy of the main subject. is there. When the number of shots is small, the amount of shake to be corrected is small. Therefore, in order to balance the shake correction of the main subject and the shake correction of the background, the weight setting unit 114 sets each area (the center area of the image with the main subject and the background). The weighting of the feature points in the image peripheral area) is made equal.

  FIG. 12 is an explanatory diagram of weighting according to the fourth embodiment, where the vertical axis indicates weighting and the horizontal axis indicates the number of shots. In FIG. 14, as the number of shots increases, the weight of feature points in the center area of the image is increased (solid line), and on the contrary, the weight of feature points in the image peripheral area is decreased (broken line).

  FIG. 13 is a flowchart summarizing the photographing operation of the main part according to the fourth embodiment of the present invention. The difference from the flow of FIG. 6 is that step # 1005 is changed to step # 3001. Except for step # 1004, other descriptions are omitted.

In step # 1004, the number of images to be photographed and the respective exposure times are obtained from the photographing conditions such as the brightness of the subject obtained in step # 1002. What are the shooting conditions here? • Subject brightness • Shooting focal length • Shooting subject distance • Shooting optical system brightness (aperture value)
-Five points of sensitivity of the imaging unit. Further, the photographing magnification is obtained from the photographing focal length and the photographing subject distance.

  Further, the number of shots is obtained by the following procedure. For example, assume that the sensitivity of the imaging unit 19 is set to ISO200. At this time, in order to measure the brightness of the subject and appropriately expose it, it is assumed that the aperture 13a is fully open (for example, f2.8) and the shutter 12a requires an exposure time 1/8.

  At this time, if the shooting focal length is 30 mm in terms of 35 mm film, there is a risk of camera shake in shooting with an exposure time of 1/8, so the exposure time is set to 1/32 with no risk of camera shake, and four shots are taken. Set to do. If the focal length is long, image deterioration due to camera shake occurs unless the exposure time is further shortened. Therefore, the exposure time is further shortened, and the number of shots is increased accordingly to perform exposure complementation.

  As described above, the exposure time in the multiple-image shooting becomes longer as the shooting subject becomes darker and the shooting lens becomes darker, becomes longer as the sensitivity of the imaging unit 19 becomes lower, and becomes shorter as the focal length of the lens becomes longer. Furthermore, the higher the shooting magnification, the shorter (the higher the shooting magnification, the greater the influence of shift shake, so the exposure time must be further shortened). The number of shots in the multiple shots increases as the shooting subject is darker and the shooting lens is darker, increases as the sensitivity of the imaging unit 19 is lower, increases as the focal length of the lens is longer, and the shooting magnification is higher. The more it becomes.

  After the above calculation is completed, the camera viewfinder and the liquid crystal display display that the image stabilization mode (multiple shooting mode) has been set, and at the same time, the obtained number of shots is displayed to notify the photographer.

  In step # 3001 of FIG. 13, the weighting setting unit 114 sets weighting for feature point shift detection in the image center region and feature points in the image peripheral region according to the number of shots. The weighting increases the weighting of the feature points in the center area of the image as the number of shots increases, and the change of the weighting according to the shooting magnification is as described with reference to FIG.

  In the above embodiment, the weighting of the central area and the peripheral area of the image is changed according to the number of shots. However, as described in the above embodiments 2 and 3, the features in the high contrast area and low image area are described. The point weight may be changed by the number of shots. Furthermore, the weighting of the feature points of the areas divided for each in-focus state may be changed by the number of shots.

  Further, weighting correction is performed by comparing the contrast ratio of the main subject and the background and the in-focus state, so that an image with a more shake correction balance can be obtained.

  The weighting control using the contrast and the in-focus state has already been described in detail in the second and third embodiments. The difference from the second and third embodiments is that depending on the shooting magnification depends on the number of shots. The explanation is omitted because it is only changed.

  As described above, the present invention relates to a camera having an electronic image stabilization system that acquires a plurality of images, detects their mutual shift amounts, and aligns and combines the images based on the results. In this camera, the weighting of feature points of different areas in the image is set (changed) according to the number of shots. This makes it possible to perform well-balanced image blur correction even during close-up photography.

  Specifically, the camera according to the fourth embodiment includes the following components. The image capturing control unit 18 and the image storage unit 113 that sequentially perform image capturing and obtain a plurality of images, and the weight setting unit 114 that sets the weights of feature points of different regions in the image according to the number of captured images. Furthermore, a shift detection unit 115 that detects a shift amount between the images from the coordinates of the feature points weighted by the weight setting unit 114, and coordinate conversion of each image based on the shift amount detected by the shift detection unit 115. And a coordinate conversion unit 116 to perform. Furthermore, an image synthesis unit 117 that synthesizes the images after coordinate transformation by the coordinate transformation unit 116 is provided. Then, the image blur between the images is corrected by the shift amount reflecting the weighting according to the number of shots, and then the composition is performed to complement the image exposure.

  The weight setting unit 114 will be described in more detail. The weighting of the feature points in the image peripheral area is increased when the number of shots is small compared to when the number of shots is large. Increase the point weight. Or, when the number of shots is small, the weight of the feature points in the first region where there are many images with low contrast is larger than when the number of shots is large. The weighting of the feature points in area 2 is increased.

  The camera also has a focus state detection means for detecting the focus state of different areas in the shooting screen. The weighting setting unit 114 increases the weighting of the feature points in the region other than the vicinity of the focus of the image when the number of shots is small compared to when the number of shots is large. When the number of shots is large, the weighting of the feature points in the in-focus vicinity region is made larger than when the number is small.

  Therefore, it is possible to perform satisfactory image blur correction even in close-up photography that complements image exposure.

  In the above-described embodiments, the camera (imaging device) has been described. However, the same effect can be obtained with an electronic device, a communication device, or the like that can execute photographing.

(Other examples)
The object of each embodiment is also achieved by a method of supplying software program code for realizing the functions of the above-described embodiments to a system or apparatus. The computer of the system or apparatus (or CPU or MPU) reads out and executes the program code. In this case, the program code itself realizes the functions of the above-described embodiment. Further, a recording medium storing the program code corresponding to the above-described procedure for providing the program code to the computer of the system or apparatus constitutes the present invention.

(Correspondence between Invention and Example)
The photographing control unit 18 and the image storage unit 113 correspond to the exposure control unit of the present invention, the weight setting unit 114 corresponds to the weight setting unit of the present invention, and the shift detection unit 115 corresponds to the shift detection unit of the present invention. Further, the coordinate conversion unit 116 corresponds to the coordinate conversion unit of the present invention, and the image composition unit 117 corresponds to the image composition unit of the present invention. Further, the part of the imaging control unit 18 that detects the focusing state of different areas in the shooting screen by the TV-AF (so-called hill climbing) method or the phase difference AF method corresponds to the focusing state detection means of the present invention.

11a Shooting lens 14b Focus drive unit 18 Shooting control unit 19 Imaging unit 113 Image storage unit 114 Weight setting unit 115 Deviation detection unit 116 Coordinate conversion unit 117 Image composition unit 119 Display unit

Claims (15)

Exposure control means for performing sequential shooting to obtain a plurality of images;
Weighting setting means for setting weights of feature points of different regions in the image based on a photographing magnification;
A deviation detecting means for detecting a deviation amount between the images from the coordinates of the feature points weighted by the weight setting means;
Coordinate conversion means for performing coordinate conversion of each image based on the shift amount detected by the shift detection means;
Image synthesizing means for synthesizing the images after coordinate transformation by the coordinate transformation means,
An image pickup apparatus characterized in that after image blur between the images is corrected by the shift amount in which weighting based on the photographing magnification is reflected, the image is combined to complement exposure of the image.   The weight setting means increases the weighting of the feature points in the image peripheral area when the shooting magnification is low compared to when it is high, and the feature points in the image center area when compared with when the shooting magnification is high. The imaging apparatus according to claim 1, wherein the weighting of the imaging apparatus is increased.   The weight setting means increases the weighting of the feature points in the first region where there are many low-contrast images when the shooting magnification is low, and compared with when the shooting magnification is high, compared to when the shooting magnification is low. The imaging apparatus according to claim 1, wherein the weighting of the feature points in the second region where there are many high images is increased.   The imaging apparatus according to claim 3, wherein the weight setting unit changes the weight of the feature points according to a result of contrast comparison between the first area and the second area.   The imaging apparatus according to claim 4, wherein the weighting setting unit increases the weighting of the feature points in the second region as the contrast comparison result between the first region and the second region is larger. In-focus state detecting means for detecting the in-focus state of the different areas in the shooting screen,
The weight setting means increases the weighting of the feature points in the region other than the vicinity of the focus of the image when the shooting magnification is low compared to when it is high, and when the shooting magnification is high compared to when the shooting magnification is low. The imaging apparatus according to claim 1, wherein the weighting of the feature points in the neighborhood region is increased.   The weighting setting unit changes the weighting of the feature points in accordance with a comparison result of in-focus states of an in-focus vicinity area of the image and an area other than the in-focus vicinity of the image. The imaging device described.   The weighting setting means increases the weighting of the feature points in the in-focus vicinity region as the comparison result of the in-focus state in the in-focus vicinity region of the image and a region other than the in-focus vicinity of the image increases. The imaging apparatus according to claim 7, wherein the imaging apparatus is characterized. Exposure control means for performing sequential shooting to obtain a plurality of images;
Weighting setting means for setting weights of feature points of different regions in the image according to the number of shots;
A deviation detecting means for detecting a deviation amount between the images from the coordinates of the feature points weighted by the weight setting means;
Coordinate conversion means for performing coordinate conversion of each image based on the shift amount detected by the shift detection means;
Image synthesizing means for synthesizing the images after coordinate transformation by the coordinate transformation means,
An image pickup apparatus characterized in that after image blur between the images is corrected by the shift amount in which weighting according to the number of shots is reflected, composition is performed to complement image exposure.   The weight setting means increases the weighting of the feature points in the image peripheral area compared to when the number of shots is small compared to when the number of shots is large, and the feature points in the center area of the image compared to when the number of shots is large The imaging apparatus according to claim 9, wherein the weighting of the imaging apparatus is increased.   The weight setting means increases the weighting of the feature points in the first area where the number of images with low contrast is large compared to when the number of captured images is small, and contrast when compared with when the number of captured images is small. The image pickup apparatus according to claim 9, wherein the weighting of the feature points in the second region having a large number of high-images is increased. In-focus state detecting means for detecting the in-focus state of the different areas in the shooting screen,
The weighting setting unit increases the weighting of the feature points in the region other than the vicinity of the focus of the image compared to when the number of shots is small, and is in focus when the number of shots is large compared to when the number is small. The imaging apparatus according to claim 9, wherein the weighting of the feature points in the neighborhood region is increased. An exposure control process that sequentially captures images to obtain a plurality of images;
A weighting setting step for setting weighting of feature points of different regions in the image based on a photographing magnification;
A displacement detection step of detecting a displacement amount between the images from the coordinates of the feature points weighted by the weight setting step;
A coordinate conversion step of performing coordinate conversion of each image based on the shift amount detected by the shift detection step;
An image synthesis step of synthesizing the images after coordinate transformation by the coordinate transformation step,
A method for controlling an imaging apparatus, comprising: combining after correcting image blur between the images with the shift amount reflecting weighting based on the photographing magnification, and complementing image exposure. An exposure control process that sequentially captures images to obtain a plurality of images;
A weighting setting step for setting weighting of feature points of different regions in the image according to the number of shots;
A displacement detection step of detecting a displacement amount between the images from the coordinates of the feature points weighted by the weight setting step;
A coordinate conversion step of performing coordinate conversion of each image based on the shift amount detected by the shift detection step;
An image synthesis step of synthesizing the images after coordinate transformation by the coordinate transformation step,
A method for controlling an image pickup apparatus, comprising: performing composition after correcting image blur between the images with the shift amount in which weighting according to the number of shots is reflected, and complementing image exposure.   The program for making a computer perform each process of the control method of the imaging device of Claim 13 or 14.

Images