JP3530696B2 - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct camera-shake in response to a zoom and a focus state in the case of generating a high definition image by synthesizing plural sets of images obtained by shifting picture elements. SOLUTION: Picture elements are shifted by moving a 2nd lens group L2 based on each reference waveform to pitch and awing directions respectively. In this case, a 1st coefficient is multiplied wit the reference waveform in response to zoom and focus and a 2nd coefficient is multiplied with a camera- shake signal detected by camera-shake sensors GRP, GRY. The 2nd lens group L2 is driven with a signal being a sum of these two multiplication outputs to conduct simultaneously picture element deviation and camera-shake correction corresponding to the states of zoom and focus.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup device,
In particular, it is suitable for use in performing pixel shifting while changing the relative position between the subject image and the image pickup means for photoelectrically converting the subject image.

[0002]

2. Description of the Related Art For cameras using silver salt film, C
A so-called electronic still camera using a solid-state image sensor such as a CD has already been commercialized. Although this electronic still camera is superior to the silver salt camera in terms of immediacy, it is inferior to the silver salt camera in terms of the resolution and dynamic range of the image sensor.

Therefore, in order to improve the above-mentioned first defect, that is, the insufficient resolution capability, conventionally, the relative position between the subject image formed by the imaging optical system and the image pickup device for photoelectrically converting the subject image is made minute. An imaging device using a so-called pixel shift technique, which obtains a high-definition image by performing plural times of photographing while changing to obtain plural sets of image signals and synthesizing the plural sets of image signals by a predetermined method, Already proposed. As prior art documents for this pixel shifting, there are, for example, the following patent publications.

In Japanese Patent Laid-Open No. 60-27278, a wedge prism in front of the photographing optical system is rotated around the optical axis to decenter the optical image on the image pickup element in parallel, and output images are combined to produce a high-definition image. Is getting In Japanese Patent Laid-Open No. 60-91774, an optical image on an image pickup device is obtained by shifting a part of the lens of the master system in a direction perpendicular to an optical axis in a photographing optical system composed of a variable power system and a master system. Are decentered in parallel and the output images are combined to obtain a high-definition image. JP-A-6
In Japanese Patent Laid-Open No. 1-232628, a parallel plane transparent body in front of an image sensor is rotated around an axis orthogonal to the optical axis to decenter the optical image on the image sensor in parallel, and output images are combined to form a high-definition image. It has gained. In Japanese Patent Laid-Open No. 7-287268,
The variable apex angle prism in front of the shooting optical system is driven based on the camera shake signal and the pixel shift signal, and the optical image on the image sensor is decentered in parallel to eliminate the image shake caused by the camera shake and improve the image definition by shifting the pixel. Achieved at the same time.

However, such a pixel shift technique, like the multiple exposure of a still camera, has a long interval between the acquisition times of the first image signal and the last image signal, and if a camera shake occurs between them, the image quality deteriorates. Higher definition cannot be expected by shifting the pixels. Therefore, as a prior art document for solving this drawback, for example, there are the following patent publications.

In Japanese Patent Laid-Open No. 7-240932, a variable apex angle prism in front of the photographing optical system or a moving lens group in the rear of the photographing optical system simultaneously achieves elimination of image blur caused by camera shake and high definition of an image by shifting pixels. is doing. Further, Japanese Patent Laid-Open No. 7-287268 also discloses that when the focal length of the photographing optical system is equal to or greater than a predetermined value, the pixel shift driving accuracy is lowered, and therefore the pixel shift control is prohibited.

Further, there are the following patent publications as prior art documents for improving the second shortcoming, which is the lack of dynamic range capability. JP-A-1-31
In 9370, a set of image pickup devices is exposed a plurality of times with different exposure amounts, and the images obtained by this operation are combined to obtain a wide dynamic range image. In Japanese Laid-Open Patent Publication No. 7-264488, a wide dynamic range image is obtained by synthesizing image signals obtained by a plurality of sets of image pickup elements having different sensitivity characteristics.

Further, as a prior art for solving the first and second drawbacks at the same time, in Japanese Patent Laid-Open No. 8-37628, at least one set of a plurality of images for pixel shifting is the other image. However, by obtaining different exposure amounts, a high-definition and wide dynamic range image is obtained.

On the other hand, in an image pickup apparatus having an image pickup element, a motion vector of an image is obtained from a time series output of the image pickup element,
It is possible to determine the image shake prior to shooting.
Therefore, as a prior art in this field, Japanese Patent Application Laid-Open No. 2-5707
In Japanese Patent Laid-Open No. 8 (1994), the motion vector of an image is continuously detected, and the image at the time when the motion vector becomes the minimum is used as a final recorded image, thereby suppressing the influence of camera shake during exposure. There is disclosure. In addition, JP-A-8-17256
Japanese Patent Publication No. 8 discloses a method for obtaining a high-definition image by obtaining a motion vector between a plurality of images for pixel shifting, removing image shake components due to camera shake or subject shake by interpolation, and then performing image synthesis. .

[0010]

However, the conventional examples disclosed in the above publications have the following drawbacks.
JP-A-60-27278, JP-A-60-91774
In Japanese Patent Laid-Open No. 61-236282 and Japanese Patent Laid-Open No. 61-236282, a high-definition image cannot be obtained under a shooting condition in which the image shake due to the hand shake is large because the shake correction mechanism is not provided. This is because the operation of obtaining multiple sets of images for pixel shifting is the same operation as multiple exposure, so the time from the first set of image capture to the final set of image capture is compared to the exposure time of normal shooting. It is much longer and, as a result, is more affected by camera shake.

In Japanese Unexamined Patent Publication No. 7-287268, since a variable vertical angle prism, which is a light beam deflection means for camera shake correction and pixel shifting, is located in front of the photographing optical system, the camera shake signal is converted into the drive amount of the variable angle prism. The coefficient to be used (camera shake correction coefficient) does not change due to zooming. Therefore, at the time of zooming, it suffices to change only the coefficient (pixel shift correction coefficient) for converting the pixel shift signal into the drive amount of the variable apex angle prism. However, if a light beam deflecting unit is provided in the image capturing optical system in order to reduce the size of the image capturing optical system, both the camera shake signal and the pixel shift signal must be converted with unique coefficients during zooming and the light beam deflecting unit must be driven. However, there is no such disclosure in this publication.

Further, in this publication, a pixel shifting mechanism is
Since it is applied to a video camera (camcorder) that records a moving image, the image capturing cycle is fixed and fixed by the field frequency of the moving image. However, when the pixel shift mechanism is applied to a so-called electronic still camera that records a still image, it is wider when an image sensor that can change the image capturing cycle according to the charge storage time (that is, subject brightness) of the image sensor is used. It can be applied to subjects in the brightness range. In this case, however, the influence of camera shake is more pronounced as the image capturing cycle is extended, so it is necessary to finely control the pixel shifting operation according to the shooting conditions. However, this publication only describes that the pixel shifting operation is changed according to the focal length of the photographing optical system. Further, there is no description about expansion of dynamic range.

In Japanese Patent Laid-Open No. 7-240932, the pixel shift is executed even when the accuracy of the camera shake correction is insufficient or the camera shake correction fails, so that not only a high-definition image cannot be obtained but also the pixel shift control is performed. An image of lower quality than the original image that does not exist may be obtained. In Japanese Patent Laid-Open No. 7-287268, the pixel shift is prohibited under the condition that a predetermined pixel shift accuracy is not expected to be obtained, but the determination is not made based on the effect of actual camera shake. Therefore, as in the above publication, when a large camera shake occurs, an image having a lower image quality than the original image may be obtained.

In Japanese Patent Laid-Open No. 60-91774, only high definition of an image is possible, and there is no description of expansion of dynamic range. JP-A-1-319370, JP-A-7-26
In each publication of No. 4488, only the dynamic range can be expanded, and there is no description of high definition.

On the other hand, in JP-A-8-37628,
Although there is a description of high definition of an image and expansion of a dynamic range, there is no detailed description of an exposure amount determination method for a plurality of sets of images obtained for expansion of a dynamic range. Therefore, it is not possible to finely respond to subject scenes having various luminance distributions existing in the world. Further, like the multiple exposure, the pixel shifting operation lengthens the apparent exposure (accumulation) time to the image pickup device, so that it is necessary to take measures against camera shake, which is not described in this publication.

In Japanese Patent Laid-Open No. 2-57078, there is no description of the pixel shift, and therefore, only the effect of reducing the influence of camera shake on a normal photographed image can be obtained, and high definition of an image by the pixel shift cannot be expected. Further, when performing high definition of an image by pixel shifting, it is necessary to control the pixel shifting amount to a predetermined amount based on the pixel interval of the image sensor. However, in Japanese Patent Laid-Open No. 172568/1996, there is no optical camera shake correction means using a variable apex angle prism or the like, so that image shake due to camera shake during pixel shifting is large and occurs randomly. Therefore, there is no guarantee that the predetermined amount of pixel shift will be performed, and it is unlikely that a high-definition image will be obtained even if image blur correction by interpolation is performed. In addition, there is no disclosure such as a warning when a desired high-definition image cannot be obtained, and selection of the next best measure.

The present invention has been made in view of the above problems, and an object of the present invention is to make it possible to appropriately perform pixel shifting in accordance with imaging conditions.

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

SUMMARY OF THE INVENTION An image pickup apparatus of the present invention comprises an image pickup optical means for forming a subject image, an image pickup means for photoelectrically converting the subject image, and a relative position between the subject image and the image pickup means. An image moving unit that changes in an image plane, a pixel shifting unit that obtains a plurality of sets of images by the image capturing unit while moving the image moving unit, and a combination of an aperture value and an exposure time as an image capturing condition of the image capturing unit. It is characterized in that an exposure control condition setting means for determining and a switching means for switching the operation of the pixel shifting means according to the setting state of the exposure control condition setting means are provided.

Another feature of the present invention is that
An image pickup optical means for forming a subject image, an image pickup means for photoelectrically converting the subject image, an image moving means for changing a relative position of the subject image and the image pickup means within an image forming plane, and the image moving means. Pixel shift means for obtaining a plurality of sets of images by the image pickup means while moving, determination means for determining the image pickup condition, and operation of the pixel shift means based on the determination result of the determination means for a first predetermined number of pixels A switching means for switching between a first pixel shifting mode for shifting and a second pixel shifting mode for shifting a pixel a second predetermined number of times different from the first predetermined number of times are provided. .

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

DETAILED DESCRIPTION OF THE INVENTION The first to tenth embodiments of the present invention will be described below. 1 to 5 are views commonly related to the first to tenth embodiments of the present invention. FIG. 2 shows an example of an image forming optical system, which has a focal length of 10 mm.
It is a 3 × zoom of ˜30 mm, the figure (a) is the wide end (f = 10 mm), and the figure (b) is the tele end (f = 30 m).
The arrangement of the lenses in m) is shown.

This image-forming optical system is composed of four groups. For zooming, the fourth group is fixed and the first, second, and third groups move, and the first group moves during focus adjustment. . Then, by displacing the second group in the direction perpendicular to the optical axis, the image on the image plane is displaced to perform pixel shift and camera shake correction.

Next, the light beam deflection effect of the second lens group will be described with reference to FIG. FIG. 3 shows each lens group of FIG. 2 in a simplified manner. In FIG. 3A, the second group has a predetermined amount d _L.
The light beam deflection effect on the image side when it is shifted downwards only
FIG. 11B also shows the light beam deflection effect on the object side when the second lens unit is shifted downward by the predetermined amount d _L. First, FIG. 3A will be described. A light beam incident on the optical axis of the first lens unit from the object side is deflected upward by the negative second lens unit that is shifted downward, passes through the third lens unit and the fourth lens unit, and is imaged on the image plane. Reach the high d _IM position. When the ratio of the second group shift amount d _L to the image displacement amount high d _IM at this time is referred to as the eccentricity sensitivity S _d , these three values are: d _IM = S _d × d _L. ) Is related.

The eccentricity sensitivity S _d changes depending on the arrangement of the second and subsequent lens groups, and therefore changes in accordance with zooming in this embodiment. On the other hand, in the present embodiment, since the front focus by the first group is adopted, the eccentricity sensitivity S _d does not change due to focusing. However, if a rear focus method that focuses on the fourth lens group is adopted, the eccentricity sensitivity S _d can be increased even by focusing.
Fluctuates. Therefore, since the eccentricity sensitivity S _d is generally expressed as a function S _d (f, R) of the focal length f and the subject distance R, the equation (1) is also expressed as d _IM = S _d (f, R) × d _L ……… Replaced with (2).

Next, the second group shift driving at the time of executing pixel shifting
The momentum will be described. Figure 4 explains the principle of pixel shifting
Fig. 2 is an enlarged view of the light receiving part of the image sensor.
It In FIG. 4, a square as a pixel is formed on the light receiving surface.
The light receiving part is lateral W_Y, Vertical direction W _PDistribute regularly at intervals
Placed. The resolution of the image formed on this light receiving surface is
Pixel spacing W_Y, W_PIs defined by
Multiple image signals are acquired while changing the relative position of
Image resolution by combining and reconstructing
Can be improved. For example, the point where the image is
When it is located at the center IM1 of the part, the first set of image signals (second order
All pixel signals of the original sensor) are fetched and stored.
The image is then moved to the right X so that the point of the image is located at IM2._Y
= W_YDisplace by / 2 and capture the second set of image signals
I remember. Similarly, move the image point to IM3, IM4
Image signals of the third and fourth groups are taken in, and a total of four pairs of image signals
The amount of information about the image is quadrupled by combining
The spatial resolution of can be doubled both horizontally and vertically.

Here, the image is moved to X _Y (= W for the pixel shift).
_Y / 2) or just to move the _{X P (= W P / 2} ) using the image movement effect of the second group shown in FIG. 3 (a). That is, in order to displace upward by X _P image in accordance with equation _{(2), d L = X} P / S d (f, R) and d _L only the second group determined by ......... (3) You can shift it down. The image displacement X _P for pixel shifting've always constant values, decentering sensitivity S _d (f, R) zooming since they depend focusing, the second group shift amount d _L
Also needs to be changed according to the state of the optical system. Therefore, in the present invention, as will be described later, data regarding the eccentricity sensitivity S _d (f, R) according to the zooming and focusing states is stored in a ROM (Read Only Memory) in a microprocessor (CPU) as a first coefficient. I remember.

Next, FIG. 3B will be described. A light beam emitted to the left from the center of the image plane along the optical axis c of the optical system passes through the fourth group and the third group, and is deflected upward by the negative second group which is shifted downward. Then, the light beam that has passed through the first group is deflected by an angle of θ _OB with respect to an axis c ′ parallel to the optical axis c of the image forming system, and is projected on the object side. If the ratio of the second group shift amount d _L and the optical axis deflection angle θ _OB at this time is called angle sensitivity Sθ, these three values are θ _OB = Sθ × d _L (4) Related by. Since the angle sensitivity Sθ changes depending on the arrangement of the lens groups before the second lens group, it changes according to zooming and focusing in this embodiment. Therefore, in general, the angle sensitivity Sθ is a function Sθ (f, f, of the focal length f and the subject distance R, like the eccentricity sensitivity S _d described above.
Since it is expressed as R), the equation (4) is also replaced with θ _OB = Sθ (f, R) × d _L (5).

Next, the shift driving amount of the second lens group during camera shake correction will be described. It is assumed that the camera body having the image forming optical system and the image pickup element shakes downward by the hand shake, that is, in the direction in which the tip of the image forming optical system faces downward by the angle θ _CAMERA . The image blur caused by camera shake at this time is equivalent to the image displacement when the subject moves upward by the angle θ _OB (= θ _CAMERA ) with respect to the camera. Therefore, referring to FIG. 3B, when the subject moves upward by the angle θ _OB , the movement of the subject image can be eliminated by shifting the second group downward by d _L. That is, based on the camera shake angle θ _CAMERA detected by the camera shake detection sensor and the formula (5), the second group is d _L obtained by the following formula: d _L = θ _CAMERA / Sθ (f, R) (6) Image shift due to camera shake can be eliminated by shifting in the downward direction.

Then, the camera shake angle θ_CAMERAChanges moment by moment
And the angle sensitivity Sθ (f, R) is also zoomed,
The amount of shift of the second lens unit d because it changes depending on the focusing.
_LAlso needs to be changed according to the state of the optical system. Book there
In the invention, the eccentricity sensitivity S described above is used. _dSimilar to (f, R),
Angle sensitivity S according to the state of focusing and focusing
Data regarding θ (f, R) is used as the second coefficient
It is stored in the ROM in the black processor.

FIG. 1 is a block diagram of the image pickup apparatus. In FIG. 1, CMR is a camera body, and LNS is a lens, which is configured as an interchangeable lens that is attachable to and detachable from the camera body CMR. First, the camera body CMR will be described. C
The CPU is an in-camera microcomputer (hereinafter abbreviated as a microcomputer), which is a one-chip microcomputer having ROM, RAM, A / D, and D / A conversion functions. Microcomputer C in camera
The CPU performs a series of camera operations such as automatic exposure control (AE), automatic focus adjustment (AF), and pixel shift control according to the camera sequence program stored in the ROM. Therefore, the in-camera microcomputer CCPU communicates with the peripheral circuits and the lens LNS in the camera body CMR to control the operation of each circuit and lens.

Four sets of connection terminals are provided on the mount portion for connecting the camera body CMR and the lens LNS. The power supply BAT in the camera supplies power to the circuits and actuators in the camera, and also the lens L via the line VCC.
It also supplies power to NS. DCL is the camera microcomputer CC
A line for transmitting a signal from the PU to the microcomputer LCPU in the lens, which will be described later, and a line DLC for transmitting a signal from the microcomputer LCPU in the lens to the microcomputer CCPU in the camera.
The camera body CMR uses the lens L through these two lines.
Control NS. Further, the grounds of the camera and the lens are also connected via the line GND.

IMS is an image pickup device such as CCD, IMDR is a driver for controlling charge accumulation, charge transfer and the like of the image pickup device IMS. The MEM is a memory for recording and storing image signal data of a photographed image, and a semiconductor memory, a magnetic disk, an optical disk or the like is used. The DISP is a display unit composed of a liquid crystal display or the like, and displays an image obtained by the image pickup device IMS and also displays the operating state of the camera and the like. BS is a beam splitter composed of a half mirror, which guides a part of the photographing light flux to the sensor SNS. Sensor S
NS comprises a focus detection sensor that detects the focus state of the imaging optical system and a photometric sensor that detects subject brightness. CN
C is a connector for connecting to an external device such as a desktop computer, and is used for transmitting the contents of the memory MEM to the outside and controlling the camera body CMR by a signal from the external device.

SWMN is a main switch, and when this switch is turned on, the microcomputer CCPU in the camera permits execution of a predetermined program relating to photographing. SW1, SW
A switch 2 is interlocked with the release button of the camera and is turned on by pressing the first stroke and the second stroke of the release button. SWSF is a pixel shift mode selection switch, and is used for permitting / not permitting pixel shift and selecting a predetermined mode from a plurality of pixel shift modes. SWIS is for image stabilization.
(Hereinafter, abbreviated as IS in the drawings) is a switch for selecting permission / non-permission of camera shake correction. SWM
OD is a photographing mode selection switch. When the photographer selects a predetermined mode, the AE mode and AF mode intended by the photographer are set, and the pixel shift mode and the camera shake correction mode are automatically set.

Next, the lens LNS side will be described. L
The CPU is the microcomputer in the lens and the microcomputer CCP in the camera
Like U, it is a one-chip microcomputer having ROM, RAM, A / D, and D / A conversion functions. Microcomputer LC in lens
PU is a signal line DCL from the microcomputer CCPU in the camera
Drive control of a focusing actuator, a zooming actuator, a diaphragm actuator, and an image blur correction actuator, which will be described later, is performed according to a command sent via the. Further, various operating conditions of the lens and parameters peculiar to the lens are transmitted to the in-camera microcomputer CCPU via the signal line DLC. L1 to L4 are lens groups corresponding to the first to fourth lens groups described with reference to FIG. 2, and constitute a zoom optical system, and a subject image is formed on the image pickup element IMS by this optical system.

FACT is a focusing actuator, which moves the first lens unit L1 forward and backward in the optical axis direction to adjust the focus, and the focus encoder FENC detects information corresponding to the position of L1, that is, the object distance, and the microcomputer LCPU in the lens detects it. Send out. ZACT is a zooming actuator, which moves the first lens unit L1 to the third lens unit L3 forward and backward in the optical axis direction by a zoom mechanism (not shown) to perform zooming, and to output the zoom information to the zoom encoder ZE.
The NC detects and sends it to the microcomputer LCPU in the lens.
DFM is a diaphragm for adjusting the amount of light, and DACT is a diaphragm actuator that drives the diaphragm DFM.

GRP and GRY are camera shake sensors such as a vibration gyro, which detect angular shakes in the vertical (pitch) direction and the horizontal (yaw) direction of the camera.
P and GRY are installed. Then, the shake detection result is transmitted to the lens microcomputer LCPU. Second lens unit L2
Are independently shiftable in a two-dimensional direction within a plane perpendicular to the optical axis. The pitch actuator I is arranged in the vertical direction with respect to the optical axis, that is, in the pitch shake correction direction.
It is driven by ACTP, and is driven by a yaw actuator IACTY in the left-right direction (the direction perpendicular to the paper surface in this figure), that is, in the yaw shake correction direction. Incidentally, this shift mechanism is described in, for example, Japanese Patent Application Laid-Open No. 6-3727 by the present applicant and is well known.

FIG. 5 is a block diagram for explaining the main operations of the pixel shift and camera shake correction of the present invention. The CCPU block surrounded by a two-dot chain line is the microcomputer CCP in the camera
The part executed by U, and the LCPU block is the part executed by the in-lens microcomputer LCPU. Reference numeral 11 denotes a photographing condition setting circuit, which sets operation modes of functions of AE, AF, pixel shift, and camera shake correction. A timing pulse generation circuit 12 generates a trigger signal for controlling the driving of the image moving optical system for shifting the pixel and the timing of capturing the image signal of the image sensor. Reference numeral 13 denotes an image sensor drive circuit, which captures an image signal at a predetermined timing and a predetermined charge storage condition in accordance with control signals from the shooting condition setting circuit 11 and the timing pulse generation circuit 12. A temporary storage circuit 14 temporarily stores the captured image signal for pixel shift processing. An image synthesizing circuit 15 synthesizes a plurality of sets of image signals obtained by the pixel shifting operation to obtain a high-definition image. A recording unit 16 corresponds to the memory MEM in FIG. 1 and records the combined high-definition image signal.

Reference numeral 21 denotes a pixel shift signal generation circuit, which generates a command signal for displacing an image for pixel shift. Two
Reference numeral 2 denotes a first coefficient generation circuit, which reads out data corresponding to the decentering sensitivity S _d (f, R) described above from the ROM of the microcomputer LCPU in the lens according to the focus and zoom information of the imaging optical system, and shifts the pixel. The shift amount command value of the second group is calculated so that the image movement amount becomes X _Y or X _P by correcting the signal magnitude.

Reference numeral 31 is a camera shake sensor, which corresponds to the above-mentioned vibration gyroscopes GRP and GRY. Reference numeral 32 denotes a camera shake signal calculation circuit, which performs filtering and integral calculation of the camera shake angular velocity signal detected by the camera shake detection sensor 31. Calculate the camera shake angle. Reference numeral 33 denotes a second coefficient generation circuit, which is used to detect the angle sensitivity S according to the focus and zoom information of the imaging optical system.
Data corresponding to θ (f, R) is transferred to the microcomputer LC in the lens.
The value is read from the ROM of the PU, the magnitude of the camera shake angle signal is corrected, and the second group shift amount command value for image shake prevention control is calculated.

Reference numeral 41 denotes a synthesizing circuit, which adds the second group shift amount command value for shifting the pixel determined in 22 and the second group shift amount command value for camera shake compensation determined in 33. 42
Is an image stabilization actuator, which is IACTP, I
The actuator is controlled so that the second lens group is driven according to the drive command value obtained by the combining circuit 41, which corresponds to ACTY. Reference numeral 43 is a block showing that the second lens group is actually being driven to shift, and the image pickup device 1 is driven by the driving.
The movement of the image is controlled to 3 above. Pixel shift and camera shake correction are simultaneously performed by the above blocks.

(First Embodiment) FIGS. 6 and 7 are flow charts showing a control flow of each of the microcomputers CCPU and LCPU in the camera body and the interchangeable lens in the first embodiment of the present invention. First, the control flow of the in-camera microcomputer CCPU will be described with reference to FIG. 1 and FIG. When the power switch (main switch) SWMN on the camera body CMR side is turned on, power supply to the in-camera microcomputer CCPU is started, and the operation from step (102) is started after step (101). In step (102), the state of SW1 which is turned on by pressing the release button in the first step is detected, and when the switch SW1 is off, the process proceeds to step (103). Then, in this step (103), an image blur correction operation (hereinafter referred to as IS (Image Stabiliz
(abbreviation of ation)) is sent. The above steps (102) and (103) are repeatedly executed until the switch SW1 is turned on or the power switch SWMN is turned off.

When the switch SW1 is turned on during execution of the above flow, steps (102) to (11)
Go to 1). In step (111), the in-camera microcomputer CCPU transmits an image blur correction start command to the in-lens microcomputer LCPU via the line DCL. In the next step (112), parameter communication is performed to acquire lens-specific parameters such as the lens open F-number and focal length from the lens microcomputer LCPU. In step (113), the brightness of the subject is measured by the sensor SNS, the image signal storage time of the image sensor and the aperture control value are calculated according to a predetermined exposure control program diagram, and the results are also transmitted to the in-lens microcomputer LCPU. In step (114), the focus state is detected by the sensor SNS and a driving command for the focusing lens is transmitted to the in-lens microcomputer LCPU.

In step (115), the state of the pixel shift mode selection switch SWSF is detected, and the pixel shift conditions such as whether or not the pixel shift can be performed and the number of pixel shifts are set based on the photometric result. In step (116), the state of SW2, which is turned on when the release button is pressed in the second stage, is detected, and the switch SW
When 2 is off, the process returns to step (111), and steps (111) to (115) are repeatedly executed. When SW2 is determined to be ON in step (116), the process proceeds to step (117).

In step (117), a counter CNT that counts the number of pixel shifts is initialized to zero. In step (118), a timing pulse serving as a trigger signal for pixel shift control is generated, and the in-lens microcomputer LCPU is generated.
Also send. In step (119), the driver IMD
Charge accumulation in the image sensor IMS and transfer / readout control of accumulated charges are performed via R. Step (120)
Then, the image signal read in the step (119) is temporarily stored in the RAM in the camera microcomputer CCPU.
In step (121), the pixel shift counter CN
Add 1 to T and update. In step (122), it is determined whether the counter CNT has reached a predetermined value N _SF . When the counter CNT has not reached the predetermined value N _SF , the process returns to step (118) to wait for the next timing pulse generation, and the pixel shift control is continued. Step (12
When the counter CNT reaches the predetermined value N _SF in 2), the process proceeds to step (123).

In step (123), the fact that the pixel shift has been completed is transmitted to the in-lens microcomputer LCPU. In step (124), a plurality of image signals obtained by pixel shifting are combined to create one high-definition image.
In step (125), the image obtained in step (124) is recorded in the memory MEM.

With the above, the photographing operation is completed, and step (10
Return to 2). Then, in this step (102), the switch S
If W1 is in the on state, the operation after step (111) is repeated, and if the switch SW1 is off, the in-lens microcomputer LCPU is instructed to stop the image blur correction operation in the step (103).

FIG. 7 is a flow chart showing the control of the microcomputer LCPU in the lens. In FIG. 7, when the power switch SWMN on the camera side is turned on to supply power to the interchangeable lens side, the process proceeds from step (131) to step (132). IS in step (132)
The start command is determined, and when the IS start command is not received from the camera body CMR, the process proceeds to step (133). In step (133), the IS stop command is determined,
When the IS stop command has not come from the camera body CMR, the process returns to step (132). When the IS stop command is received, the process proceeds to step (134) to stop the image shake correction actuator IACT in the pitch and yaw directions. If an IS start command is transmitted from the microcomputer CCPU in the camera during execution of steps (132) to (134),
The process moves from step (132) to step (141).

In step (141), the shake detection sensor G
Start RP and GRY, and input camera shake signals in the pitch and yaw directions. Step (142) is step (1) in FIG.
12), which transmits the lens-specific parameters to the camera side in accordance with the request from the camera microcomputer CCPU. In step (143), the zoom encoder ZENC and the focus encoder FENC are detected in order to detect the zoom state and the focus state of the optical system. Step (1
In step 44), the first coefficient for pixel shifting and the second coefficient for camera shake correction are read out from the ROM table based on the detection result of step (143). In step (145), the image blur correction actuator IACT is drive-controlled based on the image blur signal obtained in step (141) and the second coefficient obtained in step (144) to eliminate the image blur caused by the image blur.

In step (146), the diaphragm DFM is driven through the actuator DACT based on the photometric information obtained from the in-camera microcomputer CCPU to adjust the light amount.
In step (147), the focusing actuator FACT is driven based on the focus detection information obtained from the in-camera microcomputer CCPU to adjust the focus. Step (1
In 48), it is determined whether or not a timing pulse for pixel shifting is received. If the timing pulse has not been received, the process returns to step (141), and the camera shake correction, aperture control, and focus adjustment operation are repeatedly executed. When the reception of the timing pulse is confirmed in step (148), the process proceeds to step (149).

In step (149), a reference waveform for driving the second lens group in the pitch or yaw direction for pixel shifting is generated. In step (150), the pixel shift reference waveform generated in step (149) is multiplied by the first coefficient read in step (144) to generate a pixel shift drive waveform, and this is combined with the image stabilization signal. To do. By driving and controlling the image blur correction actuator IACT according to this combined signal, the image drive for the image blur correction and the pixel shift is simultaneously and accurately executed.

In step (151), it is judged whether or not the pixel shift completion signal is transmitted from the in-camera microcomputer CCPU. If it is not transmitted, the pixel shift is not completed, so the process returns to step (148) and the next step is executed. Waits for the reception of the timing pulse, and when step (148) to step (150) are performed a predetermined number of times and the pixel shift completion signal transmitted after the pixel shift operation is completed is received,
The process returns from step (151) to step (132). Then, if the IS start command is not received in step (132) and the IS stop command is confirmed in step (133), the image shake correction actuator IACT is stopped in step (134), and a series of lens control operations associated with imaging ends. To do.

FIG. 8 is a timing chart for explaining the operation of the camera and lens according to the flow of FIGS. 6 and 7. (A) and (b) show the states of the switches SW1 and SW2 which are interlocked with the release button of the camera.
(C) is a trigger signal for pixel shift timing control. (D) shows the charge accumulation timing of the image sensor IMS. (E) and (f) show pixel shift reference waveforms in the pitch and yaw directions. (G) and (h) are camera shake sensors GR
The shake waveforms detected by P and GRY, in which the shake displacement waveform after the detection signal is appropriately subjected to processing such as integration are shown. (I) and (j) show the pitch and the drive displacement in the yaw direction of the second lens group for image blur correction.

The flow of FIGS. 6 and 7 will be summarized below with reference to FIG. At time t ₁ , switch SW1
When is turned on, camera shake signals (g) and (h) are output. Then, according to the value obtained by multiplying this signal by the second coefficient, the second lens unit L2 is drive-controlled as in the waveforms (i) and (j), and camera shake correction is performed.

[0088] When the switch SW2 at time t ₂ is turned on, first timing pulses TP1 is generated at time t ₁₁ after a predetermined time. Then, in response to this, the image pickup device IMS accumulates charges in the light receiving unit during the period from time t ₁₂ to time t ₁₃ according to the accumulation time calculated from the photometric result.
When charge accumulation at time t ₁₃ is finished, the transfer and read the accumulated charge, a yaw direction of the pixel shifting reference waveform (f) is generated. Then, the lens displacement (j) in the yaw direction is driven by the command value obtained by adding the value obtained by multiplying the reference waveform (f) by the first coefficient and the camera shake correction waveform.

[0089] From time t ₁₁ to the predetermined time has elapsed after the time t ₂₁ second timing pulse TP2 is generated. Then, similarly, the image sensor IMS accumulates charges in the light receiving unit during the period from time t ₂₂ to time t ₂₃ . When the charge accumulation at time t ₂₃ is finished, the transfer and read the accumulated charge Pitch pixel shifting reference waveform (e) is generated. Then, the lens displacement (i) in the pitch direction is also driven by the command value obtained by adding the value obtained by multiplying the reference waveform (e) by the first coefficient and the camera shake correction waveform.

[0090] The time t ₃₁ after a predetermined time from the time t ₂₁ the third timing pulse TP3 is generated. Then, similarly, the image sensor IMS accumulates charges in the light receiving unit during the period from time t ₃₂ to time t ₃₃ . When the charge accumulation ends at time t ₃₃ , the pixel shift reference waveform (f) in the yaw direction is returned to the original value as the accumulated charge is transferred and read. Then, the lens displacement (j) in the yaw direction is driven by the command value that responds only to the shake correction waveform.

[0091] From time t ₃₁ to time t ₄₁ after a predetermined time the last moment of the timing pulse TP4 is generated. Then, similarly, the image sensor IMS accumulates charges in the light receiving unit during the period from time t ₄₂ to time t ₄₃ . When the charge storage ends at time t ₄₃ , the pixel shift reference waveform (e) in the pitch direction is returned to the original value as well as the transfer / readout of the stored charge. Then, the lens displacement (i) in the pitch direction is also driven by the command value that responds only to the shake correction waveform. The switch SW1 at time t ₅ is once turned off, the driving of the hand vibration detection and the second lens group is stopped.

The positions of the images at the times t ₁₁ , t ₂₁ , t ₃₁ , t ₄₁ , and t ₅ during the above-described camera shake correction and pixel shift control are IM1, IM2, IM3, and IM3, IM2, IM3, respectively.
IM4 and IM1 are realized, and image shifting in which the vertical and horizontal positions of each image are shifted by half the pixel interval is realized. In addition,
The image shift reference waveform is not a rectangular wave but a trapezoidal wave in order to mitigate an impact due to a sudden position change of the second lens group.

Next, the principle of synthesizing a plurality of sets of image signals obtained by pixel shifting to generate one set of high-definition image signals will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram for explaining the relative positional relationship between the image and the image sensor in the pixel shift. 9A is the same as that described in FIG. 4, and the image is IM1 → I for the pixels of the image sensor fixed in the camera.
It indicates that the mobile terminal moves in the order of M2 → IM3 → IM4 → IM1. As shown in (b) below, the position of the image sensor is IG1 → IG2 → IG3 → IG with respect to the immovable image.
It is completely equivalent to moving in the order of 4 → IG1. Therefore, the output signal of each pixel when the image sensor is located at IG1 is I
Let G1 (i, j). Here, the image sensor is an area sensor of (m × n) pixels. Similarly, IG2, IG3, IG
The output signals at the time of being positioned at 4 are IG2 (i, j), IG3
(I, j) and IG4 (i, j).

FIG. 10 illustrates the method of synthesizing these four sets of image signals. IM a new pixel signal set of (2m × 2n) pixels, which is a combination of signals of four sets of (m × n) pixels.
Let G (u, v). Then, the pixel signal IMG (u, v)
The four pixels on the upper left of FIG. 4 are the four original pixels assembled as shown in the figure. Considering the image restoration method based on this figure, IMG (u = 2i-1, v = 2j) ← IG1 (i, j) ... (7) IMG (u = 2i-1, v = 2j- 1) ← IG2 (i, j) ... (8) IMG (u = 2i, v = 2j-1) ← IG3 (i, j) ... (9) IMG (u = 2i, v = 2j) ← IG4 (i, j) ... (10) According to the equation, one set of high-definition image signals can be generated from four sets of image signals.

This method is applied to a monochrome image pickup device or a multi-plate type color image pickup device using a color separation prism. In the single-plate type color image pickup device using a mosaic type color filter, the pixel shift amount and the image synthesizing method are There are some differences in points, but the basic idea is the same.

According to the first embodiment, (1) the pixel shift signal is converted by the first coefficient, the camera shake signal is converted by the second coefficient, and the image shake correction optical system is formed by the combined signal of both. By driving, one image moving unit simultaneously performs camera shake correction and pixel shift control, so that it is possible to obtain a high-definition image signal that is free from image deterioration due to camera shake and pixel shift. (2) By using the first coefficient and the second coefficient according to the zoom and focus states, it is possible to always perform accurate camera shake correction and pixel shift control even if the zoom and focus states change. (3) Since the control circuit in the camera also performs image composition after pixel shifting, a high-definition image signal can be obtained without using a dedicated external device or the like. There is an effect.

(Second Embodiment) The first embodiment has been described as an embodiment for accurately performing camera shake correction and pixel shift at the same time. The second embodiment shown below is
An embodiment in which an optimum pixel shift mode is selected according to a shooting state of a camera will be described. FIG. 11 is a part of a control flow of the second embodiment, FIG. 12 is a principle diagram of a second pixel shift mode, and FIG. 13 is a principle explanatory diagram of image synthesis in the second pixel shift mode. The operation of the second embodiment will be described below with reference to the drawings.

The control flow of the camera of this embodiment is basically the same as the control flow of the camera of the first embodiment shown in FIG. 6, but the "pixel shift condition of step (115)" By setting the "setting" portion to the subroutine shown in FIG. 11, the operation peculiar to this embodiment is realized. The flow of this embodiment will be described below with reference to FIGS. 6 and 11, but the flow of FIG. 6 has been described above in detail, and therefore will be briefly described here.
In FIG. 6, when it is determined that SW1 is turned on in step (102), the process proceeds to step (111), and step (11)
1) In step (112), an IS start command is transmitted and parameter communication is performed. Subsequently, photometry and focus detection are performed in steps (113) and (114), and the results are also transmitted to the in-lens microcomputer LCPU.

In the next step (115), the flow shown in FIG. 11 is executed. In step (215) of FIG. 11,
The microcomputer LCPU in the lens is requested to transmit the shake angular velocity peak value ω _peak within a predetermined time. Then, the in-lens microcomputer LCPU transmits, for example, the maximum camera shake angular velocity ω _peak in the two-second pitch or the yaw direction to the in-camera microcomputer CCPU. In step (216), the maximum value δ of the image blur amount during exposure when the camera shake correction is inoperative is calculated according to the following equation δ = f × ω _peak × t _exp . Where f is the focal length of the imaging optical system, t
_exp is the exposure time of the image sensor, which is derived from the photometric result, that is, the charge storage time. Image blur amount δ derived here
Is used in the following steps as a camera shake index for determining whether or not pixel shifting should be executed.

In step (217), the magnitude of the image blur amount δ is determined. If the image blur amount δ is less than or equal to the predetermined value DEL1, it is considered that the influence of the image deterioration due to the camera shake is small and the pixel shift can achieve a sufficiently high definition, and the pixel shift number N _SF is set to 4 in step (218). To do. Here, the number of pixel shifts being 4 means the pixel shift and image synthesizing method described in the first embodiment.
This is called a first pixel shift mode. Step (2
In 17), the image shake amount δ is less than or equal to a predetermined value DEL2 (however, DE
If it is determined that L2> DEL1), the number of pixel shifts, N _SF , is set to 2 in step (219). In this mode, a slight image quality deterioration due to camera shake is expected even if camera shake correction is executed.
This is for optimizing the image quality deterioration due to camera shake and the image quality improvement due to pixel shifting. The second pixel shift mode will be described later.

If it is determined in step (217) that the image shake amount δ is larger than the predetermined value DEL2, step (220)
The pixel shift count N _{SF is set} to 1, that is, the pixel shift is not executed. Even if the camera shake correction is executed, the pixel shift is prohibited because the influence of the image quality deterioration due to the camera shake is expected to exceed the image quality improvement due to the pixel shift. After performing steps (218) to (220), the process proceeds to step (221). Step (22
In 1), the pixel shift mode determined in the above flow is transmitted to the in-lens microcomputer LCPU. Step (222)
Then, the type of the pixel shift mode is displayed on the display means DISP in the camera to inform the photographer of which mode the photographing is performed.

After execution of step (222), the process returns to step (116) in FIG. In step (116), the switch SW2 is discriminated. If SW2 is on, step (11)
Proceed to 7). Step (117) to Step (12)
In 2), the pixel shift is executed as in the first embodiment, but in the second embodiment, a plurality of pixel shift modes are selectively used in steps (215) to (222) as described above. Therefore, also in steps (117) to (122), the operation is controlled according to the pixel shift mode. Subsequently, in step (123), pixel shift completion communication is performed to the microcomputer LCPU in the lens,
Go to step (124). Also in step (124), image composition according to the plurality of pixel shift modes is selected and performed. In step (125), the pixels synthesized in the previous step are recorded, and the process returns to step (102).

FIG. 12 illustrates the operation in the second pixel shift mode, that is, when the pixel shift number is two, and corresponds to FIG. 9 in the first embodiment. 12
The upper part (a) shows the movement of the image with respect to the light receiving element. At a certain point on the image, the charge is stored / read out from the light receiving element in the state of being located on the IM 21. Then, after the image is moved to the IM 22 by the pixel shifting operation, charge accumulation / readout of the light receiving element is performed again. The image is then returned to the original position IM21. The lower diagram (b) of FIG. 12 shows the movement of the image replaced by the movement of the image pickup device.
The image sensor located at 1 moves to IG22 by the pixel shifting operation, and then returns to IG21.

FIG. 13 is a diagram for explaining an image synthesizing method in the second pixel shift mode. In this second pixel shift mode, IG21 (i, j) and IG22 (i, j)
2 sets of images, that is, (4 × m × n) IMG2 (u, composed of (2 × m × n) image signals in total.
The image signal of v) is synthesized. Therefore, as shown in FIG. 13, the pixels indicated by black circles in IMG2 (u, v) are IG21.
The pixel of (i, j) or IG22 (i, j) is used as it is, and the pixel indicated by the white circle is synthesized from the average value of the four pixels around it (or 2 or 3 pixels when there are no 4 pixels).

When this is expressed by an equation,   IMG2 (u = 2i-1, v = 2j) <-IG21 (i, j)                                                         ……… (12)   IMG2 (u = 2i, v = 2j-1) ← IG22 (i, j)                                                         ……… (13)   IMG2 (u = 2i-1, v = 2j-1) ← {IG21 (i, j) + IG22 (I, j) + IG21 (i, j-1) + IG22 (i-1, j)} / 4                                                         ……… (14)   IMG2 (u = 2i, v = 2j) ← {IG21 (i, j) + IG22 (I, j) + IG21 (i + 1, j) + IG22 (i, j + 1)} / 4                                                         ……… (15) Can be expressed as

According to the second embodiment, the first
In addition to the effects of the embodiment described above, (4) the optimum pixel shift mode is selected according to the amount of camera shake in consideration of the image quality deterioration due to the camera shake during execution of pixel shift and the pixel shift effect. Optimal pixel shift operation can be executed. There is an effect.

(Third Embodiment) In the second embodiment, the embodiment in which the optimum pixel shift mode is selected according to the camera shake is shown. The third embodiment shown below shows an embodiment in which the appropriateness of pixel shift execution is determined from the shooting mode of the camera set by the photographer, and the pixel shift mode is switched. Similar to the second embodiment, the control flow of the camera of the third embodiment is basically the same as the control flow of the camera of the first embodiment shown in FIG. By setting the "pixel shift condition setting" portion of (115) to the subroutine shown in FIG. 14, the operation peculiar to the third embodiment is realized. The flow of the third embodiment will be described below with reference to FIGS. 6 and 14, but the flow of FIG. 6 has been described in detail in the first embodiment, and therefore will be briefly described here. To do.

In FIG. 6, in step (102) SW
When it is determined that 1 is ON, the process proceeds to step (111), and in step (111) and step (112), an IS start command is transmitted and parameter communication is performed. Then step (1
13) In step (114), photometry and focus detection are performed, and the results are also transmitted to the in-lens microcomputer LCPU. In the next step (115), the flow shown in FIG. 14 is executed. In step (315) of FIG. 14, the camera CM
The state of the R shooting mode selection switch SWMOD is determined and the shooting conditions such as the exposure control mode set by the photographer are recognized.

At step (316), it is determined whether or not the photographing mode is the landscape mode. The landscape mode is an exposure control mode in which the aperture control value is set to a small aperture (large F number) to deepen the depth of field. When the landscape mode is set, it is expected that the subject is still and the camera is firmly held so that camera shake is unlikely to occur. Therefore, the process proceeds to step (319) to set the high-definition mode in which pixel shifting is performed four times. If it is determined in step (316) that the mode is not the landscape mode, the process proceeds to step (317). Step (3
In 17), it is determined whether or not the shooting mode is the portrait mode. The portrait mode is an exposure control mode in which the aperture control value is set to the aperture close to the open aperture (small F number) to reduce the depth of field. Since it is expected that the photographing mode is the same as that in the landscape mode in the portrait mode, the process proceeds to step (319). If it is determined in step (317) that the portrait mode is not set, the process proceeds to step (318).

At step (318), it is determined whether or not the photographing mode is the sports mode. Sports mode is an exposure control mode that stops the movement of the subject by setting the exposure time to high speed. When the sport mode is selected, the subject is moving, and it is expected that the camera will undergo a panning operation, that is, a camera movement similar to a large camera shake. In addition, the movement of the subject during the pixel shift operation causes subject shake, which may not be expected to provide high definition due to the pixel shift, and may result in an unnatural photograph. Therefore, when setting the sports mode, step (32
Proceed to 1) and N _SF = 1 is set, that is, pixel shifting is prohibited.
If it is determined in step (318) that the mode is not the sports mode, that is, if it is not any of landscape, portrait, and sports modes, the process proceeds to step (320), and the pixel shift count is set to two. Step (3
After executing 19) to step (321), the process proceeds to step (322).

In step (322), the pixel shift mode determined in the above flow is transmitted to the in-lens microcomputer LCPU. In step (323), the display means D in the camera
The type of the pixel shift mode is displayed on the ISP to notify the photographer of the mode in which the photographing is performed. After execution of step (323), the process returns to step (116) in FIG. In step (116), the switch SW2 is determined, and if SW2 is on, the process proceeds to step (117). In steps (117) to (122), pixel shift is executed according to the selected pixel shift mode as in the second embodiment. Then, in step (123), pixel shift completion communication is performed to the microcomputer LCPU in the lens, and the process proceeds to step (124). Also in step (124), image composition according to the plurality of pixel shift modes is selected and performed. In step (125), the image combined in the previous step is recorded, and the process returns to step (102).

According to the third embodiment described above, the first
In addition to the effects of the embodiment of (5), whether or not the pixel shift execution is appropriate is determined based on the shooting mode of the camera set by the photographer, and the pixel shift mode is switched to select a pixel suitable for the movement of the camera and the subject. The shift operation can be executed. There is an effect. Further, in the present embodiment, the exposure control mode is targeted as an example of the shooting mode, but the pixel shift mode may be switched according to the focus detection mode or the like.

(Fourth Embodiment) In the first embodiment, pixel shift and camera shake correction are performed by one image moving means using the first coefficient for pixel shift control and the second coefficient for camera shake correction. Was running. On the other hand, by using the first coefficient for camera shake correction control of a method different from the camera shake correction, the camera shake correction capability can be further enhanced. Figure 1
5 is a block diagram of a fourth embodiment for that purpose.
It is a modification of the block diagram of the first embodiment shown in FIG. In FIG. 15, the motion vector detection circuit 17 is added to the camera microcomputer CCPU, and another first coefficient generation circuit 22a is added to the lens microcomputer LCPU.
The difference is that was added.

The motion vector detection circuit 17 is a known circuit that detects a positional shift due to camera shake of two sets of images from the spatial correlation of two sets of image signals captured at different timings in time. Used for image stabilization. As the two sets of images used for motion vector detection, images obtained by pixel shifting or images periodically fetched regardless of pixel shifting are used. Alternatively, a signal from the focus detection sensor may be used. However, since the relative positions of the two sets of image signals for pixel shifting are shifted in advance by a predetermined dimension due to pixel shifting, it is necessary to correct this shift when detecting a motion belt.

By the way, in the present embodiment, since the camera shake correction system by the camera shake detection sensor utilizing the inertial force such as the vibration gyro is incorporated, the image shake is corrected while the camera shake correction system is operating, The motion vector detection circuit 17 should not detect shake. However, a shake detection sensor such as a vibration gyro utilizing the inertial force has a drawback that it cannot detect a shake in an ultra-low frequency band due to the occurrence of DC offset or drift of an output signal. Therefore, the motion vector detection circuit 17 detects low-frequency image blur even during the operation of the camera shake correction system. Therefore, this image blur signal is converted by the first coefficient generating circuit 22a, and this signal and the camera shake signal that has passed through the second coefficient generating circuit 33 are combined by the combining circuit 41. By driving the camera shake correction actuator 42 with the combined output, a camera shake correction system in a wide band from a low frequency range to a high frequency range is realized, and the camera shake correction capability is enhanced. As a result, the image blurring when the pixels are shifted is further reduced, which contributes to the high definition of the image.

16 and 17 are flow charts showing a control flow of each microcomputer in the camera body and the interchangeable lens in the fourth embodiment. First, referring to FIG. 1 and referring to FIG.
The control flow of is explained. When the power switch (main switch) SWMN on the camera body CMR side is turned on, power supply to the in-camera microcomputer CCPU is started, and the operation from step (402) is started via step (401). In step (402), the state of SW1 which is turned on by pressing the release button in the first step is detected, and when this switch SW1 is off, step (40)
Go to 3). Then, in this step (403), a command to stop the image blur correction operation is transmitted to the interchangeable lens LNS side. Steps (402) and (403)
Is repeatedly executed until the switch SW1 is turned on or the power switch SWMN is turned off.

When the switch SW1 is turned on during execution of the above flow, steps (402) to (41)
Go to 1). In step (411), the in-camera microcomputer CCPU transmits an image blur correction start command to the in-lens microcomputer LCPU via the line DCL. In the next step (412), parameter communication is performed so that the in-camera microcomputer CCPU obtains lens-specific parameters such as the lens open F-number and the focal length. In step (413), the brightness of the subject is measured by the sensor SNS, the image signal storage time of the image sensor and the aperture control value are calculated, and the result is also transmitted to the lens microcomputer LCPU. In step (414), the focus state is detected by the sensor SNS, and the result is detected by the in-lens microcomputer LC.
Also send to PU.

In step (415), the state of the pixel shift mode selection switch SWSF is detected, and the pixel shift conditions such as whether or not the pixel shift can be performed and the number of pixel shifts are set based on the photometric result. In step (416), the image sensor I
Charge accumulation of MS and transfer / readout control of accumulated charge are performed. In step (417), the control step (4
16) The image signal read out in 16)
Temporarily stored in RAM in U. In step (418), a motion vector is detected from the two sets of images stored in the RAM. When this step is executed for the first time, there is only one set of image signals, so at this time, the motion vector is zero.

In step (419), the motion vector detected in the previous step is transmitted to the in-lens microcomputer LCPU. In step (420), the state of SW2 that is turned on by pressing the release button in the second step is detected. When this switch SW2 is off, the process returns to step (411), and steps (411) to (419) are repeatedly executed. To do. When SW2 is determined to be ON in step (420), the process proceeds to step (421). In step (421), the pixel shift control similar to steps (117) to (122) of the first embodiment shown in FIG. 6 is executed. In step (422), the fact that the pixel shift is completed is transmitted to the in-lens microcomputer LCPU. In step (423), a plurality of image signals obtained by pixel shifting are combined to create one high-definition image. In step (424), the above step (423)
The image obtained in step 1 is recorded in the memory MEM.

With the above, the photographing operation is completed, and step (40
Return to 2). Then, in this step (402), the switch S
If W1 is in the on state, the operation after step (411) is repeated, and if the switch SW1 is off, the in-lens microcomputer LCPU is instructed to stop the image blur correction operation in step (403).

FIG. 17 is a flow chart showing the control of the microcomputer LCPU in the lens. In FIG. 17, when the power switch SWMN on the camera side is turned on to supply power to the interchangeable lens side, the process proceeds from step (431) to step (432). In step (432), the IS start command is determined, and the IS
When the start command has not arrived, the process proceeds to step (433).
In step (433), the IS stop command is discriminated, and when the IS stop command is not received from the camera body CMR, the process returns to step (432). When the IS stop command is received, the process proceeds to step (434) to stop the image shake correction actuator IACT in the pitch and yaw directions. When an IS start command is transmitted from the in-camera microcomputer CCPU during execution of steps (432) to (434), the process moves from step (432) to step (441).

In step (441), the shake detection sensor G
Start RP and GRY, and input camera shake signals in the pitch and yaw directions. The step (442) corresponds to the step (412) in FIG. 16, and transmits the lens-specific parameter to the camera side according to the request of the in-camera microcomputer CCPU. In step (443), the zoom encoder ZEN is used to detect the zoom state and focus state of the optical system.
C, focus encoder FENC is detected. In step (444), the first coefficient for pixel shifting and the second coefficient for camera shake correction are read out from the ROM table based on the detection result of step (443). In step (445), the image blur correction actuator IACT is drive-controlled based on the image blur signal obtained in step (441) and the second coefficient obtained in step (444) to eliminate the image blur caused by the image blur.

In step (446), the diaphragm DFM is driven through the actuator DACT based on the photometric information obtained from the in-camera microcomputer CCPU to adjust the light amount. At step (447), the microcomputer CCPU in the camera
The focusing actuator FACT is driven to adjust the focus based on the focus detection information obtained from. In step (448), the motion vector corresponding to step (419) in FIG. 16 is received. In step (449), an image shake correction signal is generated based on the motion vector signal obtained in step (448) and the first coefficient obtained in step (444), and this signal and the image shake correction signal obtained from the vibration gyro are combined. To synthesize. Then, the image blur correction actuator IACT is driven and controlled by the combined signal, so that the image blur correction in a wider band is realized.

At step (450), it is judged whether or not the timing pulse for shifting the pixel is received. If the timing pulse is not received, step (44)
Returning to 1), camera shake correction, aperture control, focus adjustment, and motion vector reception are repeatedly executed. Step (450)
After confirming the reception of the timing pulse in step (4)
Proceed to 51). In step (451), a reference waveform for driving the second group in the pitch or yaw direction for pixel shifting is generated. In step (452), the pixel shift reference waveform generated in step (451) is multiplied by the second coefficient read in step (444) to generate a pixel shift drive waveform, and this is combined with a signal for camera shake correction. . By driving and controlling the image blur correction actuator IACT according to the combined signal, the image blur correction and the image drive for pixel shifting are simultaneously and accurately executed.

At step (453), it is determined whether or not the pixel shift completion signal is transmitted from the microcomputer CCPU in the camera. If not transmitted, the pixel shift is not completed, so the process returns to step (450). Wait for the reception of the timing pulse. When step (450) to step (453) are performed a predetermined number of times and a pixel shift completion signal transmitted after the pixel shift operation is completed is received,
The process returns from step (453) to step (432). Then, if the IS start command is not received in step (432) and the IS stop command is confirmed in step (433), the image shake correction actuator IACT is stopped in step (434), and a series of lens control operations associated with imaging ends. To do.

According to the fourth embodiment, the first
(6) The motion vector signal is converted by the first coefficient, the camera shake signal is converted by the second coefficient, and the image blur correction optical system is driven by the combined signal of both. It is possible to widen the image stabilization band. (7) By using the first coefficient and the second coefficient according to the zoom and focus states, it is possible to always perform accurate camera shake correction even if the zoom and focus states change. There is an effect. The fourth embodiment is effective regardless of the presence or absence of the pixel shift function.

(Other Modifications) In the first to fourth embodiments, the lens group in the imaging optical system is arranged in the direction perpendicular to the optical axis as the image moving means for camera shake correction and pixel shifting. And the optical axis deflecting action by this was utilized. Instead, it is also possible to use a so-called variable apex angle prism, which is composed of two transparent parallel flat plates and a transparent liquid sealed therein, as a part of the imaging optical system. Further, the second and third embodiments are effective for any type of imaging optical system, and exert their effects regardless of the presence or absence of a camera shake correction mechanism.

(Fifth Embodiment) FIG. 18 shows the control flow of the microcomputer CCPU in the camera according to the fifth embodiment, except for the control flow and step (115) of the first embodiment of FIG. Substantially the same processing is performed. In step (115), the state of the pixel shift mode selection switch SWSF is detected and the step (1) is performed.
13) and (114) are used to set conditions such as the pixel shift control mode and the exposure control mode of the image sensor based on the focus detection result and the like. Details will be described later.

The control flow of the microcomputer LCPU in the lens is substantially the same as the control flow of FIG. 7 according to the first embodiment.

Next, the details of the exposure amount control method of the fifth embodiment will be described with reference to FIGS. 19 to 21. First, the light receiving characteristics of a general image sensor will be described with reference to FIG. Figure 20 shows the characteristic curve of film or image sensor.
The horizontal axis represents the exposure amount, the vertical axis represents the image density in the case of a silver salt film, and the output voltage in the case of an image sensor. When a subject is photographed with a constant aperture value and shutter speed, the horizontal axis in the figure can be regarded as the subject brightness. In FIG. 20, a broken line represents a silver salt film, and a solid line represents a characteristic curve of an image pickup device such as CCD. While the silver salt film has a wide dynamic range, the dynamic range of the image pickup device is narrow, and only a subject in the brightness range of H ₁ to H ₂ can be reproduced.

FIG. 19 is a flow chart for setting the exposure amount, which details step (115) in FIG. Step (18
Step 1) is a step of dividing the photometric subject in step 113 in FIG. 6 into blocks. The method is shown in FIG.
Will be explained. FIG. 21 is a diagram showing a detection region of the focus detection / photometry sensor SNS shown in FIG. The detection area of the sensor SNS has a detection area almost the same as the light receiving area of the image sensor IMS, and the detection area is divided into 8 × 12 = 96 areas as shown in FIG. Also, the subject brightness can be detected. The sensor SNS has, for example, a focus detection means of a secondary imaging phase difference detection system including a pair of secondary imaging optical systems and a two-dimensional image sensor provided in each secondary imaging optical system. Can be realized by

A subject image is formed on the sensor SNS as shown in FIG. 21, and from the focus detection result and the luminance measurement result of this subject image, the subject areas are BK1 to BK.
It is classified into 5 blocks of 5. Here, each block is divided into blocks so that subjects having substantially the same distance and brightness level are included.

Next, returning to FIG. 19, step (182)
Then, the main subject is analogized from the divided blocks according to a predetermined algorithm. Specifically, if the focus detection area is set by the photographer, the subject included in the area is the main subject, and the focus detection area is set to the automatic selection mode,
If the photometric area is selected by the subject / photographer near the center of the screen and at a relatively short distance, the subject included in the photometric area has the function of detecting the gaze direction of the photographer within the main subject / fanfinder. With such a camera, the main subject can be inferred by determining the subject in the gaze direction of the photographer. In the present embodiment, according to the above algorithm, the block B including a person is
It is determined that K3 is the main subject.

Subsequently, in step (183), blocks are grouped based on the divided blocks and the main object analogy result. This is because the main function of the present invention is to adjust the exposure amount when the pixel is shifted, and the number of times the exposure amount is adjusted must be equal to or less than the pixel shift number. Therefore, the large number of blocks are organized into a small number of groups, and the exposure amount adjustment number is set to be equal to or less than the pixel shift number. Specifically, as shown below, B
K1 and BK2 are set to the high brightness group GP1, the block BK3 including the main subject is set to the middle brightness group GP2, and the blocks KB4 and KB5 are set to the low brightness group GP3. BK1, BK2 → GP1 (high brightness group) BK3 → GP2 (medium brightness group) BK4, BK5 → GP3 (low brightness group)

Then, in step (184), the average luminance of the subjects in each group determined in the previous step (183) is calculated. In step (185), appropriate exposure doses E ₁ to E _n (here, n
= 3) is calculated. In step (186), the aperture value and the exposure time t _e1 to t _e1 to obtain the proper exposure amount are obtained.
_En is calculated according to a predetermined program diagram. After the execution of step (186), the process returns to step (116) of FIG. 6 to shift the pixels and expose the image sensor IMS (charge accumulation).
To execute.

FIG. 22 is a timing chart for explaining the operation of the camera and the lens according to the flow of FIGS. 6 and 19. (A) and (b) respectively show the states of the switches SW1 and SW2 which are interlocked with the release button of the camera. (C) is a trigger signal for pixel shift timing control. (D) shows the charge accumulation timing of the image sensor IMS. (E) and (f) show pixel shift reference waveforms in the pitch and yaw directions. (G) and (h) are the shake waveforms detected by the shake detection sensors GRP and GRY, and here, the shake displacement waveforms after the detection signals are appropriately subjected to processing such as integration are shown. (I) and (j) show the pitch and the drive displacement in the yaw direction of the second lens group for image blur correction.

The flow of FIGS. 6 and 19 will be summarized below with reference to FIG. Switch SW at time t ₁
When 1 is turned on, camera shake signals (g) and (h) are output. Then, the second lens group is drive-controlled in accordance with the value obtained by multiplying this signal by the second coefficient, as shown in waveforms (i) and (j), and camera shake correction is performed. When the switch SW2 is turned on at time t _2, the first timing pulse TP1 is generated at time t ₁₁ after a predetermined time. Then, in response to this, the image pickup device IMS _causes the light receiving unit during the period from time t ₁₂ to time t _{13 in} order to perform the exposure for the exposure time t _e2 suitable for the medium luminance group GP2 calculated in step (186) of FIG. Charge accumulation.

When the charge accumulation ends at time t ₁₃ , the pixel shift reference waveform (f) in the yaw direction is generated together with the transfer and reading of the accumulated charge. Then, the lens displacement (j) in the yaw direction is driven by the command value obtained by adding the value obtained by multiplying the reference waveform (f) by the first coefficient and the above-mentioned shake correction waveform. The second timing pulse TP2 is generated at time t ₂₁ after a predetermined time from the time t _11. Then, similarly, the image sensor I
The MS accumulates charges in the light receiving unit during the period from time t ₂₂ to time t ₂₃ . The exposure time here is the same as the step (1
The exposure time t _e1 suitable for the high brightness group GP1 calculated in 86) is set. When the charge accumulation at time t ₂₃ is finished, the transfer and read the accumulated charge Pitch pixel shifting reference waveform (e) is generated. Then, the lens displacement (i) in the pitch direction is also driven by a command value obtained by adding the value obtained by multiplying the reference waveform (e) by the first coefficient and the camera shake correction waveform.

[0139] The time t ₃₁ after a predetermined time from the time t ₂₁ the third timing pulse TP3 is generated. Then, similarly, the image sensor IMS accumulates charges in the light receiving unit during the period from time t ₃₂ to time t ₃₃ . The exposure time here is again set to the exposure time t _e2 suitable for the middle luminance group GP2. When the charge accumulation ends at time t ₃₃ , the pixel shift reference waveform (f) in the yaw direction is returned to the original value as the accumulated charge is transferred and read. Then, the lens displacement (j) in the yaw direction is driven by the command value that responds only to the shake correction waveform.

[0140] last timing pulse TP4 is generated from time t ₃₁ to time t ₄₁ after a predetermined time. Then, similarly, the image sensor IMS accumulates charges in the light receiving unit during the period from time t ₄₂ to time t ₄₃ . The exposure time here is shown in FIG.
Low brightness group GP3 calculated in step (186) of
The exposure time t _e3 suitable for is set. When the charge storage ends at time t ₄₃ , the pixel shift reference waveform (e) in the pitch direction is returned to the original value as well as the transfer / readout of the stored charge. Then, the lens displacement (i) in the pitch direction is also driven by the command value that responds only to the shake correction waveform. The switch SW1 at time t ₅ is once turned off, the driving of the hand vibration detection and the second lens group is stopped.

The positions of the images at the times t ₁₁ , t ₂₁ , t ₃₁ , t ₄₁ , and t ₅ during the above-described camera shake correction and pixel shift control are IM1, IM2, IM3, and IM3, IM3, and IM3, respectively.
IM4 and IM1 are realized, and the pixel shift in which the vertical and horizontal positions of each image are displaced by half the pixel interval is realized. Then, when the exposure is performed four times during the pixel shift, the exposure amount suitable for each subject brightness group is given. The pixel shift reference waveform is not a rectangular wave but a trapezoidal wave because the shock caused by a sudden change in the position of the second lens L2 is mitigated.

Next, the principle of combining a plurality of sets of image signals obtained by pixel shifting according to the present embodiment to generate one set of high-definition image signals will be described with reference to FIGS. 9 and 10. As described above, FIG. 9 is a diagram for explaining the relative positional relationship between the image and the image sensor in the pixel shift. FIG. 9A is the same as that described in FIG. 4, and the image is IM1 → IM2 → IM3 with respect to the pixels of the image sensor fixed in the camera.
→ IM4 → IM1 is moved in this order. This is completely equivalent to the position of the image pickup device moving in the order of IG1 → IG2 → IG3 → IG4 → IG1 with respect to the immovable image as shown in (b). In this embodiment, since the exposure time is changed in accordance with each step of pixel shifting, that is, the position of the image sensor, the exposure time can be changed as shown in FIG.
In (b), the size of the circle is shown.

Next, a method of synthesizing a plurality of images obtained by the above operation will be described. First, the output signal of each pixel when the image sensor is located at IG1 is IG1 (i, j). Here, the image sensor is an area sensor of (m × n) pixels. Similarly, the output signals when positioned at IG2, IG3, and IG4 are IG2 (i, j), IG3 (i, j), and IG4.
(I, j). FIG. 10 illustrates a method of synthesizing these four sets of image signals.

A new image signal set of (2m × 2n) pixels obtained by synthesizing signals of four sets of (m × n) pixels is IMG (u,
v). Then, the four pixels at the upper left of the image signal IMG (u, v) are four sets of original pixels assembled as shown in the figure. Therefore, considering an image restoration method based on this figure, IMG (u = 2i−1, v = 2j) ← IG1 (i, j) × K ₂ ... (16) IMG (u = 2i−1, v = 2j-1) ← IG2 (i, j) × K ₁ (17) IMG (u = 2i, v = 2j-1) ← IG3 (i, j) × K ₂ (18) IMG (U = 2i, v = 2j) ← IG4 (i, j) × K ₃ (19) According to the equation (19), one set of high-definition image signals can be generated from four sets of image signals. Here, K ₁ to K ₃ are coefficients for correcting a shift in the output level of each image signal set due to a difference in exposure time, and K ₁ = C / t _e1 (20) K ₂ = C / t _e2 represented by _{...... (21) K 3 = C} / t e3 ......... (22). Since C is a constant, K ₁ > K ₂ > K ₃
It becomes a big and small relationship.

FIG. 23 is for explaining this action. As in FIG. 20, the horizontal axis is the subject brightness and the vertical axis is the output voltage. Here, the image signal group IG4 with the longest exposure time
The correction coefficient K ₃ of 1 is set to 1, and IG1 and IG3 are K ₂ times, I
G2 is synthesized _1x K. Further, the effective range of the original image signal output is set to a range between V _min and V _max , and the image signal set IG
In the upper limit region of 4, the upper and lower limit regions of the image signal sets IG1 and IG3, and the lower limit region of the image signal set IG2, output signals outside the valid range are not used, and interpolation using surrounding pixels is performed after image synthesis by pixel shifting. An image signal is obtained by calculation.

That is, a pixel set having a short exposure time raises the amplification factor of the output voltage at the time of reproducing an image signal, and conversely, a pixel set having a long exposure time lowers the amplification factor of the output voltage so that the combined high voltage is obtained. The reproduction level of each pixel of a fine image is normalized. Then, in a region where one pixel group is saturated but the other pixel group is not saturated, a saturated image signal is generated by interpolation from a non-saturated image signal. By the above operation, the subject image in the luminance range from H3 to H4 can be faithfully reproduced.

In the present embodiment, of the four exposures in the pixel shift operation, two exposures have the same exposure time according to the brightness level of the main subject. This is because the number of exposures suitable for the main subject is increased so that the image information of the main subject can be obtained as much as possible. Further, in the present embodiment, the signal normalization at the time of image synthesis is controlled by the expansion and contraction of the exposure time. . Note that the pixel shift and image combining method according to the present embodiment is applied to a multi-plate color image sensor using a monochrome image sensor or a color separation prism. Although there are some differences in the pixel shift amount and the image synthesizing method, the basic idea is the same.

According to the fifth embodiment, (1) the exposure level of each image at the time of pixel shift execution is adjusted from the luminance distribution information of the object measured by the multi-division sensor,
By synthesizing these, a high-definition and wide-dynamic-range image can be obtained. (2) Since the main subject analogy estimating means is provided and the exposure level is adjusted based on the detection result, the image reproducibility of the main subject is improved. (3) Since the number of exposures suitable for the brightness level of the main subject is increased among the multiple exposures associated with pixel shifting, the image reproducibility of the main subject is further improved. There is an effect.

(Sixth Embodiment) The fifth embodiment is an embodiment in which the number of pixel shifts is fixed and the exposure level is adjusted according to the brightness condition of the subject. The sixth embodiment described below shows an embodiment in which an optimum pixel shift mode is selected according to the shooting state of the camera and an exposure level adjustment suitable for the selected pixel shift mode is performed. FIG. 24 is a part of the control flow of the sixth embodiment, which is shown in FIG.
2 is a principle diagram of a second pixel shift mode, and FIG. 13 is an explanatory diagram of a principle of image combination in the second pixel shift mode.
The operation of the sixth embodiment will be described below with reference to the drawings.

The control flow of the camera of the sixth embodiment is basically the same as the control flow of the camera of the fifth embodiment shown in FIG. 18, but step (11)
By making the "exposure / pixel shift condition setting" part of 5) a subroutine shown in FIG. 24, the operation peculiar to the sixth embodiment is realized. The flow of the sixth embodiment will be described below with reference to FIGS. 18 and 24. The flow of FIG. 18 has been described in detail in the fifth embodiment, and therefore will be briefly described here. To do.

In FIG. 18, in step (102), S
When it is determined that W1 is on, the process proceeds to step (111),
In steps (111) and (112), an IS start command is transmitted and parameter communication is performed. Subsequently, photometry and focus detection are performed in steps (113) and (114), and the results are also transmitted to the in-lens microcomputer LCPU. In the next step (115), the flow shown in FIG. 24 is executed. In step (261) of FIG. 24, the object projected on the photometric sensor SNS is divided into a plurality of blocks, as in step (181) of FIG. 19 of the fifth embodiment. Then, in step (262), the process shown in FIG.
Similar to step (182) of 9, the main subject is analogized from the plurality of blocks. In step (263),
A request for transmitting the shake angular velocity peak value ω _peak within a predetermined time is issued to the lens microcomputer LCPU. Then, the in-lens microcomputer LCPU transmits the camera shake angular velocity maximum value ω _peak in the pitch or yaw direction for 2 seconds to the in-camera microcomputer CCPU.

At step (264), the maximum value δ of the image blur amount during exposure is calculated according to the following equation: δ = f × ω _peak × t _exp (23) Where f is the focal length of the imaging optical system, t
_exp is the exposure time of the main subject derived from the photometric results, that is, the charge accumulation time. Image blur amount δ derived here
Is used in the following steps as a camera shake index for determining whether or not pixel shifting should be executed. In step (265), the magnitude of the image blur amount δ is determined. If the image shake amount δ is less than or equal to the predetermined value DEL1, it is considered that the influence of the image deterioration due to the camera shake is small and the pixel shift can achieve sufficiently high definition, and the pixel shift count N _SF is set to 4 times in step (271). To do. Here, the number of pixel shifts being 4 means the pixel shift and image combining method described in the fifth embodiment, and this is referred to as a first pixel shift mode. In the first pixel shift mode,
The step (272) and the subsequent steps are executed.

Steps (272) to (27)
In step 5), the same operations as those in steps (183) to (186) in FIG. 19 are executed to determine the exposure times t _e1 to t _{en = 3} for the four pixel shift operations. After execution of step (275), the process proceeds to step (266). If it is determined in step (265) that the image blur amount δ is less than or equal to the predetermined value DEL2 (where DEL2> DEL1), step (2)
In 81), the pixel shift number N _SF is set to 2 times. Even if image stabilization is performed, some image quality deterioration due to camera shake is expected. Therefore, select a mode with a small number of pixel shifts (this is called the second pixel shift mode) to reduce image quality degradation due to camera shake and pixel loss. This is to optimize the image quality improvement by shifting. In the second pixel shift mode, step (282) and subsequent steps are executed.

At step (282), a sub-subject, which is the second most important after the main subject, is estimated. For example, step (261)
Among the blocks divided by, the block having the largest area is assumed to be the sub-subject. In step (283), the brightness of the main subject determined in step (262) and the brightness of the sub-subject determined in step (282) are calculated. In step (284), the above step (283)
The exposure amounts E _main and E _sub suitable for the brightness of the main / sub-subject are calculated from the brightness calculated in step 1. Step (285)
Then, from the exposure amount calculated in the above step (284), the exposure times t _emain and t suitable for the brightness of the main and sub-subjects are _obtained .
_esub is calculated. After executing step (285), the process proceeds to step (266). The second pixel shift mode will be described later. When it is determined in step (265) that the image shake amount δ is larger than the predetermined value DEL2, step (29
In 1), the pixel shift count N _SF is set once, that is, the pixel shift is not executed. Even if the camera shake correction is executed, the pixel shift is prohibited because the influence of the image quality deterioration due to the camera shake is expected to exceed the image quality improvement due to the pixel shift.

In step (293), step (26
The brightness of the main subject determined in 2) is calculated. In step (294), an exposure amount E _main suitable for the brightness of the main subject is calculated from the brightness calculated in step (293). In step (295), the above step (29
From the exposure amount calculated in 4), the exposure time _temain suitable for the brightness of the main subject is calculated. After execution of step (295), the process proceeds to step (266). Step (27
5), step (285) or step (295) is executed, the process proceeds to step (266). In step (266), the pixel shift mode determined in the above flow is transmitted to the in-lens microcomputer LCPU. Step (2
In 67), the type of the pixel shift mode is displayed on the display means DISP in the camera to inform the photographer of which mode the photographing is performed. After execution of step (267), the process returns to step (116) in FIG.

At step (116), the switch SW2 is discriminated. If SW2 is on, the process proceeds to step (117). In steps (117) to (122), pixel shifting is performed as in the fifth embodiment, but in the sixth embodiment, pixel shifting is performed in steps (271) to (295) as described above. Since the mode is selectively used, the operation according to each pixel shift mode is also controlled in steps (117) to (122). Subsequently, in step (123), pixel shift completion communication is performed to the microcomputer LCPU in the lens, and step (1
Proceed to 24). Also in step (124), image composition according to the plurality of pixel shift modes is selected and performed. In step (125), the image combined in the previous step is recorded, and the process returns to step (102).

The operation in the second pixel shift mode, that is, when the number of pixel shifts is two will be described with reference to FIG. Note that this FIG. 12 corresponds to FIG. 9 in the first and fifth embodiments. FIG. 12A shows the movement of the image with respect to the light receiving element. First, a certain point on the image is located in the IM 21, and charges are accumulated / read out from the light receiving element under an exposure condition suitable for the brightness of the main subject. Then, after the image is moved to the IM 22 by the pixel shifting operation, the charge accumulation / readout of the light receiving element is performed again under the exposure condition suitable for the brightness of the subject. And the image is the original position IM
Returned to 21. In FIG. 12B, the movement of the image is replaced with the movement of the image pickup device, and the image pickup device initially located in the IG 21 is exposed for the exposure time t _main . Then, the pixel shift operation is performed to move to IG22, and the exposure time t
_After being exposed by esub, it returns to IG21.

FIG. 13 is a diagram for explaining an image synthesizing method in the second pixel shift mode. In the second pixel shift mode, 2 of IG21 (i, j) and IG22 (i, j)
IMG2 (u, v) composed of (4 × m × n) images from a set of images, that is, a total of (2 × m × n) image signals.
The image signals of are combined. Therefore, as shown in FIG. 13, the pixels indicated by black circles in IMG2 (u, v) are IG21.
The pixel of (i, j) or IG2 (i, j) is used as it is, and the pixel indicated by the white circle is synthesized from the average value of the four pixels (2 or 3 pixels when there are no 4 pixels) around it.

This is expressed by an equation: IMG2 (u = 2i-1, v = 2j) ← IG21 (i, j) × K _main ..... (24) IMG2 (u = 2i, v = 2j-1) ← IG22 (i, j) × K _sub ... (25) IMG2 (u = 2i-1, v = 2j-1) ← {IG21 (i, j) × K _main + IG22 (i, j) × K _sub + IG21 (i, j−1) × K _main + IG22 (i−1, j) × K _sub } / 4 (26) IMG2 (u = 2i, v = 2j) ← {IG21 (i, j) × K _main + IG22 (i, j) × K _sub + IG21 (i + 1, j) × K _main + IG22 (i, j + 1) × K _sub } / 4 (27) Here, K _main and K _sub are coefficients for correcting the deviation of the output level of each image signal set due to the difference of the exposure time, as in the first embodiment, and K _main = C / t _emain ... (28) K _sub = C / _{te sub} ... (29)

According to the sixth embodiment, the fifth
In addition to the effects of the embodiment of (4), the optimum pixel shift mode is selected according to the amount of camera shake, and the exposure level adjustment suitable for the selected pixel shift mode is performed. It is possible to obtain an image with well-balanced dynamic range. There is an effect.

(Seventh Embodiment) The sixth embodiment has shown the embodiment in which the optimum pixel shift and the exposure adjustment mode are selected according to the camera shake. A seventh embodiment shown below is an embodiment in which it is determined whether or not the pixel shift is to be performed based on the photographing mode of the camera set by the photographer, and the pixel shift mode and the exposure adjustment mode are switched. Similar to the sixth embodiment, the camera control flow of the seventh embodiment is basically the same as the camera control flow shown in FIG. 18, and the "exposure / exposure" of step (115) is performed.
By setting the "pixel shift condition setting" portion as a subroutine shown in FIG. 25, the operation peculiar to the seventh embodiment is realized. Hereinafter, the flow of the seventh embodiment will be described with reference to FIGS. 18 and 25. The flow of FIG. 18 has been described in detail in the fifth embodiment, and therefore will be briefly described here. To do.

In FIG. 18, in step (102), S
When it is determined that W1 is on, the process proceeds to step (111),
In steps (111) and (112), an IS start command is transmitted and parameter communication is performed. Subsequently, photometry and focus detection are performed in steps (113) and (114), and the results are also transmitted to the in-lens microcomputer LCPU. In the next step (115), the flow shown in FIG. 25 is executed. In the step (361) in the figure, as in the step (181) in FIG. 19, the subject projected on the photometric sensor SNS is divided into a plurality of blocks. Then, in step (362), step (182) in FIG.
Similarly, the main subject is analogized from the plurality of blocks. In step (363), the state of the photographing mode selection switch SWMOD of the camera CMR is determined to recognize the photographing conditions such as the exposure control mode set by the photographer.

At step (364), it is determined whether the photographing mode is the landscape mode. The landscape mode is an exposure control mode in which the aperture control value is set to a small aperture (large F number) to deepen the depth of field. When the landscape mode is set, the subject is stationary and the camera is expected to be firmly held so that hand shake is unlikely to occur, so the process proceeds to step (371) and the high-definition mode is set in which pixel shifting is performed four times. If it is determined in step (364) that the mode is not the landscape mode, the process proceeds to step (365). Step (3
In 65), it is determined whether or not the shooting mode is the portrait mode. The portrait mode is an exposure control mode in which the aperture control value is set to the aperture close to the open aperture (small F number) to reduce the depth of field. Since it is expected that the photographing mode is the same as that in the landscape mode in the portrait mode, the process proceeds to step (371). If it is determined at step (365) that the portrait mode is not set, then the processing advances to step (366).

At step (366), it is determined whether or not the photographing mode is the sports mode. Sports mode is an exposure control mode that stops the movement of the subject by setting the exposure time to high speed. When the sport mode is selected, the subject is moving, and it is expected that the camera will undergo a panning operation, that is, a camera movement similar to a large camera shake. In addition, the movement of the subject during the pixel shift operation may cause subject shake, which may not be expected to provide high detail due to the pixel shift, and may even result in an unnatural photograph. Therefore, when setting the sports mode, step (39
Proceed to 1) and N _SF = 1 is set, that is, pixel shifting is prohibited.
If it is determined in step (366) that the mode is not the sports mode, that is, if it is determined that the mode is neither landscape, portrait, or sports, the process proceeds to step (381), and the pixel shift count is set to 2.

Steps (371) to (37)
5), step (381) to step (385),
Steps (391) to (395) are shown in FIG.
The same operation as the step (271) to step (275), step (281) to step (285), and step (291) to step (295) of the sixth embodiment shown in FIG. That is, although the exposure adjustment amount is determined according to each pixel shift mode, the detailed operation has been described in the sixth embodiment, and therefore will be omitted. After executing step (375), step (385) or step (395), the process proceeds to step (367). In step (367), the microcomputer LC in the lens
The pixel shift mode determined in the above flow is transmitted to PU. In step (368), the display means DIS in the camera
The type of the pixel shift mode is displayed on P to notify the photographer of which mode the photographing is performed. After execution of step (368), the process returns to step (116) in FIG.

At step (116), the switch SW2 is discriminated. If SW2 is on, the process proceeds to step (117). In steps (117) to (122), pixel shift is executed according to the selected pixel shift mode as in the sixth embodiment. Then step (1
In 23), the pixel shift completion communication is performed to the microcomputer LCPU in the lens, and the process proceeds to step (124). Step (1
Also in 24), image composition according to the plurality of pixel shift modes is selected and performed. In step (125), the image combined in the previous step is recorded, and the process returns to step (102).

According to the seventh embodiment, the fifth
In addition to the effects of the embodiment of (5), whether or not the pixel shift execution is appropriate is determined based on the shooting mode of the camera set by the photographer, and the pixel shift mode is switched to select a pixel suitable for the movement of the camera and the subject. The shift operation can be executed. As a result, it is possible to obtain an image in which the definition and the dynamic range are well balanced according to the shooting situation. There is an effect. Further, in the present embodiment, the exposure control mode is targeted as an example of the shooting mode, but the pixel shift mode may be switched according to the focus detection mode or the like.

(Other Modifications) In the fifth to seventh embodiments, the lens group in the imaging optical system is perpendicular to the optical axis as the image moving means for camera shake correction and pixel shift. Direction, and the optical axis polarization effect by this is used, but instead of this, a part of the imaging optical system
It is also possible to use a so-called variable apex angle prism which is composed of a sheet of transparent parallel plates and a transparent liquid sealed therein. In addition, even in a mode without a camera shake correction unit,
The effects of the present invention can be exhibited equally. Further, although the exposure time is variable as a method of adjusting the exposure amount between a plurality of images when the pixels are shifted, the aperture value may be variable. Alternatively, the exposure amount may be adjusted by arranging a transmitted light amount adjusting means such as an EC (electro-chromic) element or a liquid crystal element in the photographing optical system and making the density variable.
Further, in the present embodiment, the effect of the present invention can be exhibited even in a mode in which pixel shift is not performed when a plurality of image sets having different exposure levels are obtained. In this case, the effect of increasing the detail of the image disappears, but the effect of obtaining a wide dynamic range image according to the state of the subject can be exhibited.

(Eighth Embodiment) The control flow of the microcomputer microcomputer CCPU in the eighth embodiment is shown in FIG.
The procedure is substantially the same as that of step 8, but the processing content of step (124) is different. That is, in step (124), it is a subroutine for determining image blur from a plurality of image signals obtained by pixel shifting, and synthesizing images by a predetermined method based on the determination result to create one high-definition image.
Detailed operation contents will be described with reference to FIG.

The control flow of the in-lens microcomputer LCPU is substantially the same as that shown in FIG.

The operation of the camera and the lens according to the control flow of FIGS. 18 and 7 is performed substantially in the same manner as described in the first embodiment along the timing chart shown in FIG.

Next, a method for determining image blur due to camera shake from a plurality of images obtained by pixel shifting will be described with reference to FIGS. 26 to 32. 26 and 27 are 2
The principle of calculating the relative positional deviation amount of both images from the degree of correlation of a pair of one-dimensional image signals will be described. In both figures, the horizontal axis represents the pixel coordinates of the image sensor, and the vertical axis represents the output signal value of each pixel.

In FIG. 26A, IM1 is the output of the first image signal set acquired at a certain time, IM2 is the output of the second image signal set acquired a predetermined time after that, and both images are pixel-shifted. It is assumed that the position is displaced by an integral multiple of the pixel interval due to an image blur caused by an operation or a camera shake. Figure 2
6 (b) is the second image signal IM2 shifted leftward by two pixels, and the two image signals completely match.

Here, the correlation C of the two images (Correlation)
Let R be CR = 1-Σ {ABS (IM2 (i) -IM1 (i)) / (IM1 (i) + IM2 (i))} ... (30) where ABS is an absolute value and i is a pixel number. , Σ is defined as a sum operator from i = 1 to a predetermined pixel number, the correlation CR becomes the maximum value 1 when the pixel shift amount ST is −2 in the example of FIG. This state is represented as ST _max = -2 and CR _max = 1.

FIG. 27 (a) is an explanatory diagram when the two sets of image intervals are different from the integer multiple of the pixel interval (1.5 pixels in the figure). In FIG. 27B, the second image signal IM2 is shifted to the left by one pixel, and the correlation CR becomes 0.8. Then, even if the pixel is further shifted by one pixel, the correlation C
R cannot be 1.

FIG. 28 shows the relationship between the pixel shift amount ST and the correlation CR at this time. According to FIG. 28,
When the correlation degree CR is calculated while changing the pixel shift amount ST from +2 to -3, the pixel shift amount ST becomes -1 and -2.
Then, the correlation CR becomes the maximum value of 0.8. Correlation CR
Is linearly interpolated, the pixel shift amount ST is −1.5.
It can be seen that the correlation CR has a true maximum value of 0.9 when. That is, ST _max = -1.5, CR _max = 0.9.
Becomes

The above description is a method of calculating the image shift amount in two sets of one-dimensional images.
The image shift amount in the vertical and horizontal directions in the two-dimensional image of the set can be calculated. Specifically, as disclosed in Japanese Patent Laid-Open No. 64-10787, a two-dimensional image signal is projectively compressed into one-dimensional signals in vertical and horizontal directions, and the motion vector of the image, that is, vertical The amount of image shift in the left-right direction can be detected. Alternatively, it is also possible to directly calculate the two-dimensional motion vector by performing a correlation operation while sequentially shifting the two-dimensional image in two directions.

FIG. 29 is a diagram showing the image movement situation when the accurate pixel shift control is executed without being affected by the camera shake, and shows the relative positional relationship between the image and the image pickup element described in FIG. ing. In FIG. 29, acquisition of the first image signal (charge accumulation and readout of the image sensor) is performed in a state where a point with an image is located at a point IM1 on the lower left pixel. Subsequently, the pixel position is shifted by 0.5 pixel intervals in the horizontal direction (corresponding to the yaw direction for camera shake correction) to move the image position to IM2, and the second image signal is acquired.

FIG. 30 is a diagram showing the correlation CR of the first and second image signals obtained in FIG. The horizontal axis represents the relative shift amount of the pixel when performing the correlation calculation of two sets of image signals, where the horizontal shift amount is STH and the vertical shift amount is STV. The vertical axis represents the degree of correlation CR. The CRH in the figure is a set of two sets of two-dimensional images that are relatively ST horizontal.
The degree of correlation calculated by shifting only H pixels (± 3 pixels)
CRV also indicates the degree of correlation calculated by shifting STV pixels (± 3 pixels) in the vertical direction.

Here, the pixel shift of FIG. 29 is only in the horizontal direction.
By 0.5 pixel in the vertical direction (pitch direction for camera shake correction)
Direction, the correlation CR becomes maximum.
The pixel shift amount (this is called the maximum correlation shift amount) is S
TH_max= -0.5, STV _max= 0. Again this
Maximum correlation value CRH_max, CRV_maxIs slightly below 1.
It is a value that turns. This is because the first and second images have pixel spacing
Since it was acquired with a deviation amount different from an integer multiple of
This is because the signals do not match perfectly. And this
The phenomenon also applies to the vertical direction where pixel shifting is not performed.
CRV to fit_maxIs not 1.

FIG. 31 shows an image movement situation when a large camera shake occurs during the pixel shift control and the image shake occurs even when the camera shake correction is performed. In FIG. 31, the first image signal is acquired in a state where a certain point of the image is located at the point IM1 on the lower left pixel. Subsequently, pixel shifts of 0.5 pixel intervals were performed horizontally as indicated by the broken line arrow, but when the image shake due to camera shake was added and the image position moved to IM2, the second image signal was acquired. .

FIG. 32 is a diagram showing the degree of correlation CR of the first and second image signals acquired in FIG. 31, and each symbol in the figure has the same meaning as in FIG. According to FIG. 32, the maximum correlation shift amount is STH _max = -0.7, STV _max =
It becomes -0.3. So the difference between the maximum correlation shift amount when camera shake is not in FIG. 30 (sum computational formula) [delta] _H,
When _{δ V, δ H = -0.7 +} (- 0.5) = 0.2 ......... (31) δ V = -0.3 + 0 = 0.3 ......... (32) , and the above-mentioned [delta] _H , Δ _V are image shake amounts (units are pixel intervals) due to camera shake in the horizontal and vertical directions during pixel shifting. If the image shake amounts δ _H and δ _V exceed a predetermined value, for example, δ _MAX = ± 0.1 pixel interval, the effect of high definition due to pixel shift is canceled, and at that time, the pixel shift mode is changed. To deal with.

Here, in the present embodiment, since the camera shake correction mechanism is operating, ideally image shake due to camera shake should not occur. In reality, the output error of the camera shake detection sensor and the camera shake correction capability are not exceeded. Image shake may occur due to occurrence of large shake. Image blurring due to the above causes uniformly occurs over the entire surface of the photographing screen.

Next, the principle of synthesizing a plurality of sets of image signals obtained by pixel shifting to generate one high-definition image signal will be described with reference to FIGS. 9, 10 and equation (7) in the first embodiment. ~ Based on the same method as described with (10). Note that this method is applied to a multi-plate color image sensor using a monochrome image sensor or a color separation prism, and in a single plate color image sensor using a mosaic type color filter, it is somewhat different in terms of pixel shift amount and image composition method. There are differences, but the basic idea is the same.

FIG. 33 is a flow chart showing the functions of the shake determination and the image composition during the pixel shift described in FIGS. 26 to 32 and FIGS. 9 and 10. The step (12) in FIG.
This corresponds to the image composition subroutine 4). First, in step (161), the pixel shift counter CNT is initialized to zero. In step (162), the above equation (3
0), the degree of correlation between the first and second image signal sets is calculated. In step (163), from the correlation calculation result, the maximum correlation shift amount STH _{ma x,} calculates the STV _max. In step (164), the maximum correlation value CR
Calculate H _max , CRV _max . In step (165), the image shake amounts δ _H and δ _V are calculated. Step (16
In 6), 1 is added to the counter CNT to update it.

At step (167), the counter CNT
And a predetermined value N representing the number of pixel shifts _{science fiction}Compare with.
For example N_{science fiction}= 4, step (162) or step
(165), the first and second image signal pairs,
Between the second and third image signal pairs, and between the third and fourth image signal pairs
It will be done three times in total. Therefore, CNT is N_{science fiction}Reached -1
If not, return to step (162) and send the next image
Step (162) through step (16)
The calculation of 6) is repeatedly executed. CNT is N_{science fiction}Reached -1
Then, it progresses to step (168).

In step (168), the above steps are performed.
Predetermined for all image shake amounts calculated in (165)
Value δ_maxCompare with. And all the image shake amount
Fixed value δ _maxHigher definition by shifting pixels if smaller
Since it can be expected, proceed to step (169) and see FIG.
Image synthesis is performed by the method shown in. Step (17
0) shows that the normal pixel shift operation is executed.
Display on the display means DISP. And step (171)
Then, the process returns to the main flow of FIG. On the other hand
Up (168), the predetermined value δ_maxThe above image shake
If the existence is confirmed, jump to step (172)
However, the fact that image composition by pixel shifting will not be performed is notified.
The warning display is also displayed on the display means DISP. That
In step (171), the main flow shown in FIG.
Make a turn. At this time, the original image that is not a composite image, for example,
The first image signal set becomes the final image signal.

According to the eighth embodiment, (1) the image blur due to camera shake is extracted from the correlation calculation result of a plurality of image signals obtained by the pixel shift operation, and the pixel shift is performed according to the magnitude of the image blur. Since it is selected whether or not to execute image composition by, it is possible to suppress the influence of image deterioration due to camera shake and obtain a high-definition image signal by pixel shifting. (2) Since the photographer is informed of the result of whether or not the pixel shift can be performed using the display means, the photographer can confirm the definition of the obtained image and, if the pixel shift is not performed due to camera shake, the photographing is redone. It is possible to take measures such as. (3) Since the image shake calculation is performed every time the image is moved due to the pixel shift, it is possible to detect the image shake with high accuracy, and it is possible to reliably eliminate the failure of the pixel shift operation due to the camera shake. There is an effect.

(Ninth Embodiment) In the eighth embodiment, image blur detection is performed every time the image is moved by pixel shifting.
In the embodiment, the pixel shift is prohibited if a large image blur occurs even once. The ninth embodiment shown below is
An embodiment is shown in which the image is returned to the initial position after the pixel shift is executed, a preliminary image signal for image shake determination is acquired, and image shake detection is performed using this.

FIG. 34 is an image movement locus during the pixel shifting operation of the ninth embodiment, and FIGS. 35 and 36 are control flows of the camera. The operation of the ninth embodiment will be described below with reference to the drawings. In FIG. 34, the image is moved with respect to the image sensor according to the trajectory of the broken line, and five sets of image signals are acquired. However, the actual image is IM1 → IM2 due to camera shake.
→ IM3 → IM4 → IM5. While four sets of images IM1 to IM4 are used for high definition by shifting pixels, two sets of images IM1 and IM5 are used for correlation calculation for image blur detection.
That is, if there is no camera shake, the two images of IM1 and IM5 completely match, STH _max = STV _max = 0, and δ _H = δ.
Whereas _V = 0 and CRH _max = CRV _max = 1
When camera shake occurs, the image shake amount integrated value during the entire process of the pixel shift operation appears as δ _H and δ _V.

FIG. 35 is a main flow of the camera. This flow is basically the same as the control flow of the camera of the eighth embodiment shown in FIG. 18, but in step (122) of FIG. 18, the pixel shift execution count is determined and the conditional branch is performed. The part to do is different. That is, in the eighth embodiment of FIG. 18, after acquisition of four sets of image signals by pixel shifting is completed, the process branches to step (123). On the other hand, FIG.
In the fifth embodiment, the image position is returned to the initial position after the acquisition of four sets of image signals, and the fifth set of image signals is acquired for image shake detection. Therefore, in step (122), after the counter CNT for measuring the number of pixel shifts becomes N _SF + 1 = 5, the process branches to step (123). Step (12
3) After execution, in step (124), image composition is performed by the subroutine shown in FIG. Since the steps other than the above are the same as the operation of the eighth embodiment, the description thereof will be omitted.

FIG. 36 is a flow chart showing the image composing subroutine of step (124) in the main flow of FIG. 35. As in the case of FIG. 33 of the eighth embodiment, there is a shake determination during pixel shift. Execute image composition. First, in step (461), the degree of correlation between the first and fifth image signal sets is calculated according to the above-mentioned equation (30). Step (4
In 62), from the function calculation result, the maximum correlation shift amount STH _{ma x,} calculates the STV _max. Step (46
In 3), the maximum correlation values CRH _max and CRV _max are calculated. In step (464), the image shake amounts δ _H and δ _V
However, in this embodiment, the maximum correlation shift amounts STH _max and STV calculated in step (462) are calculated.
_max becomes the image blur amounts δ _H and δ _{V as} they are.

At step (465), the image shake amount calculated at step (464) is compared with the _first predetermined value δ ₁ . The image shake amount is the first predetermined value δ.
If it is smaller than ₁ , image shake due to camera shake is extremely small and high definition due to pixel shift can be expected. Therefore, the process proceeds to step (466), and four sets of images are combined by the method shown in FIG. In step (467), the fact that the pixel shift by the four-image synthesis has been executed is displayed on the display unit DIS.
Display on P. Then proceed to step (472) in FIG.
Return to the main flow of 5.

On the other hand, if it is determined in step (465) that the image shake amount is equal to or larger than the first predetermined value δ ₁ , the process proceeds to step (468). In step (468), the image shake amount is compared with the _second predetermined value δ ₂ which is larger than the first predetermined value δ ₁ . If the image shake amount is smaller than the second predetermined value δ ₂ , the image shake due to the camera shake is slightly large, but the high definition due to the pixel shift can be expected to some extent. Therefore, the process proceeds to step (469) and the two sets of images are processed. Perform synthesis. This means that the increase in the number of pixel shifts contributes to higher definition of the image, but the image acquisition time increases and the image quality is easily affected by camera shake, and in some cases, it is lower than the original image without pixel shift. In some cases, the image quality will end up. Therefore, in the present embodiment, when a slight image blur occurs,
The image of the entire group is not used, and the influence of camera shake is eliminated at the expense of higher definition of the image. Specifically, the first and second image signal sets are combined to improve the resolution only in the horizontal direction, or the first and third image signal sets are combined,
There is a method of generating a blank portion where pixels are not filled by interpolating surrounding image signals.

After the predetermined image combination is performed in step (469), the process proceeds to step (470), and it is displayed on the display means DISP that the pixel shift by the two image combination is executed. Then, the process proceeds to step (472) to return to the main flow of FIG. On the other hand, if it is confirmed in the step (468) that the image shake is equal to or larger than the _second predetermined value δ ₂ , the image shake is considerably large, and the image quality may be deteriorated by the pixel shift.
A jump is made to 1), and a warning display for notifying that image composition by pixel shifting is not performed is also displayed on the display means DISP. Then, in step (472), the process returns to the main flow of FIG. At this time, the original image that is not the composite image, for example, the first image signal set becomes the final image signal.

According to the ninth embodiment, the eighth
In addition to the effects (1) and (2) of the embodiment, (4) the image shake is detected from the two sets of images during the entire time of the pixel shift operation, so that the image shake determination time can be shortened. (5) Since the correlation calculation is performed between the first image for recording and the image returned to the original position after the pixel shift is completed, the relative shift amount of the image obtained from the correlation calculation result is equal to the image shake amount due to camera shake. Therefore, the image blur detection calculation becomes simple. (6) Since the optimum pixel shift mode is selected from a plurality of pixel shift modes according to the amount of image blur, failure of the pixel shift operation due to camera shake can be reliably eliminated. (7) Since the selected pixel shift mode is displayed, the photographer can know to what extent a high-definition image has been obtained. There is an effect.

(Tenth Embodiment) The eighth and ninth embodiments
The above embodiment is an embodiment in which the image blur is calculated from the correlation calculation of the plural images for pixel shift, and the pixel shift is prohibited when the image blur is equal to or more than a predetermined value. A tenth embodiment shown below shows an embodiment in which even if the image blur is equal to or larger than a predetermined value, a pixel shift can be performed by applying an image blur correction measure.

FIG. 37 is an image movement locus at the time of pixel shifting operation of the tenth embodiment, FIG. 38 is an explanatory diagram of a correlation calculation result, and FIG. 39 is a control flow of the camera. The operation of the tenth embodiment will be described below with reference to the drawings. In FIG. 37, the image is horizontally moved along the trajectory of the broken line by 0.5 pixel intervals with respect to the image sensor. However, due to camera shake, the actual image looks like a solid arrow, IM1 → IM2.
Shall move like.

FIG. 38 shows the two sets of images IM1 acquired in FIG.
And IM2 are correlation calculation results. From this FIG. 38, ST
H _max = -1.5, STV _max = 0, δ _H = 1, δ _V =
It can be seen that 0, CRH _max = CRV _max = 0.94. Here, the image blur δ _V in the vertical direction is 0, but there is no problem, but the image blur δ _{H in} the horizontal direction is one pixel interval. Therefore, if the image synthesis is performed as it is, the resolution is higher than that of the original image without pixel shift. Low image quality. However, if the amount of image blur between the two sets of images is approximately equal to an integer multiple of the pixel interval, then by combining one image by shifting the image in the opposite direction from the image blur, this image blur will occur. Can be canceled. For example, in FIG. 37, the second image I
M2 is shifted from the first IM1 by 1.5 pixel intervals in the horizontal direction due to pixel shift and camera shake, but the second image is shifted by one pixel interval in advance when the images are combined after the pixel shift. By synthesizing from, the effect of camera shake is eliminated,
Only the shift of 0.5 pixel interval corresponding to the pixel shift remains.

Specifically, first, n _H = INT (δ _H ) ... (33) n _V = INT (δ _V ) ... (34) However, INT () is rounded off the value in (). Image blur amount δ _H , δ _V
Is converted into integers n _H and n _V. Then, the above equation (8)
When the image signals i and j of the image signal set IG2 (i, j) in FIG. 2 are updated according to i ← i + n _H ... (35) j ← j + n _V. The relative position moves and the image blur can be canceled.

The above operation is performed by the image signal sets IM3 and IM4.
This is also performed for (calculated IG3 and IG4). In this case, it is better to perform the correlation calculation for the image blur calculation based on the first image signal set IM1. After performing the above operations, if image synthesis is performed according to equations (7) to (10), even if image blurring of an image interval or more occurs between image signal groups, high definition of an image can be realized by pixel shifting. it can.

The main flow of the camera according to the tenth embodiment is the same as the main flow of the eighth embodiment shown in FIG. 18, and only the image composing subroutine of step (124) is different. Shows a subroutine of the tenth embodiment. First, in step (561), the pixel shift counter CNT is initialized to zero.
In step (562), the degree of correlation between the first and second image signal sets is calculated according to the above equation (30). In step (563), from the correlation calculation result, the maximum correlation shift amount STH _{ma x,} calculates the STV _max. Step (5
In 64), the maximum correlation values CRH _max and CRV _max are calculated. In step (565), the image shake amounts δ _H and δ _V
To calculate. In step (566), the image blur amount δ _H ,
Convert δ _V into an integer.

At step (567), the size of the image blur amount is judged. If the image blur amount is smaller than a predetermined value, for example, 5 pixel intervals, the process proceeds to step (568). On the other hand, if the amount of image blur is equal to or greater than the interval of 5 pixels, there is a high possibility that image blur will occur even during the accumulation of one set of images.
The display jumps to 75) and a warning display showing that the image blur is large is displayed on the display unit DISP. Then step (5
Since the process returns to the main routine in 76), the images are not combined by shifting the pixels, and the original image is recorded in the recording device in the next step (125) of the main routine.

From step (567) to step (56
Proceeding to 8), the correlation degree maximum CRH _{ma x,} performs a magnitude determination of the CRV _max, correlation magnitude maximum value is a predetermined value, for example zero.
If larger than 5, proceed to step (569). on the other hand,
When the maximum value of the degree of correlation is low, it means that the reliability of the image blur amount obtained from the correlation calculation is low. Therefore, if image blur correction is performed based on an erroneous image blur result and image synthesis is performed, the image quality may be deteriorated. Therefore, when the maximum correlation value is 0.5 or less, the process jumps to step (575), the warning is displayed in step (575), and the process returns in step (576) as in the case described above. If the image blur amount is less than or equal to the predetermined value and the maximum correlation value is greater than or equal to the predetermined value, the process proceeds to step (567), step (568) and step (569).

In step (569), the above equation (3
5) Image blur correction is performed according to (36). Step (5
In 70), 1 is added to the counter CNT to update it. Su
In step (571), counter CNT and pixel shift times
Predetermined value N that represents a number _{science fiction}Compare with. For example N_{science fiction}= 4
If, then step (562) through step (569)
Is calculated as the first and second image signal sets, the first and third image signals.
Perform a total of 3 times between the signal set and the first and fourth image signal sets.
Becomes Therefore, CNT is N_{science fiction}If it has not reached -1
Return to step (562) to step (562) through
The operation of step (569) is repeatedly executed. CNT
Is N_{science fiction}When it reaches -1, it proceeds to step (572). Su
In step (572), the image is created by the method shown in FIG.
Perform synthesis. In step (573), image blur correction is performed.
After that, a display showing that the image composition with pixel shifting was performed
To do. Then, in step (574), the main flow of FIG.
To return.

According to the tenth embodiment, in addition to the effects (1) and (2) of the eighth embodiment, (8) the image blur between a plurality of images is detected and corrected to shift the pixels. Since the image composition is performed, even when a large image blurring of one pixel interval or more occurs, it is possible to improve the definition of the image by shifting the pixels. (9) Since it is determined from the magnitude of the degree of correlation (coincidence degree) between a plurality of images whether or not image composition by pixel shifting can be performed, it is possible to prevent deterioration of image quality due to failure of pixel shifting. There is an effect.

(Other Modifications) The present invention can also be used for an image pickup apparatus which acquires a plurality of image signals within a predetermined time for a purpose other than pixel shifting and combines them. For example, when a plurality of image signal groups are obtained by giving different exposure amounts to an image sensor and then combined to obtain a wide dynamic range image, when applied to an image capturing apparatus, a wide dynamic range image can be obtained without being affected by camera shake. You can Alternatively, when image signals of the same scene acquired at different times are combined and applied to an image pickup apparatus that obtains a multiple exposure effect, a moving subject is multiple-imaged and a still subject is reproduced without shaking.

[0208]

As described above, according to the present invention, it is possible to select the pixel shift mode suitable for the exposure control condition set by the photographer, and the photographer does not have to perform a complicated operation and can obtain a high-definition image. Images can be obtained.

According to another feature of the present invention, it is possible to appropriately select the number of pixel shifts according to the image pickup conditions, which balances the deterioration of image quality due to camera shake and the high definition due to pixel shift. It is possible to obtain a high-definition image without the operator performing complicated operations.

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216]

[0217]

[0218]

[0219]

[0220]

[0221]

[0222]

[0223]

[0224]

[0225]

[0226]

[Brief description of drawings]

FIG. 1 is a configuration diagram of an image pickup apparatus according to the present invention.

FIG. 2 is a configuration diagram of an image forming optical system used in the present invention.

FIG. 3 is a configuration diagram illustrating a light beam deflection action of an imaging optical system used in the present invention.

FIG. 4 is a configuration diagram illustrating a pixel shift principle of the present invention.

FIG. 5 is a control block diagram of a main part of the present invention.

FIG. 6 is a control flowchart of the camera of the first embodiment of the present invention.

FIG. 7 is a control flowchart of the lens of the first, fifth and eighth embodiments of the present invention.

FIG. 8 is a timing chart of control according to the first and eighth embodiments of the present invention.

FIG. 9 is a configuration diagram illustrating a pixel shifting method according to the first and fifth embodiments of the present invention.

FIG. 10 is a configuration diagram illustrating an image synthesizing method according to the first and fifth embodiments of the present invention.

FIG. 11 is a control flowchart of the camera of the second embodiment of the present invention.

FIG. 12 is a configuration diagram illustrating a pixel shifting method according to a second embodiment of the present invention.

FIG. 13 is a configuration diagram illustrating an image composition method according to a second embodiment of the present invention.

FIG. 14 is a control flowchart of the camera of the third embodiment of the present invention.

FIG. 15 is a control block diagram according to a fourth embodiment of the present invention.

FIG. 16 is a control flowchart of the camera of the fourth embodiment of the present invention.

FIG. 17 is a control flowchart of the lens according to the fourth embodiment of the present invention.

FIG. 18 is a control flowchart of the camera of the fifth and eighth embodiments of the present invention.

FIG. 19 is an exposure amount adjustment control flowchart of the fifth embodiment of the present invention.

FIG. 20 is a characteristic diagram illustrating a characteristic curve of an image sensor.

FIG. 21 is a configuration diagram illustrating subject division according to the fifth embodiment of the present invention.

FIG. 22 is a timing chart of control according to the fifth embodiment of the present invention.

FIG. 23 is a configuration diagram illustrating a dynamic range expansion method according to a fifth embodiment of the present invention.

FIG. 24 is a flowchart showing a part of a control flow of the camera of the sixth embodiment of the present invention.

FIG. 25 is a flowchart showing a part of a control flow of the camera in the seventh embodiment of the present invention.

FIG. 26 is a characteristic diagram illustrating a correlation calculation according to the eighth embodiment of this invention.

FIG. 27 is a characteristic diagram illustrating a correlation calculation according to the eighth embodiment of this invention.

FIG. 28 is a characteristic diagram illustrating the correlation degree according to the eighth embodiment of this invention.

[Fig. 29] Fig. 29 is a configuration diagram illustrating an image locus when there is no shake in the eighth embodiment of the present invention.

FIG. 30 is a characteristic diagram illustrating the degree of correlation when there is no shake in the eighth embodiment of the present invention.

[Fig. 31] Fig. 31 is a configuration diagram for describing an image locus when there is shake in the eighth embodiment of the present invention.

FIG. 32 is a characteristic diagram illustrating the degree of correlation when there is a shake in the eighth embodiment of the present invention.

FIG. 33 is a subroutine control flowchart for the camera of the eighth embodiment of the present invention.

[Fig. 34] Fig. 34 is a configuration diagram for describing an image locus when there is shake in the ninth embodiment of the present invention.

FIG. 35 is a main control flowchart of the camera of the ninth embodiment of the present invention.

FIG. 36 is a subroutine control flowchart for the camera of the ninth embodiment of the present invention.

[Fig. 37] Fig. 37 is a configuration diagram for describing an image locus when there is shake in the tenth embodiment of the present invention.

FIG. 38 is a characteristic diagram illustrating the degree of correlation when there is a shake in the tenth embodiment of the present invention.

FIG. 39 is a subroutine control flowchart for the camera of the tenth embodiment of the present invention.

[Explanation of symbols]

CMR camera body CCPU Microcomputer in camera IMS image sensor MEM memory DISP display CNC connector SWMOD shooting mode selection switch LNS lens LCPU Lens microcomputer L2 Second lens group (shake correction lens group) ZENC zoom encoder FENC focus encoder GRP, GRY hand shake sensor IACTP, IACT shake compensation actuator SNS metering / focus detection sensor

Claims (3)

(57) [Claims] 1. An image pickup optical means for forming a subject image, an image pickup means for photoelectrically converting the subject image, and an image moving means for changing a relative position between the subject image and the image pickup means within an image forming plane thereof. Pixel shifting means for obtaining a plurality of sets of images by the image pickup means while moving the image moving means; exposure control condition setting means for determining a combination of an aperture value and an exposure time as an image pickup condition of the image pickup means; An image pickup apparatus comprising: a switching unit that switches the operation of the pixel shifting unit according to the setting state of the control condition setting unit. 2. The switching means comprises a first pixel shift mode in which the pixel shift is performed a first predetermined number of times, and the first pixel shift mode.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is switched to a second pixel shift mode in which the pixel shift is performed a second predetermined number of times different from the predetermined number of times. 3. An image pickup optical means for forming a subject image, an image pickup means for photoelectrically converting the subject image, and an image moving means for changing a relative position between the subject image and the image pickup means within an image forming plane. A pixel shift means for obtaining a plurality of sets of images by the image pickup means while moving the image moving means, a determination means for determining an image pickup condition, and an operation of the pixel shift means based on the determination result of the determination means. Switching means for switching between a first pixel shift mode in which the pixel shift is performed a predetermined number of times and a second pixel shift mode in which the pixel shift is performed a second predetermined number of times different from the first predetermined number. An imaging device characterized by the above.