JPH10191136A - Image pickup device and image synthesizer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate the effect of an object-shake and a camera-shake in the case of synthesizing plural images obtained by pixel deviation. SOLUTION: An object is picked up by an image pickup element IMS while conducting pixel deviation by moving a lens group L2 and the plural obtained images are synthesized to obtain one high definition image. In this case, an image pickup area is divided into small areas and a shake amount by an object shake for every small area is checked as to each image. When an object-shake is large in the case of image synthesis, the image synthesis is not executed in the area where the object-shake occurs and the image synthesis is executed for high definition in the area where the object-shake does not occur.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for obtaining a high-definition image by slightly changing the relative position between a subject image formed by an imaging optical system and an image pickup device for photoelectrically converting the subject image. The present invention relates to an image pickup apparatus provided and an image synthesis apparatus suitable for use in the image pickup apparatus.

[0002]

2. Description of the Related Art Conventionally, a plurality of sets of image signals are obtained by performing a plurality of shootings while slightly changing the relative position between a subject image formed by an image forming optical system and an image sensor for photoelectrically converting the subject image. An image pickup apparatus using a so-called pixel shift technique has been proposed in which a single high-definition image is obtained by combining a plurality of sets of image signals by a predetermined method. However, this pixel shifting technique, like the multiple exposure of a still camera, has a long interval between the acquisition time of the first image signal and the last image signal, and the image quality deteriorates if camera shake or subject shake occurs during that time. High definition cannot be expected by shifting. Therefore, as a prior art document for solving this drawback, there is, for example, the following patent publication.

In Japanese Patent Application Laid-Open No. Hei 7-240932, the elimination of image blur due to camera shake and the high definition of an image due to pixel shift are simultaneously performed by a variable apex angle prism in front of the photographing optical system or a moving lens group behind the photographing optical system. Have achieved.
Also, in Japanese Patent Application Laid-Open No. 7-287268, a variable apex angle prism in front of a photographing optical system is driven based on a camera shake signal and a pixel shift signal, and an optical image on an image sensor is decentered in parallel to reduce image shake due to camera shake. Resolution and high-definition of the image by pixel shift are achieved at the same time. Further, when the focal length of the photographing optical system is equal to or longer than a predetermined value, the driving accuracy of the pixel shift is reduced, so that the pixel shifting control is prohibited.

On the other hand, in an image pickup apparatus having an image pickup device, a motion vector of an image is obtained from a time-series output of the image pickup device.
It is possible to determine the image blur prior to shooting.
Therefore, as a prior art in this field, Japanese Patent Laid-Open Publication No.
In Japanese Patent Laid-Open No. 8, the influence of camera shake during exposure is suppressed by continuously detecting the motion vector of a pixel and using the image at the time when the motion vector is minimized as the final recorded image. To the effect. Also, JP-A-8-1725
Japanese Patent Publication No. 68 discloses that a motion vector between a plurality of images for pixel shift is obtained, an image blur component due to a camera shake or a subject shake is removed by interpolation, and then image synthesis is performed to improve the definition of the image. .

[0005]

However, the above conventional example has the following disadvantages. JP-A-7-2
In JP-A-40932 and JP-A-7-287268, the quality of an acquired image is not determined simply by using an image stabilization mechanism to reduce image quality deterioration due to camera shake when shifting pixels. Further, image blur due to the movement of the subject cannot be excluded.

In Japanese Patent Application Laid-Open No. 2-57078,
Since there is no description of pixel shift, the effect of reducing the influence of camera shake on a normal captured image is merely reduced, and it is not possible to increase the definition of an image by pixel shift. Further, image blur due to the movement of the subject cannot be excluded.

Further, in Japanese Patent Application Laid-Open No. 8-172568, uniform image shake due to camera shake and subject shake can be eliminated to some extent, but partial image shake due to subject movement cannot be eliminated. It is also impossible to obtain an image with a special effect that dynamically expresses the movement of a subject.

A first object of the present invention is to provide an image pickup apparatus which eliminates the influence of subject shake during pixel shift control and obtains a high-definition image. A second object of the present invention is to accurately detect a subject shake area in the process of executing pixel shift,
It is an object of the present invention to provide an imaging apparatus that prevents image quality deterioration due to subject shake during pixel shift. A third object of the present invention is to provide an imaging apparatus that accurately detects a subject shake area and a subject shake amount during a pixel shift execution process, and prevents image quality degradation due to subject shake during pixel shift.

A fourth object of the present invention is to provide an image pickup apparatus which performs image composition according to a subject shake and prevents the image quality from being degraded by the pixel shift control when the subject shake occurs. It is. A fifth object of the present invention is to provide an image pickup apparatus that prevents an unnatural image from occurring in an area where a subject shake has occurred. A sixth object of the present invention is to provide an image pickup apparatus that prevents a sense of incongruity of an image generated in a region where a subject shake has occurred, and obtains a high-definition image in a region where the subject shake does not occur.

[0010] A seventh object of the present invention is to provide an image pickup apparatus which applies a special effect to an area where a subject shake has occurred to reduce a sense of discomfort in an image. An eighth object of the present invention is to provide an image pickup apparatus which displays an image synthesis mode executed according to a subject shake and informs a photographer whether or not a desired image can be obtained. A ninth object of the present invention is to provide an imaging apparatus which eliminates the influence of subject shake when obtaining a high-quality image by synthesizing a plurality of images obtained at different times and obtains a high-quality image. is there.

[0011] A tenth object of the present invention is to provide an imaging apparatus capable of obtaining a high-quality image by eliminating the effects of camera shake and subject shake when obtaining a high-quality image by combining a plurality of images obtained at different times. It is to provide. An eleventh object of the present invention is to provide a high-quality image by synthesizing a plurality of images acquired at different times, to effectively suppress image shake due to camera shake and to eliminate the influence of subject shake, An object of the present invention is to provide an imaging device capable of obtaining a quality image.

[0012] A twelfth object of the present invention is to provide an image synthesizing apparatus capable of obtaining a high-quality image by eliminating a sense of incongruity due to subject shake when a plurality of images are synthesized to obtain a high-quality image. is there. A thirteenth object of the present invention is to provide an image synthesizing apparatus capable of eliminating a partial deterioration in image quality due to erroneous synthesis when obtaining a high quality image by synthesizing a plurality of images and obtaining a high quality image over the entire screen. It is to provide.

[0013]

According to the first aspect of the present invention, there is provided a photographing optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, and a relative position between the subject image and the imaging unit. Image moving means for changing within the image plane, pixel shifting means for obtaining a plurality of sets of images by the imaging means while moving the image moving means, and obtaining a high definition image by synthesizing the plurality of sets of images. Image synthesizing means, image blur detecting means for detecting partial image blur occurring between the plurality of sets of images due to movement of a subject, and operation of the image synthesizing means based on a detection result of the image blur detecting means. Operation control means for performing control.

Further, in the invention according to claim 2, the image blur detecting means divides each of the plurality of sets of images into a plurality of predetermined regions, and detects image blur for each of the divided regions. I have. In the invention according to claim 3, the image blur detecting means has a correlation calculating means for the plurality of sets of images,
A partial motion area of the subject image is detected from the output of the correlation operation means, and a motion vector of the motion area is detected. In the invention according to claim 4, the image synthesizing means has a plurality of image synthesizing modes, and the operation control means selects the image synthesizing mode based on the determination result of the magnitude of the image blur. I have.

According to a fifth aspect of the present invention, when the partial image blur is larger than a predetermined value, the operation control means may determine whether the partial image blur is greater than a predetermined value.
Image synthesis in a predetermined area in which image blur has been detected is prohibited. In the invention according to claim 6, the operation control means, when the partial image blur is larger than a predetermined value,
In the area where image blur is detected and the area where image blur is not detected,
Different image synthesis is performed.

According to a seventh aspect of the present invention, the operation control means performs image generation by sweep processing of an acquired image in a region where image blur is detected. Claim 8
In the invention, a display unit is provided, and the operation control unit controls the display unit according to an operation control mode of the image combining unit.

According to a ninth aspect of the present invention, there is provided a photographing optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, and a plurality of image obtaining units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit. Image synthesizing means for synthesizing the plurality of sets of images to obtain one set of images; area dividing means for dividing the imaging area of the imaging means into a plurality of areas; A relative relationship detecting means for detecting a relative relationship between the plurality of areas, a comparing means for comparing the relative relationship detection results for each of the plurality of regions, and an operation control means for controlling the operation of the image synthesizing means based on the comparison result of the comparing means. Is provided.

According to a tenth aspect of the present invention, there is provided a photographing optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, and a plurality of image obtaining units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit. Image synthesizing means for synthesizing the plurality of sets of images to obtain one set of images; and camera shake for detecting uniform image shake between the plurality of images over the entire photographing screen due to camera shake occurring in the photographing optical means. Detecting means, subject shake detecting means for detecting image shake occurring in a partial area of the photographing screen between the plurality of images due to movement of the subject, and detecting the image based on the detection results of the hand shake detecting means and the subject shake detecting means. Operation control means for controlling the operation of the combining means.

According to an eleventh aspect of the present invention, there is provided a photographing optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, and a plurality of image obtaining units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit. Image synthesizing means for synthesizing the plurality of sets of images to obtain one set of images; and optically or uniformly converting a uniform image shake between the plurality of images over the entire shooting screen due to camera shake generated in the shooting optical means. A camera shake correction unit for mechanically correcting the image, a subject shake detection unit for detecting an image shake occurring in a partial area of the shooting screen between the plurality of images due to the movement of the subject, and a detection result of the subject shake detection unit. Operation control means for controlling the operation of the image synthesizing means.

According to a twelfth aspect of the present invention, there is provided an image synthesizing means for synthesizing a plurality of sets of images to obtain one set of images, and an image blur detecting means for detecting a partial image blur region generated between the plurality of sets of images. A region detection unit, and a motion vector detection unit that detects a motion vector between the plurality of sets of images in the image blur region, wherein the image synthesizing unit converts the image blur region into a first method using the motion vector. And combining the areas other than the image blur area by the second method not using the motion vector.

According to a thirteenth aspect of the present invention, there is provided an image synthesizing means for synthesizing a plurality of sets of images to obtain one set of images, and a first image area in which a relative relation between the plurality of sets of images is a first relation. And an image area dividing means for dividing the image into a second image area having a second relation different from the first relation, wherein the image synthesizing means is different between the first image area and the second image area. Image synthesis is performed.

[0022]

According to the first aspect of the present invention, the influence of subject shake is eliminated by controlling the operation of image synthesis based on the detection result of partial image shake occurring between a plurality of sets of images,
In addition, the resolution of an image is improved by pixel shifting.

According to the second aspect of the present invention, an area where the subject shake has occurred is extracted, and an image is synthesized according to the subject shake. According to the third aspect of the present invention, the region where the subject shake has occurred and the subject shake amount are extracted from the correlation calculation between the plurality of images, and the images are synthesized according to the subject shake. According to the fourth aspect of the present invention, image synthesis in which the subject shake has been corrected is performed in an area where the subject shake of a predetermined value or more has occurred.

According to the fifth aspect of the present invention, in a region where a subject shake of a predetermined value or more has occurred, synthesis of a plurality of images by pixel shift is prohibited, and a final image signal is generated from a single image signal. According to the sixth aspect of the present invention, image synthesis for preventing an unnatural feeling due to subject shake is performed for an area where image shake is detected, and high-definition image synthesis is performed for an area where image shake is not detected.

According to the seventh aspect of the present invention, in the area where the image blur has been detected, image synthesis for giving a dynamic effect to the subject image in accordance with the movement of the subject is performed. According to the invention of claim 8, the operation mode of the image synthesizing means is displayed and notified to the photographer.

According to the ninth aspect of the present invention, by comparing the detection results of the relative relationship between a plurality of sets of images for each of a plurality of regions, and controlling the image synthesis operation based on the comparison results,
Eliminate the effects of subject shake and improve image quality by image synthesis.

According to the tenth aspect of the present invention, the image synthesizing operation is controlled based on the detection result of the image shake occurring in a partial area of the photographing screen between a plurality of images, so that the image shake due to the camera shake and the object shake are obtained. Is independently detected, and the operation of image synthesis is switched or prohibited based on the detection result.

According to the eleventh aspect of the present invention, the image blurring caused by the camera shake is optically controlled by controlling the image synthesizing operation based on the detection result of the image blur occurring in a partial area of the photographing screen between a plurality of images. After that, image synthesis is performed in which image shake due to subject shake has been corrected.

According to the twelfth aspect of the present invention, a motion vector between the plurality of sets of images is detected, and at the time of image synthesis, the image blur area is synthesized by the first method using the motion vector. By combining the areas other than the shake area by the second method not using the motion vector, the subject shake occurrence area performs image synthesis using the motion vector of the subject image,
In a region where the subject shake does not occur, a synthesis different from the synthesis is performed.

According to the thirteenth aspect of the present invention, the first image area in which the relative relationship of the plurality of sets of images is the first relationship, and the second image area in which the second relationship is different from the first relationship. By performing different image synthesis in the first image area and the second image area, an area different from the normal image synthesis is performed in an area in which image quality is deteriorated when normal image synthesis is performed.

[0031]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, first to fourth embodiments of the present invention will be described. (First Embodiment) FIGS. 1 to 25 show a first embodiment of the present invention.
It is a figure concerning an embodiment. FIG. 2 shows an example of the image forming optical system, and the focal length is 3 mm of 10 mm to 30 mm.
FIG. 11A shows a wide-angle zoom (f = 10 m).
m) and FIG. 13B shows the lens arrangement at the telephoto end (f = 30 mm). FIGS. 2, 3 and 4 show the first embodiment.
Commonly used in the fourth to fourth embodiments.

The image forming optical system is composed of four groups. In zooming, the fourth group is fixed and the first, second, and third groups move, and the first group moves during focus adjustment. . Then, by displacing the second lens unit in a direction perpendicular to the optical axis, the image on the image forming plane is displaced to perform pixel shift and camera shake correction.

Next, the light beam deflecting effect of the second lens group will be described with reference to FIG. FIG. 3 is a simplified view of each lens group in FIG. 2. FIG. 3A shows that the second group has a predetermined amount d _L.
The light beam deflection effect on the image side when shifting downward only
FIG (b) is the same second group shows the light beam deflection effect on the object side when the shifted downward by a predetermined amount d _L. First, FIG. 3A will be described. Light rays incident on the optical axis of the first group from the object side are deflected upward by the negative second group shifted downward, pass through the third and fourth groups, and are imaged on the image plane. Reach the position of high d _IM . If the ratio of the second group shift amount d _L to the image displacement amount d _IM at this time is referred to as eccentric sensitivity S _d , these three values are represented by d _IM = S _d × d _L (1) ).

Since the eccentric sensitivity S _d changes according to the arrangement of the second and subsequent lens units, in this embodiment, it changes according to zooming. On the other hand, the present embodiment because it uses a front focus of the first group, decentering sensitivity S _d is not changed by focusing. However, if a rear focus method in which focusing is performed by the fourth lens unit or the like is adopted, the eccentric sensitivity S _d can be obtained even by focusing.
Fluctuates. Then, since the eccentric sensitivity S _d is generally expressed as a function S _d (f, R) of the focal length f and the subject distance R, the equation (1) is also expressed as d _IM = S _d (f, R) × d _L ... (2)

Next, the second group shift drive at the time of executing the pixel shift is performed.
The momentum will be described. Figure 4 illustrates the principle of pixel shifting
This is an enlarged view of the light receiving section of the image sensor.
You. In FIG. 4, a square as a pixel is formed on the light receiving surface.
Light receiving section is lateral W_Y, Vertical direction W _PAt regular intervals
Is placed. The resolution of the image formed on this light receiving surface is
Pixel interval W_Y, W_P, But pixels and images
Captures multiple image signals while changing the relative position of
Image resolution by combining and restoring
Can be improved. For example, the point in the image is
When located at the center IM1 of the section, the first set of image signals (secondary
All pixel signals of the original sensor) are captured and stored.
Next, the image is moved in the right direction X so that the point of the image is located at IM2._Y
= W_Y/ 2 displacement and capture the second set of image signals
Just remember. Similarly, move the image point to IM3 and IM4
The third and fourth sets of image signals are taken in, and a total of four sets of image signals are obtained.
The information amount about the image is quadrupled by combining
Can be doubled in both horizontal and vertical directions.

Here, the image is shifted to X _Y (= W
_Y / 2) or just to move the _{X P (= W P / 2} ) using the image movement effect of the second group shown in FIG. 3 (a). That is, in order to displace upward by X _P image in accordance with equation _{(2), d L = X} P / S d (f, R) and d _L only the second group determined by ......... (3) What is necessary is just to shift downward. The image displacement X _P for pixel shifting've always constant values, decentering sensitivity S _d (f, R) zooming since they depend focusing, the second group shift amount d _L
Needs to be changed according to the state of the optical system. Therefore, in the present invention, as will be described later, data on the eccentric sensitivity S _d (f, R) according to the zooming and focusing states is set as a first coefficient in a ROM (Read Only Memory) in a microprocessor (CPU). I remember.

Next, FIG. 3B will be described. Light rays emitted to the left from the center of the image plane along the optical axis c of the optical system pass through the fourth and third groups, and are deflected upward by the negative second group shifted downward. The light beam that has passed through the first group is deflected by an angle of θ _OB with respect to an axis c ′ parallel to the optical axis c of the imaging system, and is projected on the object side. At this time, the ratio between the second group shift amount d _L and the optical axis deflection angle θ _OB is referred to as angle sensitivity Sθ. These three values are represented by θ _OB = Sθ × d _L (4) Are related by Since the angle sensitivity Sθ changes depending on the arrangement of the lens groups before the second group, in the present embodiment, it changes according to zooming and focusing. So generally angular sensitivity S.theta also similar to the decentering sensitivity S _d described above, the focal length f and the object distance R of the function S.theta (f,
R), equation (4) is also replaced by θ _OB = Sθ (f, R) × d _L (5).

Next, the shift drive amount of the second lens group during camera shake correction will be described. It is assumed that the camera body having the imaging optical system and the imaging element has caused rotational shake by an angle θ _{CAMERA in a} downward direction due to camera shake, that is, in a direction in which the tip of the imaging optical system faces downward. The image blur caused by the camera shake at this time is equivalent to the image displacement when the object moves upward by an angle θ _OB (= θ _CAMERA ) with respect to the camera. Therefore, referring to FIG. 3B, when the subject moves upward by the angle θ _OB , the second group can be shifted downward by d _L to eliminate the movement of the subject image. That is, the hand shake angle theta _CAMERA camera shake detection sensor detects, based on the equation (5), the following _{equation, d L = θ CAMERA / Sθ} (f, R) ......... d L only the second group which is obtained by (6) Is shifted downward, image blur due to camera shake can be eliminated.

Then, the camera shake angle θ_CAMERAChanges from moment to moment
And the angle sensitivity Sθ (f, R) is zoomed,
The second group shift amount d is changed by focusing.
_LNeeds to be changed according to the state of the optical system. So book
In the invention, the eccentric sensitivity S described above is used. _dSimilar to (f, R),
Angle sensitivity S according to the zooming and focusing conditions
The data on θ (f, R) is used as the second coefficient,
It is stored in the ROM in the microprocessor.

FIG. 1 is a configuration diagram of an imaging apparatus according to a first embodiment of the present invention. In FIG. 1, CMR is a camera body, LNS is a lens, and is configured as an interchangeable lens that can be attached to and detached from the camera body CMR. First, the camera body CMR will be described. CCPU is a microcomputer in the camera (hereinafter abbreviated as microcomputer), ROM,
This is a one-chip microcomputer having a RAM, an A / D, and a D / A conversion function. The microcomputer CCPU in the camera performs a series of operations of the camera, such as automatic exposure control (AE), automatic focus adjustment (AF), and pixel shift control, in accordance with the camera sequence program stored in the ROM. For this purpose, the microcomputer CCPU in the camera communicates with peripheral circuits in the camera body CMR and the lens LNS to control the operation of each circuit and lens.

Four sets of connection terminals are provided on a mount portion for connecting the camera body CMR and the lens LNS. The power supply BAT in the camera supplies power to each circuit and actuator in the camera, and the lens L through the line VCC.
Power is also supplied to NS. DCL is microcomputer CC in the camera
A line for transmitting a signal from the PU to a microcomputer LCPU in the lens described later, and a line DLC for transmitting a signal from the microcomputer LCPU in the lens to the microcomputer CCPU in the camera.
Through these two lines, the camera body CMR is the lens L
Control NS. In addition, the grounds of the camera and the lens are also connected via the line GND.

IMS is an image pickup device such as a CCD, and IMDR is a driver for controlling charge accumulation and charge transfer of the image pickup device IMS. The MEM is a memory for recording and storing image signal data of a captured image, and includes a semiconductor memory, a magnetic disk, an optical disk, and the like. The DISP is a display unit composed of a liquid crystal display or the like, and displays an image obtained by the imaging device IMS and also displays an operation state of the camera. BS is a beam splitter composed of a half mirror, and guides a part of the photographing light beam to the sensor SNS. Sensor S
NS includes a focus detection sensor that detects a focus state of the imaging optical system, and a photometry sensor that detects subject brightness. CN
C is a connector for connecting to an external device such as a desktop computer, and is used to transmit the contents of the memory MEM to the outside and to control the camera body CMR by a signal from the external device.

SWMN is a main switch. When this switch is turned on, the microcomputer CCPU in the camera permits execution of a predetermined program relating to photographing. SW1, SW
Reference numeral 2 denotes a switch linked to a release button of the camera, which is turned on when the first and second strokes of the release button are pressed. SWSF is a pixel shift mode selection switch which is used to enable / disable pixel shift and to select a predetermined mode from a plurality of pixel shift modes. SWIS is Image Stabilization
, Hereinafter referred to as IS) is a selection switch for selecting permission / non-permission of camera shake correction. SWM
OD is a photographing mode selection switch. When the photographer selects a predetermined mode, the AE mode and the AF mode intended by the photographer are set, and the pixel shift mode and the camera shake correction mode are automatically set.

Next, the lens LNS side will be described. L
CPU is microcomputer in lens, microcomputer CCP in camera
Like the U, it is a one-chip microcomputer having a ROM, RAM, A / D, and D / A conversion functions. Microcomputer LC in lens
PU is a signal line DCL from the microcomputer CCPU in the camera.
In accordance with the command sent through the controller, the drive control of a focusing actuator, a zooming actuator, a diaphragm actuator, and an image blur correction actuator, which will be described later, is performed. Further, it transmits various operation states of the lens and parameters unique to the lens to the microcomputer CCPU in the camera via the signal line DLC. L1 to L4 are lens groups corresponding to the first to fourth lens groups described with reference to FIG. 2 and constitute a zoom optical system, and a subject image is formed on the image sensor IMS by the optical system.

FACT is a focusing actuator which adjusts the focus by moving the first lens unit L1 back and forth in the optical axis direction. The focus encoder FENC detects the position of L1, ie, information corresponding to the subject distance, and sends the information to the microcomputer LCPU in the lens. Send out. ZACT is a zooming actuator that performs zooming by moving the first lens unit L1 to the third lens unit L3 back and forth in the optical axis direction by a zoom mechanism (not shown), and transmits the zoom information to a zoom encoder ZE.
The NC detects it and sends it to the microcomputer LCPU in the lens.
DFM is a diaphragm for adjusting the amount of light, and DACT is a diaphragm actuator for driving the diaphragm DFM.

GRP and GRY are camera shake sensors such as a vibrating gyroscope. The same sensor GR is used to detect the camera shake in the vertical (pitch) direction and the horizontal (yaw) direction.
P and GRY are installed. The camera shake detection result is transmitted to the microcomputer LCPU in the lens. Second lens unit L2
Are configured to be independently shiftable in a two-dimensional direction within a plane perpendicular to the optical axis. In the vertical direction with respect to the optical axis, that is, in the pitch shake correction direction, the pitch actuator I
It is driven by ACTP, and is driven by a yaw actuator IACTY in the left-right direction (the direction perpendicular to the plane of the paper in this figure), that is, the yaw shake correction direction. This shift mechanism is disclosed and known in Japanese Patent Application Laid-Open No. 6-3727 by the present applicant.

FIG. 5 is a block diagram for explaining the main operation of pixel shift and camera shake correction according to the present invention. A CCPU block surrounded by a two-dot chain line is a microcomputer CCP in the camera.
The portion executed by U, similarly, the LCPU block is the portion executed by the microcomputer LCPU in the lens. An imaging condition setting circuit 11 sets the operation mode of each function of AE, AF, pixel shift, and camera shake correction. Reference numeral 12 denotes a timing pulse generation circuit which generates a trigger signal for controlling driving of an image moving optical system for pixel shift and timing for capturing an image signal of an image sensor. Reference numeral 13 denotes an image sensor driving circuit that captures an image signal at a predetermined timing and under a predetermined charge accumulation condition in accordance with control signals of the imaging condition setting circuit 11 and the timing pulse generation circuit 12. Reference numeral 14 denotes a temporary storage circuit for temporarily storing the captured image signal for pixel shifting processing. An image synthesizing circuit 15 synthesizes a plurality of sets of image signals obtained by the pixel shifting operation to obtain a high-definition image. A recording unit 16 corresponds to the memory MEM of FIG. 1 and records a synthesized high-definition image signal.

A pixel shift signal generating circuit 21 generates a command signal for displacing an image for pixel shift. 2
Reference numeral 2 denotes a first coefficient generation circuit which reads data corresponding to the eccentric sensitivity S _d (f, R) from the ROM of the microcomputer LCPU in the lens according to the focus and zoom information of the imaging optical system, and shifts the pixel. image movement amount is corrected to the magnitude of the signal to calculate the X _Y, or shift amount command value of the second group such that X _P.

Reference numeral 31 denotes a camera shake sensor, which corresponds to the vibration gyros GRP and GRY described above. Reference numeral 32 denotes a camera shake signal calculation circuit which performs filtering and integration calculation of the camera shake angular velocity signal detected by the camera shake detection sensor 31. Calculate the camera shake angle. Reference numeral 33 denotes a second coefficient generating circuit, which has the aforementioned angle sensitivity S according to the focus and zoom information of the imaging optical system.
The data corresponding to θ (f, R) is stored in the microcomputer LC in the lens.
It reads from the ROM of the PU, corrects the magnitude of the camera shake angle signal, and calculates a second group shift amount command value for image shake prevention control.

Reference numeral 41 denotes a synthesizing circuit for adding the second group shift amount command value for pixel shifting obtained at 22 and the second group shift amount command value for camera shake correction obtained at 33. 42
Is an image stabilizing actuator, and IACTP and I in FIG.
The actuator is controlled such that the second lens group is driven according to the drive command value obtained by the combining circuit 41. Reference numeral 43 denotes a block indicating that the second lens group is actually shifted and driven.
The movement of the image is controlled on 3. Pixel shift and camera shake correction are performed simultaneously by the above blocks.

FIGS. 6 and 7 show the microcomputers CC in the camera body and the interchangeable lens according to the first embodiment of the present invention.
It is a flowchart which shows the control flow of PU and LCPU. First, the control flow of the microcomputer CCPU in the camera will be described with reference to FIG. When the power switch (main switch) SWMN on the camera body CMR side is turned on, power supply to the microcomputer CCPU in the camera is started, and after step (101), step (102) is performed.
Start operation from. In step (102), the switch S is turned on by pressing the release button in the first stage.
The state of W1 is detected, and when the switch SW1 is off, the process proceeds to step (103). Then, in this step (103), an image blur correction operation (hereinafter referred to as IS (Image Stabilization)) toward the lens LNS.
Send an instruction to stop. The above steps (102) and (103) are repeatedly executed until the switch SW1 is turned on or the power switch SWMN is turned off.

When the switch SW1 is turned on during the execution of the above flow, steps (102) to (11) are executed.
Go to 1). In step (111), the camera microcomputer CCPU transmits an image blur correction start command to the lens microcomputer LCPU via the line DCL. In the next step (112), parameter communication for acquiring parameters unique to the lens, such as the open F number of the lens and the focal length, from the microcomputer LCPU in the lens is performed. In step (113), the luminance of the object is measured by the sensor SNS, the image signal accumulation time of the image sensor and the aperture control value are calculated according to a predetermined exposure control program diagram, and the results are also transmitted to the microcomputer LCPU in the lens. In step (114), the focus state is detected by the sensor SNS, and a driving command for the focusing lens is transmitted to the microcomputer LCPU in the lens.

In step (115), the state of the above-described pixel shift mode selection switch SWSF is detected, and pixel shift conditions such as whether or not pixel shift can be executed and the number of times of pixel shift are set based on the photometric results and the like. In step (116), the state of SW2, which is turned on when the release button is pressed in the second stage, is detected.
When 2 is off, the process returns to step (111), and steps (111) to (115) are repeatedly executed. If it is determined in step (116) that SW2 is on, the process proceeds to step (117).

In step (117), a counter CNT for counting the number of pixel shifts is initialized to zero. In step (118), a timing pulse serving as a trigger signal for pixel shifting control is generated, and the microcomputer LCPU in the lens is generated.
Also to send. In step (119), the driver IMD
Via R, charge accumulation of the image sensor IMS and transfer / readout control of the accumulated charge are performed. Step (120)
Then, the image signal read in the step (119) is temporarily stored in the RAM in the microcomputer CCPU in the camera.
In step (121), the pixel shift number counter CN
Add 1 to T and update. Step (122) the counter CNT is equal to or it has reached the predetermined value N _SF. And if the counter CNT has not reached the predetermined value N _SF waits for the next timing pulse generator returns to step (118), to continue the pixel shifting control. Step (12
When the counter CNT reaches a predetermined value N _SF 2), the process proceeds to step (123).

In step (123), the completion of the pixel shift is transmitted to the microcomputer LCPU in the lens. Step (124) is a subroutine for determining image blur due to camera shake from a plurality of image signals obtained by pixel shifting. Detailed operation contents will be described with reference to FIG. Step (125) is a subroutine for determining image shake due to subject shake from a plurality of image signals obtained by pixel shifting. Detailed operation contents will be described with reference to FIG. Step (126) corresponds to steps (124) and (12).
This is a subroutine for synthesizing a plurality of sets of images by a predetermined method based on the results of the camera shake and the subject shake obtained in 5) to obtain one set of images. Detailed operation contents will be described with reference to FIG. 25 and the like.
In step (127), the image obtained in step (126) is recorded in the memory MEM.

With the above, the photographing operation is completed, and step (10)
Return to 2). Then, in this step (102), the switch S
If W1 is on, the operation from step (111) is repeated, and if switch SW1 is off, in step (103), the microcomputer LCPU in the lens is instructed to stop the image blur correction operation.

FIG. 7 is a flowchart showing the control of the microcomputer LCPU in the lens. In FIG. 7, when power is also supplied to the interchangeable lens side by turning on the power switch SWMN on the camera side, the process proceeds from step (131) to step (132). In step (132), IS
The start command is determined, and when the IS start command has not been received from the camera body CMR, the process proceeds to step (133). In step (133), an IS stop instruction is determined.
If the IS stop command has not been received from the camera body CMR, the process returns to step (132). If the IS stop command has been received, the flow advances to step (134) to stop the image shake correction actuator IACT in the pitch and yaw directions. If an IS start command is transmitted from the microcomputer CCPU in the camera during execution of the steps (132) to (134),
The process moves from step (132) to step (141).

In step (141), the shake detection sensor G
Activate RP and GRY, and input hand shake signals in the pitch and yaw directions. Step (142) corresponds to step (1) in FIG.
This corresponds to 12), and transmits parameters specific to the lens to the camera in accordance with a request from the microcomputer CCPU in the camera. In step (143), the zoom encoder ZENC and the focus encoder FENC are detected to detect the zoom state and the focus state of the optical system. Step (1
In step 44), the first coefficient for pixel shift and the second coefficient for camera shake correction are read from the ROM table based on the detection result of step (143). In step (145), the image blur correction actuator IACT is drive-controlled based on the camera shake signal obtained in step (141) and the second coefficient obtained in step (144) to eliminate image blur due to camera shake.

In step (146), the aperture DFM is driven via the actuator DACT based on the photometric information obtained from the microcomputer CCPU in the camera, and the light amount is adjusted.
In step (147), the focusing actuator FACT is driven to adjust the focus based on the focus detection information obtained from the microcomputer CCPU in the camera. Step (1
At 48), it is determined whether or not a timing pulse for pixel shift is received. If the timing pulse has not been received, the process returns to step (141) to repeatedly execute the camera shake correction, the aperture control, and the focus adjustment operation. When the reception of the timing pulse is confirmed in step (148), the process proceeds to step (149).

In step (149), a reference waveform for driving the second lens group in the pitch or yaw direction for pixel shift is generated. In step (150), a pixel shift drive waveform is generated by multiplying the pixel shift reference waveform generated in step (149) by the first coefficient read in step (144), and this is combined with a camera shake correction signal. I do. By driving and controlling the image blur correction actuator IACT according to the synthesized signal, the image drive is simultaneously and accurately performed for the camera shake correction and the pixel shift.

In step (151), it is determined whether or not a pixel shift completion signal has been transmitted from the microcomputer CCPU in the camera. If the signal has not been transmitted, the pixel shift has not been completed and the process returns to step (148). Waiting for the reception of the timing pulse of step (a), and performing steps (148) to (150) a predetermined number of times, and receiving a pixel shift completion signal transmitted after the pixel shift operation is completed,
The process returns from step (151) to step (132). Then, if the IS start command is not received in step (132) and the IS stop command is confirmed in step (133), the image blur correction actuator IACT is stopped in step (134), and a series of lens control operations associated with photographing is completed. I do.

FIG. 8 is a timing chart for explaining the operation of the camera and the lens according to the flow charts of FIGS. 6 and 7. (A) and (b) show the states of the switches SW1 and SW2 respectively linked to the release button of the camera.
(C) is a trigger signal for pixel shift timing control. (D) shows the charge accumulation timing of the image sensor IMS. (E) and (f) show pixel shift reference waveforms in the pitch and yaw directions. (G) and (h) are camera shake sensors GR
P and GRY show the hand-shake waveforms, and here show the shake displacement waveforms after the detection signal is subjected to processing such as integration as appropriate. (I) and (j) show the pitch and the drive displacement in the yaw direction of the image blur correction second lens group.

Hereinafter, the flow of FIGS. 6 and 7 will be summarized and summarized with reference to FIG. Switch SW1 at time t ₁
Is turned on, camera shake signals (g) and (h) are output. Then, according to the value obtained by multiplying this signal by the second coefficient, the drive of the second lens unit L2 is controlled as shown in waveforms (i) and (j), and the camera shake is corrected.

[0064] When the switch SW2 at time t ₂ is turned on, first timing pulses TP1 is generated at time t ₁₁ after a predetermined time. Then In response to this, it performs the charge accumulation of the light receiving portion during the image pickup device IMS in accordance with the accumulation time which is calculated from the result of photometry, the time t ₁₃ from the time t _12.
When charge accumulation at time t ₁₃ is finished, the transfer and read the accumulated charge, a yaw direction of the pixel shifting reference waveform (f) is generated. Then, the lens displacement (j) in the yaw direction is driven by a command value obtained by adding a value obtained by multiplying the reference waveform (f) by the first coefficient and the camera shake correction waveform.

[0065] From time t ₁₁ to the predetermined time has elapsed after the time t ₂₁ second timing pulse TP2 is generated. Then Similarly imaging device IMS performs the charge accumulation of the light-receiving unit during the period of time t ₂₃ from the time t _22. When the charge accumulation at time t ₂₃ is finished, the transfer and read the accumulated charge Pitch pixel shifting reference waveform (e) is generated. Then, the lens displacement (i) in the pitch direction is also driven by a command value obtained by adding a value obtained by multiplying the reference waveform (e) by the first coefficient and the camera shake correction waveform.

[0066] The time t ₃₁ after a predetermined time from the time t ₂₁ the third timing pulse TP3 is generated. Then Similarly imaging device IMS performs the charge accumulation of the light-receiving unit during the period of time t ₃₃ from the time t _32. When the charge accumulation at time t ₃₃ is finished, the transfer and read the accumulated charge, a yaw direction of the pixel shifting reference waveform (f) is returned to the original value. Then, the lens displacement (j) in the yaw direction is driven by a command value responding only to the camera shake correction waveform.

[0067] The time t ₄₁ after a predetermined time from the time t ₃₁ last timing pulse TP4 is generated. Then Similarly imaging device IMS performs the charge accumulation of the light-receiving unit during the period of time t ₄₃ from the time t _42. When the charge accumulation at time t ₄₃ is finished, the transfer and read the accumulated charge Pitch pixel shifting reference waveform (e) it is also returned to the original value. Then, the lens displacement (i) in the pitch direction is also driven by the command value responding to only the camera shake correction waveform. The switch SW1 at time t ₅ is once turned off, the driving of the hand vibration detection and the second lens group is stopped.

The positions of the images at times t ₁₁ , t ₂₁ , t ₃₁ , t ₄₁ , and t ₅ during the camera shake correction and the pixel shift control are IM1, IM2, IM3, IM3,
IM4 and IM1, and the pixel shift in which the vertical and horizontal positions of each image are shifted by half the pixel interval is realized. The reason why the pixel shift reference waveform is not a rectangular wave but a trapezoidal wave is to reduce an impact due to a sudden change in the position of the second lens group.

Next, a method for determining image blur due to camera shake from a plurality of images obtained by shifting pixels will be described with reference to FIGS. FIGS. 9 and 10 show the principle of calculating the relative displacement between the two images based on the correlation between the two sets of one-dimensional image signals. In both figures, the horizontal axis is the pixel coordinates of the image sensor, and the vertical axis is the output signal value of each pixel.

In FIG. 9A, IM1 is the output of the first image signal set obtained at a certain time, IM2 is the output of the second image signal set obtained a predetermined time after that, and both images are shifted by pixels. It is assumed that the image is displaced by an integral multiple of the pixel interval due to image shake due to operation or camera shake. FIG.
(B) is a result of shifting the second image signal IM2 to the left by two pixels, and both image signals completely match.

Here, the correlation (Correlation) C of the two images
R is expressed by: CR = 1− {ABS (IM2 (i) −IM1 (i)) / (IM1 (i) + IM2 (i))} (7) where ABS is an absolute value and i is a pixel number , Σ are defined as a sum operator from i = 1 to a predetermined pixel number. In the example of FIG. 9, when the pixel shift amount ST is −2, the correlation degree CR is the maximum value 1
Becomes This state is represented as ST _max = −2 and CR _max = 1.

FIG. 10A is an explanatory diagram when the image interval between two sets is different from an integral multiple of the pixel interval (1.5 pixels in FIG. 10). FIG. 10B shows the second image signal IM2 shifted leftward by one pixel, and the degree of correlation CR is 0.8. Even if it is shifted by one more pixel, the correlation C
R does not equal 1.

FIG. 11 shows the relationship between the pixel shift amount ST and the degree of correlation CR at this time. According to FIG.
When the correlation degree CR is calculated while changing the pixel shift amount ST from +2 to -3, the pixel shift amount ST becomes -1 and -2.
, The maximum correlation value CR becomes 0.8. Then the correlation degree CR
Is interpolated by linear regression, the pixel shift amount ST becomes −1.5
It can be seen that the correlation degree CR becomes the true maximum value 0.9 at the time. That is, ST _max = -1.5, CR _max = 0.9
Becomes

The above description is a method for calculating the amount of image shift in two sets of one-dimensional images.
It is also possible to calculate image shift amounts in the vertical and horizontal directions in a set of two-dimensional images. More specifically, as disclosed in Japanese Patent Application Laid-Open No. 64-10787, a two-dimensional image signal is projected and compressed into one-dimensional signals in the up, down, left and right directions, and the motion vector of the image, that is, up and down, is calculated by respective correlation operations. , The amount of image shift in the left-right direction can be detected. Alternatively, it is also possible to directly calculate a two-dimensional motion vector by performing a correlation operation while sequentially shifting the two-dimensional image in two directions.

FIG. 12 is a diagram showing an image movement situation when accurate pixel shift control is performed without being affected by camera shake, and shows the relative positional relationship between the image and the image pickup device also described in FIG. ing. In FIG. 12, the first image signal is obtained (charge storage and readout of the image sensor) in a state where a certain point of the image is located at the point IM1 on the lower left pixel. Subsequently, the image position is moved to IM2 by shifting the pixels by 0.5 pixel intervals in the horizontal direction (corresponding to the yaw direction of camera shake correction), and the second image signal is acquired.

FIG. 13 is a diagram showing the degree of correlation CR between the first and second image signals obtained in FIG. The horizontal axis is the relative shift amount of pixels when performing the correlation operation of two sets of image signals, and the horizontal shift amount is STH and the vertical shift amount is STV. The vertical axis is the degree of correlation CR. The CRH in the figure indicates that two sets of two-dimensional images are relatively shifted in the horizontal direction by ST.
The correlation calculated by shifting by H pixels (± 3 pixels) is
CRV also indicates the degree of correlation calculated by shifting the STV pixel (± 3 pixels) in the vertical direction.

Here, the pixel shift in FIG. 12 is performed only in the horizontal direction.
0.5 pixels in the vertical direction (the pitch
Direction), the correlation degree CR becomes maximum.
The pixel shift amount (this is referred to as the maximum correlation shift amount) is S
TH_max= -0.5, STV _max= 0. Also this
Maximum correlation value CRH at the time_max, CRV_maxIs slightly below 1
It is a value that rotates. This means that the first and second images are spaced
Since it was acquired with a shift amount different from the integral multiple of
This is because the signals do not completely match. And this
This phenomenon also applies to the vertical direction without pixel shift.
CRV to fit_maxAlso does not equal 1.

FIG. 14 shows an image movement situation when a large camera shake occurs during the pixel shift control and the image shake occurs even after the camera shake correction is performed. In FIG. 14, the first image signal is obtained in a state where a certain point of the image is located at the point IM1 on the lower left pixel. Subsequently, the pixels were shifted by 0.5 pixel intervals in the horizontal direction as indicated by the dashed arrow. However, when the image shake due to camera shake was added and the image position moved to IM2, the second image signal was obtained. .

FIG. 15 is a diagram showing the degree of correlation CR between the first and second image signals obtained in FIG. 14, and each symbol in the figure represents the same meaning as in FIG. According to FIG. 15, the maximum correlation shift amount is STH _max = −0.7 and STV _max =
-0.3. So the difference between the maximum correlation shift amount when camera shake is not in FIG. 13 (sum computational formula) [delta] _H,
When _{δ V, δ H = -0.7 +} (- 0.5) = 0.2 ......... (8) δ V = -0.3 + 0 = 0.3 ......... (9) , and the above-mentioned [delta] _H , _ΔV are image shake amounts (units are pixel intervals) due to camera shake in the horizontal and vertical directions during pixel shift. If the image shake amounts δ _H and δ _V exceed predetermined values, for example, δ _MAX = ± 0.1 pixel intervals, the effect of high definition by pixel shift is negated. Take action.

In this embodiment, since the camera shake correction mechanism is operating, ideally, no image shake due to camera shake should occur, or in reality, an output error of the camera shake detection sensor and a camera shake correction capability exceeding the camera shake correction capability. Image shake may occur due to occurrence of large shake or the like. The image blur due to the above-mentioned cause uniformly occurs over the entire surface of the photographing screen. Therefore, by performing the above-described correlation calculation over almost the entire photographing screen, it is possible to determine the presence or absence of image shake due to camera shake.

Next, a method of detecting an image shake due to the movement of the object, that is, a method of detecting the object shake will be described with reference to FIGS. FIG. 16 is a diagram showing a subject image formed on the imaging area of the imaging device IMS.
A person corresponding to the BM is followed by a sub subject OBS
However, the background BKG is projected further back. It is assumed that a part of the person, that is, the arm is moving in the direction of the arrow.

FIG. 17 shows an image of the subject shown in FIG. 16 taken four times while shifting the pixels, and synthesized without subject shake correction. Since the pixel shift shooting performs the same operation as the multiple exposure, even if the image shake of the entire screen is corrected by the camera shake correction, the movement of the subject during the multiple exposure cannot be eliminated. Therefore, as shown in FIG. 17, the arms of the person become four shifted images,
In addition, a special image in which the subordinate object OBS behind it is seen through becomes a special image, and a photograph greatly different from a normal photographing intention is obtained.

FIG. 18 is a view for explaining a method of dividing an image pickup area for detecting a partial image shake caused by the movement of the subject as described above. In the figure, the imaging area is divided into a plurality of small areas AR (k), for example, horizontal = 18 × vertical = 12 = 216. (K = 1 to K, K = 216) Each area AR (k) includes a plurality of pixels as shown in FIG.
Therefore, for each area AR (k), the correlation calculation and the image blur calculation are performed based on the method described with reference to FIGS. If the image shake in all areas is substantially zero, neither camera shake nor subject shake occurs, and if the image shake in all areas is substantially equal and is equal to or larger than a predetermined value, camera shake has occurred. When it is large, it can be determined that partial subject shake has occurred.

FIG. 19 shows an image obtained by photographing the object shown in FIG. 16 while shifting the pixels, and judging the image shake for each area AR (k). The image shake of the area ARMV surrounded by the solid line including the moving arm is large. Is shown.

FIG. 20 is a diagram showing a result of synthesizing a plurality of image sets obtained by shifting pixels based on the above-described image blur detection result. In FIG. 19, in a region other than the region surrounded by the same solid line as in FIG. 19, a high-definition image is obtained by combining four sets of image signals obtained by shifting pixels. On the other hand, in the area ARMV surrounded by the solid line, image synthesis is performed using only a predetermined one of the four sets of image signals obtained by shifting the pixels, for example, only the first set of image signals. As a result, the image in the area ARMV cannot achieve high definition by shifting the pixels, but the multiple exposure effect due to the shake is prevented, and the high resolution of the still object area other than the area ARMV can be achieved by shifting the pixels.

Next, the principle of generating a set of high-definition image signals by combining a plurality of sets of image signals obtained by shifting pixels will be described with reference to FIGS. 21 and 22. FIG. 21 is a view for explaining the relative positional relationship between the image and the image sensor when shifting pixels. (A) of FIG. 21 is also described in FIG.
The image is IM for the pixels of the image sensor fixed in the camera.
1 → IM2 → IM3 → IM4 → IM1. This is because the position of the image sensor is IG1.fwdarw.IG2.fwdarw.IG3 as shown in FIG.
It is completely equivalent to moving in the order of → IG4 → IG1.
Therefore, the output signal of each pixel when the image sensor is located at IG1 is defined as IG1 (i, j). Here, the image sensor is (mx
n) An area sensor for pixels. Similarly, IG2, IG
3, the output signal when located at IG4 is IG2 (i,
j), IG3 (i, j) and IG4 (i, j).

FIG. 22 illustrates a method of synthesizing these four sets of image signals. A new pixel signal set of (2m × 2n) pixels obtained by combining four (m × n) pixel signals is defined as IM
Let G (u, v). And the pixel signal IMG (u, v)
The upper left four pixels are obtained by assembling four sets of original pixels as shown in the figure. Therefore, considering an image restoration method based on this figure, IMG (u = 2i-1, v = 2j) ← IG1 (i, j) (10) IMG (u = 2i-1, v = 2j− 1) ← IG2 (i, j)... (11) IMG (u = 2i, v = 2j-1) ← IG3 (i, j)... (12) IMG (u = 2i, v = 2j) ← IG4 (i, j) (13) According to the equation (13), one set of high definition image signals can be generated from four sets of image signals.

The above-described method is an image synthesizing method in a case where there is no camera shake or subject shake and the entire screen can be made high-definition by shifting pixels. In the present invention, the photographed screen is divided into small areas and In order to perform image blur detection and image synthesis, image synthesis according to the above equations (10) to (13) is also performed for each of the small areas. Then, in an area in which image blur due to subject blur has occurred, image formation using only one image is performed instead of combining four images. At this time, the above equation (1
The following equation is used instead of 0) to (13): IMG (u = 2i-1, v = 2j) ← IG1 (i, j) (14) IMG (u = 2i-1, v = 2j-1) ) ← IG1 (i, j) (15) IMG (u = 2i, v = 2j-1) ← IG1 (i, j) (16) IMG (u = 2i, V = 2j) ← IG1 (i, j) (17) An image is formed.

That is, there is no subject shake, and equation (10)
In the region where image synthesis according to (13) is performed, the information amount for a predetermined subject image, that is, the number of pixels constituting the subject image is quadrupled. On the other hand, since the subject shake has occurred, the number of pixels constituting the subject image has also increased four times in the area where the image synthesis according to Equations (14) to (17) has been performed, but the information amount has increased. No, that is, the definition of the image is the same as the original image before performing the pixel shift.

The above synthesizing method is applied to a monochrome image pickup device or a multi-chip color image pickup device using a color separation prism. Although there are some differences in the image synthesis method, the basic idea is the same.

FIG. 23 is a flowchart showing the operation of the image blur determination due to the camera shake during the pixel shift described in FIGS. 9 to 15, and corresponds to the camera shake determination subroutine of step (124) in FIG. First, in step (161), a pixel shift counter CNT is initialized to zero. In step (162), the degree of correlation between the first and second sets of image signals is calculated according to the above-described equation (7). Step (16
In 3), the maximum correlation shift S
TH _{ma x,} to calculate the STV _max.

In step (164), the maximum correlation value C
RH_max, CRV_maxIs calculated. Step (165)
Then, the image shake amount δ_H, Δ_VIs calculated. Step (16
In 6), the counter CNT is updated by adding 1 to it. Stay
In step (167), the counter CNT and the number of pixel shifts
A predetermined value N representing _{science fiction}Compare with. For example, N_{science fiction}= 4
In step (162) to step (165),
The operation is performed by first and second image signal sets, and second and third image signals.
No. 3 times between the 3rd and 4th image signal sets
Become. Therefore, CNT is N_{science fiction}If it does not reach -1,
Returning to step (162), the step for the next image signal set is performed.
Repeat steps (162) through (166)
And execute. CNT is N_{science fiction}If you reach -1, step
Proceed to (168).

[0093] At step (168), for all of the image blur amount calculated in step (165), is compared with the predetermined value [delta] _1. And if less than all of the image blur amount is the predetermined value [delta] _1, image blur due to camera shake is determined to be approximately zero, and sets a flag FLHD to zero representing the occurrence of camera shake in step (169). On the other hand, in step (168), of the calculated image blur amounts,
Larger than the predetermined value [delta] ₁ even one to store one hand shake flag FLHD proceeds to step (170) if any. After execution of step (169) or step (170), the process proceeds to step (171), and returns to the main flow of FIG. According to the above flow, image blur due to camera shake is determined.

FIG. 24 is a flowchart showing the operation of the image shake determination due to the object shake during the pixel shift described with reference to FIGS. 18 and 19, and corresponds to the object shake determination subroutine of step (125) in FIG. First, step (176)
Then, as shown in FIG. 18, the imaging area is changed to K small areas AR.
(K). In step (177), image blur calculation is performed on the divided predetermined area AR (k). Specifically, the operations in steps (161) to (167) of FIG. 23 are performed on the corresponding small area AR (k), and the horizontal and vertical image blurs in that area are calculated.

[0095] At step (178), the image blur amount calculated in step (177), is compared with the predetermined value [delta] _2. If the image shake amount is smaller than the predetermined value δ _2, it is determined that the image shake due to the object shake in the small area is substantially zero, and the flag FLOB (k) indicating the occurrence of the object shake is set in step (179). Set to zero. On the other hand, in step (178), if at least one of the calculated image shake amounts is larger than the predetermined value δ _{2, the} flow advances to step (180) to move the subject shake flag FLOB.
1 is stored in (k). In step (181), it is determined whether or not steps (177) to (180) have been executed for all the divided areas. If the execution has not been completed, the process returns to step (177) to calculate the unexecuted area. On the other hand, when the image blur determination of all the areas is completed, the process proceeds from step (181) to step (182), and returns to the main flow of FIG. According to the flow described above, image shake due to subject shake in each area is determined.

FIG. 25 is a flowchart showing the image synthesizing operation described with reference to FIGS. 20 to 22, and corresponds to the image synthesizing subroutine of step (126) in FIG. First, in step (186), it is determined whether or not image blur due to camera shake has occurred between a plurality of images acquired for pixel shifting. Specifically, the camera shake flag FLH set in step (169) or step (170) in FIG.
The content of D is determined. And the shake flag FLHD is 1
If the camera shake is large, the program jumps to step (194) to give a camera shake warning, and returns to the main routine in step (195). When it is determined in step (186) that the camera shake flag FLHD is zero, that is, the camera shake is small, the process proceeds to step (187).

In step (187), the argument k for performing subject shake determination for each of the small areas AR (k) divided in FIG. 18 is reset to zero. Step (18
In 8), the magnitude of the subject shake in the small area AR (k) is determined. Specifically, the subject shake flag F set in step (179) or step (180) in FIG.
The content of LOB (k) is determined. When the subject shake flag FLOB (k) is 1 and the subject shake is large, the process proceeds to step (189), and image synthesis using only one image using the equations (14) to (17) is performed.

On the other hand, if it is determined in step (188) that the subject shake flag FLOB (k) is zero, that is, it is determined that the subject shake is small, the process proceeds to step (190), where the equation (1)
Image synthesis using four images using (0) to (13) is performed. After execution of step (189) or (190), the process proceeds to step (191), where 1 is added to the argument k and updated.
In step (192), it is determined whether or not the argument k matches the number of divisions K of the small area AR (k). If k and K do not match, the process returns to step (188) to repeatedly perform subject shake determination in the small area AR (k). If k and K match, the process proceeds to step (193). Step (19
In 3), the contents of the image combination mode executed in step (189) or (190) are displayed on the display unit DISP, and the photographer is notified of what kind of image combination has been performed. Then, the process returns to the main routine in step (195).

According to the above flow, when camera shake is large, even if a plurality of images are obtained by shifting pixels, the images are not combined. When the subject shake is large, the image synthesis is not performed on the area where the subject shake occurs, and the image synthesis for high definition is performed on the area where the subject shake does not occur.
When neither camera shake nor subject shake occurs, image synthesis for high definition is performed over the entire screen.

According to the first embodiment, (1) an image shake due to a subject shake is extracted from a result of a correlation operation between a plurality of image signals obtained by a pixel shift operation, and a pixel is extracted according to the magnitude of the image shake. Since the method of synthesizing images by shifting is changed, it is possible to prevent an image having a sense of incongruity from being obtained by subject shake, and to obtain a high-definition image by shifting pixels. (2) Only a region where a subject shakes is generated by a single image instead of a normal image synthesis by pixel shift, so that a moving subject cannot be made high-definition. In addition to giving expression, a high-definition image can be obtained for a still subject.

(3) Since the execution result of the image synthesis after the pixel shift is notified to the photographer using the display means, the photographer can check the definition of the obtained image and how the moving subject is photographed. It is possible to grasp whether or not a desired image has been obtained, and to take measures such as re-taking an image when a desired image cannot be obtained. (4) Since the optical camera shake correction means is provided, it is possible to prevent image quality deterioration due to camera shake at the time of pixel shift. This has the effect.

(Second Embodiment) In the first embodiment, in a region where a subject shakes, an image is formed by a single image instead of synthesis by a plurality of sets of images. This is an embodiment in which The second embodiment described below shows an embodiment in which a region where a subject shake has occurred is subjected to a connecting process of a plurality of images to give a dynamic expression.

FIGS. 26 to 28 show the image signal generation method and effects of the second embodiment, and FIG. 29 shows the control flow of the camera. Hereinafter, the operation of the second embodiment will be described with reference to the drawings. As described with reference to FIG. 16, when a partially moving subject is photographed with a pixel shift, as shown in FIG. 17, the image of a plurality of images in a moving subject portion largely shifts, and a non-moving subject portion corresponds to a pixel shift control amount. An image having a slight shift is obtained. So first, the first group and the second
A region ARMV surrounded by a solid line in FIG. 26 is obtained by performing a correlation operation on the image of the group and extracting a region having image blur.

Next, a correlation operation is performed again on the image in the area ARMV, and a moving subject area is extracted. As a result, it is determined that the first set of motion images is OBMV ₁ and the second set of motion images is OBMV ₂ . Subsequently, a motion vector between the two motion images is detected. In this case, when calculating the correlation between the two images, the correlation calculation is performed while performing not only the image shift in the parallel movement direction of the two images, but also the image shift in the rotational movement direction. The motion of the second set of images is detected. In FIG. 26, the motion image is G _MV
Is determined to have been rotated by an angle θ _MV around.

FIG. 27 illustrates a method of performing image joining processing based on motion image areas and motion vectors in two sets of images detected by the above method. First, FIG.
In the first motion image OBMV ₁ of, the boundary line on the starting point side of the motion vector is extracted. This is a line SH ₁ of Figure 27. Subsequently, similarly, in the second motion image OBMV ₂ , a boundary line on the starting point side of the motion vector is extracted and
Expressed as H ₂ . Then, while sweeping the boundary line SH ₁ according to the motion vector, an image signal of the sweep area is generated,
To end the sweeping operation in the place that has reached the boundary line SH _2.

The above-described sweeping operation is performed between the images of the second and third sets, and between the images of the third and fourth sets. Use as is.
On the other hand, in an area without subject shake, normal image synthesis suitable for pixel shifting is performed.

FIG. 28 shows a final image obtained by the above-described image synthesizing operation. The part where the subject shake has occurred is the sweep area ARSP and the fourth set of motion images OBMV ₄
The final image also expresses a stop motion while expressing the movement of the subject. The sweeping process can also prevent the background image from being seen through.

FIG. 29 is a flowchart showing a subroutine of the camera according to the second embodiment. The main flow, the camera shake determination subroutine, the subject shake determination subroutine, and the lens control flow of the camera according to the present embodiment are the same as those in FIGS. 6, 23, 24, and 7 of the first embodiment.
Since the new part in the present embodiment is the image synthesizing subroutine of step (126) in the main flow of the camera in FIG. 6, this part is shown in FIG.

First, in step (286), it is determined whether or not image blur due to camera shake has occurred between a plurality of images acquired for pixel shifting. Specifically, as in the first embodiment, the content of the camera shake flag FLHD set in step (169) or step (170) in FIG. 23 is determined. If the camera shake flag FLHD is 1 and the camera shake is large, the process jumps to step (294) without performing image synthesis by shifting the pixels, and displays on the display device DISP that image synthesis is not to be performed, and then proceeds to step (297). To return to the main routine.

At step (286), the camera shake flag FLH
When D is zero, that is, when it is determined that the camera shake is small,
Proceed to step (287). In step (287),
A determination is made as to whether or not the subject has shaken. Specifically, the content of the subject shake flag FLOB (k) set in step (179) or step (180) in FIG. 24 is determined, and
It is determined whether there is a subject shake occurrence area. If it is determined that there is no subject shake occurrence area, step (2)
Jump to 95). In step (295), the four sets of images obtained by the pixel shift control are combined using equations (10) to (13) to obtain a high-definition image over the entire area of the captured image. Subsequently, at step (296), the fact that high definition has been achieved by shifting the pixels over the entire image area is displayed on the display unit DISP. Then, in step (297), the process returns to the main routine.

On the other hand, in step (287), if at least one region with subject shake is confirmed, the flow advances to step (288). In step (288), a moving subject portion is extracted as described with reference to FIG. Step (2
At 89), the motion vector of the extracted motion image is calculated. In step (290), as described with reference to FIG. 27, the boundaries of the motion image are extracted. Step (291)
Then, sweep processing of an area where subject shake occurs is performed from the extracted motion image and motion vector. The fourth set of images is set at the end position of the sweep processing. Step (29
In 2), image synthesis using Formulas (10) to (13) is performed on an area without subject shake. Step (29
In 3), the display unit DISP displays that the dynamic synthesis processing has been performed on the subject shake occurrence area. Then, in step (297), the process returns to the main routine.

According to the above flow, an area where the subject shake is large is detected, and a sweeping process is performed on the area to give a dynamic feeling to the subject showing a partial motion, and a high definition is provided for an area where the subject shake does not occur. Image synthesis for image conversion.

According to the second embodiment, the first
In addition to the advantages (1), (3), and (4) of the embodiment, (5) the moving images are interpolated by a connection (sweep) process to give a moving feeling, and the final image of the movement is also used. In addition, a moving person, a car, and the like can be expressed without a sense of incongruity, and a still subject image with no movement can be improved in definition by shifting pixels. This has the effect.

(Third Embodiment) In the second embodiment, the moving process of a moving subject is connected by a sweep process.
In addition, the embodiment has been described in which the image at the end of the movement is incorporated to give a dynamic expression while retaining the still image information of the moving subject. The third embodiment described below is an embodiment in which all moving areas are connected by a sweep process to reproduce an image suitable for panning and the like.

FIGS. 30 and 31 are diagrams showing an image signal generation method and effects of the third embodiment, and FIG. 32 is a control flow of the camera. Hereinafter, the operation of the third embodiment will be described with reference to the drawings. FIG. 30 is a view showing the state of FIG.
26, and shows a boundary line extraction result of the motion image extracted in FIG. Here, in the second embodiment, in the extracted motion image, only the boundary line on the starting point side of the motion vector is extracted, but in the present embodiment, both the boundary line on the starting point side and the ending point side of the motion vector are extracted. Has been extracted.
That is, in FIG. 30, in the case where the moving image processing is performed on the subject moved from the image signals of the first set and the second set, the first
Start side boundary line SH ₁₁ set th motion images and the destination side boundary line S
H ₁₂ , and four boundary lines of a start point side boundary line SH ₂₁ and an end point side boundary line SH ₂₂ of the second set of motion images are extracted.

Subsequently, two start point side boundary lines SH ₁₁ and SH ₂₁
An image is generated by a sweeping process as indicated by the arrow indicated by the solid line. Then generated image by sweeping process as arrows between the indicated by the dashed line between the end point boundary line SH ₁₂ and SH _22. The above sweep operation is performed between the images of the second and third sets and between the images of the third and fourth sets. And
By performing superposition by averaging the swept image of the start-side boundary line and the swept image of the end-side boundary line at the end, the subject continuously flows in the motion vector direction over the entire region where the subject shake has occurred. . On the other hand, in an area where there is no subject shake, a normal image synthesis suitable for pixel shift is performed as in the second embodiment.

FIG. 31 shows a final image obtained by the above-described image synthesizing operation. In the portion where the subject shakes, the entire area becomes the sweep area ARSP, and in this area, the subject flows continuously and is photographed, and the image is more dynamic. Further, by the above-described sweeping process, it is possible to prevent the background image from being seen through as in the second embodiment.

FIG. 32 is a flowchart showing a subroutine of the camera according to the third embodiment. The control flow of the camera according to the present embodiment is obtained by changing a part of the image synthesizing subroutine of the second embodiment shown in FIG. 29. Therefore, only the changed points will be described. In the image synthesizing subroutine of the second embodiment shown in FIG. 29, the boundary line extraction in step (290) and the sweep processing in step (291) are performed in step (390) in the third embodiment of FIG. In place of the boundary line extraction and the sweep processing of step (391), step (3)
The difference is that the averaging process of 92) is added. Hereinafter, the above changes will be described.

In the boundary line extraction in step (390), as shown in FIG. 30, the start point side and end point side boundary lines of the moving image are extracted. In step (391), a sweeping process of the extracted start point side and end point side boundary lines is performed. In step (392), the sweep images on the start point side and the end point side are averaged.

According to the above flow, an area where the subject shake is large is detected, and a sweeping process is performed on the entire area to give a large dynamic feeling to the subject showing a partial movement, and an area where the subject shake does not occur. For, image synthesis for higher definition is performed.

According to the third embodiment, in addition to the effects (1), (3), and (4) of the first embodiment, (6) the entire moving image area is connected (sweep) processing. Since the motion is given by interpolation, a moving person, a car, and the like can be expressed without a sense of incongruity, and the background image with no motion can be made high definition by pixel shift. In particular, the present embodiment is suitable for so-called panning, in which an image is taken while following a moving subject, and the moving subject is stopped and the background is flown. This has the effect.

(Fourth Embodiment) The first to third embodiments
In the above-described embodiment, the camera automatically selects and executes an image combining method according to the state of camera shake and subject shake. The fourth embodiment described below shows an embodiment in which a photographer can select a desired image synthesis method after a camera acquires a plurality of images for pixel shift.

In this embodiment, a predetermined program is executed by using the image synthesis mode selection switch SWCMP and the image synthesis start switch SWST of the camera shown in FIG. The image synthesis mode selection switch SWCMP of FIG. 1 has a plurality of positions connected to the microcomputer CCPU in the camera.
For example, with a switch having four positions, the photographer can use this switch SWCMP to obtain a desired image synthesis method.
Is set at a predetermined position. Image synthesis start switch SW
ST is a push switch also connected to the microcomputer CCPU in the camera. When this switch is turned on by the photographer, image synthesis is started according to the image synthesis mode selected by the image synthesis mode selection switch SWCMP.

FIG. 33 is a control flow chart according to the fourth embodiment. The operation of the fourth embodiment will be described below with reference to FIG. After the subject is photographed according to any one of the first to third embodiments, when the photographer turns on the image composition start switch SWST, execution of the composition flow is started from step (401).

In step (402), the state of the image synthesis mode selection switch SWCMP is determined, and the selected image synthesis mode is recognized. Step (403)
Then, the selected image composition mode is displayed on the display unit DISP.
To be displayed. In step (404), a branching process according to the selected image synthesis mode is performed. In this step, if it is determined that the mode in which the multiple images obtained by the pixel shift control are not combined (this mode is represented by number zero) is selected, the image combining is not performed and the step (4) is performed.
08), and the original image is directly stored in the memory MEM.
To record. If the stop motion mode (represented by the number 1) has been selected in step (404), the flow advances to step (405). In step (405), a flow equivalent to that in FIGS. 24 and 25 of the first embodiment is executed to obtain the composite image shown in FIG. Thereafter, in step (408), the synthesized image is stored in the memory M
Record in EM.

In step (404), if the dynamic mode (represented by number 2) has been selected, step (4)
Go to 06). In step (406), a flow equivalent to that in FIG. 29 of the second embodiment is executed to obtain the composite image shown in FIG. Then, in step (408),
The combined image is recorded in the memory MEM. If the panning mode (represented by the number 3) has been selected in step (404), the flow advances to step (407).
In step (407), a flow equivalent to that in FIG. 32 of the third embodiment is executed to obtain the composite image shown in FIG. Thereafter, in step (408), the combined image is recorded in the memory MEM.

Image synthesis according to each mode is performed in the above steps, and after the synthesized image is recorded, step (40)
In step 9), the composite image signal is transmitted to an external desktop computer or printer via the connector CNC.
Then, in step (410), the control of image composition and output is completed.

According to the fourth embodiment, (7) after acquiring a plurality of sets of images for pixel shifting in one photographing operation, by performing an independent image synthesizing command operation independent of the photographing operation, A composite image can be obtained by different combining methods. Therefore, there is no need to select an image synthesis mode at the time of shooting, shooting can be performed without missing a photo opportunity, and a desired image type can be selected after shooting. (8) By switching the image synthesizing mode and repeating the image synthesizing operation, images with different video effects can be obtained by one photographing, and the memory area in the camera can be saved, and the application range of the acquired image is expanded. . This has the effect.

(Other Modifications) The present invention can also be used for an image pickup apparatus which acquires a plurality of image signals within a predetermined time for a purpose other than the pixel shift and combines them. For example, if the present invention is applied to an imaging apparatus that obtains a wide dynamic range image by obtaining a plurality of image signals by giving different exposure amounts to an imaging device and combining them to obtain a wide dynamic range image, a wide dynamic range can be prevented while preventing the occurrence of discomfort due to subject shake. Images can be obtained.
Alternatively, when an interline readout CCD developed for capturing moving images is used as an image sensor, two-field images acquired at different times are synthesized to complete a high-definition one-frame image. In addition, it is possible to obtain a high-definition frame image while preventing a sense of incongruity caused by subject shake occurring between two fields.

[0130]

As described above, according to the first aspect of the present invention, since the image synthesizing operation of the pixel shift is controlled according to the subject shake between the plurality of pixel shifting images, the subject shake is large and the image synthesis is performed. In contrast, if the image quality deteriorates or an uncomfortable image is obtained, the method of synthesizing the image by changing the pixel is changed or prohibited.If there is no subject shake, a high-definition image is obtained. It is possible to provide an image pickup apparatus that can prevent adverse effects of pixel shift due to subject shake when there is subject shake.

According to the second aspect of the present invention, the photographing area is divided into a plurality of areas, and the image synthesizing operation is controlled based on the subject shake detection result for each area. In addition, it is possible to provide an image pickup apparatus capable of realizing high definition by shifting pixels in an area without subject shake. According to the third aspect of the present invention, the region where the subject shake has occurred and the subject shake amount are extracted from the correlation calculation between a plurality of images, and image synthesis is performed in accordance with the subject shake, so that accurate subject shake detection and correction can be performed. Thus, it is possible to provide an imaging apparatus capable of reliably eliminating the failure of the pixel shift due to the subject shake. According to the fourth aspect of the present invention, since an image synthesis in which the subject shake has been corrected is performed in an area in which the subject shake of a predetermined value or more has occurred, it is possible to provide an imaging apparatus in which a sense of discomfort generated in the subject shake area is prevented.

According to the fifth aspect of the present invention, in a region where a subject shake exceeding a predetermined value has occurred, synthesis of a plurality of images by pixel shift is prohibited and a final image signal is generated from a single image signal. Discomfort that occurs in the shake area can be prevented. According to the invention of claim 6, in a region where the subject shake has occurred, a sense of discomfort due to the subject shake is prevented,
A high-definition image can be obtained in an area where the subject shake is not detected.

According to the seventh aspect of the present invention, it is possible to provide an image pickup apparatus which not only prevents a sense of incongruity due to subject shake, but also provides a dynamic effect for expressing the movement of the subject. According to the invention of claim 8, the photographer can grasp what kind of image can be obtained, and when a desired image cannot be obtained, measures such as re-photographing can be taken.

According to the ninth aspect of the present invention, when a plurality of images having different image information are synthesized to obtain a high-quality image having a large amount of information, it is possible to prevent occurrence of a sense of incongruity due to subject shake. An imaging device for obtaining a high-quality image can be provided.

According to the tenth aspect of the present invention, when a plurality of images having different image information are synthesized to obtain a high-quality image having a large amount of information, image quality degradation due to camera shake and uncomfortable feeling due to subject shake may occur. It is possible to provide an imaging device that obtains a high-quality image while preventing both.

According to the eleventh aspect of the present invention, when a plurality of images having mutually different image information are synthesized to obtain a high-quality image having a large amount of information, image blur due to camera shake is optically corrected. In addition, it is possible to provide an imaging device that can prevent a sense of incongruity due to subject shake and obtain a high-quality image.

According to the twelfth aspect of the present invention, when a plurality of images having different image information are synthesized to obtain a high-quality image having a large amount of information, a motion vector due to the motion of a subject between the plurality of images is obtained. Since the image synthesis in which the subject shake is corrected based on the detection result is performed, it is possible to realize an image combining apparatus that obtains a high-quality image without causing a sense of incongruity due to the subject shake.

Further, according to the thirteenth aspect, when a plurality of images having mutually different image information are synthesized to obtain a high quality image having a large amount of information, a correlation operation result between the plurality of images is obtained. Since the image synthesis is performed, it is possible to prevent the image quality from being degraded due to erroneous image synthesis and to realize an image synthesis apparatus that obtains a high-quality image.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an imaging device according to the present invention.

FIG. 2 is a configuration diagram of an imaging optical system used in the present invention.

FIG. 3 is a configuration diagram illustrating a light beam deflecting action of an imaging optical system used in the present invention.

FIG. 4 is a configuration diagram illustrating the principle of pixel shifting according to the present invention.

FIG. 5 is a control block diagram of a main part of the present invention.

FIG. 6 is a main control flowchart of the camera according to the first embodiment of the present invention.

FIG. 7 is a control flowchart of the lens according to the first embodiment of the present invention.

FIG. 8 is a timing chart of control according to the first embodiment of the present invention.

FIG. 9 is a configuration diagram illustrating a correlation operation according to the present invention.

FIG. 10 is a configuration diagram illustrating a correlation operation according to the present invention.

FIG. 11 is a configuration diagram illustrating the degree of correlation according to the present invention.

FIG. 12 is a configuration diagram illustrating an image trajectory when there is no shake according to the present invention.

FIG. 13 is a configuration diagram illustrating a correlation degree when there is no shake according to the present invention.

FIG. 14 is a configuration diagram illustrating an image trajectory when there is a shake according to the present invention.

FIG. 15 is a configuration diagram illustrating a correlation degree when there is a shake according to the present invention.

FIG. 16 is a configuration diagram illustrating an example of a moving subject.

FIG. 17 is a configuration diagram illustrating a synthesized image obtained by normal image synthesis in pixel shifting.

FIG. 18 is a configuration diagram illustrating division of an imaging region according to the first embodiment of the present invention.

FIG. 19 is a configuration diagram illustrating an image blur area detection result according to the first embodiment of the present invention.

FIG. 20 is a configuration diagram illustrating an image after composition according to the first embodiment of the present invention.

FIG. 21 is a configuration diagram illustrating a pixel shifting method according to the first embodiment of this invention.

FIG. 22 is a configuration diagram illustrating a pixel synthesizing method according to the first embodiment of this invention.

FIG. 23 is a flowchart of a camera shake determination subroutine according to the first embodiment of the present invention.

FIG. 24 is a flowchart of a subject shake determination subroutine according to the first embodiment of this invention.

FIG. 25 is a flowchart illustrating an image composition subroutine according to the first embodiment of this invention. .

FIG. 26 is a configuration diagram illustrating a motion vector detection result according to the second embodiment of the present invention.

FIG. 27 is a configuration diagram illustrating sweep processing of a motion image according to the second embodiment of the present invention.

FIG. 28 is a configuration diagram illustrating an image after combination according to the second embodiment of the present invention.

FIG. 29 is a flowchart of an image composition subroutine according to the second embodiment of this invention.

FIG. 30 is a configuration diagram illustrating a sweep process of a motion image according to the third embodiment of the present invention.

FIG. 31 is a configuration diagram illustrating an image after combination according to the third embodiment of the present invention.

FIG. 32 is a flowchart illustrating an image composition subroutine according to the third embodiment of this invention.

FIG. 33 is a flowchart of an image composition subroutine according to the fourth embodiment of the present invention.

[Explanation of symbols]

 CMR camera main unit CCPU microcomputer in camera IMS image sensor MEM memory DISP display unit CNC connector SWMOD shooting mode selection switch SWCMP image synthesis mode selection switch SWST image synthesis start switch LNS interchangeable lens LCPU microcomputer in lens L2 second lens group (shake correction lens group) ) ZENC Zoom Encoder FENC Focus Encoder GRP, GRY Shake Sensor IACTP, IACTY Shake Correction Actuator

Claims (13)

[Claims] 1. An imaging optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, and an image moving unit for changing a relative position between the subject image and the imaging unit in an image forming plane. Pixel shifting means for obtaining a plurality of sets of images by the imaging means while moving the image moving means; image synthesizing means for synthesizing the plurality of sets of images to obtain a high-definition image; Image blur detecting means for detecting partial image blur occurring between the images, and operation control means for controlling the operation of the image synthesizing means based on the detection result of the image blur detecting means. Imaging device. 2. The image pickup apparatus according to claim 1, wherein said image blur detecting means divides each of said plurality of sets of images into a plurality of predetermined areas and detects image blur for each of the divided areas. . 3. The image blur detecting means has a correlation calculating means for the plurality of sets of images, detects a partial moving area of a subject image from an output of the correlation calculating means, and detects a motion of the moving area. 2. The method according to claim 1, further comprising detecting a vector.
An imaging device according to any one of the preceding claims. 4. The image synthesizing unit has a plurality of image synthesizing modes, and the operation control unit selects the image synthesizing mode based on a determination result of the magnitude of the image blur. Item 2. The imaging device according to Item 1. 5. The image pickup apparatus according to claim 1, wherein said operation control means prohibits image synthesis of a predetermined area in which the image shake is detected when said partial image shake is larger than a predetermined value. apparatus. 6. The image processing apparatus according to claim 1, wherein when the partial image blur is larger than a predetermined value, the operation control unit performs different image synthesis in an area where the image blur is detected and in an area where the image blur is not detected. The imaging device according to claim 1. 7. The imaging apparatus according to claim 6, wherein said operation control means performs image generation by sweep processing of the acquired image in an area where image blur is detected. 8. Display means is provided, wherein said operation control means comprises:
2. The imaging apparatus according to claim 1, wherein the display unit is controlled according to an operation control mode of the image combining unit. 9. An imaging optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, a plurality of image acquisition units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit, Image synthesizing means for synthesizing images to obtain a set of images; area dividing means for dividing the imaging area of the imaging means into a plurality of areas; detecting a relative relationship between the plurality of sets of images for each of the plurality of areas Relative relationship detection means, comparison means for comparing the relative relationship detection results for each of the plurality of areas, and operation control means for controlling the operation of the image synthesizing means based on the comparison results of the comparison means. Imaging device. 10. An imaging optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, a plurality of image acquisition units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit; Image synthesizing means for synthesizing images to obtain a set of images; camera shake detecting means for detecting uniform image shake across the entire photographing screen between the plurality of images due to camera shake occurring in the photographing optical means; Subject shake detecting means for detecting image shake occurring in a partial area of the photographing screen between the plurality of images due to movement; controlling the operation of the image synthesizing means based on the detection results of the hand shake detecting means and the subject shake detecting means An image pickup apparatus, comprising: an operation control unit that performs operation control. 11. An imaging optical unit for forming a subject image, an imaging unit for photoelectrically converting the subject image, a plurality of image acquisition units for obtaining a plurality of sets of images at predetermined time intervals by the imaging unit; Image synthesizing means for synthesizing images to obtain a set of images; and camera shake for optically or mechanically correcting uniform image shake between the plurality of images over the entire photographing screen due to camera shake occurring in the photographing optical means. Correction means, subject shake detection means for detecting image shake occurring in a partial area of the shooting screen between the plurality of images due to movement of the subject, and image synthesis means based on the detection result of the subject shake detection means. An imaging apparatus, comprising: an operation control unit that controls an operation. 12. An image synthesizing unit that synthesizes a plurality of sets of images to obtain one set of images, an image blur region detecting unit that detects a partial image blur region generated between the plurality of sets of images, Motion vector detecting means for detecting a motion vector between the plurality of sets of images in the image blur area, wherein the image synthesizing means synthesizes the image blur area by a first method using the motion vector, An image synthesizing apparatus, wherein an area other than the image blur area is synthesized by a second method not using the motion vector. 13. An image synthesizing means for synthesizing a plurality of sets of images to obtain one set of images, a first image area in which a relative relationship between the plurality of sets of images is a first relation, and the first relation Image region dividing means for dividing the image into a second image region having a second relationship different from the image region, wherein the image combining device performs different image combining between the first image region and the second image region. An image synthesizing apparatus characterized by the above-mentioned.