JP6000862B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus such as a digital camera or a video camera.

  In Patent Document 1, for each vertical period of a moving image, a plurality of images captured with an exposure time shorter than one vertical period are shifted and superimposed to output one image, and the focus is based on the image obtained by capturing An imaging apparatus that performs focus adjustment by driving a lens is disclosed.

JP 2007-81477 A

  However, in Patent Document 1, the calculation load increases because image alignment for each photographed frame must be performed for the number of combined images.

  An object of the present invention is to perform highly accurate focus adjustment with a small calculation load.

  An image pickup apparatus according to the present invention includes an image pickup device that photoelectrically converts an optical image, an image alignment unit that aligns a plurality of images obtained from the image pickup device, and an image that is aligned by the image alignment unit. Storage means for storing, image combining means for combining the plurality of images aligned by the image alignment means, and information on overlapping areas of the plurality of images combined by the image combining means A focus state detection unit that detects a focus state based on the image capturing direction, and a detection unit that detects a change in the shooting direction. When the detection unit does not detect the change in the shooting direction, the image alignment unit Aligns the pre-alignment image with the plurality of aligned images stored in the storage means, and the detection means detects the change in the photographing direction. In this case, the image alignment unit is configured to store the image before the alignment on the image before alignment based on an alignment amount for aligning one image stored in the storage unit with the image before alignment. A plurality of registered images stored in the image are aligned.

  According to the present invention, high-precision focus adjustment can be performed with a small calculation load.

It is a block diagram of the imaging device of this embodiment. It is a figure for demonstrating operation | movement of CPU shown in FIG. It is a figure explaining the alignment and synthesis | combination of the image for a focus state detection by CPU shown in FIG. It is a figure explaining the alignment and synthesis | combination of the image for a focus state detection by CPU shown in FIG. 1 when an imaging | photography direction is changed. It is a figure explaining position alignment and composition of an image for recording. It is a flowchart for demonstrating the alignment composition method of the image for a focus state detection. It is a flowchart for demonstrating the registration composition method of the image for recording.

  FIG. 1 is a block diagram of a digital camera (imaging device, optical device) in the present embodiment, and 11 is a photographing optical axis. In the digital camera, the lens barrel 14 is fixed to the camera body 12, but the imaging device may be a digital single-lens reflex camera or a mirrorless camera in which the lens barrel 14 can be attached to and detached from the camera body 12. Moreover, a digital still camera or a digital video camera may be used.

  The lens barrel 14 houses a photographing optical system that forms an optical image of an object (subject). The photographing optical system includes a fixed lens 13a, a variable magnification lens 13b, an automatic focus adjustment (autofocus: AF) lens 13c, and an optical low-pass filter 13d along the optical path from the object side to the image side. The photographing optical system is disposed on the object side of the image sensor 13e. The variable magnification lens (zoom lens) 13b is moved in the optical axis direction to change the focal length. The AF lens 13c is driven in the direction of the optical axis (arrow 15b) by the motor 15a to adjust the focus.

  The shutter 16 is disposed between the zoom lens 13b and the AF lens 13c, and guides the light flux from the subject to the image sensor 13e by opening the shutter blade 17 during an appropriate exposure period. Reference numeral 18 denotes a liquid crystal monitor that is located on the back of the camera body 12 and displays live view display, captured images, and various information. The image sensor 13e is a photoelectric conversion unit that photoelectrically converts an optical image.

  The CPU 19 built in the camera main body 12 is shown outside the camera main body 12 as a block diagram only for elements related to the present embodiment. The CPU 19 is composed of a microcomputer (processor). The CPU 19 includes an image processing unit 110, an exposure control unit 111, an image alignment unit 113, a storage unit 114, a reference image selection unit 115, an image composition unit 117, a reproduction image recording unit 118, and a focus state detection unit 119. The image composition unit 117 includes a reproduction image composition unit 117a and an AF image composition unit 117b.

  The exposure control means 111 performs opening / closing drive control of the shutter blades 17 via the shutter drive means 112. That is, the exposure control unit 111 sets an exposure time suitable for shooting a still image or a moving image based on the subject information from the image processing unit 110. At the time of still image shooting, the shutter blades 17 are opened for a time corresponding to the shooting conditions, and imaging is performed by guiding the subject light flux to the image sensor 13e. During moving image shooting, the exposure control means 111 always keeps the shutter blades 17 open during moving image shooting, and controls an electronic shutter function (not shown) of the image sensor for every given frame length (for example, 1/30 seconds). Update the image.

  The image aligning unit 113 aligns a plurality of images (digital image data obtained by A / D converting the signal output from the image sensor 13e) obtained from the image sensor 13e.

  The storage unit 114 stores a single image 21a and a plurality of images (aligned images) aligned by the image alignment unit 113.

  The shooting state detection unit 116 detects a change in shooting direction.

  The reference selection unit 115 selects which of the plurality of aligned images stored in the storage unit 114 and the image before alignment is used as the alignment reference by the image alignment unit 113. That is, the reference selection unit 115 selects a reference image (or reference line) when aligning a plurality of images. Further, the reference selection unit 115 changes the alignment reference based on the shooting state (for example, change of shooting direction). Further, the reference selection unit 115 selects different reference images for the synthesis of the recording image and the synthesis of the focus state detection image.

  For example, when synthesizing a focus state detection image, it is as follows. That is, the reference selection unit 115 selects a plurality of registered images stored in the storage unit 114 as a reference image when the shooting state detection unit 116 does not detect a change in shooting direction (or a plurality of positions). Select the reference line used to align the aligned images). As a result, when the shooting state detection unit 116 does not detect a change in the shooting direction, the image registration unit 113 adds the image before registration to the plurality of registered images stored in the storage unit 114. Will be aligned.

  On the other hand, when the pre-shooting state detection unit 116 detects a change in the shooting direction, the reference selection unit 115 selects an image before alignment as a reference image (or alignment set to the image before alignment). Select the baseline for). As a result, when the photographing state detection unit 116 detects a change in the photographing direction, the image alignment unit 113 aligns a plurality of alignment images stored in the storage unit with the image before the alignment. It will be. At that time, an alignment amount for aligning one of a plurality of registered images stored in the storage unit with the image before alignment is obtained, and the alignment amount is applied to the remaining aligned images. This reduces the calculation load.

  The image synthesizing unit 117 synthesizes a plurality of images aligned by the image alignment unit 113.

  The focus state detection unit 119 detects the focus state based on the information of the overlapping area of the plurality of images synthesized by the AF image synthesis unit 117b of the image synthesis unit 117. The recording image synthesized by the reproduced image synthesizing unit 117 a is recorded by the reproduced image recording unit 118.

  The image recorded by the reproduction image recording unit 118 is displayed on the liquid crystal monitor 18 via the image reproduction unit 120. The focus state detection unit 119 drives the motor 15a via the AF driving unit 121 and drives the AF lens 13c in the direction of the arrow 15b.

  Hereinafter, the operations of the image alignment unit 113, the storage unit 114, the reference image selection unit 115, the image composition unit 117, the reproduction image recording unit 118, and the focus state detection unit 119 will be described with reference to FIGS.

  In FIG. 2, reference numerals 21a to 21d denote images acquired in time series, and reference numeral 22 denotes a main photographing subject recorded in the images 21a to 21d. If camera shake occurs only in the direction of arrow 23 when images 21a to 21d are acquired, the main subject changes in the direction of arrow 23 as time passes.

  The alignment means 113 shifts the images 21a to 21d so that the positions of the main subjects are aligned. In FIG. 2, the images 21a to 21d are shifted so that the eyes of the main subject 21 are aligned with the reference line 24. Shift like this.

  The image synthesizing unit 117 synthesizes the images 21 a ′ to 21 d ′ aligned by the alignment unit 113 to obtain a synthesized image 21 ′ as indicated by an arrow 25. The composite image 21 ′ indicated by a thick frame is an image obtained by extracting only the overlapping areas of the images 21 a ′ to 21 d ′.

  The focus state detection unit 119 detects the focus state of the photographing optical system 13 from the contrast of the composite image 21 ′ by a known TV-AF technique. That is, focus detection is performed by detecting the peak position of the contrast of the subject image formed by the image sensor 13e while performing a scan that changes the relative position of the focus position formed by the AF lens 13c and the image sensor 13e.

  Next, a process of aligning and synthesizing a plurality of images at predetermined time intervals for focusing will be described with reference to FIG.

  First, the image processing means 110 acquires images 21a and 21b. The image alignment means 113 aligns the image 21b with respect to the subject of the image 21a (reference line 24 with the pupil as a reference) to obtain an image 21b '. The AF image synthesizing unit 117b obtains an image 31a by synthesizing the images 21a and 21b '. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image.

  Next, the image processing means 110 acquires the image 21c. The image alignment means 113 aligns the image 21c with the subject of the image 21a stored in the storage means 114 as a reference (reference line 24 with reference to the pupil) to obtain an image 21c '. The AF image synthesizing unit 117b obtains an image 31b by synthesizing the images 21a and 21b 'stored in the storage unit 114 and the image 21c' currently aligned. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31b.

  Next, the image processing means 110 acquires the image 21d. The image alignment means 113 aligns the image 21d with the subject of the image 21a stored in the storage means 114 as a reference (reference line 24 with reference to the pupil) to obtain an image 21d '. The AF image synthesizing unit 117b obtains an image 31c by synthesizing the images 21a, 21b ', 21c' and the currently aligned image 21d 'stored in the storage unit 114. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31c.

  Next, the image processing means 110 acquires the image 21e. The image alignment means 113 aligns the image 21e with the subject of the image 21b 'stored in the storage means 114 as a reference (reference line 24 with reference to the pupil) to obtain an image 21e'. The AF image synthesizing unit 117b obtains an image 31d by synthesizing the images 21b ', 21c', 21d 'stored in the storage unit 114 and the image 21e' currently aligned. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31d.

  Next, the image processing means 110 acquires the image 21f. The image alignment means 113 aligns the image 21f with the subject of the image 21c 'stored in the storage means 114 as a reference (reference line 24 with reference to the pupil) to obtain an image 21f'. The AF image combining unit 117b combines the images 21c ', 21d', and 21e 'stored in the storage unit 114 and the currently aligned image 21f' to obtain an image 31e. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31e.

  As described above, when the number of aligned images read from the storage unit 114 reaches a predetermined number (four in this embodiment), the CPU 19 reads the registered image read from the storage unit 114 every time an image before alignment is acquired. Update and erase the aligned images in chronological order. As a result, since the total number of images stored in the storage unit 114 does not exceed the predetermined number, the capacity of the storage unit 114 does not become insufficient. In addition, since each alignment image is aligned based on the image 21a, it is not necessary to consider the influence of camera shake when detecting the focus state.

  FIG. 4 is a diagram for explaining the alignment and composition method when the photographing direction is changed (panning or tilting). The description will be continued after obtaining the image 31e of FIG.

  The image processing unit 110 acquires the image 21g. However, it is assumed that the shooting direction is changed at the timing indicated by the vertical line 41. The shooting state detection means 116 detects a change in the shooting direction based on the fact that the amount of motion vector of the image is large or the output of an angular velocity meter mounted on the shooting device. Therefore, the subject composition in the image 21g is significantly different from the previous images (only the head of the main subject is in the composition in the image 21g). When the photographing direction is changed, it is preferable to detect the focus state with reference to the composition.

  When the shooting direction is changed, based on an instruction from the reference selection unit 115, the image alignment unit 113 uses the subject of the image 21g as a reference (reference line 42 with respect to the pupil) as the images 21d ′ and 21e ′. , 21f ′ are aligned to obtain images 21d ″, 21e ″, and 21f ″.

  However, since these three images 21d ′, 21e ′, and 21f ′ have already been aligned, for example, the images 21d ′ and 21e ′ are shifted by the alignment amount obtained by aligning the image 21f ′ with the image 21g. 21d "and 21e" are obtained. Therefore, the load for calculating the alignment amount is small. The AF image synthesizing unit 117b obtains an image 31f by synthesizing the images 21d ", 21e", 21f ", 21g. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31f.

  Next, the image processing means 110 acquires the image 21h. The image aligning unit 113 aligns the image 21h with the subject of the image 21e ″ stored in the storage unit 114 as a reference (reference line 42 based on the pupil) to obtain an image 21h ′. AF image combining unit 117b Synthesizes the images 21e ″, 21f ″, 21g, and 21h ′ to obtain the image 31g. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image 31g.

  Next, the image processing means 110 acquires the image 21i. The image alignment unit 113 aligns the image 21i with the subject of the image 21f ″ stored in the storage unit 114 as a reference (reference line 42 with reference to the pupil) to obtain an image 21i ′. AF image synthesis unit 117b Synthesizes the images 21f ″, 21g, 21h ′, and 21i ′ to obtain the image 31h. The focus state detection unit 119 detects the focus state from the subject information within the thick black frame of the image of the image 31h.

  In this way, when the shooting direction is changed, the focus state detection can always be performed with an appropriate composition by aligning and synthesizing the images before and after the acquired image as a reference.

  3 and 4 show image alignment and composition methods for focus state detection. This method is suitable for focus state detection even when the subject moves from the center of the screen (composition) to the end, and the subject is always in the center of the screen. Is not suitable because there is no movement. Therefore, in the present embodiment, the alignment reference is different between the focus state detection image and the recorded image.

  FIG. 5 is a diagram for explaining registration and composition of recording images.

  First, the image processing means 110 acquires images 21a and 21b. The images 21a and 21b acquired here are the same as the focus state detection images described in FIG. The image alignment means 113 aligns the image 21a with the subject of the image 21b as a reference (reference line 24A with the pupil as a reference) to obtain an image 21a '. The reproduced image synthesizing unit 117a obtains an image 51a by synthesizing the images 21b and 21a '. The image 51a differs from the image 31a obtained in FIG. 3 in the alignment reference line (reference lines 24 and 24A). The reproduction image recording unit 118 records the image 51a, and the image reproduction unit 120 displays the image 51b on the liquid crystal monitor 18.

  Note that the image 51a recorded in the reproduced image recording means 118 is a composite image of the images 21a 'and 21b, and the portion of the image 21a' that extends beyond the black frame is trimmed with reference to the frame of the image 21b indicated by a thick black frame. Since the gain of the image changes in the portion where the images 21a 'and 21b do not overlap within the black frame, the CPU 19 electrically corrects the amount.

  Next, the image processing means 110 acquires the image 21c. The image alignment means 113 aligns the images 21a 'and 21b stored in the storage means 114 with the image 21c as a reference (reference line 24B with reference to the pupil) to obtain images 21a "and 21b'. Since the two images 21a 'and 21b have already been aligned, the calculation for determining the alignment amount when aligning with the image 21c is sufficient. The reproduced image combining unit 117a combines the images 21a "and 21b 'and the image 21c stored in the storage unit 114 to obtain the image 51b. The reproduced image recording unit 118 records the image 51b, and the image reproducing unit 120 displays the image 51b on the liquid crystal monitor 18.

  Note that the image 51b recorded in the reproduced image recording unit 118 is a composite image of the images 21a '', 21b ', and 21c, and the black frame in the images 21a' 'and 21b' is based on the frame of the image 21c indicated by a thick black frame. The part that protrudes is trimmed. The portion of the black frame where the images 21 a ″, 21 b ′, and 21 c do not overlap has an image gain change, so the CPU 19 electrically corrects that amount.

  Next, the image processing means 110 acquires the image 21d. The image aligning unit 113 aligns the images 21a ″, 21b ′, and 21c stored in the storage unit 114 with the image 21d as a reference (a reference line 24C with respect to the pupil) to obtain an image 21a ′ ″, 21b '' and 21c 'are obtained. Since the three images 21 a ″, 21 b ′, and 21 c have already been aligned, the calculation for determining the alignment amount when aligning with the image 21 d is sufficient. The reproduced image synthesizing unit 117a obtains an image 51c by synthesizing the images 21a "" and 21b "and the images 21c 'and 21d. The reproduced image recording unit 118 records the image 51c, and the image reproducing unit 120 displays the image 51c on the liquid crystal monitor 18.

  Note that the image 51c recorded in the reproduced image recording unit 118 is a composite image of the images 21a ′ ″, 21b ″, 21c ′, and 21d, and the image 21a ″ is based on the frame of the image 21d indicated by a thick black frame. The portion that protrudes from the black frame in ', 21b' ', 21c is trimmed. The portion of the black frame where the images 21a "" and 21b ", 21c ', 21d do not overlap changes the gain of the image, so the CPU 19 electrically corrects that amount.

  Next, the image processing means 110 acquires the image 21e. The image aligning unit 113 aligns the images 21b ″, 21c ′, and 21d stored in the storage unit 114 with the image 21e as a reference (a reference line 24D based on the pupil), thereby obtaining an image 21b ′ ″, 21c '' and 21d 'are obtained. Since the three images 21 b ″, 21 c ′, and 21 d have already been aligned, it is only necessary to calculate the alignment amount when aligning with the image 21 e. The reproduced image synthesizing unit 117a synthesizes the images 21b '' and 21c '' and the images 21d 'and 21e to obtain the image 51d. The reproduction image recording unit 118 records the image 51d, and the image reproduction unit 120 displays the image 51d on the liquid crystal monitor 18.

  Note that the image 51d recorded in the reproduced image recording unit 118 is a composite image of the images 21b ″ ′ and 21c ″, 21d ′, and 21e. The images 21b ″ ′ and 21c are based on the frame of the image 21e indicated by a thick black frame. ”, The portion that protrudes from the black frame at 21d ′ is trimmed. Since the gain of the image changes in a portion where the images 21 b ″ ″ and 21 c ″, 21 d ′, 21 e do not overlap within the black frame, the CPU 19 electrically corrects that amount.

  Next, the image processing means 110 acquires the image 21f. The image aligning unit 113 aligns the images 21c ″, 21d ′, and 21e stored in the storage unit 114 with the image 21f as a reference (a reference line 24E with respect to the pupil) to obtain an image 21c ′ ″, 21d '' and 21e 'are obtained. Since the three images 21 c ″, 21 d ′, and 21 e have already been aligned, it is only necessary to calculate the alignment amount when aligning with the image 21 f. The reproduction image combining unit 117a combines the images 21c '' and 21d '' and the images 21e 'and 21f to obtain the image 51e. The reproduction image recording unit 118 records the image 51e, and the image reproduction unit 120 displays the image 51e on the liquid crystal monitor 18.

  Note that the image 51e recorded in the reproduced image recording means 118 is a composite image of the images 21c ″ ′, 21d ″, 21e ′, and 21f. The images 21c ″ ′ and 21d are based on the frame of the image 21f indicated by a thick black frame. ", The portion of the frame 21e 'that extends beyond the black frame is trimmed. The portion of the black frame where the images 21 c ″ ″ and 21 d ″, 21 e ′, 21 f do not overlap has a change in the image gain, so the CPU 19 electrically corrects that amount.

  When the number of registered images read from the storage unit 114 reaches a predetermined number, the CPU 19 updates the registered image read from the storage unit 114 each time the pre-alignment image is acquired, and deletes the registered images in chronological order. . As a result, the total number of images stored in the storage unit 114 does not exceed a predetermined number.

  Unlike the case of focus state detection, the last acquired image is used as a reference, and a plurality of images are aligned in the storage unit 114, synthesized, and recorded. In the alignment, since the plurality of images stored in the recording unit 114 have already been aligned, the calculation of the alignment amount with respect to the reference image is performed for any one of the plurality of images, and the rest Therefore, the calculation load is small. In this embodiment, the image for obtaining the alignment amount is an image acquired at the timing closest to the reference image in time, but is not limited to this.

  FIG. 6 is a flowchart for explaining the alignment and composition of the focus state detection image executed by the CPU 19, and this flow starts with the start of moving image shooting. The alignment composition method shown in FIG. 6 can be embodied as a program for causing a computer to realize the function of each step, and the same applies to FIG.

  In step # 601, the image processing unit 110 acquires an image. In step # 602, the image stored in the storage unit 114 is read out. In step # 603, the imaging state detection unit 116 determines whether or not the camera is in the panning state from the vibration detection unit output such as the image signal and the angular velocity meter. If the camera is not in the panning state, the process proceeds to step # 604. If so, the process proceeds to step # 609.

  In step # 604, the image alignment unit 113 aligns the latest acquired image with reference to the image read in step # 602. In step # 605, the latest image aligned by the image alignment unit 113 is stored in the storage unit 114. In step # 606, the AF image compositing means 117b obtains the latest image and the image read out in step # 602, and the image for which the alignment has been completed is combined into an image for focus state detection.

  In step # 607, the focus state detection unit 119 detects the focus state from the subject information of the focus state detection image. In step # 608, the AF lens 13c is driven to detect the next focus state or to focus in accordance with the detected focus state, and the process returns to step # 601.

  If it is detected in step # 603 that the camera is in a panning state, in step # 609, the image is read in step # 602 based on the latest image acquired by the image alignment unit 113 in accordance with an instruction from the reference selection unit 115. Align images. Thereafter, step # 605 and subsequent steps are performed.

  FIG. 7 is a flowchart for explaining registration and composition of the recording image executed by the CPU 19, and this flow starts with the start of moving image shooting. Steps similar to those in FIG. 6 are denoted by the same reference numerals. After step # 601 and step # 602, step # 609 is performed, and then step # 605 is performed. Next, in step # 701, the reproduction image synthesizing unit 117a synthesizes the image read out in step # 602 and the latest acquired image to form a recording image. Next, in step # 702, the recording image created in step # 701 is recorded in the reproduction image recording means 118.

  In this way, the image 51 synthesized by the reproduction image synthesizing unit 117a is a synthesis of a plurality of different images, so there is little noise, and there is a movement because the alignment reference is changed every time an image is acquired. Can be a video.

  On the other hand, the image 31 synthesized by the AF image synthesizing unit 117b is also composed of a plurality of images, so that there is little noise, and since the first acquired image is used as the alignment reference, there is no movement in the focus state detection It is possible to make an image suitable for. Further, when panning or tilting occurs, it is possible to detect the focus state adapted to the new composition by switching the alignment reference.

  As mentioned above, although this embodiment was described, this invention is not limited to this embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The imaging device can be applied to a digital camera or a digital video.

13e ... Image sensor, 19 ... CPU, 113 ... Image alignment means, 114 ... Storage means, 116 ... Shooting state detection means, 117 ... Image composition means, 117a ... Reproduction image composition means, 117b ... AF image composition means, 118 ... Reproduced image recording means, 119 ... focus state detecting means

Claims (5)

An image sensor that photoelectrically converts an optical image;
Image alignment means for aligning a plurality of images obtained from the image sensor;
Storage means for storing the image aligned by the image alignment means;
Image combining means for combining the plurality of images aligned by the image alignment means;
A focus state detecting means for detecting a focus state based on information of overlapping regions of the plurality of images synthesized by the image synthesizing means;
Detection means for detecting a change in the shooting direction;
Have
If the detection means does not detect the change in the shooting direction, the image alignment means aligns the image before alignment with the plurality of aligned images stored in the storage means,
When the detection unit detects a change in the photographing direction, the image alignment unit is based on an alignment amount for aligning one image stored in the storage unit with the image before the alignment. An image pickup apparatus characterized by aligning a plurality of aligned images stored in the storage unit with the image before alignment. It further has a recording means for recording an image for recording,
The image alignment means is stored in the storage means in the image before alignment based on an alignment amount for aligning one image stored in the storage means with the image before alignment. 2. The imaging apparatus according to claim 1, wherein a plurality of registered images are aligned, and the recording unit records the image combined by the image combining unit as the recording image.   The recording image recorded in the recording unit has a frame of the image before the alignment, and a portion protruding from the frame of the plurality of images stored in the storage unit is trimmed. The imaging device according to claim 2.   The imaging apparatus according to claim 3, wherein a gain of a portion where a plurality of images do not overlap within a frame of the image before alignment in the recording image is corrected. When alignment has been image stored in said storage means reaches a predetermined number, wherein said each time to get the alignment previous image, characterized by also erase registration-completed image to be read out from said memory means in chronological order Item 5. The imaging device according to any one of Items 1 to 4.