JP2013085140A - Imaging apparatus and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a composite image with little blurring by accurately aligning plural images even when moving bright points such as shooting-up firework are photographed as being handheld.SOLUTION: An imaging apparatus includes: plural image acquisition means 20 which sequentially takes photographs and acquires plural images; bright point extraction means 204 which extracts bright points in plural taken images; bright point determination means 20 which makes a determination on whether a bright point is a moving bright point or a stationary bright point from the size of the bright point extracted by the bright point extraction means; motion vector calculation means 201 which calculates a motion vector in plural images with the stationary bright point as a reference determined by the bright point determination means; image composition means 202 which forms a composite image on the basis of the motion vector calculated by the motion vector calculation means.

Description

  The present invention relates to an imaging apparatus having a function of synthesizing a plurality of captured images and capturing an image without shake, and a control method thereof.

  When shooting fireworks, it is difficult to shoot with the aim of the moment when the fireworks open, so the camera is usually fixed and long exposure is performed.

  In general, since fireworks shine for about 2 to 3 seconds, it is difficult to shoot by hand without fixing the camera.

  Japanese Patent Application Laid-Open No. 2004-151561 describes a photographing method for combining a plurality of images taken under different exposure conditions to expand the dynamic range, and further, combining the images while correcting misalignment between the images. An imaging method is disclosed.

  In Patent Document 2, a detection area for detecting a motion vector between a plurality of images is set in advance by an image acquired from an output signal from an image sensor before the start of shooting, and a photographer can A technique is disclosed that enables a plurality of images to be accurately aligned by shooting including a subject that is a reference point.

Japanese Patent No. 3110797 Japanese Patent No. 4324234

  2. Description of the Related Art Conventionally, there has been proposed a photographing method that performs alignment by adding correlations of images photographed a plurality of times continuously during long-time exposure, and performs addition synthesis.

  However, since sparks spread radially, it has been a problem to accurately align a plurality of images taken continuously.

  As one method for solving this problem, a method is known in which a feature point for aligning a plurality of images is set as a subject other than fireworks, and a photographer sets a detection area including the feature point in advance. Yes.

  However, since it is necessary to set a detection area for detecting a feature point in advance before shooting, it is necessary to set a detection area including the feature point every time the angle of view or framing is changed.

(Object of invention)
An object of the present invention is to provide an imaging apparatus capable of obtaining a composite image without shake by accurately aligning and synthesizing a plurality of images even when hand-held shooting of a moving bright spot such as a fireworks display and its It is to provide a control method.

  In order to achieve the above object, an imaging apparatus according to the present invention includes a plurality of image acquisition means for acquiring a plurality of images by sequentially photographing, a bright spot extraction means for extracting a bright spot appearing in the plurality of images, Based on the size of the bright spot extracted by the bright spot extracting means, the bright spot determining means for determining whether the bright spot is a moving bright spot or a stationary bright spot, and the stationary bright spot determined by the bright spot determining means as a reference A motion vector calculating unit that calculates a motion vector between the plurality of images, and an image combining unit that combines a single composite image based on the motion vector calculated by the motion vector calculating unit. It is what.

  According to the present invention, even if a moving bright spot such as a fireworks display is photographed by hand, a composite image without shake can be obtained by accurately aligning and synthesizing a plurality of images.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 6 is a diagram illustrating image composition processing in Embodiment 1. It is a figure explaining the process which extracts the stationary luminescent spot in Example 1 from the picked-up image. 3 is a flowchart illustrating a firework shooting operation of the first embodiment. 3 is a flowchart illustrating a feature point detection operation according to the first exemplary embodiment. 3 is a flowchart illustrating a feature point detection region setting operation according to the first embodiment. It is a block diagram which shows schematic structure of the imaging device of the compound eye which concerns on Example 2 of this invention. It is a figure explaining the exposure timing of a prior art and Example 1 and 2. FIG. FIG. 10 is a diagram illustrating an image composition process in Embodiment 2. 6 is a flowchart illustrating a firework shooting operation of the second embodiment.

  The mode for carrying out the invention is as described in Examples 1 and 2 below.

  FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera as an imaging apparatus according to Embodiment 1 of the present invention.

  In FIG. 1, an imaging apparatus 100 includes a variable power lens (hereinafter referred to as zoom lens) 10, a focus adjustment lens (hereinafter referred to as focus lens) 12, an aperture shutter unit 13 incorporating an aperture and a shutter, and converts an optical image into an electrical signal. The imaging device 14 is provided.

  The imaging apparatus 100 further includes a gain amplifier 15 that amplifies the analog signal output of the image sensor 14 and sets the sensitivity of the image sensor 14, and an A / D converter 16 that converts the analog signal output of the image sensor 14 into a digital signal. Prepare.

  In addition, the imaging apparatus 100 includes a timing generation circuit 18 that supplies a clock signal and a control signal to the imaging element 14, the A / D converter 16, and the D / A converter 26. The timing generation circuit 18 is controlled by the memory control circuit 22 and the system control circuit 50.

  In addition, the imaging apparatus 100 includes an image processing circuit 20. The image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22.

  Further, the image processing circuit 20 performs a predetermined calculation process using the captured image data. The system control circuit 50 controls the exposure control circuit 40 and the focus adjustment circuit 42 on the basis of the obtained calculation result. AF (auto focus) processing of TTL (through the lens) method, AE (automatic) Exposure) processing and EF (flash pre-emission) processing are performed.

  Further, the image processing circuit 20 performs predetermined arithmetic processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained arithmetic result.

  The image processing circuit 20 also extracts a feature point detection unit 200 that extracts a common feature point for each of a plurality of images taken in succession, and adds the image by calculating a motion vector based on the feature point. A motion vector calculation unit 201 and an image synthesis processing unit 202 that perform a synthesis process are provided. The motion vector calculation unit 201 constitutes a motion vector calculation unit.

  The feature point detection processing unit 200 includes a feature point detection region setting processing unit 203 that sets, in each image, a detection region including feature points suitable for adding and combining a plurality of images, and a feature point detection region setting processing unit. A bright spot extraction processing unit 204 that extracts bright spots as feature points from the detection area set in 203 is provided. Note that the feature point detection processing unit 200 extracts feature points. However, if an effect can be obtained, the feature point detection processing unit 200 may use another vector detection method such as a block matching method instead of feature point extraction. good.

  With respect to the feature point detection processing unit 200, it is determined whether to execute the process according to the shooting mode set by the operation unit 63.

  The memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32.

  The data of the A / D converter 16 is sent via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is sent directly via the memory control circuit 22 and the image display memory 24 or the memory 30. Is written to.

  A system control circuit 50 that controls the entire imaging apparatus 100 calculates an appropriate exposure value based on the brightness level measured by the TTL via the memory control circuit 22 and controls the exposure control circuit 40.

  The display image data written in the image display memory 24 is displayed on the image display unit 28 including a TFT, an LCD, and the like via the D / A converter 26. If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized.

  The memory 30 is for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. This makes it possible to write a large amount of images to the memory 30 at high speed even in panoramic shooting and continuous shooting in which a plurality of still images are continuously shot.

  The memory 30 can also be used as a work area for the system control circuit 50. The memory 30 functions as a storage unit that stores relative information of the focus adjustment circuit 42 with respect to the operation of the zoom control circuit 44 as a scaling unit for scaling the subject image.

  A compression / decompression circuit 32 that compresses / decompresses image data by adaptive discrete cosine transform (ADCT) or the like reads an image stored in the memory 30, performs compression processing or expansion processing, and writes the processed data to the memory 30.

  The memory 66 stores constants, variables, programs, etc. for operating the system control circuit 50. The exposure control circuit 40 controls the aperture shutter unit 13 having an aperture function and a shutter function.

  The focus adjustment circuit 42 controls focusing of the focus lens 12. The zoom control circuit 44 controls zooming of the zoom lens 10.

  The exposure control circuit 40 and the focus adjustment circuit 42 are controlled using the TTL method, and the system control circuit 50 performs the exposure control circuit 40 and the focus adjustment based on the calculation result obtained by calculating the captured image data by the image processing circuit 20. The circuit 42 is controlled.

  The display unit 64 includes a liquid crystal display panel, a speaker, and the like that display an operation state, a message, and the like using characters, images, sounds, and the like according to execution of a program by the system control circuit 50. One or a plurality of display units 64 are installed at positions where the display unit 64 is easily visible in the vicinity of the operation unit 63 of the imaging apparatus 100.

  The electrically erasable / recordable nonvolatile memory 65 is, for example, a flash ROM.

  Next, the operation members 60 and 61 and the operation unit 63 for inputting various operation instructions of the system control circuit 50 are configured by switches and dials. Here, a specific description thereof will be given.

  The shutter switch SW1 (60) is turned on during operation of a shutter switch member (not shown), and AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (strobe preflash) processing. Instruct the start of shooting preparation operation.

  The shutter switch SW2 (61) is turned on when the operation of a shutter switch member (not shown) is completed, and instructs the start of a series of processes.

  The series of processes is an exposure process in which a signal read from the image sensor 14 is written to the memory 30 via the A / D converter 16 and the memory control circuit 22, and further an arithmetic operation in the image processing circuit 20 and the memory control circuit 22. Development processing using The series of processes is a recording process in which image data is read from the memory 30, compressed by the compression / decompression circuit 32, and written to the recording medium 70.

  The operation unit 63 including various buttons and a touch panel can be used for various shooting scenes such as an auto mode, a program mode, an aperture priority mode, a shutter speed priority mode, a night view mode, a child shooting mode, a fireworks shooting mode, and an underwater shooting mode. You can select the appropriate setting.

  The flash light emitting unit 62 controls the light emission by calculating an appropriate light emission amount based on the luminance level measured by the system control circuit 50.

  The power supply control circuit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like.

  The power source 86 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The power supply control circuit 80 and the power supply 86 are connected via connectors 82 and 84.

  The recording medium 70 includes a recording unit 73 composed of a semiconductor memory, a magnetic disk, or the like, an interface (I / F) 72, and a connector 71. The interface 72 is connected to the interface 90 of the imaging device 100 via the connector 71 and the connector 92 of the imaging device 100.

  Hereinafter, the operation of the imaging apparatus 100 will be described.

  In general, when the subject is dark, in order to obtain a proper exposure with a single exposure, the aperture is opened and the exposure time is long exposure.

  In this case, if the image is taken by hand, the photograph will be shaken. Therefore, in many cases, the exposure is increased to shorten the exposure time as much as possible. However, when shooting with increased sensitivity, the image quality deteriorates due to the influence of the dark current of the image sensor 14 and the noise of the gain amplifier 15 that amplifies the signal of the image sensor 14.

  In view of this, a method for capturing an image that has little image quality degradation and is not shaken even with a dark subject has been proposed. First, there is an imaging method in which the exposure time is divided in such a short time as not to cause camera shake, and a plurality of images taken continuously by the divided number are added and combined while being aligned.

  However, when the subject moves so as to spread radially like a fireworks display, it has been difficult to accurately align the images to be synthesized with this photographing method.

  The following is a method based on image composition addition photography in which a plurality of images taken with a short exposure time are added and synthesized while being aligned, and the imaging apparatus 100 synthesizes an image at a desired position without shaking the fireworks. explain.

  FIG. 2 is an example of a plurality of images acquired by sequentially shooting up fireworks that are moving bright spots by image composition addition shooting. A more natural fireworks image can be obtained by shooting with a shooting interval as short as possible.

  2A to 2D are diagrams in which images are taken in time series when continuous shooting is performed from when the fireworks are launched until the fireworks are most open.

  FIG. 2A is a photographed image immediately after the fireworks spread, and the sparks are hardly spread. FIG. 2 (b) spreads a little, and the fireworks spread greatly with time as shown in FIG. 2 (c) and FIG. 2 (d).

  Here, when performing addition synthesis for each image, it is clear that it is difficult to use fireworks as feature points that serve as a reference for image alignment because the size and shape of the sparks of the fireworks are different.

  FIG. 2E shows an example of an image that has failed due to position alignment when the images are combined. An example is shown in which fireworks in each image are aligned as feature points, but cannot be accurately combined.

  In order to accurately align the images of (a) to (d) in FIG. 2, it is better to use the light (stationary luminescent spot) of the building window reflected in the background rather than the fireworks (moving luminescent spot) as feature points. Obviously, until now, fireworks and night lights often overlapped, and it has been difficult to extract and process only the bright spots of night scenes.

  In the first embodiment, accurate positioning is possible by extracting bright points of a background night scene reflected in a place other than fireworks without using fireworks as feature points and using them as feature points of image synthesis.

  First, grid-like lines that are overwritten in the images (a) to (d) of FIG. 2 are image division areas. When combining the images, feature point detection area setting and motion vector calculation are performed in units of the divided areas.

  Note that the division size and the number of divisions of the divided area may be dynamically changed depending on the image size, the focal length of the imaging optical system, the ratio of the subject on the screen, and the like.

  As a feature point detection area setting method of the first embodiment, the captured image is the image with the lowest average luminance of the screen in FIG. 2 (a) to FIG. 2 (d) and the average of the screen. The image with high brightness is compared with FIG.

  When the average brightness of the screen is the lowest, there is a high possibility that it is just before or after the fireworks are launched, and it has a feature that it is easy to extract a bright spot such as lighting that shines in the night view.

  Here, when there are a plurality of images having the lowest average luminance of the images, an image with a fast shooting order is selected.

  On the other hand, when the average brightness of the image is the highest, there is a high possibility that the fireworks are in the largest spread state, and it is easy to extract areas where the sparks of the fireworks do not overlap with the bright spots such as lighting that shines in the night view There is. Here, when there are a plurality of images having the highest average luminance of the images, an image with a slow shooting order is selected.

  In these two images, a divided area in which the luminance level difference between the corresponding divided areas is smaller than a predetermined threshold (second threshold) and each luminance level is larger than the predetermined threshold (first threshold). When extracted, it is possible to set a feature point detection region for all images taken continuously.

  Here, there are cases where a clear image may not be obtained in the peripheral area of the image due to lens distortion or lack of peripheral light quantity, so it is not suitable for accurately aligning and synthesizing the image even if feature points are detected. is there. Therefore, the peripheral area or the four corner areas of the image may not be set as the feature point detection area.

  When the feature point detection areas are set for (a) to (d) in FIG. 2 under the above conditions, the four divided areas enclosed in the figure can be used as the feature point extraction areas.

  Depending on the photographed composition, a bright point such as a night view illumination may not be included, and therefore a feature point detection region may not be set. In this case, the entire area of the screen or the central divided area excluding the divided areas of the peripheral area may be set as the feature point detection area. May be stored in the recording unit.

When it is determined that it is difficult to accurately align a plurality of continuous images, it is desirable to store only one image having the best condition among the plurality of continuous images. The image with the highest average luminance of the image is likely to be a state where the fireworks are most widely opened or a moment when a plurality of fireworks are reflected on the entire screen. Therefore, by selecting an image having the highest average luminance of the images from the continuous images, it can be expected that the probability of storing the failed photo is reduced.
FIG. 2F is an example of an image that has been accurately aligned when the images are combined.

  Next, a stationary bright spot is detected as a feature point from the feature point detection area set for each image. A process of determining a moving bright spot (moving bright spot) like a spark of fireworks and a bright spot of a subject that does not move (still bright spot) in the feature point detection area is performed.

FIG. 3 is a diagram illustrating an example of a spark trajectory α of fireworks and a light β of a building window.
When the fireworks are launched, the sparks expand radially, so if the image is taken with an exposure time longer than a predetermined time, the locus of the spark is photographed in a narrow shape (α in FIG. 3).
On the other hand, in the case of the light β of the building window, the shape of the bright spot does not change regardless of the exposure time.

In the first embodiment, utilizing this feature, a bright spot determination method for determining bright spots of a moving subject and a stationary subject is proposed.
For the bright spot extracted from the image, the number of pixels constituting the bright spot and the peripheral length of the bright spot are detected and used as an index of size.

Here, a bright spot smaller than a predetermined size (threshold) with respect to the extracted bright spot size is determined as a stationary bright spot.
In addition to the size of the bright spot, the bright spot length of the subject may be extracted more accurately by adding the length of the bright spot to the determination condition.

  First, the vertical and horizontal lengths of the bright spots are detected from the captured image. Thereby, the length of the bright spot is calculated.

C = √ (A ² + B ² )
c = √ (a ² + b ² )

  If the lengths of C and c are shorter than the predetermined length D, it is determined as a feature point, and if it is longer, it is determined not to be a feature point. At this time, the predetermined length D is determined according to the captured image size and the exposure time at the time of shooting.

C ≦ D, c ≦ D: C and c are feature points C> D, c> D: C and c are not feature points

  Furthermore, although it has been described that the bright spot length determination condition is shorter than the predetermined length D, a predetermined length E shorter than the predetermined length D is set, and the bright spot length is longer than E. May be added to the determination condition.

E ≦ C ≦ D, E ≦ c ≦ D: C and c are feature points C> D, c> D: C and c are not feature points Calculate motion vectors of feature points detected between consecutive images, According to the motion vector, the position of each image is matched and combined.

  Thus, it is possible to shoot an image with little image quality degradation without hand shaking even during hand-held shooting from when the fireworks are launched until the fireworks spread the most.

Hereinafter, the flow of the above process will be described with reference to FIG.
FIG. 4 is a diagram illustrating a firework shooting sequence according to the first embodiment. The shooting mode of the imaging apparatus 100 is set to the fireworks shooting mode, and the shutter switch 60 (SW1) is pressed (S100). When the shutter switch 60 is pressed, the brightness of the subject is measured to prepare for shooting. Here, the exposure time is determined based on the photometric result, and the number n of consecutive shots for image synthesis is determined (S101). When shooting starts (S102), shooting continues until the shooting counter reaches n (No in S103). The image processing circuit 20 or the like that executes step S102 corresponds to a multiple image acquisition unit. When the number of shots reaches n (Yes in S103), feature point detection processing for image composition is performed (S104). The feature point detection process will be described later.

  At this time, if the feature point can be detected (Yes in S105), a motion vector is calculated based on the feature point of each image (S106), and the calculated motion vector is used for alignment of each image to synthesize an image ( S107).

  When the feature point cannot be detected (No in S105), an image with the maximum average luminance of the image is extracted from n consecutively captured images and recorded on the recording medium 70 (S108).

The feature point detection process performed in step S104 described above will be described in detail with reference to FIG.
In step S200, the bright spot extracted from the continuously photographed image is analyzed into a bright spot moving like fireworks and a stationary bright spot such as a night scene illumination (S201).

  First, the size of the bright spot is determined. At this time, the size of the bright spot is the number of pixels constituting one bright spot, or the size of the bright spot is expressed by expressing the length around the bright spot by the number of pixels. If the size of the bright spot is less than or equal to a predetermined size (Yes in S202), the length of the bright spot is then calculated. The calculation method calculates the length from the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the bright spot, and if it is equal to or shorter than the predetermined length (Yes in S203), it is determined as a feature point (S204). If the size or the length of the bright spot is larger than the predetermined number of pixels (No in S202, No in S203), this bright spot is not determined as a feature point (S206). The image processing circuit 20 that executes steps S202 and S203 corresponds to a bright spot determination unit.

  A series of these processes is performed on n consecutively photographed images, and the process proceeds to the next process according to the detection result of the feature points in step S105 of FIG. Here, the determination process of feature point detection may be determined only by the length of the bright spot, may be determined only by the size of the bright spot, or may be determined by another color such as the color or brightness of the bright spot. You may judge on different conditions.

  Further, in the bright spot extraction process performed in step S200 of FIG. 5, a detection area in which the feature point is included in the image is set from the captured image, and the feature point is detected from the detection area, thereby speeding up the process. May be performed.

This feature point detection area setting method will be described in detail with reference to FIG.
In step S300 of FIG. 6, the average luminance is calculated for n consecutively photographed images. Among these, an image with the lowest average luminance (S301) and an image with the highest average luminance (S302) are extracted, and each image is divided into a plurality of divided areas (S303). Next, there is a problem that the accuracy of the feature points detected from the divided areas in the peripheral part of the image is lowered when the images are aligned due to the influence of the distortion of the image or the decrease in resolution. For this reason, it is good also as an invalid area | region not using for a feature point detection (S304). The image of the minimum average luminance and the image of the maximum average luminance extracted in step S301 and step S302 are compared for each divided area (S305), and a divided area with a small luminance change between the two images is extracted (S306). . If the extracted image of the divided area with a small luminance change is equal to or higher than the predetermined luminance (Yes in S307), this divided area is set as the feature point detection area (S308), and the extracted image of the divided area with a small luminance change is displayed. If it is less than the predetermined luminance (No in S307), this divided area is not set as the feature point detection area.

  Although the above-described first embodiment realizes a shooting method that does not shake even if the fireworks are shot by hand, depending on the type of fireworks, there is a time during which no exposure is performed between images that have been continuously shot. May be taken without being continuously connected. Therefore, a method for capturing fireworks by the imaging apparatus according to the second embodiment will be described.

  FIG. 7 is a block diagram illustrating a schematic configuration of an imaging apparatus including a plurality of imaging optical systems according to the second embodiment of the present invention.

In FIG. 7, the imaging apparatus 400 includes a main imaging optical system 300 and a sub imaging optical system 301. Here, one imaging optical system is illustrated as the sub imaging optical system 301, but there is no problem even if a plurality of sub imaging optical systems are provided.
The main imaging optical system 300 has the same configuration as the imaging optical system in FIG.

  The main imaging optical system 300 includes a variable power lens (hereinafter referred to as zoom lens) 10, a focus adjustment lens (hereinafter referred to as focus lens) 12, an aperture shutter unit 13 in which an aperture and a shutter are incorporated, and imaging that converts an optical image into an electrical signal. An element 14 is provided.

  Further, a gain amplifier 15 that amplifies the analog signal output of the image sensor 14 and sets the sensitivity of the image sensor 14, and an A / D converter 16 that converts the analog signal output of the image sensor 14 into a digital signal are provided.

  The sub-imaging optical system 301 includes a variable power lens (hereinafter referred to as zoom lens) 310, a focus adjustment lens (hereinafter referred to as focus lens) 312, an aperture shutter unit 313 in which an aperture and a shutter are incorporated, and imaging that converts an optical image into an electrical signal. An element 314 is provided.

  In addition, a gain amplifier 315 that amplifies the analog signal output of the image sensor 314 to set the sensitivity of the image sensor 314 and an A / D converter 316 that converts the analog signal output of the image sensor 314 into a digital signal are provided.

  The system control circuit 50 that controls the entire imaging apparatus 400 calculates an appropriate exposure value based on the luminance level measured by the TTL via the memory control circuit 22 and controls the exposure control circuit 340.

  The exposure control circuit 340 controls an aperture shutter unit 313 having an aperture function and a shutter function and a gain amplifier 315 that sets the sensitivity of the image sensor 314 by amplifying the analog signal output of the image sensor 314 (AE processing).

  The system control circuit 50 controls focusing of the focus lens 312 with respect to the focus adjustment circuit 342 using information calculated from the image data captured by the image processing circuit 20 (AF processing).

Here, when AF processing or AE processing is performed in the main imaging optical system 300, the sub imaging optical system 301 may be controlled using the processing result of the main imaging optical system 300.
The zoom control circuit 344 controls zooming of the zoom lens 310.

  The memory 30 functions as a storage unit that stores relative information of the focus adjustment circuit 342 with respect to the operation of the zoom control circuit 344 as a scaling unit for scaling the subject image.

  The system configuring the imaging apparatus 400 in FIG. 7 differs from the imaging apparatus 100 in FIG. 1 in the configuration relating to the control of the sub imaging optical system 301 and the sub imaging optical system, but for other configurations, the imaging apparatus in FIG. The same as 100.

  FIG. 8 is a chart of exposure timing of the photographing method according to the second embodiment. In the conventional fireworks photography method, exposure is started before the fireworks are launched, and the exposure is finished so that the moment when the fireworks are most open is included (exposure chart a). For this reason, the exposure time is necessarily long. However, in the image composition shooting by the image pickup optical system according to the first embodiment of the present invention (exposure chart b), continuous shooting is performed at a shutter speed that does not shake and that allows a firework locus to be captured. As a result, similarly to the conventional fireworks photography method, it is possible to fit the fireworks up to the most open moment in one photographed image without shaking.

  However, when continuous shooting is performed by the imaging apparatus 100, there is a time during which exposure is not performed. For this reason, depending on the type of fireworks, a break occurs during the time when the exposure is not performed.

  Therefore, an example of an imaging method using the imaging apparatus 400 including the plurality of imaging optical systems 300 and 301 will be described. As shown in the exposure chart c, the exposure of the main imaging optical system 300 is started, and there is no camera shake. Take a photo with an exposure time sufficient to show the trail of fireworks. The sub imaging optical system 301 starts exposure in synchronization with the exposure end timing of the main imaging optical system 300. Similarly, exposure of the main imaging optical system 300 is started in synchronization with the timing of completion of exposure of the sub imaging optical system 301. As described above, by repeatedly taking pictures in accordance with the exposure timings of the plurality of imaging optical systems, it becomes possible to take pictures without interruption of exposure during fireworks photography.

  A continuous image captured by a plurality of imaging optical systems is detected by detecting a feature point of each image, calculating a motion vector, and synthesizing and aligning the images in the same manner as the image composition photographing by the image capturing apparatus 100. One image is obtained.

  In the above description, in the image composition shooting by the compound eye imaging optical system, the timing of the exposure end and the exposure start is adjusted so as not to be interrupted. However, if the exposure timing overlaps, the overlapped portion in the composite image becomes bright. Do not overlap the exposure period.

  Although not shown, in an imaging apparatus including a compound-eye imaging optical system, parallax occurs in an image due to binocular parallax even when images are taken with the same angle of view. Since the angle of convergence is negligible in fireworks shooting, the convergence angle can be ignored, but when more precise alignment is required, such as when shooting with a telephoto lens, the effect of parallax is corrected by image correction such as affine transformation. It may be solved by.

  FIG. 9 is a diagram showing an example for explaining a composite image FIG. 9A photographed by the main imaging optical system 300 and a composite image FIG. 9B photographed by the sub-imaging optical system 301. 9 (a) and FIG. 9 (b) are aligned and combined, it is possible to obtain a seamless fireworks display image as shown in FIG. 9 (c).

  Here, it has been described that a composite image of the main imaging optical system 300 and a composite image of the sub imaging optical system 301 are combined to complete one captured image. A plurality of images captured by the imaging optical system 301 may be arranged in time series and the respective images may be aligned.

Hereinafter, an example of the flow of the above processing will be described with reference to FIG.
FIG. 10 is a diagram illustrating a firework shooting sequence using the compound-eye imaging optical system according to the second embodiment.
In the fireworks shooting mode, the shutter switch 60 is pressed (S400), and photometry is performed. Based on the photometric result, the number of images to be captured and the shutter speed are determined (S401). As an example of processing, it is assumed that n images are captured by the main imaging optical system 300 and m images are captured by the sub imaging optical system 301. At this time, the number of shots of each imaging optical system is m = n or m = n + 1.

  Exposure is started by the main imaging optical system 300 (S402), and exposure is performed at the shutter speed determined in step S401. When the exposure of the main imaging optical system 300 is finished (S403), the exposure of the sub imaging optical system 301 is started so that the exposure is not interrupted (S405). When the exposure is finished in the sub imaging optical system 301 (S406), if the number of shots has not reached m (N0 in S407), the main imaging optical system 300 starts exposure so that the exposure is not interrupted (S403). A series of these processes is repeated, and when the number of shots of the sub-imaging optical system 301 reaches m, shooting is terminated (Yes in S407), and feature points for image synthesis are detected from the shot images (S408).

  At this time, if a feature point can be detected (Yes in S409), a motion vector is calculated based on the feature point of each image (S410), and the calculated motion vector is used for alignment of each image to synthesize an image ( S411).

  If a feature point cannot be detected (No in S409), an image having the maximum average luminance of the image is extracted from n + m consecutively captured images and recorded on the recording medium 70 (S412).

  As mentioned above, although preferable Example 1 and Example 2 of this invention were described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

  The present invention has been described by taking a digital camera as an example of the imaging apparatus in the embodiments. However, the present invention may be any device provided with an apparatus capable of imaging. For example, it may be a mobile phone with a camera or an electronic device.

DESCRIPTION OF SYMBOLS 100 Imaging device 50 System control part 20 Image processing part 200 Feature point detection processing part 201 Motion vector calculating part 202 Image composition processing part 203 Feature point detection area setting processing part 300 Main imaging optical system 301 Sub imaging optical system 400 Imaging apparatus

Claims (9)

A plurality of image acquisition means for acquiring a plurality of images by sequentially photographing;
Bright spot extraction means for extracting bright spots appearing in the plurality of images,
Bright spot determination means for determining whether the bright spot is a moving bright spot or a stationary bright spot from the size of the bright spot extracted by the bright spot extracting means;
Motion vector calculating means for calculating a motion vector between the plurality of images with reference to the stationary bright spot determined by the bright spot determining means;
An image pickup apparatus comprising: an image combining unit that combines a single composite image based on the motion vector calculated by the motion vector calculating unit.   The imaging according to claim 1, wherein the bright spot determining unit determines that the bright spot extracted by the bright spot extracting unit is a stationary bright spot when the size of the bright spot is smaller than a predetermined size. apparatus.   The imaging according to claim 2, wherein the bright spot determination unit determines that the bright spot extracted by the bright spot extraction unit is a stationary bright spot when the length of the bright spot is shorter than a predetermined length. apparatus.   The motion vector calculating means extracts an image having the lowest luminance level and an image having the highest luminance level from the plurality of images, divides the extracted image into a plurality of divided areas, and luminances of the corresponding divided areas The imaging apparatus according to claim 1, wherein a feature point detection area for calculating the motion vector is set by comparing levels.   The motion vector calculation means has a luminance level greater than a first threshold among the divided areas corresponding to the image with the lowest luminance level and the image with the highest luminance level, and a difference in luminance level. The imaging apparatus according to claim 4, wherein the divided area smaller than a second threshold is set as the feature point detection region.   The imaging apparatus according to claim 4, wherein the motion vector calculation unit does not make a peripheral area of the plurality of images the feature point detection area.   The imaging apparatus according to claim 4, wherein the motion vector calculating unit changes the size of the divided area according to a focal length of an imaging optical system. Having a plurality of imaging optical systems,
The plurality of image acquisition means acquires a plurality of images taken by the plurality of imaging optical systems so that exposure is not interrupted and exposure is not overlapped. The imaging device described in 1. A multiple image acquisition step of acquiring a plurality of images by sequentially shooting;
A bright spot extracting step for extracting bright spots in the plurality of images;
A bright spot determining step for determining whether the bright spot is a moving bright spot or a stationary bright spot from the size of the bright spot extracted in the bright spot extracting step;
A motion vector calculating step for calculating a motion vector between the plurality of images with reference to the stationary bright spot determined in the bright spot determining step;
An image synthesizing method comprising: an image synthesizing step of synthesizing one synthesized image based on the motion vector calculated in the motion vector calculating step.

Images