JP2013141105A - Image processing device, electronic camera, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correctly align a subject when overlapping images with a large difference in luminance.SOLUTION: An image processing device comprises: an image input unit that inputs a plurality of images photographed temporally successively which are composed of two types of images different in presence of light emission by an auxiliary light source, and before and after one type of an image out of the two types of images, include the other type of at least one image each, which is different in presence of light emission; a calculation unit that compares one type of a plurality of images out of the two types of images with each other to calculate a misalignment amount between the one type of the plurality of images; and an estimation unit that estimates a relative misalignment amount of the other type of the image out of the two types of images on the basis of the misalignment amount calculated by the calculation unit.

Description

  The present invention relates to an image processing technique.

  Conventionally, a method for synthesizing an image obtained by photographing a bright part with appropriate exposure and an image obtained by photographing a dark part with appropriate exposure is known. This technique is used, for example, when dealing with the problem that bright areas are over-exposure and dark areas are black-out when an image exceeding the dynamic range that can be imaged with an electronic camera is taken. For example, in the invention of Patent Document 1, a technique for aligning the subject so that the subject overlaps accurately by dividing the image and performing feature detection in the process of superimposing the images and then aligning the image positions is studied. Has been.

Japanese Patent No. 4136044

  However, the composition method according to the invention of Patent Document 1 sometimes fails to accurately align the subject when superimposing images with greatly different luminances. If alignment is not performed accurately, there is a further problem that the image quality of the image generated by the synthesis process is lowered.

  An object of the present invention is to accurately align a subject when superimposing images having greatly different luminances.

  The image processing apparatus according to the present invention includes two types of images that differ in the presence or absence of light emission by an auxiliary light source, and the other type of images that differ in the presence or absence of light emission before and after one of the two types of images. An image input unit for inputting a plurality of images taken continuously in time including at least one image and a plurality of images of one type of the two types of images, Based on the shift amount calculated by the calculation unit and a shift unit that calculates the shift amount between the plurality of types of images, the relative shift of the image of the other type among the two types of images. An estimation unit for estimating the quantity.

  The image input unit includes at least one light-emitting photographed image taken by emitting the auxiliary light source, and at least one picture taken without emitting the auxiliary light source before and after the at least one light-emitting photographed image. A plurality of images taken continuously in time including each non-light-emitting photographed image, and the calculation unit compares the plurality of non-light-emitting photographed images with each other to obtain a plurality of the non-light-emitting photographed images. The shift amount may be calculated in between, and the estimation unit may estimate a relative shift amount of the at least one light-emitting photographed image based on the shift amount calculated by the calculation unit.

  Further, the image processing apparatus may further include an image composition unit that performs composition processing for compositing the plurality of images input by the image input unit based on the calculation result by the calculation unit and the estimation result by the estimation unit. .

  In addition, the estimation unit may estimate the deviation amount by performing an interpolation process on the relationship between the shooting time of the plurality of images input by the image input unit and the position of each image. .

  In addition, the image input unit inputs motion information of the imaging device when the plurality of images are captured, together with the plurality of images, and the calculation unit takes into account the motion information and the shift amount The estimation unit may estimate the shift amount in consideration of the motion information.

  Further, the estimation unit performs a predetermined gain adjustment on at least one image captured by causing the auxiliary light source to emit light among the two types of images, and causing the auxiliary light source after the gain adjustment to emit light. Even if at least one photographed image is compared with at least one image photographed without causing the auxiliary light source to emit light among the two types of images, the deviation amount is estimated by taking the comparison result into consideration good.

  In addition, the estimation unit may include an image of a high-luminance portion in at least one image captured without causing the auxiliary light source to emit light, and at least one image captured by emitting the auxiliary light source after the gain adjustment. You may compare with the image of the part corresponding to the said high-intensity part.

  An electronic camera having any of the image processing apparatuses described above and having the image input unit composed of an imaging unit and a light emitting unit is also effective as a specific aspect of the present invention.

  An image processing program for causing a computer to execute the functions of any of the above-described image processing apparatuses is also effective as a specific aspect of the present invention.

  According to the present invention, it is possible to accurately align a subject when superimposing images having greatly different luminances.

It is a figure which shows the structural example of the electronic camera 101 which concerns on this embodiment. It is a figure which shows the outline | summary of a synthetic | combination process. It is another figure which shows the outline | summary of a synthetic | combination process. 3 is a diagram illustrating a configuration example of a control unit 108 and an image processing unit 107. FIG. It is a flowchart which shows the imaging | photography process in "composite imaging | photography mode." It is a flowchart which shows the synthetic | combination process in "composite photography mode." It is a figure explaining the alignment process. It is a figure explaining the estimation process of a position.

  Embodiments of an image processing apparatus, an electronic camera, and an image processing program according to the present invention will be described below in detail with reference to the drawings. In the following embodiment, an example of an electronic camera that includes the image processing apparatus according to the present invention and uses the image processing program according to the present invention will be described.

[Configuration Example of Electronic Camera 101]
FIG. 1 is a diagram illustrating a configuration example of an electronic camera 101 according to the present embodiment. In FIG. 1, an electronic camera 101 includes an optical system 102, a mechanical shutter 103, an image sensor 104, an A / D conversion unit 105, an image buffer 106, an image processing unit 107, a control unit 108, and a memory 109. A display unit 110, an operation unit 111, a memory card IF (interface) 112, an auxiliary light source 113, and a gyro sensor 114.

  In FIG. 1, the subject light incident on the optical system 102 is incident on the light receiving surface of the image sensor 104 via the mechanical shutter 103. Here, the optical system 102 includes a plurality of lenses such as a zoom lens and a focus lens, a lens driving unit, an aperture mechanism, and the like, and the control unit 108 controls the zoom lens position, the focus lens position, the aperture value, and the like. The

  The image sensor 104 is configured by, for example, a CMOS solid-state image sensor, and outputs an image signal corresponding to the amount of received light by a photoelectric conversion unit of a pixel arranged two-dimensionally on the light receiving surface.

  The A / D converter 105 converts the image signal output from the image sensor 104 into digital image data for each pixel, and captures the image signal in the image buffer 106.

  The image buffer 106 is composed of, for example, a volatile high-speed memory. The image buffer 106 not only temporarily stores the image data output from the A / D conversion unit 105 but also is used as a buffer memory when the image processing unit 107 performs image processing. Alternatively, it is also used as a display memory when an image at the time of photographing or a photographed image stored in the memory card 112 a connected to the memory card IF 112 is read and displayed on the display unit 110.

  The image processing unit 107 performs image processing such as white balance processing, color interpolation processing, gamma correction processing, saturation enhancement processing, and contour enhancement processing on the image data captured in the image buffer 106. Further, the image processing unit 107 performs compression processing on the image data after image processing using an image compression method compliant with the JPEG standard or the like. Furthermore, in the electronic camera 101 according to the present embodiment, as will be described later, the image processing unit 107 performs an image composition process for compositing a plurality of continuously shot images.

  The control unit 108 is configured by a CPU that operates according to a program stored therein, for example. The control unit 108 controls the operation of the electronic camera 101. For example, the shooting mode selection dial 111a of the operation unit 111 sets the shooting mode of the electronic camera 101, or when the release button 111b is pressed, lens control and aperture control of the optical system 102 are performed. The mechanical shutter 103 is opened and closed, and a subject image is captured by the image sensor 104. Then, the control unit 108 performs control to read out the image signal from the image sensor 104 with a preset resolution, and to import the image data converted into a digital value by the A / D conversion unit 105 into the image buffer 106. Furthermore, the control unit 108 instructs the image processing unit 107 to perform predetermined image processing on the image data captured in the image buffer 106, and adds a predetermined file name to the image data after image processing (for example, JPEG data). And header information is added and stored in the memory card 112a via the memory card I / F 112. Alternatively, the image data captured in the image buffer 106 is displayed on the display unit 110 as a captured image. In particular, in the electronic camera 101 according to the present embodiment, when the “composite shooting mode” is selected with the shooting mode selection dial 111 a, the control unit 108 connects a plurality of images set in advance with the image sensor 104. A copy is taken into the image buffer 106. Here, the control unit 108 shoots the auxiliary light source 113 to emit only the image at a preset position when continuously shooting a plurality of images. Then, the control unit 108 instructs the image processing unit 107 to synthesize a plurality of captured images. Note that the image composition processing of the image processing unit 107 will be described in detail later.

  The memory 109 is composed of a nonvolatile semiconductor memory such as a flash memory, for example. Parameters such as a shooting mode, exposure information, and focus information of the electronic camera 101 are stored. The control unit 108 controls the operation of the electronic camera 101 with reference to these parameters. In particular, in the electronic camera 101 according to the present embodiment, parameters relating to the “composite shooting mode” are stored in the memory 109 as described later. Note that these parameters are appropriately updated according to user operations performed via the operation unit 111.

  The display unit 110 includes a liquid crystal monitor or the like. Then, the control unit 108 displays a captured image, a setting menu necessary for operating the electronic camera 101, and the like. In particular, in the electronic camera 101 according to the present embodiment, when the setting menu button 111c of the operation unit 111 is pressed, a menu screen is displayed on the display unit 110, and the user sets a parameter setting item of “composite shooting mode” with the cross key 111d. You can select to change the settings.

  The operation unit 111 includes a shooting mode selection dial 111a, a release button 111b, a setting menu button 111c, a cross key 111d, and the like. The user operates these operation buttons to use the electronic camera 101. For example, the shooting mode selection dial 111a performs “auto shooting mode”, “high sensitivity shooting mode”, “composite shooting mode”, and the like. The operation information of the operation unit 111 is output to the control unit 108, and the control unit 108 controls the operation of the electronic camera 101 according to the operation information input from the operation unit 111.

  The memory card IF 112 is an interface for connecting the memory card 112 a to the electronic camera 101. Then, the control unit 108 reads and writes image data to and from the memory card 112a via the memory card IF 112. In particular, the electronic camera 101 according to the present embodiment stores an image synthesized in the “composite shooting mode”. It should be noted that it may be set on the setting menu screen of the “composite shooting mode” so that a plurality of images before being combined in the “composite shooting mode” are stored in the memory card 112a. In this case, when the release button 111b is pressed in the “composite shooting mode”, a plurality of continuously shot images are stored in the memory card 112a without being combined. Alternatively, both the composite image and a plurality of continuously shot images may be stored in the memory card 112a. Further, in order to manage a plurality of continuously shot images collectively, for example, when the number of continuous shots is 3, “gousei-1.jpg”, “gousei-2.jpg”, “gousei-3. “jpg” is associated with the same name, and a number indicating the photographing order is added to the end, and the result is stored in the memory card 112a. Alternatively, a folder may be created every time continuous shooting is performed in the “composite shooting mode”, and a plurality of continuously shot images may be stored together in one folder. Alternatively, unique information that associates a plurality of images may be added as header information of the image file. Further, in this case, information regarding whether or not the image is taken by emitting light from the auxiliary light source 113 may be added. This makes it possible to perform similar processing using this information when performing the synthesis processing later. Information based on the output from the gyro sensor 114 may also be added. This makes it possible to perform similar processing using this information when performing the synthesis processing later.

  The auxiliary light source 113 corresponds to a flash mounted on a general camera, and is composed of a light emitting element that emits a flash to a subject during photographing. Then, a flash is emitted at a timing commanded by the control unit 108.

  The gyro sensor 114 is a sensor that detects the attitude of the electronic camera 101, and detects the movement of the electronic camera 101 with an index of angle or angular velocity. Since the configuration of the gyro sensor 114 is the same as that of a known technique, the description thereof is omitted. It should be noted that the gyro sensor 114 may be provided only for acquiring information used for a composition process to be described later, or may be provided for use in a camera shake correction function or the like.

  The above is the configuration and basic operation of the electronic camera 101. In the case of configuring a single image processing apparatus according to the present invention instead of the electronic camera 101, in FIG. 1, the optical system 102, the mechanical shutter 103, the image sensor 104, the A / D conversion unit 105, the auxiliary light source 113, and The processing of the control unit 108 that reads out image data from the memory card 112a to the image buffer 106 corresponds to the processing of the image input unit of the image processing device. . Then, the control unit 108 (corresponding to the control unit of the image processing apparatus) reads captured image data from the memory card 112a to the image buffer 106, performs image processing in the image processing unit 107, and displays the image on the display unit 110. Alternatively, the image data after image processing is stored again in the memory card 112a. Here, in the case of an image processing apparatus, the operation unit 111 is provided with a playback button for reading a captured image from the memory card 112a into the image buffer 106 in place of the release button 111b, and the imaging displayed on the display unit 110. An image on which image processing is to be performed is selected from among the completed images with the cross key 111d. Further, an image processing selection dial for selecting image processing contents is provided instead of the shooting mode selection dial 111a, and the image processing selected with the image processing selection dial is performed on the image selected with the cross key 111d. For example, when “compositing processing mode” is selected with the image processing selection dial, a plurality of continuously shot image data corresponding to the image selected with the cross key 111 d is read from the memory card 112 a to the image buffer 106. The process of combining the plurality of continuously shot image data read out to the image buffer 106 by the image processing unit 107 is the same as in the case of the electronic camera 101. As described above, the image processing apparatus can be realized as a dedicated image processing apparatus that reads out image data from the memory card 112a in which a plurality of images photographed in the “composite photographing mode” of the electronic camera 101 is stored, and performs composition processing.

  Note that the image processing apparatus may be realized by a computer. In this case, the control unit 108 corresponds to the CPU of the computer, reads image data from the memory card 112a connected to the memory card IF of the computer, and displays it on the monitor instead of the display unit 110. It is assumed that the image composition program is installed in the computer.

[Composite shooting mode]
Next, the “composite shooting mode” of the electronic camera 101 according to the present embodiment will be described. In general, when shooting a subject (such as a night view) that exceeds the dynamic range that can be shot with an electronic camera, white portions in bright portions and black portions in dark portions occur. Therefore, an image in which a bright part is photographed with appropriate exposure and an image in which a dark part is photographed with appropriate exposure are combined to create an image in which both a bright part and a dark part are clearly photographed.

  For example, in FIG. 2, as shown in the example of the subject 151, when a person is photographed against a night scene (shooting composition 152), an image 153 shot with a flash and an image 154 shot without a flash are combined, It is possible to obtain an image 155 in which both the person and the background are combined so as to have good exposure (appropriate or close to appropriate exposure).

  In the composition process for superimposing the image 153 and the image 154, alignment is performed prior to the superposition. Since the brightness of the main subject is greatly different between the image 153 and the image 154, the position detection of the main subject or the determination of whether or not they are the same subject cannot be accurately performed by comparing the colors or the contours. It becomes difficult to align the main subject of the two images to be superimposed.

  In particular, as shown in FIG. 2, when a person is photographed with a night view as a background, the image 153 photographed with a flash shows the person brightly but the background night scene appears dark. On the other hand, in the image 154 taken without the flash, the night scene of the background appears bright, but the person appears dark. Therefore, when aligning these images, there are not enough feature points suitable for comparison, and there is a possibility that sufficient alignment accuracy cannot be obtained. When the alignment accuracy is low, when the processing of adding or averaging the signals of these images (referred to as simple combining) is performed, the position of the main subject is combined and the image generated by the combining processing Image quality may be degraded.

  In view of this, in the electronic camera 101 according to the present embodiment, the position adjustment method is devised to reduce the position of the main subject even when there are images with significantly different luminances in a plurality of continuously shot images. Can be synthesized.

[Image composition method]
Next, an image composition method in the electronic camera 101 according to the present embodiment will be described. FIG. 3 shows an example of a photographed image when the number of continuous shots is five as an example of a plurality of continuously shot images. In FIG. 3, the image 161 is the first (first) image, the image 162 is the second image, the image 163 is the third image, and the image 164 is 4 in the order of shooting. An image captured on the first sheet, image 165, is an image captured on the fifth (last) image. Four images 161, 162, 164, and 165 are images taken without a flash, and an image 163 is an image taken with a flash. Here, in shooting without flash, the aperture value and the shutter speed are controlled so that the brightness of the subject as the background is adequately exposed in the image generated by the synthesis process.

  Note that the example in FIG. 3 is an example. If images taken without a flash are included before and after an image taken with a flash, any combination of the number of images and the order of shooting are included. It may be.

  As shown in the example of FIG. 3, four images (image 161, image 162, image 164, and image 165) photographed without a flash and an image photographed with a flash (image 163) are shown. The subject is very different. Therefore, it is difficult to perform alignment based on the image information of these images. On the other hand, when four images (image 161, image 162, image 164, and image 165) photographed without a flash are compared with each other, the photographed subject is substantially the same. Therefore, it is relatively easy to perform alignment based on the image information of these images.

  In view of this, in the electronic camera 101 according to the present embodiment, as in the example of FIG. 3, a method of alignment when combining a plurality of continuously shot images is devised. First, four images (image 161, image 162, image 164, and image 165) photographed without a flash are compared to perform alignment. Next, the relative position of the image photographed with the flash is estimated based on the alignment result of the four images photographed without the flash.

[Configuration Example of Image Processing Unit 107]
Next, a configuration example of the image processing unit 107 will be described with reference to FIG. In FIG. 4, the same reference numerals as those in FIG. In FIG. 4, the image processing unit 107 includes a synthesis unit 201. Further, the synthesis unit 201 includes an alignment processing unit 251, a position estimation unit 252, and a synthesis processing unit 253. Further, the control unit 108 includes a composite image capturing control unit 301 that executes a process of generating a composite image when the “composite capturing mode” is selected by the operation unit 111.

  First, the photographer selects “composite shooting mode” with the shooting mode selection dial 111 a of the operation unit 111. When the photographer presses the release button 111 b, the composite image capturing control unit 301 of the control unit 108 starts capturing with reference to the “composite capturing mode” parameter stored in advance in the memory 109. Here, the “composite shooting mode” parameters stored in the memory 109 are, for example, the number of continuous shots: 5 and the flash shooting position: 3rd.

  When the release button 111b is pressed, the composite image capturing control unit 301 instructs the image capturing unit 104 to start continuous shooting. Here, after two images that are temporally continuous are captured in the image buffer 106, the auxiliary light source 113 is caused to emit light, and a third image is captured and captured in the image buffer 106. Further, two images that are temporally continuous are taken into the image buffer 106 without causing the auxiliary light source 113 to emit light. At this stage, the five captured images captured in the image buffer 106 may be stored in the memory card 112a.

  Next, the composite image capturing control unit 301 instructs the image processing unit 107 to instruct the image processing unit 107 to synthesize the five captured images captured in the image buffer 106. As described with reference to FIG. 3, the alignment processing unit 251 in the composition unit 202 of the image processing unit 107 performs alignment by comparing four images captured without a flash.

  The position estimation unit 252 estimates the relative position of the image captured with the flash based on the result of the alignment performed by the alignment processing unit 251.

  Based on the result of the alignment performed by the alignment processing unit 251 and the estimation result of the relative position of the image taken with the flash performed by the position estimation unit 252, the composition processing unit 253 A synthesis process for synthesizing the five photographed images captured in 106 is performed.

  Next, a process for capturing an image in the “composite shooting mode” and a detailed process for combining captured images will be described.

  FIG. 5 is a flowchart showing processing when an image is captured in the “composite shooting mode”. 5 is performed by the composite image capturing control unit 301 of the control unit 108.

  (Step S101) The photographer selects “composite shooting mode” with the shooting mode selection dial 111a of the operation unit 111 before starting shooting.

  (Step S102) The composite image capturing control unit 301 determines whether or not the photographer has pressed the release button 111b of the operation unit 111, and proceeds to the next step when the release button 111b is pressed. At this time, the composite image capturing control unit 301 sets the count value of the counter indicating the number of the next image to be captured during continuous shooting to 1 (first image). The counter is a software variable of an image processing program executed by the control unit 108.

  (Step S103) The composite image shooting control unit 301 refers to the “synthetic shooting mode” parameter stored in the memory 109, and the count value of the counter indicates the image position for flash shooting (in this example, the third frame). ). If it is an image position for flash photography, the process proceeds to step S104. If it is not an image position for flash photography, the process proceeds to step S105.

  (Step S <b> 104) The composite image shooting control unit 301 controls the aperture value of the optical system 102 and the shutter speed of the mechanical shutter 103 to cause the auxiliary light source 113 to emit light, and the image captured by the image sensor 104 is A / D conversion unit 105. To the image buffer 106 (shooting with flash). Then, the composite image capturing control unit 301 increments the count value of the counter that indicates how many images are to be captured next. In this flash photography, the exposure is controlled so as to achieve an appropriate exposure.

  (Step S <b> 105) The composite image capturing control unit 301 controls the aperture value of the optical system 102 and the shutter speed of the mechanical shutter 103, and images captured by the image sensor 104 via the A / D conversion unit 105. (Shooting without flash). Then, the composite image capturing control unit 301 increments the count value of the counter that indicates how many images are to be captured next. In this photographing, the composite image photographing control unit 301 performs photographing with reference to the “synthetic photographing mode” parameter stored in the memory 109. In addition, it is assumed that the proper exposure in shooting without flash is obtained from a photometric value when the release button 111b is pressed (for example, photometry from the output of the image sensor 104).

  (Step S106) The composite image capturing control unit 301 reduces the number of captured images indicated by the count value of the counter (the count value is the number of the next captured image, which is one more than the number of captured images, so it is reduced by one). It is determined whether or not (the number of images) matches the number of consecutive shots in the “composite shooting mode” stored in the memory 109. If the continuous shooting has been completed, the “composite shooting mode” is ended. If the continuous shooting has not been completed, the process returns to step S103 to repeat the same shooting process. For example, if the continuous shooting number of the “composite shooting mode” parameter is 5 and the count value of the counter indicating the number of the next image to be shot is 6 (the number), the shooting is ended. If the count value is less than or equal to the fifth sheet, the process returns to step S103.

  In this way, when the “composite shooting mode” is selected with the shooting mode selection dial 111 a, when the release button 111 b is pressed, a predetermined composite mode is stored according to the “composite shooting mode” parameter stored in the memory 109. Flash shooting is performed at the image position, and a predetermined number of images taken continuously are taken into the image buffer 106.

  (Step S107) The composite image capturing control unit 301 stores the continuous shot image captured in the image buffer 106 in Step S106 in the memory card 112a via the memory card IF 112. Alternatively, the composite image is displayed on the display unit 110. At this time, the composite image shooting control unit 301 records the shooting time in association with each image. Note that the shooting time is, for example, for an image shot without a flash, an intermediate time from the start of exposure to the end of exposure is set as the shooting time, and for an image shot with flash, the time when the flash emission is started is shot. Time. Furthermore, the composite image capturing control unit 301 also records the output value of the gyro sensor 114 in association with each image.

  The memory card 112a can be connected to a computer, a dedicated image processing device, or the like, and a plurality of images shot in the “composite shooting mode” can be combined by the computer or the image processing device. In this case, the computer and the image processing apparatus are equipped with an image processing program for executing image composition processing described later.

  Next, a process of combining a plurality of images continuously shot in the “composite shooting mode” will be described with reference to the flowchart of FIG. The processing in FIG. 6 is performed by the image processing unit 107 in accordance with a command from the composite image capturing control unit 301 of the control unit 108. Note that the image processing unit 107 includes the processing blocks described with reference to FIG.

  (Step S <b> 201) The alignment processing unit 251 reads out a plurality of images shot in the “composite shooting mode” from the image buffer 106. In the case of a computer or an image processing apparatus, a plurality of images shot in the “composite shooting mode” are read from the memory card 112a.

  (Step S202) The alignment processing unit 251 performs an alignment process between four images captured without a flash among the plurality of images read out in step S201. The alignment processing between the four images (image 161, image 162, image 164, and image 165 in FIG. 3) may be performed in any order. For example, the alignment of the image 161 and the image 162, the alignment of the image 162 and the image 164, and the alignment of the image 164 and the image 165 may be sequentially performed. The image 161 and the image The alignment of 162, the alignment of the images 161 and 164, and the alignment of the images 161 and 165 may be sequentially performed.

  The alignment process is performed based on the image information as in the known technique. Note that the four images (image 161, image 162, image 164, and image 165 in FIG. 3) shot without the flash are all images shot without the flash, and therefore the exposure conditions of the main subjects are similar. Thus, it is possible to accurately detect the position of the main subject and determine whether or not the subject is the same subject by comparing the colors and the luminance.

  The alignment processing unit 251 may add the output value of the gyro sensor 114 to the alignment process. For example, as shown in FIG. 7, the movement amount of the image is obtained based on the output value of the gyro sensor 114, and the deviation amount is obtained so as to cancel the movement amount. If the electronic camera 101 has a lens shift function or a sensor shift function, the shift amount may be obtained by adding the shift amount of the lens or sensor to the output value of the gyro sensor 114. It is also possible to limit the range in which the alignment process based on the image information is performed by performing the alignment process based on the image information described above after performing the alignment process based on the output value of the gyro sensor 114. It is. That is, it is possible to perform rough alignment by alignment processing based on the output value of the gyro sensor 114, and to perform detailed alignment processing based on image information after limiting the range.

  (Step S203) The position estimation unit 252 estimates the relative position of the image captured with the flash based on the result of the alignment process performed in step S202.

  The position estimation unit 252 estimates the relative position of an image captured with a flash by the following three methods A to C.

(A) Position Estimation Based on Output of Gyro Sensor 114 The position estimation unit 252 includes at least one of four images (image 161, image 162, image 164, and image 165 in FIG. 3) captured without a flash, For the image taken with the flash (image 163 in FIG. 3), alignment processing is performed in the same manner as the processing described in step S202 based on the output of the gyro sensor 114, and the position of the image taken with the flash is estimated. To do. At this time, the comparison target may be any of the four images captured without the flash, but it is preferable to compare the images 162 and 164 that are close in time.

(B) Position Estimation Based on Position Registration Processing Result of Image Taken without Flash The position estimation unit 252 has four images (image 161, image 162, image in FIG. 3) taken without a flash described in step S202. 164 and the image 165) are used to estimate the position of the image taken with the flash.

  FIG. 8 is an example showing the relationship between the shooting time and position of each image. The horizontal axis in FIG. 8 indicates the shooting time (T1 to T5) of each image, and the vertical axis indicates the position in the X direction of each image. In FIG. 8, among the elapsed times (ΔT1 to ΔT4), only ΔT2 is longer than the other elapsed times because the third frame is shot with a flash and it takes time to prepare for shooting.

  As a result of the alignment processing in step S202, the shooting time of the first image = T1, the position in the X direction = P1, the shooting time of the second image = T2, the position in the X direction = P2, and 4 If the shooting time of the first image = T4, the position in the X direction = P4, the shooting time of the fifth image = T5, and the position in the X direction = P4, three images taken with the flash The position in the X direction of the eye image (shooting time = T3) is unknown.

  Therefore, the position estimation unit 252 first plots the relationship between the shooting time and the position in the X direction for four images shot without a flash, and creates a spline curve L that passes through these points. On the spline curve L, the position in the X direction corresponding to the shooting time T3 is set as the position in the X direction (= P3) of the third image, which is an image shot with flash.

  If there are two images taken without a flash, temporary interpolation may be performed instead of the spline curve.

  Also, the position estimation unit 252 performs the same process as the X direction for the Y direction, and estimates the position of the third image, which is an image taken with a flash, in the Y direction.

(C) Position Estimation Based on Image Information The position estimation unit 252 includes at least one of four images (image 161, image 162, image 164, and image 165 in FIG. 3) taken without a flash and a flash. With respect to the photographed image (image 163 in FIG. 3), alignment processing is performed based on the image information in the same manner as a known technique, and the position of the photographed image with the flash is estimated. At this time, the comparison target may be any of the four images captured without the flash, but it is preferable to compare the images 162 and 164 that are close in time.

  Note that the position estimation unit 252 may perform the alignment process after performing a predetermined gain adjustment on the image captured with the flash prior to the alignment process. The predetermined gain adjustment described above is an adjustment for adjusting the luminance level of the background portion of the image photographed with the flash to the luminance level of the background portion of the image photographed without the flash. By performing such gain adjustment, highlight high-intensity parts such as point light sources in the background part of images taken with a flash so that they can be compared with the equivalent parts of images taken without a flash. Can do. The gain adjustment may be performed on a copy of an image photographed with a flash, or may be performed on an image itself photographed with a flash. When gain adjustment is performed on the image itself taken with the flash, the gain adjustment in the reverse direction may be performed after the position is estimated.

  The position estimation unit 252 may estimate the position of an image captured with a flash by performing any one of the three methods described above, or may combine two or more methods. For example, by limiting the processing range of at least one of the estimation processes (B) and (C) based on the estimation result described in (A), the processing load is reduced and the estimation process is efficiently performed. It can be carried out. Similarly, by limiting the processing range of the estimation process of (C) based on the estimation result described in (B), the processing load can be reduced and the estimation process can be performed efficiently.

  Further, of the various methods described above, which method is used for the estimation process may be specified based on a user operation via the operation unit 111.

  (Step S204) The composition processing unit 253 performs the composition processing of the plurality of images read out in step S201 based on the result of the alignment process performed in step S202 and the result of the position estimation performed in step S203. Do.

  The synthesizing process is performed in the same manner as a known technique. For example, the composition processing unit 253 determines a composition position based on the result of alignment or position estimation, and adds or averages the signals of the corresponding pixels of each image. In the case of a color image, the same processing is performed for each color.

  Note that the angle of view of the image generated by the synthesis process is shifted due to the influence of camera shake or the like, so that there may be a feeling of strangeness in the synthesized image. In such a case, as a post-combination process, a trimming process is performed to cut out overlapping portions of the angle of view from the image 161 to the image 165, and the trimmed image is stored in the memory card 112a as a final image. Also good.

  (Step S <b> 205) The image processing unit 107 outputs an image generated by the composition processing by the composition processing unit 253. Note that the combined image is temporarily held in the image buffer 106 until the next shooting is performed. Then, the control unit 108 displays the combined image held in the image buffer 106 on the display unit 110 and a file name (for example, a serial number and a shooting date / time) determined in advance for the image data of the combined image. Is automatically added and stored in the memory card 112a via the memory card IF 112.

  In this way, the electronic camera 101 according to the present embodiment compares a plurality of non-light-emitting captured images with each other, calculates a shift amount between the plurality of non-light-emitting captured images, and based on the calculated shift amount, A relative shift amount of the luminescence image is estimated. Therefore, the subject can be accurately aligned when superimposing images with greatly different luminances.

  In addition, the electronic camera 101 according to the present embodiment performs a combining process for combining a plurality of images based on a shift amount between a plurality of non-light-emitting captured images and an estimated relative shift amount of the light-emitting captured images. Do. Therefore, the background has less noise, and a composite image having an appropriate brightness on the entire screen can be obtained.

(Modification)
In the above-described embodiment, the number of continuous shots is five, and only the third image is shot with flash, and the previous and second images are shot without flash. However, the present invention is limited to this example. Not. For example, the number of continuous shots includes both shooting with flash and shooting without flash, and may be any number as long as it is three or more.

  For example, in the case of three consecutive shots in order of shooting without flash, shooting with flash, and shooting without flash, there is a flash in the middle based on the result of alignment processing between images generated by shooting without flash before and after It is possible to estimate a relative shift amount of an image generated by photographing. On the other hand, when three consecutive shots are taken in the order of shooting with flash, shooting without flash, and shooting with flash, the middle flash is based on the result of registration processing between the images generated by shooting with flash before and after. It is possible to estimate a relative shift amount of an image generated by non-photographing. In any case, alignment processing may be performed by comparing images in which the captured subjects are substantially the same, and the relative position of an image whose position is unknown may be estimated based on the result.

  Further, two or more images generated by shooting with flash may be used. When images generated by shooting with flash are temporally continuous, first, alignment processing between images generated by shooting with flash is performed, and then synthesis processing is performed. And the relative position of the image produced | generated by the imaging | photography with flash | flush after a synthesis process should just be estimated similarly to the said embodiment. In addition, when images generated by shooting with flash are generated discretely (not temporally continuous), each image generated by shooting with flash is based on images generated by shooting without flash before and after the shooting. The same processing as in the above embodiment may be performed.

  In the above-described embodiment, the electronic camera 101 is provided with the “composite shooting mode”, and the series of processing is performed when the “composite shooting mode” is executed. However, the present invention is not limited to this example. For example, when a user selects a shooting mode in which the processing of the present invention is effective, such as “night scene mode”, a series of processing may be automatically performed. In the configuration in which the shooting mode is set on the camera side based on subject information and the like, a series of processing may be automatically performed when it can be determined that the processing of the present invention is effective. The series of processes described in the above embodiment may be executed in combination with other processes.

  The configuration of the electronic camera described in the above embodiment is an example, and the present invention is not limited to this example. For example, a single-lens reflex camera or a camera mounted on an information terminal such as a mobile phone may be used. In any case, a program that can execute the above-described series of processes may be stored in advance. A program for realizing the processing described in the above embodiment by a computer is also effective as a specific aspect of the present invention. This program may be recorded on a medium, or may be downloaded from a server or the like via communication such as the Internet.

  As described above, the image processing apparatus, the electronic camera, and the image processing program according to the present invention have been described by way of example in each embodiment. However, in various other forms without departing from the spirit or main features thereof. Can be implemented. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The present invention is defined by the claims, and the present invention is not limited to the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

DESCRIPTION OF SYMBOLS 101 ... Electronic camera, 104 ... Image pick-up element, 107 ... Image processing part, 108 ... Control part, 113 ... Auxiliary light source, 114 ... Gyro sensor, 201 ... Composition part, 251 ... Positioning process part, 252 ... Position estimation part, 253 ... Composition processing unit 301 ... Composite image photographing control unit

Claims (15)

It consists of two types of images that differ in the presence or absence of light emission by the auxiliary light source, and includes at least one image of the other type that differs in the presence or absence of the light emission before and after one of the two types of images. An image input unit for inputting a plurality of images taken continuously,
A calculation unit that compares the plurality of images of one type of the two types of images with each other, and calculates a shift amount between the plurality of images of the one type;
An image processing apparatus comprising: an estimation unit configured to estimate a relative shift amount of the other type of the image among the two types of images based on the shift amount calculated by the calculation unit. . The image processing apparatus according to claim 1.
The image input unit includes at least one light-emitting photographed image taken by emitting the auxiliary light source, and at least one piece photographed without emitting the auxiliary light source before and after the at least one light-emitting photographed image. Enter multiple images taken continuously in time, including non-flash images,
The calculation unit compares a plurality of the non-light-emitting captured images with each other, calculates the shift amount between the plurality of non-light-emitting captured images,
The image processing apparatus, wherein the estimating unit estimates a relative shift amount of the at least one light-emitting photographed image based on the shift amount calculated by the calculation unit. The image processing apparatus according to claim 1 or 2,
An image synthesizing unit that performs a synthesizing process for synthesizing the plurality of images input by the image input unit based on a calculation result by the calculation unit and an estimation result by the estimation unit. Processing equipment. The image processing apparatus according to any one of claims 1 to 3,
The estimation unit estimates the shift amount by performing an interpolation process on the relationship between the shooting time of the plurality of images input by the image input unit and the position of each image. Image processing device. The image processing apparatus according to any one of claims 1 to 4,
The image input unit inputs movement information of the imaging device when the plurality of images are taken together with the plurality of images,
The calculation unit calculates the shift amount in consideration of the motion information,
The said estimation part estimates the said deviation | shift amount in consideration of the said motion information. The image processing apparatus characterized by the above-mentioned. The image processing apparatus according to any one of claims 1 to 5,
The estimation unit performs a predetermined gain adjustment on at least one image captured by emitting the auxiliary light source among the two types of images, and captured the auxiliary light source after the gain adjustment. Comparing at least one image and at least one image of the two types of images taken without causing the auxiliary light source to emit light, and estimating the amount of deviation in consideration of the comparison result. An image processing apparatus. The image processing apparatus according to claim 6.
The estimation unit includes the high brightness portion in an image of a high-intensity portion in at least one image captured without causing the auxiliary light source to emit light, and in at least one image captured by emitting the auxiliary light source after the gain adjustment. An image processing apparatus that compares an image of a portion corresponding to a luminance portion. An imaging unit that captures a plurality of images continuously in time;
A light emitting section for emitting an auxiliary light source;
Of the plurality of images, the auxiliary light source of the light emitting unit is caused to emit light when taking an image at a preset time position, and the two types of images are different in the presence or absence of light emission by the auxiliary light source. A control unit for generating, by the imaging unit, a plurality of images taken continuously in time before and after one type of image, including at least one image of the other type with or without the emission of light. ,
A calculation unit that compares the plurality of images of one type of the two types of images with each other, and calculates a shift amount between the plurality of images of the one type;
An electronic camera comprising: an estimation unit configured to estimate a relative shift amount of the other type of the image among the two types of images based on the shift amount calculated by the calculation unit. The electronic camera according to claim 8,
The control unit includes at least one light-emitting photographed image taken by emitting the auxiliary light source and at least one non-photographed image taken without causing the auxiliary light source to emit light before and after the at least one light-emitting photographed image. A plurality of images that are continuously captured in time including a flash image are generated by the imaging unit,
The calculation unit compares a plurality of the non-light-emitting captured images with each other, calculates the shift amount between the plurality of non-light-emitting captured images,
The electronic camera is characterized in that the estimation unit estimates a relative shift amount of the at least one light-emitting photographed image based on the shift amount calculated by the calculation unit. The electronic camera according to claim 8 or 9,
An electronic camera further comprising: an image composition unit that performs composition processing for compositing the plurality of images generated by the imaging unit based on a calculation result by the calculation unit and an estimation result by the estimation unit. . The electronic camera according to any one of claims 8 to 10,
The estimation unit estimates the shift amount by performing an interpolation process on the relationship between the shooting time of the plurality of images generated by the imaging unit and the position of each image. camera. The electronic camera according to any one of claims 8 to 11,
A detection unit for detecting movement of the electronic camera during shooting by the imaging unit;
The calculation unit calculates the shift amount in consideration of motion information detected by the detection unit,
The said estimation part estimates the said deviation | shift amount in consideration of the said motion information. The electronic camera characterized by the above-mentioned. The electronic camera according to any one of claims 8 to 12,
The estimation unit performs a predetermined gain adjustment on at least one image captured by emitting the auxiliary light source among the two types of images, and captured the auxiliary light source after the gain adjustment. Comparing at least one image and at least one image of the two types of images taken without causing the auxiliary light source to emit light, and estimating the amount of deviation in consideration of the comparison result. Electronic camera. The electronic camera according to claim 13.
The estimation unit includes the high brightness portion in an image of a high-intensity portion in at least one image captured without causing the auxiliary light source to emit light, and in at least one image captured by emitting the auxiliary light source after the gain adjustment. An electronic camera characterized by comparing an image of a portion corresponding to a luminance portion. It consists of two types of images that differ in the presence or absence of light emission by the auxiliary light source, and includes at least one image of the other type that differs in the presence or absence of the light emission before and after one of the two types of images. An image input procedure to input multiple images taken consecutively,
A calculation procedure for comparing the plurality of images of one type of the two types of images with each other and calculating a shift amount between the plurality of images of the one type;
A computer is configured to execute an estimation procedure for estimating a relative shift amount of the other type of the two types of images based on the shift amount calculated by the calculation unit. Image processing program.

Images