JP2012195660A - Image processing apparatus and electronic camera, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate alignment and composition of subjects greatly different from each other in luminance.SOLUTION: An image processing apparatus comprises: an image input part which receives input of plural sheets of images captured temporally consecutively which include an image captured with light emission of an auxiliary light source; a reference image selection part which selects as a first reference image the light-emission captured image captured with the light emission of the auxiliary light source and, as a second reference image, no-light-emission captured image which is the closest temporally to the first reference image among plural sheets of no-light-emission captured images which are captured without light emission of the auxiliary light source; and an image composition part which performs first composition for composing the plural sheets of no-light-emission captured images to the second reference image and second composition for composing the first-composed image to the first reference image.

Description

  The present invention relates to an image processing technique.

  Conventionally, when shooting an image that exceeds the dynamic range that can be captured with an electronic camera, the bright part is overexposed and the dark part is blackened. A method of combining a photographed image is used. In this case, it is necessary to perform processing for superimposing images, and image processing for aligning the positions of the subjects so that the subjects overlap accurately is performed. For example, a technique for aligning an image position after performing feature detection by dividing an image has been studied (for example, see Patent Document 1).

Japanese Patent No. 4136044

  However, in the conventional synthesis method, when superimposing images with greatly different luminances, it is difficult to accurately align the subject, requiring complicated arithmetic processing and an expensive processing device, and processing time is increased. There was a problem.

  An object of the present invention is to provide an image processing apparatus, an electronic camera, and an image processing program capable of accurately aligning a subject when superimposing images having greatly different luminances.

  An image processing apparatus according to the present invention captures an image input unit that inputs a plurality of images that are continuously captured in time including an image that is captured with an auxiliary light source, and an image that is captured with the auxiliary light source emitted. A light emission captured image is a first reference image, and a non-light emission captured image that is temporally closest to the first reference image among a plurality of non-light emission captured images captured without causing the auxiliary light source to emit light is a second reference image, A reference image selection unit for selecting each of the images, a first combining process for combining the plurality of non-light-emitting photographed images with the second reference image, and an image after the first combining process for combining with the first reference image. It has an image composition part which performs the 2nd composition processing.

  Further, the first composition processing detects a position of a main subject set in advance for each of the plurality of non-light-emitting photographed images, and performs the non-light-emitting photographing so as to match the position of the main subject of the second reference image. The position of the main subject in the image is adjusted and combined, and the second combining process combines the first reference image with the image combined by the first combining process at the same pixel position.

  Further, the plurality of non-light-emitting photographed images are photographed with underexposure on the main subject, and the light-emitting photographed images are photographed with appropriate exposure on the main subject.

  In particular, the first reference image photographed by emitting the auxiliary light source is the first or last image among the images photographed continuously in time.

  Alternatively, the first reference image captured by causing the auxiliary light source to emit light is an image other than the first or last image among the images continuously captured in time, and the reference image selection unit includes the plurality of reference images. Two non-light-emitting photographic images temporally preceding and following the first reference image are selected as a front second reference image and a rear second reference image from among the non-light-emitting photographic images, and the image composition unit As the first synthesis process, the plurality of non-light-emitting photographed images temporally before the first reference image are converted into the front second reference image, and the plurality of temporally subsequent images from the first reference image. The non-luminous photographed images are combined with the rear second reference image, respectively, and the second combining process is the front second reference combined with the first reference image by the first combining process. Each of the image and the rear second reference image Characterized by combining in one pixel location.

  An electronic camera according to the present invention is an electronic camera having the image processing device, wherein the image input unit includes an imaging unit that captures a plurality of images continuously in time, and the plurality of images. And a light emitting unit that emits an auxiliary light source when taking an image at a preset time position, and the image synthesizing unit further includes a recording unit that records the synthesized image on a storage medium.

  An image processing program according to the present invention is an image processing program for causing a computer to execute the function of the image processing apparatus.

  The image processing apparatus, the electronic camera, and the image processing program according to the present invention can accurately align a subject by changing the order in which the images are combined when superimposing images having greatly different luminances. it can.

It is a figure which shows the structural example of the electronic camera 101 which concerns on this embodiment. It is a figure which shows the outline | summary of a synthetic | combination process. It is a figure which shows an example of a synthetic | combination order. 3 is a diagram illustrating a configuration example of a control unit 108 and an image processing unit 107. FIG. It is a figure which shows the mode of a 1st synthetic | combination process. It is a figure which shows the mode of a 2nd synthetic | combination process. It is a flowchart which shows the imaging | photography process in "composite imaging | photography mode." It is a flowchart which shows the synthetic | combination process in "composite imaging | photography mode". It is a figure which shows the modification 1 of a synthetic | combination order. It is a figure which shows the modification 2 of a synthetic | combination order.

  Embodiments of an image processing apparatus, an electronic camera, and an image processing program according to the present invention will be described below in detail with reference to the drawings. In the following embodiment, an example of an electronic camera that includes the image processing apparatus according to the present invention and uses the image processing program according to the present invention will be described.

[Configuration Example of Electronic Camera 101]
FIG. 1 is a diagram illustrating a configuration example of an electronic camera 101 according to the present embodiment. In FIG. 1, an electronic camera 101 includes an optical system 102, a mechanical shutter 103, an image sensor 104, an A / D conversion unit 105, an image buffer 106, an image processing unit 107, a control unit 108, and a memory 109. A display unit 110, an operation unit 111, a memory card IF (interface) 112, and an auxiliary light source 113.

  In FIG. 1, the subject light incident on the optical system 102 is incident on the light receiving surface of the image sensor 104 via the mechanical shutter 103. Here, the optical system 102 includes a plurality of lenses such as a zoom lens and a focus lens, a lens driving unit, an aperture mechanism, and the like, and the control unit 108 controls the zoom lens position, the focus lens position, the aperture value, and the like. The

  The image sensor 104 is configured by, for example, a CMOS solid-state image sensor, and outputs an image signal corresponding to the amount of received light by a photoelectric conversion unit of a pixel arranged two-dimensionally on the light receiving surface.

  The A / D converter 105 converts the image signal output from the image sensor 104 into digital image data for each pixel, and captures the image signal in the image buffer 106.

  The image buffer 106 is composed of, for example, a volatile high-speed memory. The image buffer 106 not only temporarily stores the image data output from the A / D conversion unit 105 but also is used as a buffer memory when the image processing unit 107 performs image processing. Alternatively, it is also used as a display memory when an image at the time of photographing or a photographed image stored in the memory card 112 a connected to the memory card IF 112 is read and displayed on the display unit 110.

  The image processing unit 107 performs image processing such as white balance processing, color interpolation processing, gamma correction processing, saturation enhancement processing, and contour outline enhancement processing on the image data captured in the image buffer 106. Further, the image processing unit 107 performs compression processing on the image data after image processing using an image compression method compliant with the JPEG standard or the like. Furthermore, in the electronic camera 101 according to the present embodiment, as will be described later, the image processing unit 107 performs an image composition process for compositing a plurality of continuously shot images.

  The control unit 108 is configured by a CPU that operates according to a program stored therein, for example. The control unit 108 controls the operation of the electronic camera 101. For example, the shooting mode selection dial 111a of the operation unit 111 sets the shooting mode of the electronic camera 101, or when the release button 111b is pressed, lens control and aperture control of the optical system 102 are performed. The mechanical shutter 103 is opened and closed, and a subject image is captured by the image sensor 104. Then, the control unit 108 performs control to read out the image signal from the image sensor 104 with a preset resolution, and to import the image data converted into a digital value by the A / D conversion unit 105 into the image buffer 106. Furthermore, the control unit 108 instructs the image processing unit 107 to perform predetermined image processing on the image data captured in the image buffer 106, and adds a predetermined file name to the image data after image processing (for example, JPEG data). And header information is added and stored in the memory card 112a via the memory card I / F 112. Alternatively, the image data captured in the image buffer 106 is displayed on the display unit 110 as a captured image. In particular, in the electronic camera 101 according to the present embodiment, when the “composite shooting mode” is selected with the shooting mode selection dial 111 a, the control unit 108 connects a plurality of images set in advance with the image sensor 104. A copy is taken into the image buffer 106. Here, the control unit 108 shoots the auxiliary light source 113 to emit only the image at a preset position when continuously shooting a plurality of images. Then, the control unit 108 instructs the image processing unit 107 to synthesize a plurality of captured images. The image composition processing of the image processing unit 107 will be described in detail later.

  The memory 109 is composed of a nonvolatile semiconductor memory such as a flash memory, for example. Parameters such as a shooting mode, exposure information, and focus information of the electronic camera 101 are stored. The control unit 108 controls the operation of the electronic camera 101 with reference to these parameters. In particular, in the electronic camera 101 according to the present embodiment, parameters relating to the “composite shooting mode” are stored in the memory 109 as described later. Note that these parameters are appropriately updated according to user operations performed via the operation unit 111.

  The display unit 110 includes a liquid crystal monitor or the like. Then, the control unit 108 displays a captured image, a setting menu necessary for operating the electronic camera 101, and the like. In particular, in the electronic camera 101 according to the present embodiment, when the setting menu button 111c of the operation unit 111 is pressed, a menu screen is displayed on the display unit 110, and the user sets a parameter setting item of “composite shooting mode” with the cross key 111d. You can select to change the settings.

  The operation unit 111 includes a shooting mode selection dial 111a, a release button 111b, a setting menu button 111c, a cross key 111d, and the like. The user operates these operation buttons to use the electronic camera 101. For example, the shooting mode selection dial 111a performs “auto shooting mode”, “high sensitivity shooting mode”, “composite shooting mode”, and the like. The operation information of the operation unit 111 is output to the control unit 108, and the control unit 108 controls the operation of the electronic camera 101 according to the operation information input from the operation unit 111.

  The memory card IF 112 is an interface for connecting the memory card 112 a to the electronic camera 101. Then, the control unit 108 reads and writes image data to and from the memory card 112a via the memory card IF 112. In particular, the electronic camera 101 according to the present embodiment stores an image synthesized in the “composite shooting mode”. It should be noted that a plurality of images before being combined in the “composite shooting mode” may be set on the setting menu screen of the “composite shooting mode” so as to be stored in the memory card 112a. In this case, when the release button 111b is pressed in the “composite shooting mode”, a plurality of continuously shot images are stored in the memory card 112a without being combined. Alternatively, both the composite image and a plurality of continuously shot images may be stored in the memory card 112a. Further, for example, when the number of continuous shots is 3, so that a plurality of continuously shot images can be managed collectively, it is possible to use “gousei-1. jpg, gousei-2. jpg, gousei-3. As jpg, it associates with the same name, adds a number indicating the photographing order to the end, and stores it in the memory card 112a. Alternatively, a folder may be created every time continuous shooting is performed in the “composite shooting mode”, and a plurality of continuously shot images may be stored together in one folder. Alternatively, unique information that associates a plurality of images may be added as header information of the image file. Further, in this case, information regarding whether or not the image is taken by emitting light from the auxiliary light source 113 may be added. As a result, there is no need to fix the order of images to be shot with light emission, and even when the order of images to be shot with light emission is changed for each shooting, light is emitted from a plurality of images read from the memory card 112a. Can be used to identify the image.

  The auxiliary light source 113 corresponds to a flash mounted on a general camera, and is composed of a light emitting element that emits a flash to a subject during photographing. Then, a flash is emitted at a timing commanded by the control unit 108.

  The above is the configuration and basic operation of the electronic camera 101. When a single image processing apparatus according to the present invention is configured instead of the electronic camera 101, the optical system 102, the mechanical shutter 103, the image sensor 104, the A / D conversion unit 105, and the auxiliary light source 113 in FIG. The processing of the control unit 108 that reads out image data from the memory card 112a to the image buffer 106 corresponds to the processing of the image input unit of the image processing device. Then, the control unit 108 (corresponding to the control unit of the image processing apparatus) reads captured image data from the memory card 112a to the image buffer 106, performs image processing in the image processing unit 107, and displays the image on the display unit 110. Alternatively, the image data after image processing is stored again in the memory card 112a. Here, in the case of an image processing apparatus, the operation unit 111 is provided with a playback button for reading a captured image from the memory card 112a into the image buffer 106 in place of the release button 111b, and the imaging displayed on the display unit 110. An image on which image processing is to be performed is selected from among the completed images with the cross key 111d. Further, an image processing selection dial for selecting image processing contents is provided instead of the shooting mode selection dial 111a, and the image processing selected with the image processing selection dial is performed on the image selected with the cross key 111d. For example, when “compositing processing mode” is selected with the image processing selection dial, a plurality of continuously shot image data corresponding to the image selected with the cross key 111 d is read from the memory card 112 a to the image buffer 106. The process of combining the plurality of continuously shot image data read out to the image buffer 106 by the image processing unit 107 is the same as in the case of the electronic camera 101. As described above, the image processing apparatus can be realized as a dedicated image processing apparatus that reads out image data from the memory card 112a in which a plurality of images photographed in the “composite photographing mode” of the electronic camera 101 is stored, and performs composition processing.

  Note that the image processing apparatus may be realized by a personal computer. In this case, the control unit 108 corresponds to the CPU of the personal computer, reads the image data from the memory card 112a connected to the memory card IF of the personal computer, and displays it on the monitor instead of the display unit 110. It is assumed that the image composition program is installed in the personal computer.

[Composite shooting mode]
Next, the “composite shooting mode” of the electronic camera 101 according to the present embodiment will be described. In general, when shooting a subject that exceeds the dynamic range that can be shot with an electronic camera, white areas in bright areas and black areas in dark areas occur. Therefore, an image in which a bright part is photographed with appropriate exposure and an image in which a dark part is photographed with appropriate exposure are combined to create an image in which both a bright part and a dark part are clearly photographed.

  For example, in FIG. 2, as shown in the example of the subject 151, when a person is photographed against a night scene (shooting composition 152), an image 153 shot with a flash and an image 154 shot without a flash are combined, It is possible to obtain an image 155 in which both the person and the background are combined so as to have good exposure (appropriate or close to appropriate exposure).

  However, in the composition processing for superimposing the image 153 and the image 154, it is necessary to align the main subject (the subject that the photographer wants to photograph most beautifully) as a reference for superimposition, but the image 153 and the image 154 However, since the brightness of the main subject is greatly different, it is impossible to accurately detect the position of the main subject or to determine whether or not the subject is the same by comparing the colors and the contours. It becomes difficult to align the main subject.

  In particular, when a plurality of images (for example, five images) that are continuously shot are combined, the positional shift of the main subject between temporally adjacent images is small, but the first image and the fifth image There is a possibility that the position of the main subject is greatly deviated due to camera shake or the like. For this reason, when a process of adding or averaging signals at the same pixel position of two images (referred to as simple synthesis) is performed, there is a possibility that the position of the main subject is greatly blurred and synthesized.

  Therefore, in the electronic camera 101 according to the present embodiment, by changing the composition order and composition method, even when there are images with greatly different luminances in a plurality of continuously shot images, the main subject is misaligned. It is possible to synthesize so as to reduce it.

[Image composition method]
Next, an image composition method in the electronic camera 101 according to the present embodiment will be described. FIG. 3 shows an example of a photographed image when the number of continuous shots is five as an example of a plurality of continuously shot images. In FIG. 3, the image 161 is the first (first) image, the image 162 is the second image, the image 163 is the third image, and the image 164 is 4 in the order of shooting. An image captured on the first sheet, image 165, is an image captured on the fifth (last) image. In addition, four images 161, 162, 163, and 164 are images taken without a flash, and an image 165 is an image taken with a flash. Here, in normal shooting without flash, the aperture value and the shutter speed are controlled so as to obtain an appropriate exposure according to the brightness of the subject, but in the “composite shooting mode” of the electronic camera 101 according to the present embodiment. Images shot without a flash are shot with underexposure (underexposure) than normal shooting.

  In the example of FIG. 3, the state in which the position of the person (here, the main subject) is slightly shifted to the left side due to camera shake or the like is exaggerated. Actually, the shift of the main subject is hardly noticeable between temporally adjacent images (for example, the image 161 and the image 162), but between the first captured image 161 and the last captured image 165. The position shift of the main subject becomes large, and there is a problem that when the simple composition is performed, the main subject is greatly shaken and synthesized. In such a case, it is necessary to align the main subject between the image 161 and the image 165. However, it is possible to detect the position of the main subject by comparing the colors and the luminance, and determining whether or not they are the same subject. Therefore, it is difficult to align the main subjects of the two images.

  Therefore, in the electronic camera 101 according to the present embodiment, the composition order and composition method when compositing a plurality of continuously shot images as in the example of FIG. 3 are devised. First, an image 165 obtained by flash shooting the main subject with appropriate exposure is set as a reference image (first reference image). Next, an image 164 that is temporally closest to the image 165 is set as a reference image (second reference image) of the combining process (first combining process) performed first.

  In the first compositing process, the image 164 is composed of three images 161 to 163 taken by flash. At this time, since there is a misalignment between the position of the main subject of the image 164 and the position of the main subject of the image 161, the image 161 is positioned so that the position of the main subject of the image 161 matches the position of the main subject of the image 164. 161 is shifted to the left to compose the image 164. In the synthesis processing method, signals at the same pixel position after alignment are added or averaged. In the case of a color image, the same processing is performed for each color.

  Next, the image 162 is combined with the image 164 after combining the image 161. Also in this case, after aligning the main subject of the image 161 with reference to the position of the main subject of the image 164, the image 162 is combined with the image 164.

  Similarly, the image 163 is combined with the image 164 after combining the image 161 and the image 162. Also in this case, after aligning the main subject of the image 163 with reference to the position of the main subject of the image 164, the image 163 is combined with the image 164.

  In this manner, the three images 161 to 163 can be combined with the position of the main subject in the image 164 as a reference. Note that the four images from image 161 to image 164 are all images taken without a flash, so the exposure conditions of the main subject are similar, and the position of the main subject by color comparison, luminance comparison, etc. It is easy to detect, and it is possible to accurately determine whether or not they are the same subject.

  Next, a process of combining the image 164 after the three images 161 to 163 are combined with the flash-captured image 165 in which the main subject is captured with appropriate exposure (second combining process). I do.

  In the second combining process, the signals at the same pixel position are simply added or averaged without aligning the main subject between the image 164 and the image 165 after the three images 161 to 163 are combined. Perform simple synthesis. In this case, as described above, since the image 164 and the image 165 are temporally continuous, the positional deviation is small. Further, the four images from image 161 to image 164 are taken with no flash and underexposure, so the main subject is dark, and the slight displacement of the main subject in the combined image is hardly noticeable. Note that the processing for aligning and synthesizing the image of the auxiliary light source 113 and the main subject that is captured with the image of the auxiliary light source 113 that has not been light-emitted is a feature amount such as color tone or luminance. However, the processing is difficult, and if it is to be realized, complicated arithmetic processing and an expensive processing device are required. On the other hand, after the remaining non-light-emitting images are synthesized with a non-light-emitting image that is temporally adjacent to the image captured with light emission and having a small positional deviation as a second reference image, as in the present embodiment. By simply synthesizing the luminescent image, it is possible to generate a synthesized image that is inexpensive and inconspicuous in positional deviation. It should be noted that the processing for detecting and aligning the main subject between images with similar brightness (non-light-emitting images) reduces the processing load compared to the alignment processing between images with significantly different brightness. Even an inexpensive CPU with inferior performance can be performed relatively easily.

  In this way, an image 166 obtained by synthesizing five images 161 to 165 can be obtained. In addition, since the angle of view of the image 166 after synthesis is shifted due to the influence of camera shake or the like, there may be a feeling of strangeness in the image after synthesis. Therefore, as a post-combination process, a trimming process may be performed to cut out overlapping portions of the angle of view from the image 161 to the image 165, and the trimmed image 167 may be stored in the memory card 112a as a final image.

  In the “composite shooting mode”, an image without flash is shot with underexposure. In this case, a numerical value of underexposure is set in advance in the memory 109 as a parameter of “composite shooting mode”. Here, the numerical value for underexposure is the number of adjustment steps for the aperture value and shutter speed for proper exposure. For example, the underexposure is performed by reducing the aperture value by 1 step from the aperture value for proper exposure or by increasing the shutter speed by 1 step. Alternatively, the underexposure value may be automatically set according to the number of images shot without a flash. For example, when the number of captured images without flash is four, the brightness of the image after addition is quadrupled, so that the aperture value for proper exposure is reduced by two steps, or the shutter speed is increased by two steps. As a result, the image after the four non-flash photographed images are combined has a brightness close to proper exposure. Furthermore, since noise is averaged by the synthesis process, an effect of noise removal can be obtained. In addition, since the shutter speed when shooting an image without a flash can be increased, camera shake can be prevented. This facilitates the detection and alignment of the main subject and improves the image quality after synthesis.

  As described above, the electronic camera 101 according to the present embodiment can obtain a background image (such as a night view) with less noise by synthesizing a plurality of non-flash images taken by underexposure (first synthesis process). . Furthermore, by synthesizing a flash photographed image in which the main subject is photographed at an appropriate exposure with the composite image (first composite image) of the photographed image without flash (second composition processing), both the main subject and the night view are clean. A photographed composite image (second composite image) can be obtained.

[Configuration Example of Image Processing Unit 107]
Next, a configuration example of the image processing unit 107 will be described with reference to FIG. In FIG. 4, the same reference numerals as those in FIG. In FIG. 4, the image processing unit 107 includes a reference image selection unit 201 and a synthesis unit 202. Furthermore, the synthesis unit 202 includes a synthesis order determination unit 251, a first synthesis unit 252, and a second synthesis unit 253. Further, the control unit 108 includes a composite image capturing control unit 301 that executes a process of generating a composite image when the “composite capturing mode” is selected by the operation unit 111.

  First, the photographer selects “composite shooting mode” with the shooting mode selection dial 111 a of the operation unit 111. When the photographer presses the release button 111 b, the composite image capturing control unit 301 of the control unit 108 starts capturing with reference to the “composite capturing mode” parameter stored in advance in the memory 109. Here, the “composite shooting mode” parameters stored in the memory 109 are the number of continuous shots: 5 and the flash shooting position: 5th. When the numerical value of underexposure is also set as a parameter of “composite shooting mode”, for example, it is assumed that the underexposure with the shutter speed increased by two steps is set.

  When the release button 111b is pressed, the composite image capturing control unit 301 instructs the image capturing unit 104 to start continuous shooting. Here, after four images that are temporally continuous are captured in the image buffer 106, the auxiliary light source 113 is caused to emit light, and the final fifth image is captured and captured in the image buffer 106. At this stage, the five captured images captured in the image buffer 106 may be stored in the memory card 112a.

  Next, the composite image capturing control unit 301 instructs the image processing unit 107 to instruct the image processing unit 107 to synthesize the five captured images captured in the image buffer 106. The reference image selection unit 201 of the image processing unit 107 selects an image without flash photography that is temporally closest to the image (first reference image) that has been photographed with flash from among the five photographed images captured in the image buffer 106. Select as the second reference image. Note that it is possible to know which image the flash photographed image is by referring to the “composite photographing mode” parameter stored in the memory 109. Alternatively, when a plurality of images that have been shot in the “composite shooting mode” stored in the memory card 112a are read out and processed, an image with a relatively large average luminance level (or relative) is selected from the plurality of images. In other words, it may be determined that the image having the highest average luminance level) is the image captured by flash.

  When the second reference image is selected by the reference image selecting unit 201, the combining order determining unit 251 of the combining unit 202 determines the order of combining a plurality of continuously shot images as described with reference to FIG. . First, in the process of combining a plurality of non-flash images excluding the second reference image with the second reference image, the combining order determination unit 251 first, for example, combines a plurality of non-flash images in time series order, reverse time series order, Alternatively, it is determined to be random. It is assumed that whether to combine in time series order, reverse time series order, or random order is determined in advance. Alternatively, it may be stored in the memory 109 as a “composite shooting mode” parameter and can be changed in the setting menu.

  In the example of FIG. 3, since the fifth image 165 is the first reference image and the fourth image 164 is the second reference image, the synthesis order determination unit 251 selects the three images 161 to 163 as the image 164. Determine the order of synthesis. For example, when the composition order is time-series order, the image 161 is first synthesized with the image 164 as a reference, and then the image 162 is synthesized with the synthesized image 164 as a reference, and the image 163 with the synthesized image 164 as a reference. Is output to the first combining unit 252 so as to be combined. In the case of reverse time series order, the image 163 is first synthesized with the image 164 as a reference, then the image 162 is synthesized with the synthesized image 164 as a reference, and the image 161 is taken with the synthesized image 164 as a reference. The result is output to the first combining unit 252 so as to be combined. Alternatively, in the case of random order, for example, first, the image 162 is synthesized with the image 164 as a reference, then the image 161 is synthesized with the synthesized image 164 as a reference, and the image 163 is synthesized with the synthesized image 164 as a reference. The result is output to the first combining unit 252 so as to be combined.

  The first synthesizing unit 252 performs a process of synthesizing the non-flash photographed image excluding the second reference image with the second reference image (first synthesis process). In the first combining process, a preset main subject is detected from each of the two images to be combined, and the positions of the main subjects of both images are adjusted. Here, the position shift of the main subject between two images that are temporally adjacent in a continuously shot image can be almost ignored if the continuous shooting interval is short, but between images that are taken apart in time. The position shift of the main subject cannot be ignored when combining. For example, in the example of FIG. 3, the displacement of the main subject between the image 164 and the image 163 or between the image 164 and the image 165 can be almost ignored, but the displacement of the main subject between the image 164 and the image 161 is large. It becomes a problem at the time of synthesis. Therefore, the first combining unit 252 detects the main subject at the time of combining and detects the position shift of the detected main subject, and performs the processing after matching the position shift of the main subject between the two images to be combined. This is shown in FIG. Here, the main subject is a person. The main subject detection method is a well-known technology already implemented by an electronic camera with a face detection function, so detailed explanations are omitted, but in the case of a person, the outline of the face, the arrangement of eyes, nose and mouth It is possible to detect using a preset feature amount. Also, between images taken without flash, the subject's color tone, brightness, etc. are similar, so there is little variation between images in terms of whether the main subject is the same subject or the location of the main subject. Can be detected. As described above, the main subjects detected between the image 165 captured with the flash and the image 164 captured without the flash differ greatly in color tone and brightness, so whether or not they are the same main subject. However, it is difficult to accurately detect the position of the main subject.

  FIG. 5 is a diagram showing how the main subject is aligned when the image 161 and the image 164 shown in FIG. 3 are combined. In FIG. 5, a process of matching the main subject 171 detected in the image 161 with the position of the main subject 174 detected in the image 164 is performed. In this case, first, the pixel coordinates of the central portion of the main subject 171 of the image 161 (the nose position at the center of the human face in the example of FIG. 5) and the central portion of the main subject 174 of the image 164 (similarly, the position of the nose) ) To obtain the positional deviation amount d from the pixel coordinates. Then, the image 161 is simply combined with the image 164 of the second reference image at a position shifted by the positional shift amount d. Thereby, the horizontal central axis 180 of the main subject 171 of the image 161 and the main subject 174 of the image 164 can be matched. In the example of FIG. 5, an example in which the position is shifted in the horizontal direction is shown. However, even when the position is shifted in the vertical direction, the center axis 181 in the vertical direction of the image 161 and the center in the vertical direction of the image 164 are the same. A positional deviation amount with respect to the axis 184 is obtained, and the image 161 is shifted in the vertical direction and synthesized with the image 164. Further, even when there is a positional shift in both the horizontal direction and the vertical direction, the above two alignments can be combined and combined.

  Here, although the case where the image 161 is combined with the image 164 has been described in the above example, the process of combining the image 162 and the image 163 with the image 164 can be performed in the same manner.

  In this way, the first synthesizing unit 252 uses the position of the main subject of the image 164 as the second reference image as a reference, and images taken without the flash excluding the second reference image (images 161 to 163 3). Two images) can be combined (first combining process).

  Note that the four images from image 161 to image 164 are all images taken without a flash, so the exposure conditions of the main subject are similar, and the position of the main subject by color comparison, luminance comparison, etc. It is possible to accurately detect and determine whether or not the subject is the same.

  Next, the second synthesis unit 253 performs a process of synthesizing the first reference image with the image synthesized by the first synthesis process (second synthesis process). In the second combining process, signals at the same pixel position in two images are simply combined without performing alignment.

  FIG. 6 is a diagram illustrating a state when the image 165 illustrated in FIG. 3 and the image 164 after the first combining process are combined by the first combining unit 252. In the example of FIG. 6, the image 165 and the first combined image 164 'are simply combined as they are at the same pixel position.

  Here, since the first reference image (image 165) and the second reference image (image 164) are temporally continuous images, it can be considered that there is almost no positional deviation when the continuous shooting interval is short. Even in simple composition, the positional deviation of the main subject after composition is hardly noticeable.

  In this way, the second combining unit 253 combines the image 165 of the first reference image taken with flash and the image 164 ′ after combining the plurality of images without flash with the second reference image. Processing can be performed. As a result, it is possible to obtain a composite image (second composite image) in which both the main subject and the night view are beautifully photographed.

  Next, a process for capturing an image in the “composite shooting mode” and a process for combining captured images will be described.

  FIG. 7 is a flowchart illustrating processing when an image is captured in the “composite shooting mode”. 7 is performed by the composite image capturing control unit 301 of the control unit 108.

  (Step S101) The photographer selects “composite shooting mode” with the shooting mode selection dial 111a of the operation unit 111 before starting shooting.

  (Step S102) The composite image capturing control unit 301 determines whether or not the photographer has pressed the release button 111b of the operation unit 111, and proceeds to the next step when the release button 111b is pressed. At this time, the composite image capturing control unit 301 sets the count value of the counter indicating the number of the next image to be captured during continuous shooting to 1 (first image). The counter is a software variable of an image processing program executed by the control unit 108.

  (Step S103) The composite image capturing control unit 301 refers to the “composite capturing mode” parameter stored in the memory 109, and the count value of the counter indicates the image position for flash capturing (in this example, the fifth image) ). If it is an image position for flash photography, the process proceeds to step S104. If it is not an image position for flash photography, the process proceeds to step S105.

  (Step S <b> 104) The composite image shooting control unit 301 controls the aperture value of the optical system 102 and the shutter speed of the mechanical shutter 103 to cause the auxiliary light source 113 to emit light, and the image captured by the image sensor 104 is A / D conversion unit 105. To the image buffer 106 (shooting with flash). Then, the composite image capturing control unit 301 increments the count value of the counter that indicates how many images are to be captured next. In this flash photography, the exposure is controlled so as to achieve an appropriate exposure.

  (Step S <b> 105) The composite image capturing control unit 301 controls the aperture value of the optical system 102 and the shutter speed of the mechanical shutter 103, and images captured by the image sensor 104 via the A / D conversion unit 105. (Shooting without flash). Then, the composite image capturing control unit 301 increments the count value of the counter that indicates how many images are to be captured next. In this photographing, the composite image photographing control unit 301 refers to the “composite photographing mode” parameter stored in the memory 109 and photographs the underexposure value lower than the appropriate exposure. In addition, it is assumed that the proper exposure in shooting without flash is obtained from a photometric value when the release button 111b is pressed (for example, photometry from the output of the image sensor 104).

  (Step S106) The composite image capturing control unit 301 reduces the number of captured images indicated by the count value of the counter (the count value is the number of the next captured image, which is one more than the number of captured images, so it is reduced by one). It is determined whether or not (the number of images) matches the number of consecutive shots in the “composite shooting mode” stored in the memory 109. If the continuous shooting has been completed, the “composite shooting mode” is ended. If the continuous shooting has not been completed, the process returns to step S103 to repeat the same shooting process. For example, if the continuous shooting number of the “composite shooting mode” parameter is 5 and the count value of the counter indicating the number of the next image to be shot is 6 (the number), the shooting is ended. If the count value is less than or equal to the fifth sheet, the process returns to step S103.

  In this way, when the “composite shooting mode” is selected with the shooting mode selection dial 111 a, when the release button 111 b is pressed, a predetermined composite mode is stored according to the “composite shooting mode” parameter stored in the memory 109. A predetermined number of images taken continuously by flash photography at the image position are taken into the image buffer 106.

  (Step S107) The composite image capturing control unit 301 stores the continuous shot image captured in the image buffer 106 in Step S106 in the memory card 112a via the memory card IF 112. Alternatively, the composite image is displayed on the display unit 110.

  The memory card 112a can be connected to a personal computer, a dedicated image processing apparatus, or the like, and a personal computer or an image processing apparatus can perform composition processing of a plurality of images photographed in the “composite photographing mode”. In this case, the personal computer and the image processing apparatus are equipped with an image processing program for executing an image composition process as described below.

  Next, a process of combining a plurality of images continuously shot in the “composite shooting mode” will be described with reference to the flowchart of FIG. Note that the processing in FIG. 8 is performed by the image processing unit 107 in accordance with a command from the composite image capturing control unit 301 of the control unit 108. Note that the image processing unit 107 includes the processing blocks described with reference to FIG.

  (Step S <b> 201) The reference image selection unit 201 reads a plurality of images shot in the “composite shooting mode” from the image buffer 106. In the case of a personal computer or an image processing apparatus, a plurality of images shot in the “composite shooting mode” are read from the memory card 112a.

  (Step S202) The reference image selection unit 201 selects a reference image from a plurality of images read out in step S201. As described above, the reference image selected by the reference image selection unit 201 is the first reference image that is temporally closest to the first reference image among the first reference image that has been shot with flash and the image that has not been shot with flash. Two of the two reference images.

  (Step S203) The composition order determination unit 251 determines the order in which a plurality of continuously shot images are to be combined. Since the photographed image with flash is combined last, the processing here determines the order in which a plurality of images without flash excluding the second reference image are combined with the second reference image. As described above, the method for determining the composition order is, for example, a plurality of non-flash images in time series order, reverse time series order, or random.

  (Step S204) The first combining unit 252 combines a plurality of non-flash photographed images according to the combining order determined in step S203. As described above, the composition method executes the first composition processing for performing composition by aligning the main subject.

  (Step S205) The second synthesizing unit 253 synthesizes the image obtained by synthesizing the plurality of non-flash photographed images and the flash photographed image (first reference image) in Step S204. Note that, as described above, the composition method executes the second composition processing for performing composition by aligning the pixel main subjects at the same position.

  (Step S206) The image processing unit 107 outputs the image after the second combining unit 253 combines. Note that the combined image is temporarily held in the image buffer 106 until the next shooting is performed. Then, the control unit 108 displays the combined image held in the image buffer 106 on the display unit 110, or a file name (for example, a serial number or shooting date / time) determined in advance for the image data of the combined image. Is automatically added and stored in the memory card 112a via the memory card IF 112.

  In this way, the electronic camera 101 according to the present embodiment can synthesize a plurality of images that are continuously shot in the “composite shooting mode”. In particular, by synthesizing a flash image in which the main subject is captured at an appropriate exposure with a composite image of an image captured without a flash, it is possible to obtain a composite image in which both the main subject and the night scene are clearly captured. . Further, in the electronic camera 101 according to the present modification, the composition order is determined based on the flash-captured image captured by emitting the auxiliary light source 113, so that the composition of the main subject is small without performing complicated processing. An image can be generated. Furthermore, since the electronic camera 101 according to the present embodiment combines a plurality of non-flash images taken with underexposure, the noise component is canceled and a background image (particularly a night view) having a high S / N ratio can be obtained. .

(Modification 1 of composition order)
Next, a modification of the composition order of the embodiment described above will be described with reference to FIG. In the example of FIG. 3, the image position for flash shooting is the last image in continuous shooting, but in this modification, the first image 261 is flash-shot as shown in FIG. 9, and the remaining four images (images) are displayed. 262 to image 265) without flash.

  In FIG. 9, the reference image selection unit 201 of the image processing unit 107 selects an image 261 as a first reference image and an image 262 that is temporally closest to the first reference image as a second reference image.

  Then, the composition order determination unit 251 first determines the order in which the non-flash captured image excluding the second reference image is combined with the second reference image. Here, as described above, the composition order is, for example, a plurality of non-flash images in time series order, reverse time series order, or random.

  In the example of FIG. 9, the first image 261 is the first reference image, and the second image 262 is the second reference image. Therefore, the composition order determination unit 251 converts the three images 263 to 265 into the image 262. Determine the order of synthesis. For example, when the composition order is time-series order, first, the image 263 is synthesized with the image 262 as a reference, then the image 264 is synthesized with the synthesized image 262 as a reference, and the image 265 with the synthesized image 262 as a reference. Are determined to be combined and output to the first combining unit 251. Note that the order of combining the photographed images 261 with flash is combined last regardless of the position of the image captured with flash.

  The first synthesizing unit 252 synthesizes a plurality of images taken without a flash, excluding the second reference image, according to the synthesis order determined by the synthesis order determining unit 251 with the second reference image. Note that the main subject of the two images to be combined is aligned and combined.

  Then, the second synthesis unit 253 simply synthesizes the image 261 captured by the first synthesis unit 252 with the image that has been flash captured.

  As described above, the electronic camera 101 according to the present modification can perform the composition processing similarly to the case of FIG. 3 even when the image position to be captured with the flash is the first image in the continuous shooting, and can be captured without the flash. By synthesizing a flash photographed image in which the main subject is photographed at an appropriate exposure with the composite image of the obtained images, a composite image in which both the main subject and the night scene are photographed beautifully can be obtained. In particular, in the electronic camera 101 according to the present modification, the composition order is determined based on the flash-captured photographed image taken by the auxiliary light source 113, so that a composite image with little subject displacement is generated. Can do.

(Modification 2 of composition order)
Next, modification 2 of the composition order will be described with reference to FIG. In the example of FIGS. 3 and 9, the image position to be photographed with the flash is the first or last image in the continuous shooting, but in this modification, as shown in FIG. 10, the image in the middle of continuous shooting (for example, the image 363) is flashed. The remaining images before and after (for example, image 361, image 362, image 364, and image 365) are captured without flash.

  In FIG. 10, the reference image selection unit 201 of the image processing unit 107 sets the image 363 as the first reference image, and the two images before and after the first reference image that are temporally closest to the first reference image (the image 362 in the example of FIG. 10). Image 364) is selected as the second reference image. Here, the difference from the previous embodiment is that the closest image in time is selected from the images of the flashless images captured before the imaging time of the first reference image, and the first reference image The image closest to the time is selected from images taken without a flash taken after the shooting time. The first compositing unit 252 then combines the non-flash photographed image portion before the first reference image photographing time and the non-flash photographed image composition processing portion after the first reference image photographing time. The synthesis process is performed separately. That is, the first synthesis process described in the previous embodiment is executed in two stages. Here, for the sake of convenience, the composition processing of the non-flash photographed image in the portion before the photographing time of the first reference image is the first composition processing A, and the composition processing of the non-flash photographed image in the portion after the photographing time of the first reference image. Is referred to as a first synthesis process B.

  In FIG. 10, the first combining unit 252 combines an image 361 with the image 362 of the second reference image in the first combining process A. In this case, the composition is performed by performing processing for matching the position of the main subject of the image 361 with the position of the main subject of the image 362. In the example of FIG. 10, there are only two images to be combined. However, when there are a plurality of images separated in time as in the example of FIG. 3, the combining order determination unit 251 determines the order of combining, The images are combined after the main subject is aligned between the images to be combined. In this way, in the first synthesis process A, an image A in which the image 361 is synthesized so as to match the position of the main subject of the second reference image 362 is obtained.

  Similarly, the first combining unit 252 combines the image 365 with the image 364 of the second reference image in the first combining process B. In this case, the composition is performed after the position of the main subject in the image 365 is aligned with the position of the main subject in the image 364. In this case as well, there are only two images to be combined. However, when there are a plurality of images that are separated in time as in the example of FIG. 3, the combining order determination unit 251 determines the combining order and combines them. Composite after alignment of main subject between images. In this way, in the first combining process B, an image B in which the image 365 is combined so as to match the position of the main subject in the second reference image 364 is obtained.

  Then, the second synthesis unit 253 performs a process of synthesizing the image A and the image B after the synthesis of a plurality of images taken without flash and the image 363 taken with flash. Also in this case, the second synthesis process described in the previous embodiment is executed in two stages. Here, for convenience, the process of combining the image A with the first reference image is referred to as a second combining process A, and the process of combining the image B with the first reference image is referred to as a second combining process B. As described above, the synthesis method of the second synthesis process A and the second synthesis process B is a simple synthesis that synthesizes signals at the same pixel position. In this way, it is possible to obtain an image obtained by combining the composite image A and the image B of a plurality of images photographed without flash and the image 363 photographed with flash.

  As described above, the electronic camera 101 according to the present modification can perform the composition processing similarly to the case of FIG. 3 even when the image position to be captured with the flash is an image in the middle of continuous shooting, and is captured without the flash. By synthesizing a flash photographed image in which the main subject is photographed with appropriate exposure to the composite image of the obtained images, it is possible to obtain a composite image in which both the main subject and the night scene are photographed beautifully. In particular, in the electronic camera 101 according to the present modification, the composition order is determined based on the flash-captured photographed image taken by the auxiliary light source 113, so that a composite image with little subject displacement is generated. Can do.

  As described above, the image processing apparatus, the electronic camera, and the image processing program according to the present invention have been described by way of example in each embodiment. However, in various other forms without departing from the spirit or main features thereof. Can be implemented. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The present invention is defined by the claims, and the present invention is not limited to the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

DESCRIPTION OF SYMBOLS 101 ... Electronic camera; 102 ... Optical system; 103 ... Mechanical shutter; 104 ... Image pick-up element; 105 ... A / D conversion part; 106 ... Image buffer; Processing unit; 108 ... Control unit; 109 ... Memory; 110 ... Display unit; 111 ... Operation unit; 111a ... Shooting mode selection dial; 111b ... Release button; 111d ... cross key; 112 ... memory card IF; 112a ... memory card; 113 ... light emitting unit; 201 ... reference image selection unit; 202 ... compositing unit; 251 ... Composition order determination unit; 252 ... First composition unit; 253 ... Second composition unit; 301 ... Composite image photographing control unit

Claims (7)

An image input unit for inputting a plurality of images photographed continuously in time including images photographed by emitting an auxiliary light source;
A light emission photographed image taken by emitting the auxiliary light source is a first reference image, and a non-light emission photographed image taken without causing the auxiliary light source to emit light is closest in time to the first reference image. A reference image selection unit for selecting each of the light emission captured images as a second reference image;
An image combining unit that performs a first combining process for combining the plurality of non-light-emitting captured images with the second reference image and a second combining process for combining the image after the first combining process with the first reference image. And an image processing apparatus. The image processing apparatus according to claim 1.
The first combining process detects a position of a main subject that is set in advance for each of the plurality of non-light-emitting captured images, and adjusts the position of the non-light-emitting captured image so as to match the position of the main subject of the second reference image. Adjust the position of the main subject and synthesize,
The image processing apparatus, wherein the second combining process combines the first reference image at the same pixel position with the image combined by the first combining process. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the plurality of non-light-emitting captured images are captured with underexposure on the main subject, and the light-emitting captured images are captured with appropriate exposure with respect to the main subject. The image processing apparatus according to any one of claims 1 to 3,
The first reference image captured by emitting the auxiliary light source is the first or last image among the images captured continuously in time. The image processing apparatus according to any one of claims 1 to 3,
The first reference image captured by emitting the auxiliary light source is an image other than the first or last image among the images continuously captured in time.
The reference image selecting unit selects two non-light-emitting photographed images temporally before and after the first reference image among the plurality of non-light-emitting photographed images, respectively, as a front second reference image and a rear second reference. Select as image,
In the first combining process, the image combining unit converts the plurality of non-light-emitting captured images temporally before the first reference image into the front second reference image in terms of time from the first reference image. The plurality of non-light-emitting photograph images after are combined with the rear second reference image,
In the second combining process, the front second reference image and the rear second reference image combined by the first combining process are combined with the first reference image at the same pixel position. A featured image processing apparatus. An electronic camera comprising the image processing device according to claim 1,
The image input unit includes: an imaging unit that captures a plurality of images continuously in time; and a light-emitting unit that emits an auxiliary light source when capturing an image at a preset time position among the plurality of images. Configure
An electronic camera, further comprising: a recording unit configured to record an image synthesized by the image synthesizing unit on a storage medium.   An image processing program for causing a computer to execute the function of the image processing apparatus according to any one of claims 1 to 5.

Images