JP4497001B2 - Image processing apparatus, electronic camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an electronic camera, and an image processing program that combine images generated by a plurality of shootings to generate a combined image.

2. Description of the Related Art Conventionally, there has been known an electronic camera having a configuration in which a plurality of images generated by repeated imaging by divided exposure are combined to finally obtain one combined image (see, for example, Patent Document 1).
In the electronic camera disclosed in Patent Document 1, a plurality of images with a limited dynamic range are generated, and these are combined to obtain an image with an extended dynamic range. Further, when images are combined, alignment is performed in order to correct displacement of each image due to camera vibration or the like during shooting when imaging is repeated a plurality of times.
JP 2004-159292 A

  However, the above-described correction of misalignment by alignment is effective in that the alignment calculation takes time, for example, when the movement of the subject itself is large, especially when the shape changes during multiple shooting. In addition to being unable to combine, a plurality of differently shaped subjects appear on the finally obtained image, resulting in a new inconvenience due to the combination.

  An object of the present invention is to provide an image processing apparatus, an electronic camera, and an image processing program that can generate a high-quality composite image by effectively combining images according to a subject.

An image processing apparatus according to the present invention includes: a determination unit that determines a reference image of one frame from a predetermined number or more images obtained by repeatedly capturing an image of an object scene; and For each comparison image composed of images excluding the reference image, an evaluation unit that evaluates the degree of approximation using the reference image as a comparison target, and from the comparison image in descending order of evaluation by the evaluation unit (the predetermined number) -2) It is characterized by comprising a synthesis unit that selects a frame image as a selected image and generates a synthesized image using the selected image and the reference image.

The synthesizing unit performs an alignment process on each of the selected images with reference to the reference image, and an addition process between each of the selected images subjected to the alignment process and the reference image The composite image may be generated by
In addition , the evaluation unit, the amount of movement of each of the comparative images of the main subject in the object scene, the degree of deformation of each of the comparative images of the main subject, the focus state of each of the comparative images, the evaluation may me a line as a factor of at least one of the respective brightness of the comparison image.

Further , the evaluation unit divides each of the comparison images into a plurality of regions, the amount of movement of the subject in each region, the degree of deformation of the subject in each region, the focus state in each region, the evaluation of at least one as a factor of the brightness distribution in the region may be I lines.

The determination unit may determine the reference image based on at least one of brightness, contrast, and high frequency component in each of the predetermined number or more of images .
In addition , the image capturing apparatus may include an imaging unit that repeatedly captures an image of the object scene and generates the predetermined number or more images, and the determination unit uses an image captured at the beginning of the repeated imaging operation as the reference image. determined, the evaluation unit, in parallel with repetitive imaging operation by the imaging unit, the evaluation of the approximation degree can be I line.

In addition , the imaging unit includes an imaging unit that repeatedly captures an image of the object scene to generate the predetermined number of images and an illumination unit that performs preliminary light emission, and the imaging unit is in focus during preliminary light emission by the illumination unit. After the adjustment, the focus state may be fixed until the image of the object scene is repeatedly captured and the generation of the predetermined number of images or more is completed .

In addition , the image capturing apparatus includes an imaging unit that repeatedly captures an image of the object scene and generates the predetermined number or more images, and the determination unit selects a predetermined image from among images captured at the beginning of the repeated imaging operation. set the object, the imaging unit, while the being performed repetitive imaging operation, to the predetermined subject may be I line continuous automatic focusing operation.

An electronic camera according to the present invention is an electronic camera including any one of the above-described image processing apparatuses, repeatedly performing an image capturing operation of an object scene and performing a preliminary image capturing operation for continuously generating a plurality of images, From the imaging unit that performs the main imaging operation for repeatedly capturing the image of the object scene and generating the predetermined number of images, and the plurality of images continuously generated by the preliminary imaging operation When the amount of change is smaller than a predetermined threshold value, the extraction unit that extracts the feature amount of the image is compared with each of the feature amounts extracted by the extraction unit. The determination unit determines that the image is to be combined by the combining unit, the determining unit is configured to transmit one frame from the predetermined number or more of images obtained by the main imaging operation. Determine the reference image The evaluation unit evaluates the degree of approximation with respect to each of the comparison images including the reference image of the predetermined number or more images excluding the reference image, and the synthesis unit includes: The (predetermined number−2) frame image is selected as a selection image from the comparison image in descending order of evaluation by the evaluation unit, and a composite image is generated using the selection image and the reference image. And

It is also effective as a specific aspect of the present invention to express the configuration related to the above invention by converting it into an image processing program for realizing image processing on an image to be processed by a computer.

  According to the image processing apparatus, the electronic camera, and the image processing program of the present invention, it is possible to effectively synthesize images according to the subject and generate a high-quality synthesized image.

<< First Embodiment >>
The first embodiment of the present invention will be described below.
As shown in the functional block diagram of FIG. 1, the electronic camera 1 according to the first embodiment includes a control unit (circuit) 2, an imaging unit (circuit) 3, an image processing unit (circuit) 4, a recording unit (circuit) 5, An illumination unit (circuit) 6 and a photometry unit (circuit) 8 are provided, and these units are connected to each other via a bus. The electronic camera 1 also includes an operation unit 7 including a power button, a release button, and the like (not shown).

  The control unit 2 functioning as a controller includes a CPU and a memory (not shown), and a program for controlling each unit is recorded in the memory in advance. The control unit 2 also performs determination of a reference image, evaluation of an image, selection of an image to be combined, and the like, which are features of the present invention (details will be described later). In addition, the control unit 2 includes a counter (not shown) that counts the number of photographing. Further, the control unit 2 sets an imaging parameter (for example, an aperture value) by the imaging unit 3 based on the photometric data by the photometric unit 8.

  The imaging unit 3 includes an imaging lens (not shown), a shutter, an imaging device, an aperture, and the like, and generates an image by capturing a subject image with the imaging parameters set by the control unit 2. By adjusting the time for opening a shutter (not shown), it is possible to adjust the exposure amount to be exposed to the image sensor. Note that this shutter may be realized by an electronic shutter function of the image sensor or may be a combination of both.

In addition, the image processing unit 4 performs predetermined image processing such as white balance adjustment on the image signal generated by the imaging unit 3 and combines the images that are the characteristics of the present invention. The recording unit 5 records image data that has been subjected to image processing by the image processing unit 4, and a removable memory such as a memory card can be attached and detached.
The operation at the time of shooting of the electronic camera 1 having the above-described configuration will be described with reference to the flowchart of FIG. 2 and the timing chart of FIG. Below, the process in the case of synthesize | combining the image which is the characteristics of this invention is demonstrated.

  When an instruction to start shooting is given by the user via the operation unit 7, the control unit 2 determines the total exposure time and the number N of divisions in step S1. The control unit 2 determines the total exposure time and the number of divisions N, the photometric data obtained by the photometric unit 8, the image signal level obtained by the image pickup device of the image pickup unit 3, the sensitivity and noise performance of the image pickup device of the image pickup unit 3, and the divided exposure. It is determined according to various conditions such as the length of time (ie, shutter speed). Here, the description will be made assuming that the number N of exposure times is determined to be N = 4, for example, under such various conditions. This corresponds to, for example, a case where the signal level of the subject image picked up by the image pickup device is about ¼ of the appropriate level by a certain shooting parameter.

In step S2, the control unit 2 initializes a counter (not shown) in the control unit 2, and sets the count value n to n = 0.
In step S3, the control unit 2 prohibits image signal output from the imaging unit 3 at the timing of the pulse signal A shown in FIG. That is, the control unit 2 outputs the pulse signal shown in FIG. 3 to the imaging unit 3 at a constant cycle (for example, 32 ms). Then, for example, the imaging unit 3 reads the accumulated charge from the imaging device of the imaging unit 3 at the timing of the pulse signal B, and outputs an image signal. In the example of FIG. 3, when the shutter is in the open state, the stored charge is not read and the image signal is not output even if the pulse signal is input. The readout of the stored charge and the image signal at the timing is prohibited.

In step S <b> 4, the control unit 2 closes a shutter (not shown) of the imaging unit 3 after opening the time t. Time t is the time obtained by dividing the total exposure time by the division number N determined in step S1. In the present embodiment, an example of equally dividing by the division number N is shown, but this is not restrictive.
In step S5, the control unit 2 permits the readout of the accumulated charges and the output of the image signal at the timing of the pulse signal following the shutter closing operation. This is because the stored charge is read out and the image signal is output at the timing of the pulse signal B shown in FIG. For example, when the shutter opening time is 50 ms, the readout of the accumulated charge and the output of the image signal are prohibited at the timing of the pulse signal B, and the readout of the accumulated charge is performed at the timing of the pulse signal C following the pulse signal B. The output of the image signal is permitted.

  In step S6, the control unit 2 causes the imaging unit 3 to read out stored charges at the timing of the pulse signal B and output an image signal, thereby recording image data. The imaging unit 3 outputs an image signal of the imaging element of the imaging unit 3 at the timing of the pulse signal B. That is, at the timing of the next pulse signal after the shutter is closed, the accumulated charge accumulated in the image sensor before the shutter is closed is read and output as an image signal. The image signal output from the imaging unit 3 is subjected to signal processing such as A / D conversion, and then input to the image processing unit 4. Image data output from the image signal is not shown in the control unit 2. Recorded in the memory.

In step S7, the control unit 2 increments the count value of a counter (not shown) in the control unit 2 by 1, so that n = n + 1.
In step S8, the control unit 2 determines whether n = N × 2. Here, “N × 2” is an example of “a predetermined number or more” in the claims. In the present embodiment, “N + 1” is an example of the “predetermined number” in the claims. The reason for n = N × 2 is that an image including a spare is required when selecting an image suitable for composition in evaluation and selection of an image to be described later (details will be described later). In this embodiment, N × 2 is used, but this is not restrictive. The controller 2 repeats the processing from step S3 to step S8 until n = N × 2, and when n = N × 2, the process proceeds to step S9. At this time, an image of (N × 2) frames is recorded in a memory (not shown) in the control unit 2.

  In step S9, the control unit 2 determines a reference image. The control unit 2 obtains at least one of the image data level (brightness), the contrast, and the high-frequency component of the spatial frequency for each of the continuously shot images recorded in the memory in the control unit 2, and performs the most composition. A suitable image is determined as a reference image. For example, the brightest image whose image data level has not reached the saturation level, the image with the highest contrast value, the image with the highest high-frequency component, etc. are set as the reference image.

  In step S10, the control unit 2 evaluates the image. The control unit 2 uses the reference image determined in step S9 as a comparison target, and among the (N × 2) frame images recorded in the memory (not shown), the reference of each image (comparison image) excluding the reference image. The degree of approximation to the image is evaluated using factors such as the amount of motion of the main subject in the field, the focus state of the image, the degree of deformation of the subject, and the brightness of the image. The control unit 2 evaluates each factor numerically. The main subject is set by a method in which the user instructs and selects the reference image from the reference image, or the subject located at the center of the reference image is automatically set as the main subject.

-About the amount of movement of the main subject When the main subject (person) is moving against the stationary background, the entire image is aligned based on the main subject based on the movement information of the main subject. In the case of obtaining, the position of the main subject can be made to coincide with each other, but a composite image in which the background becomes a multiple image is obtained accordingly. The interval between the multiple images increases as the amount of movement of the main subject increases, and the image quality of the composite image deteriorates. Therefore, the smaller the amount of motion is, the higher the evaluation value is. Note that the motion amount may be evaluated not only in the main subject area on the screen but also in an area including other areas on the screen.

-About the focus state When the focus is blurred, the accuracy of block matching in image alignment (motion compensation processing) is lowered, and as a result, a blurred composite image is obtained. The block matching process is a general technique often used in the motion compensation process. For example, Japanese Patent Laid-Open No. 5-40828 discloses an example of the technique. Therefore, the evaluation value is increased as the change in the focus state is smaller. Note that the change of the focus state may be evaluated not only in the main subject area but also in an area including other areas.

Further, in order to suppress the change in the focus state, after the counter described in step S2 is initialized, preliminary illumination is performed by the illumination unit 6 to perform focus adjustment, and then in step S8, until n = N × 2. The focus state may be fixed. By maintaining the focus state in this way, changes in the focus state can be suppressed. Such processing may be performed automatically or may be performed in response to a user operation.
-Deformation degree of subject If the configuration including the enlargement / reduction processing is included in the comparison processing of the block of the reference image by the block matching process, if the degree of deformation of the subject is not large, the deformation (change in size) will occur over time. It is also possible to combine a plurality of frames of images including a subject to be included.

However, if the degree of deformation (including change in size) of the subject in the remaining image with respect to the subject in the reference image is large, there is a high possibility that an error will occur in the block matching process. A composite image that is a multiple image is obtained.
In this way, the probability that the block matching process cannot be handled increases as the degree of deformation (magnification) of the subject increases. Therefore, the evaluation value is increased as the deformation degree (magnification degree) of the subject is smaller.

  The degree of deformation of the subject is calculated by, for example, calculating a difference value between image data of a zoomed subject image obtained by subjecting the subject in the reference image to a scaling process and a subject image in the comparison target image. The determination is made based on the processing magnification and the difference value of the image data. When the magnification of the scaling process is 1 and the difference value of the image data is 0, it indicates that the subject in the reference image is the same as the subject in the image to be compared, and the evaluation value is Highest set.

This evaluation may be performed for all subjects in the reference image in order to improve the quality of the composite image, or only for some subjects such as the main subject in order to shorten the calculation time. Also good.
・ Brightness of the image In the comparison process of the block of the reference image by the block matching process, if there is a change in brightness between the reference image and the image to be compared, the block matching process is performed normally. Disappear.

For this reason, the brightness of the image to be compared is evaluated based on the brightness of the reference image, the evaluation value when the brightness is the same is set to the highest value, and the evaluation value is increased as the amount of change in brightness increases. Set low. This evaluation may be performed for the average brightness of the entire screen, or may be performed only for a part of the subject area in the screen, such as an area including the main subject.
In step S11, the control unit 2 selects an image to be synthesized according to the evaluation result performed in step S10. The control unit 2 selects an image (selected image) of 3 frames (that is, (N-2) frames) in order of highest evaluation from (N × 2) frame images recorded in a memory (not shown), The four-frame image is selected as an image to be combined with the reference image and supplied to the image processing unit 4.

In step S <b> 12, the control unit 2 combines the four frames of images determined in step S <b> 11 after performing alignment processing using the reference image as a reference.
The reference image is divided into a plurality of blocks, and a block matching process is performed between each of the divided blocks and the remaining image used for synthesis, and a motion vector of each block of the reference image is added to each image. Against it.

Based on the motion vector thus determined, the remaining image areas used for synthesis are associated with a certain area of the reference image, and the areas of the images associated with the motion vectors are determined. The image data is added to each pixel in the area after moving to the area position of the corresponding reference image.
Here, for example, when the size of the block is reduced and the enlargement / reduction process is performed in the comparison process with the block of the reference image by the block matching process, and the area of each image is moved to the area position of the corresponding reference image If the configuration is such that the interpolation processing is performed, it is also possible to synthesize a plurality of frames of images including a subject that deforms (including a change in size) with time.

In step S13, the control unit 2 records the composite image obtained as a result of the processing in step S12 in the recording unit 5, and ends a series of processing.
As described above, according to the first embodiment, a reference image of one frame is determined from a predetermined number or more images obtained by repeatedly capturing an image of the object scene, and a predetermined number or more of images are determined. For each of the comparison images including the images excluding the reference image, the degree of approximation is evaluated using the reference image as a comparison target. Then, an image of (the predetermined number −2) frames is selected as a selected image in the descending order of evaluation from the comparative image, and a composite image is generated using the selected image and the reference image. Further, the composite image is generated by performing alignment processing on each selected image with reference to the reference image, and adding each selected image subjected to the alignment processing to the reference image. Therefore, by selecting and synthesizing images in order from the image suitable for synthesis (in descending order of evaluation), it is possible to effectively synthesize the images according to the subject and generate a high-quality synthesized image. For example, when the amount of motion increases in one frame image among images captured continuously and returns to the original state in the next image, an image with a large amount of motion has a lower evaluation order, As a result, it can be excluded from the object of synthesis.

  The block matching process described above requires enormous operations. If the image used for composition is selected in descending order of evaluation value, the image used for composition is selected in the order of least movement (deformation) with respect to the reference image, so that the search area for block matching can be reduced. The calculation time can be shortened. In addition, the frame image used for composition is selected in the order of the smallest change in focus state, the smallest amount of change in brightness, or the smallest degree of deformation (scaling) of the subject, improving the accuracy of block matching processing. Can be made.

Furthermore, an image with sufficient brightness can be generated, and a random noise removal effect by adding a plurality of frames of image data can be expected.
Further, according to the first embodiment, the amount of movement in each of the comparison images of the main subject in the field, the degree of deformation in each of the comparison images of the main subject, the respective focus states of the comparison image, and the comparison image The evaluation is performed using at least one of the lightness of each as a factor. Therefore, an image used for composition can be selected in accordance with the movement of the subject, the change of the subject, and the like. For example, using these evaluation values, it is determined whether or not image alignment processing is possible, and only the images that are determined to be alignment processing can be used to perform alignment processing and composition processing. good.

  In addition, according to the first embodiment, a reference image serving as a reference for alignment of images of a plurality of frames is determined based on at least one of brightness, contrast, and high frequency components in each of a predetermined number of images. To do. Therefore, since the reference image can be appropriately determined, the image quality of the image generated by the synthesis can be improved. When light is emitted by the illumination unit at the time of acquiring a plurality of frames of images, the image captured at this time is used as a reference image, and the brightness of the comparison image (or selected image) is corrected to match the reference image. If the configuration is such that the alignment process is performed, the bright image is the reference, so the accuracy of the block matching process is improved.

  In addition, according to the first embodiment, the illumination unit that performs preliminary light emission is provided, focus adjustment is performed at the time of preliminary light emission by the illumination unit, and then an image of the object scene is repeatedly captured to generate a predetermined number of images or more. The focus state is fixed until the end. Therefore, since the focus state can be kept constant during the generation of a plurality of images used for composition, an image suitable for composition can be generated.

  Of course, the automatic focusing operation may be continuously performed while tracking the subject of interest (main subject) on the screen. With such a configuration, an image focused on the subject of interest is obtained in each frame image, an image of the subject of interest suitable for synthesis is obtained, and the accuracy of the block matching process is improved. In this case, it is desirable that the target subject is set by the control unit 2 from among images captured at the beginning of the imaging operation equal to or greater than the division number.

  Note that multiple frames of low-resolution dark images are acquired, a strobe is emitted at the end of the divided exposure to obtain a high-resolution image of the main subject (person), and each image is added and combined to obtain an S / N ratio. The present embodiment can also be applied when improving and obtaining a bright image of the main subject. In such a case, each process described in the present embodiment is performed on an image excluding a high-resolution image of a main subject (person) obtained by emitting a strobe at the end of divided exposure, and the combined image is displayed. What is necessary is just to synthesize | combine this high resolution image.

In the first embodiment, an example in which at least one of the image data level (brightness), contrast, and high-frequency component is obtained and an image most suitable for synthesis is determined as a reference image has been described. The largest image may be used as the reference image, the reference image may be determined according to the shooting order of continuous shooting, or several methods may be combined.
For example, if the first image of continuous shooting is the reference image, the imaging operation (steps S3 to S8) in FIG. 2 and the image evaluation operation (step S10) can be performed in parallel. It is possible to shorten the time until generation of the composite image. Further, if an image taken at approximately the center time during the continuous shooting period is used as a reference image, an image taken before and after the time can be used as an image for synthesis. With such a configuration, for example, a frame image with a small amount of motion, deformation (magnification), and change in brightness is synthesized as compared with a case where an image obtained at the end of continuous shooting is used as a reference image. Can be used.

<< Second Embodiment >>
The second embodiment of the present invention will be described below.
The electronic camera of the second embodiment has the same configuration as the electronic camera 1 of the first embodiment. Therefore, illustration and description of the configuration are omitted, and the same reference numerals as those in the first embodiment are used below.

  First, the process before the start of photographing, which is a feature of the second embodiment, will be described using the flowchart of FIG. In the second embodiment, it is determined whether or not to combine images before starting continuous shooting for acquiring images used for combining. For example, when a ball (main subject) rolls in a stationary background, if the background portion is aligned and image synthesis is performed in order to perform shake correction, the background portion may correct the shake by alignment. it can. However, since the rolling ball is synthesized with an image that continues to move during the divided exposure, a plurality of balls appear in the synthesized image depending on the velocity of the ball. Similarly, the same can be said when the main subject is deformed, when the brightness changes abruptly, or when the distance to the subject changes abruptly. Therefore, in the second embodiment, the situation of the subject is grasped before the start of shooting, and it is determined whether or not to synthesize an image according to the situation.

When a power button (not shown) of the electronic camera 1 is turned on, the control unit 2 detects this, and starts preliminary imaging via the imaging unit 3 in step S21. Preliminary imaging is imaging of a so-called through image for composition confirmation, and the imaging unit 3 repeatedly captures an image of the object scene and continuously generates a plurality of images.
In step S22, the control unit 2 extracts feature amounts from each of the plurality of images. When the control unit 2 generates images of a predetermined number of frames (for example, 4 frames) via the imaging unit 3, the motion amount of each main subject in the object scene as the feature amount from these images, and the subject At least one of the degree of deformation (including zooming), the focus state of each image, and the brightness of each image is extracted. The main subject can be extracted by a method similar to the method described in the first embodiment.

  For example, the amount of movement is calculated based on how much the same point of the main subject has moved after detecting the main subject moving from multiple images and associating each region of the main subject between multiple images. To do. This is obtained by, for example, an average of the amount of motion between images in a plurality of main subjects, an average of the amount of motion between images in the entire screen, or a similar calculation. In addition, the degree of deformation is obtained by obtaining boundary information between the method described in the first embodiment and a certain region in an image with another region, and the degree of change of the boundary information among a plurality of images, for example, the aspect ratio and the approximation ratio. It is calculated from the number of vertices of the square. In addition, the focus state calculates the focus position at a plurality of positions in the object scene, and calculates the change of the focus position between the plurality of images. The focus position is obtained by analyzing the sensor output for autofocus. The obtained focus position information can be associated with each image and recorded in the recording unit 5 together with each image data. Further, the focus state of each image may be obtained from the image data of the image generated by the imaging unit 3. The control part 2 calculates | requires each feature-value numerically.

In step S23, the control unit 2 obtains a change amount of the feature amount.
In step S24, the control unit 2 determines whether or not the amount of change is equal to or less than a threshold value. When the amount of change is larger than the threshold value, it is expected that unnaturalness such as the above-described ball example will occur as a result of image synthesis. In determining the threshold value, it is preferable to set the boundary of whether the change of the subject is conspicuous or inconspicuous in consideration of the result of the experiment.

If it is determined in step S24 that the amount of change is equal to or less than the threshold value, the control unit 2 turns on the image composition execution flag in the control unit 2 in step S25.
On the other hand, if it is determined in step S24 that the amount of change is not less than or equal to the threshold value, in step S26, the control unit 2 turns off the image composition execution flag. The control unit 2 prohibits image synthesis at the time of main shooting subsequent to preliminary imaging (through image imaging) by turning off the image synthesis execution flag.

Next, the operation of the electronic camera 1 during actual photographing will be described using the flowchart of FIG. It is assumed that each process of the flowchart of FIG. 4 described above is performed before the flowchart of FIG. 5 is executed.
When the user gives an instruction to start actual shooting via the operation unit 7, in step S31, the control unit 2 determines whether or not the image composition execution flag is ON.

If it is determined in step S31 that the image synthesis execution flag is ON, in step S32, the control unit 2 performs imaging and image synthesis.
In step S32, the control unit 2 performs a series of processes described using the flowchart of FIG. 2 in the first embodiment. However, the following processing is performed for the image evaluation in step S10. That is, in step S10, the control unit 2 divides each image of the (N × 2) frame recorded in a memory (not shown) into a plurality of areas, the degree of deformation of the main subject in each area, and the brightness distribution in each area. And the reference image determined in step S9 is evaluated as a comparison target. The control unit 2 evaluates each factor numerically.

・ Deformation (magnification) degree of main subject Calculates motion vectors for each area, obtains motion vector size variation and motion vector direction variation as variances, and based on this variance, transforms the main subject Find the degree. The greater the degree of deformation, the lower the accuracy of block matching in composition, resulting in a blurred composite image. Therefore, the smaller the change in the degree of deformation, the higher the evaluation value.

-About brightness distribution The image data level (brightness) is calculated | required for every area | region, and the change of brightness distribution is detected based on the difference with another image. The greater the change in the lightness distribution, the lower the accuracy of block matching in composition, resulting in a blurred composite image. Therefore, the evaluation value is increased as the change in brightness distribution is smaller.

When evaluating each area, the evaluation of each area may be averaged to obtain the evaluation value of the entire screen, or the evaluation values of some areas in the screen may be reflected on the entire screen. Anyway.
On the other hand, if it is determined in step S31 that the image composition execution flag is OFF, the control unit 2 performs normal photographing without performing image composition in steps S33 to S36. That is, an image of the object scene is captured by continuously performing exposure for the entire exposure time, and an image of one frame is generated.

In step S <b> 33, the control unit 2 prohibits reading of accumulated charges and image signals at the timing of the pulse signal A, as in step S <b> 3 of FIG. 2.
In step S34, the control unit 2 closes the shutter after releasing the time t1 as in step S4 of FIG. However, the time t1 is the total exposure time determined based on the shutter speed or the like.

In step S <b> 35, the control unit 2 permits the readout of the accumulated charges and the output of the image signal at the timing of the pulse signal B as in step S <b> 5 of FIG. 2.
In step S <b> 36, the control unit 2 records the image data in the recording unit 5 by reading out the accumulated charges and outputting the image signal at the timing of the pulse signal B, similarly to step S <b> 6 in FIG. 2.

In step S31, when it is determined that the image composition execution flag is OFF, it may be configured to notify that the image composition is not performed. If the notification is made in step S26 in FIG. 4, it is even more preferable because the preliminary imaging stage is notified.
As described above, according to the second embodiment, each of the comparison images is divided into a plurality of regions, and the evaluation is performed using the degree of deformation of the subject in each region and the brightness distribution in each region as factors. Therefore, it is possible to select an image to be used for composition in descending order of evaluation according to the movement of the subject or the change of the subject.

  In addition, according to the second embodiment, a predetermined image is obtained from each of a plurality of images continuously generated by preliminary imaging by repeatedly capturing images of an object scene and continuously generating a plurality of images. These feature amounts are extracted, and the extracted feature amounts are compared. If the change amount is smaller than a predetermined threshold value, it is determined that the images are to be combined. If it is determined that the images are to be combined, the image of the object scene is repeatedly captured to generate the predetermined number or more images, and a reference image of one frame is determined from the obtained predetermined number or more images. Then, for each of the comparison images formed by removing the reference image from among the predetermined number of images, the degree of approximation is evaluated using the reference image as a comparison target. -2) A frame image is selected as a selected image, and a composite image is generated using the selected image and the reference image. Therefore, division exposure and image synthesis can be performed only when the subject is suitable for synthesis.

Further, according to the second embodiment, as the predetermined feature amount, the amount of movement of each main subject image in the object scene, the degree of deformation of each subject image, the focus state of each image, Extract at least one of brightness. Therefore, it is possible to determine whether or not to perform synthesis using a feature amount corresponding to the movement of the subject or the change in the subject.
In the second embodiment, the predetermined threshold can be determined according to the total exposure time, the number of divisions, the shutter speed, the shooting interval of each shooting in continuous shooting, and the like. Therefore, it is possible to set a threshold value for determining whether or not to combine images to an appropriate value.

According to the second embodiment, when it is not determined that the images are to be combined, the exposure is continuously performed for the entire exposure time, thereby capturing an image of the object scene and generating a one-frame image. . Therefore, when it is not suitable for synthesis, it is possible to perform shooting with reduced inconvenience by performing normal shooting. Moreover, it is possible to perform shooting according to the user's intention such as panning.
In the second embodiment, it is determined whether or not to synthesize an image based on the result of preliminary imaging. However, a mode (such as shutter priority mode) or a shutter speed may be added to the determination. good.

In preliminary imaging, the exposure time as described in the first embodiment may be divided by a predetermined number of divisions to perform division exposure.
The first embodiment and the second embodiment have been described using an electronic camera as an example of the present invention. However, a part of the processing performed by these electronic cameras (for example, steps S9 to S9 in FIG. 2). The processing in step S12 may be realized by a computer. In this case, the image processing program of the present invention may be installed in advance in the computer.

Moreover, you may make it perform combining each process demonstrated in the said 1st Embodiment and 2nd Embodiment. For example, in the second embodiment, when evaluating an image, the amount of movement of the main subject, the focus state of the image, and the like may be used as factors as in the first embodiment.
Further, in the above-described embodiment, the apparatus for adding the image data after performing the alignment of the images of the plurality of frames obtained by the divided exposure has been described. However, after performing the alignment of the images of the plurality of frames. The present invention can be similarly applied to an image processing apparatus such as a noise reducer that averages or weights averages image data.

2 is a functional block diagram of the electronic camera 1. FIG. 3 is a flowchart showing the operation of the electronic camera 1. 4 is a timing chart in processing of the electronic camera 1. 6 is another flowchart showing the operation of the electronic camera 1. 6 is another flowchart showing the operation of the electronic camera 1.

Explanation of symbols

1, electronic camera 2, control unit 3, imaging unit 4, image processing unit 5, recording unit 6, illumination unit 7, operation unit 8, photometry unit

Claims (14)

A determining unit that determines a reference image of one frame from a predetermined number of images obtained by repeatedly capturing an image of the object scene;
An evaluation unit that evaluates the degree of approximation for each of the comparison images including the reference image of the predetermined number or more images excluding the reference image;
A synthesis unit that selects an image of the (predetermined number−2) frames as a selection image in descending order of evaluation by the evaluation unit from the comparison image, and generates a synthesis image using the selection image and the reference image. An image processing apparatus comprising: The image processing apparatus according to claim 1.
The synthesizing unit performs an alignment process on each of the selected images with the reference image as a reference, and adds each of the selected images subjected to the alignment process and the reference image to perform the alignment process. An image processing apparatus for generating a composite image. The image processing apparatus according to claim 2,
The evaluation unit includes a movement amount of each of the comparison images of the main subject in the object scene, a degree of deformation of each of the comparison images of the main subject, a focus state of each of the comparison images, and the comparison. An image processing apparatus, wherein the evaluation is performed using at least one of the brightness of each image as a factor. The image processing apparatus according to claim 2,
The evaluation unit divides each of the comparison images into a plurality of regions, the amount of movement of the subject in each region, the degree of deformation of the subject in each region, the focus state in each region, and the amount in each region The image processing apparatus, wherein the evaluation is performed using at least one of the brightness distribution as a factor. The image processing apparatus according to claim 1.
The determination unit determines the reference image based on at least one of brightness, contrast, and high-frequency component in each of the predetermined number of images. The image processing apparatus according to claim 1.
An image capturing unit that repeatedly captures the image of the object scene and generates the predetermined number or more images;
The determining unit determines an image captured at the beginning of the repeated imaging operation as the reference image,
The said evaluation part evaluates the said approximation degree in parallel with the repeated imaging operation by the said imaging part. The image processing apparatus characterized by the above-mentioned. The image processing apparatus according to claim 1.
An imaging unit that repeatedly captures an image of the object scene and generates the predetermined number of images;
An illumination unit that performs preliminary light emission,
The imaging unit performs focus adjustment at the time of preliminary light emission by the illuminating unit, and then repeatedly captures the image of the object scene and fixes the focus state until the generation of the predetermined number of images or more is completed. An image processing apparatus. The image processing apparatus according to claim 1.
An image capturing unit that repeatedly captures the image of the object scene and generates the predetermined number or more images;
The determination unit sets a predetermined subject from among images captured at the beginning of the repeated imaging operation,
The image processing apparatus, wherein the imaging unit continuously performs an automatic focusing operation on the predetermined subject while the repeated imaging operation is performed. An electronic camera comprising the image processing device according to any one of claims 1 to 5,
A main imaging operation that repeatedly captures images of the object scene and performs a preliminary image capturing operation that continuously generates a plurality of images, and repeatedly captures images of the object scene and generates the predetermined number of images or more. An imaging unit for performing
An extraction unit that extracts a predetermined feature amount from each of the plurality of images continuously generated by the preliminary imaging operation;
A determination unit that compares the feature amounts extracted by the extraction unit and determines that the image is to be combined by the combining unit when the amount of change is smaller than a predetermined threshold;
When the determination unit determines to combine the images by the combining unit, the determination unit determines a reference image of one frame from the predetermined number or more images obtained by the main imaging operation, The evaluation unit evaluates the degree of approximation for each of the comparison images made up of images excluding the reference image among the predetermined number or more images, with the reference image as a comparison target, and the combining unit performs the comparison image From the highest evaluation by the evaluation unit, an image of (the predetermined number−2) frames is selected as a selection image, and a composite image is generated using the selection image and the reference image. camera. An image processing program for realizing image processing on an image to be processed by a computer,
An acquisition step of acquiring a predetermined number or more images obtained by repeatedly capturing an image of the object scene;
A determining step of determining a reference image of one frame from the predetermined number of images;
An evaluation step for evaluating the degree of approximation for each of the comparison images including the reference image of the predetermined number or more images excluding the reference image;
A synthesis step of selecting an image of the (predetermined number−2) frames as a selection image from the comparison image in descending order of evaluation in the evaluation step, and generating a composite image using the selection image and the reference image. When
An image processing program comprising: The image processing program according to claim 10,
In the synthesizing step, alignment processing is performed on each of the selected images with reference to the reference image, and the selection image that has been subjected to the alignment processing is added to the reference image to perform the alignment processing. Generate a composite image
An image processing program characterized by that. The image processing program according to claim 11,
In the evaluation step, the amount of movement of each of the comparison images of the main subject in the object scene, the degree of deformation of each of the comparison images of the main subject, the focus state of each of the comparison images, and the comparison The evaluation is performed using at least one of the brightness of each image as a factor.
An image processing program characterized by that. The image processing program according to claim 11,
In the evaluation step, each of the comparison images is divided into a plurality of regions, the amount of movement of the subject in each region, the degree of deformation of the subject in each region, the focus state in each region, and the amount in each region The evaluation is performed using at least one of the brightness distribution as a factor.
An image processing program characterized by that. The image processing program according to claim 10,
In the determining step, the reference image is determined based on at least one of brightness, contrast, and high-frequency component in each of the predetermined number or more images.
An image processing program characterized by that.

Images