JP2014039222A - Image processing apparatus, control method and control program of the same and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy when a light emission image and a non-light emission image are positioned.SOLUTION: When a light emission image obtained by emitting light at the time of photographing a subject and a non-light emission image group which are continuously obtained without emitting light are positioned and synthesized, a positioning section 107 obtains a first positioning conversion coefficient used when a non-light emission image except for a reference non-light emission image is positioned to the reference non-light emission image in the non-light emission image group, and obtains a second positioning conversion coefficient used when the reference non-light emission image is positioned to the light emission image. The positioning section positions the light emission image and the non-light emission image group in accordance with the first and second positioning conversion coefficients. An image synthesis section 114 synthesizes the positioned light emission image and the non-light emission image group to obtain a synthesized image.

Description

  The present invention relates to an image processing apparatus, a control method thereof, a control program, and an imaging apparatus, and more particularly, to an image processing apparatus that performs image processing on both a subject and a background in an appropriate exposure state when shooting night scenes.

  In general, when shooting both the main subject (mainly a person) and the background with an appropriate exposure under low illumination such as a night view, the exposure time and the like are set so that the background has an appropriate exposure. A main subject such as a person is photographed with an appropriate exposure set by an auxiliary illumination light such as a flash. Such shooting is called slow sync shooting.

  However, in slow sync photography, the exposure time to be set is long because the background is low in illuminance. For this reason, it is necessary to perform so-called tripod photography in order to prevent the background from blurring.

  In order to prevent blurring in the background, a single image shot with flash firing (hereinafter referred to as a flash image) and a plurality of images (hereinafter referred to as non-flash images) shot in a time that does not cause camera shake due to non-flash firing. Is called continuous shooting.

  Then, the light-emitting image and the non-light-emitting image are aligned, the light-emitting image is used for the main subject portion, and a plurality of non-light-emitting images are added and averaged for the background portion. This realizes slow sync photography by so-called hand-held photography.

  For example, the distance to the main subject is detected in advance, and when the main subject is far away, alignment of the light-emitting image and the plurality of non-light-emitting images is performed in the background, and when the main subject is close, alignment is performed. Is performed on a main subject (see Patent Document 1).

  Also, there is an apparatus in which an area irradiated with a strobe is detected as a foreground area from a pre-photographed emission image, and image alignment between the emission image and the non-emission image is performed in the foreground area (see Patent Document 2). ).

JP 2009-17606 A JP 2010-114566 A

  As described above, in Patent Document 1, when it is determined that the main subject is close, the alignment of the light-emitting image and the non-light-emitting image is performed on the main subject. However, the luminance of the main subject is generally greatly different between the light-emitting image and the non-light-emitting image, and if the alignment is performed with reference to the main subject having such a different luminance, the accuracy of the alignment inevitably decreases. Resulting in.

  On the other hand, as described above, in Patent Document 2, the area irradiated with the strobe is set as a near view area, and the alignment of the light emission image and the non-light emission image is performed in the foreground area. However, as in Patent Document 1, since the brightness of the foreground area is greatly different between the light-emitting image and the non-light-emitting image, the alignment accuracy is lowered.

  Accordingly, an object of the present invention is to provide an image processing apparatus, a control method thereof, a control program, and an imaging apparatus capable of improving the accuracy when aligning a luminescent image and a non-luminescent image. .

  In order to achieve the above object, an image processing apparatus according to the present invention positions a luminescent image obtained by emitting light when photographing a subject and a non-luminescent image group obtained continuously without emitting light. An image processing apparatus for combining and synthesizing a non-light-emitting image excluding the reference non-light-emitting image as a reference non-light-emitting image as a reference non-light-emitting image in the non-light-emitting image group And a first alignment conversion coefficient used for aligning the reference non-light-emitting image with the light-emitting image on the basis of the light-emitting image. Second calculating means to be obtained; and positioning means for aligning the light emitting image and the non-light emitting image group according to the first alignment conversion coefficient and the second alignment conversion coefficient; And having and an image synthesizing means for the synthesized and the synthesized image of aligned light emitting image and the non-emission image group by the alignment means.

  An imaging apparatus according to the present invention, together with the image processing apparatus described above, emits light to shoot a subject to obtain a luminescent image, and continuously shoots the subject without performing the luminescence to obtain a non-luminous image group. And imaging means.

  The control method according to the present invention is an image processing apparatus that aligns and combines a light-emitting image obtained by emitting light when photographing a subject and a non-light-emitting image group obtained continuously without light emission. In the non-light emitting image group, a first non-light emitting image is used as a reference non-light emitting image, and a non-light emitting image excluding the reference non-light emitting image is aligned with the reference non-light emitting image. A first calculation step for obtaining an alignment conversion coefficient, and a second calculation step for obtaining a second alignment conversion coefficient used when aligning the reference non-light-emitting image with the light-emitting image on the basis of the light-emitting image. An alignment step of aligning the light emitting image and the non-light emitting image group according to the first alignment conversion coefficient and the second alignment conversion coefficient; An image synthesizing step combined aligned the luminescent image and the non-emission image group in step combined to a composite image, characterized by having a.

  The control program according to the present invention is an image processing apparatus that aligns and combines a light-emitting image obtained by emitting light when photographing a subject and a non-light-emitting image group obtained continuously without light emission. A control program to be used, wherein a non-light-emitting image excluding the reference non-light-emitting image as a specific non-light-emitting image in the non-light-emitting image group as a reference non-light-emitting image in the computer included in the image processing device A first calculation step for obtaining a first alignment conversion coefficient used when aligning the light emitting image, and a second calculating step used when aligning the reference non-light emitting image with the light emitting image on the basis of the light emitting image. A second calculating step for obtaining an alignment conversion coefficient; and the emission image and the first alignment conversion coefficient and the second alignment conversion coefficient in accordance with the second alignment conversion coefficient and the second alignment conversion coefficient. An alignment step of aligning the non-light emitting image group, and an image combining step of combining the light emitting image and the non-light emitting image group aligned in the alignment step to form a composite image. Features.

  According to the present invention, it is possible to improve the accuracy at the time of performing alignment between the light emitting image and the non-light emitting image, and to improve the image quality.

It is a block diagram which shows an example of the imaging device with which the image processing apparatus by embodiment of this invention was used. It is a figure for demonstrating operation | movement of the imaging part shown in FIG. 1, a strobe light emission control part, and an exposure condition control part. It is a flowchart for demonstrating the process performed in the flash irradiation area | region calculation part shown in FIG. It is a flowchart for demonstrating the process performed by the nonluminous image conversion coefficient calculation part shown in FIG. It is a figure for demonstrating calculation of the conversion factor shown in FIG. It is a flowchart for demonstrating the process performed in the light emission image conversion coefficient calculation part shown in FIG. It is a figure for demonstrating calculation of the conversion coefficient (projection conversion coefficient) by the alignment conversion coefficient calculation part shown in FIG. It is a figure for demonstrating conversion of the image conversion coefficient by the alignment conversion coefficient calculation part shown in FIG. It is a flowchart for demonstrating the process performed in the image synthetic | combination part shown in FIG. It is a figure which shows the relationship between the difference absolute value used by the image composition part shown in FIG. 1, and an image composition ratio.

  Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  It is a block diagram which shows an example of the imaging device with which the image processing apparatus by the 1st Embodiment of this invention was used.

  An imaging apparatus (hereinafter simply referred to as a camera) includes an imaging unit 101. Although not shown, the imaging unit 101 includes an imaging lens group, a semiconductor imaging device such as a CMOS sensor or a CCD (hereinafter simply referred to as an imaging device), and a strobe light emitting unit.

  Here, the imaging unit 101 continuously performs a plurality of shootings on the same subject. At this time, the strobe light emission control unit 102 controls whether or not the strobe light is emitted every time shooting is performed. Further, the exposure condition control unit 103 determines the exposure condition for each photographing.

  An analog signal (video signal or image signal) obtained as a result of shooting by the imaging unit 101 is given to the A / D conversion unit 104. The A / D converter 104 A / D converts the video signal as image data and sends the image data to the signal processor 105.

  The signal processing unit 105 performs predetermined signal processing such as gradation conversion and noise reduction on the image data, and outputs the image data to the image processing unit 106. In the illustrated example, the image processing unit 106 is the image processing apparatus according to the first embodiment.

  As described above, the image processing unit 106 includes the alignment unit 107 that performs alignment of image data obtained as a result of a plurality of shootings, and combines the image data that has undergone the alignment process by the alignment unit 107. An image composition unit 114 that performs processing is included.

  The output (synthesized image data) of the image composition unit 114 is given to the image display unit 115 and the image recording unit 116. The image display unit 115 displays an image corresponding to the composite image data, for example, on a liquid crystal monitor provided in the imaging apparatus. The image recording unit 116 records the composite image data on a recording medium (not shown).

  Here, the imaging unit 101, the strobe light emission control unit 102, and the exposure condition control unit 103 illustrated in FIG. 1 will be described.

  FIG. 2 is a diagram for explaining operations of the imaging unit 101, the strobe light emission control unit 102, and the exposure condition control unit 103 shown in FIG.

  Here, the number of times of shooting is five, and shooting with strobe emission is performed for the first time. It is assumed that shooting is performed without firing the strobe for the second and subsequent times. In the following description, a photographed image obtained by the first photographing is referred to as a light emitting image 0, and photographed images obtained by the second and subsequent photographing are referred to as non-light emitting images 1 to 4. The non-light emitting images 1 to 4 are collectively referred to as a non-light emitting image group.

  Further, in FIG. 2, when the shooting sensitivity of the luminescent image 0 is set to SON, the shooting sensitivity SOFF of the non-luminescent image group is set to SOFF = 2 × SON. In this way, the shooting sensitivity is set in consideration of the averaging of the non-light emitting image group in the subsequent processing in consideration of the noise of the person (subject) of the non-light emitting image after the averaging and the light emitting image 0. This is to match the level.

Incidentally, unlike the shutter speed when capturing a luminescence image 0 and non-emission image group, the shutter speed in luminescent image 0 is a T _ON, the shutter speed in the non-emission image group is set to T _OFF.

Further, the aperture value at the time of shooting the light emitting image 0 and the non-light emitting image group is different, the aperture value is A _ON in the light emitting image 0, and the aperture value is A _OFF in the non-light emitting image group.

  In general, when N (N is an integer of 2 or more) images are averaged, the noise level of the image after the averaging is (1 / √N) times. Here, since four non-light-emitting images are added and averaged, the noise level of the image after the averaging is (1/2) times.

  Referring to FIG. 1 again, the alignment unit 107 includes a memory unit 108, a strobe irradiation area calculation unit 109, a non-light emission image conversion coefficient group calculation unit 110, a light emission image conversion coefficient calculation unit 111, and an alignment conversion coefficient calculation unit 112. And an image deformation unit 113.

  The memory unit 108 temporarily stores image data (that is, a light emitting image and a non-light emitting image group) subjected to predetermined signal processing by the signal processing unit 105. The strobe irradiation area calculation unit 109 calculates a region irradiated with a strobe (irradiation region) and a region not irradiated (non-irradiation region) for the image data stored in the memory unit 108 before performing processing for alignment. Determine.

  FIG. 3 is a flowchart for explaining processing in the strobe irradiation area calculation unit 109 shown in FIG.

  When the irradiation area determination process is started, the strobe irradiation area calculation unit 109 inputs the light emission image 0 and the non-light emission image 1 from the memory unit 108, and adjusts the luminance level in these two images (step S301).

  When adjusting the brightness level, for example, the exposure conditions (aperture value (Av), shutter speed (Tv), photographing sensitivity (Sv)) of the light-emitting image 0 and the non-light-emitting image 1 are compared. If there is an exposure difference, the strobe irradiation area calculation unit 109 applies a gain for correcting the exposure difference to a relatively dark image to align the luminance levels in the two images.

  By performing such processing, it is possible to eliminate the influence of the difference in the brightness level due to the exposure difference when the strobe irradiation determination described later is performed using the brightness difference.

  Subsequently, the strobe irradiation area calculation unit 109 divides the two images into a plurality of blocks (small blocks) and integrates the luminance in the small blocks in each image to obtain a luminance integrated value (step S302). Then, the strobe irradiation area calculation unit 109 obtains a difference in luminance integrated value (luminance difference value) between the two images (step S303).

  Next, the strobe irradiation area calculation unit 109 compares the absolute value of the luminance difference value with a predetermined first threshold value (threshold value 1) and determines whether absolute value> threshold value 1 is satisfied (step S304). When the absolute value is equal to or smaller than the first threshold value (absolute value ≦ threshold value 1) (NO in step S304), the strobe irradiation area calculation unit 109 affects the strobe for the small block (small block with absolute value ≦ threshold value 1). The degree is calculated (step S305).

  Here, the strobe influence E is expressed by the following equation (1), where SZ is the absolute value and SS is the integrated luminance value of the non-light-emitting image 1.

E = SZ / SS (1)
Subsequently, the strobe irradiation area calculation unit 109 determines whether or not strobe influence degree> threshold value 2 by comparing the strobe influence degree (light emission influence degree) with a predetermined second threshold value (threshold value 2) ( Step S306). When the stroboscopic influence degree ≦ threshold 2 (NO in step S306), the stroboscopic irradiation area calculation unit 109 regards the small block as being a small block in which no stroboscope has been irradiated and no change in luminance has occurred. A strobe non-irradiation block is set (step S307).

  On the other hand, if absolute value> threshold value 1 (YES in step S304), the strobe irradiation area calculation unit 109 regards that the small block has changed in brightness due to the strobe irradiation, and determines that the small block is a strobe irradiation block. (Step S308).

  Similarly, if the stroboscopic influence degree> threshold value 2 (YES in step S306), the stroboscopic irradiation area calculation unit 109 determines that the small block is not illuminated by the stroboscopic irradiation although the absolute value of the luminance difference value is not large. It is regarded as a block whose luminance is increased in comparison. Then, the flash irradiation area calculation unit 109 proceeds to the process of step S308 and sets the corresponding small block as a flash irradiation block.

  Subsequently, following the processing in step S307 or S308, the strobe irradiation area calculation unit 109 determines whether or not strobe irradiation or non-irradiation determination processing has been performed for all small blocks (step S309). If the determination process has not been completed for all the small blocks (NO in step S309), the strobe irradiation area calculation unit 109 returns to the process of step S304 and performs the determination process for the next small block.

  When the discrimination processing is completed for all the small blocks (YES in step S309), the strobe irradiation area calculation unit 109 sets the set of strobe irradiation blocks as the strobe irradiation area and sets the set of non-strobe irradiation blocks as the strobe non-irradiation area. To do. Then, the strobe irradiation area calculation unit 109 ends the irradiation area determination process.

  FIG. 4 is a flowchart for explaining processing performed by the non-light emitting image conversion coefficient calculation unit 110 shown in FIG.

  The non-light emission image conversion coefficient calculation unit 110 detects a positional deviation between the non-light emission image groups, and calculates a non-light emission image conversion coefficient (projection conversion coefficient: first alignment conversion coefficient) for alignment. .

  First, the non-light emission image conversion coefficient calculation unit 110 reads a non-light emission image group from the memory unit 108 via the strobe irradiation area calculation unit 109. Then, the non-light emission image conversion coefficient calculation unit 110 sets a reference image (reference non-light emission image) for conversion coefficient calculation from the non-light emission image group. Here, it is assumed that the non-light-emitting image 1 is set as a reference image (reference non-light-emitting image).

  Further, the non-light emission image conversion coefficient calculation unit 110 selects an image other than the reference image from the non-light emission image group as a comparison image (step S401). Here, the non-light-emitting images 2 to 4 are selected as comparison images.

  Next, the non-light emission image conversion coefficient calculation unit 110 calculates a positional deviation between the reference image and the comparison image. For example, for each small block obtained as a result of the division by the strobe irradiation area calculation unit 109, the non-light emission image conversion coefficient calculation unit 110 detects a motion vector between the reference image and the comparison image (step S402).

  When detecting a motion vector, for example, a pattern matching process using the sum of absolute differences as an evaluation value is used.

  By the way, when a composite image is finally performed, a background portion that is not irradiated with a strobe is obtained from the non-light-emitting image. For this reason, it is desirable to calculate the conversion coefficient for aligning the non-light emitting images according to the motion vector detected in the strobe non-irradiation region.

  Furthermore, if the movement of the strobe illumination area, that is, the movement of the main subject area where the strobe is illuminated represents the same movement as the strobe non-irradiation area, the alignment can be achieved by calculating the conversion coefficient using the motion vector of the strobe illumination area. Improves accuracy.

  However, in particular, when the main subject is a person and the difference in distance between the person and the background is large, the movement indicated by the flash irradiation area and the non-flash area due to the movement of the person and the influence of the distance difference, etc. May be different. In such a case, if the conversion coefficient is calculated using the motion vector of the strobe irradiation area, the alignment accuracy of the background portion may be lowered.

  Accordingly, in the illustrated example, the movement of the background portion of the conversion coefficient H calculated from only the strobe non-irradiation area and the strobe irradiation area and the strobe non-irradiation area, that is, the conversion coefficient H ′ calculated from the entire image, is more accurately determined. The conversion coefficient shown is used.

  FIG. 5 is a diagram for explaining the calculation of the conversion coefficient shown in FIG.

  4 and 5, after detecting a motion vector as described above, the non-light emission image conversion coefficient calculation unit 110 responds to the plurality of motion vectors detected in the small blocks belonging to the strobe non-irradiation region. Then, a conversion coefficient H (first conversion coefficient) representing the positional deviation of the image is calculated (step S403). Here, the least square method is used as a method of calculating the conversion coefficient H from a plurality of motion vectors.

  Subsequently, the non-luminous image conversion coefficient calculation unit 110 calculates a conversion coefficient H ′ (second conversion coefficient) according to a plurality of motion vectors detected in the entire image (step S404).

  Next, the non-luminous image conversion coefficient calculation unit 110 evaluates the conversion coefficients H and H ′. First, the non-light emitting image conversion coefficient calculation unit 110 converts the coordinates of the point of interest of the small block belonging to the strobe non-irradiation region using the conversion coefficients H and H ′ (step S405). Here, the point of interest represents a central point in each of the small blocks.

  As shown in FIG. 5, the non-light-emitting image has a strobe irradiation area 501 and a strobe non-irradiation area 502. Then, when the process of step S405 is performed on the motion vector 503 detected in one small block, motion vectors 504 and 505 are generated with the point of interest 506 as the starting point and the point of interest 507 after coordinate conversion as the end point.

  The motion vector 504 is a motion vector obtained using the conversion coefficient H, and the motion vector 505 is a motion vector obtained using the conversion coefficient H ′.

  Next, the non-light emitting image conversion coefficient calculation unit 110 calculates errors ErrH and ErrH ′ between the motion vector 503 and the motion vectors 504 and 505 generated using the conversion coefficients H and H ′ in the small block (step S406). ). Here, errors ErrH and ErrH 'indicate the distance between the end point coordinates indicated by the motion vector.

  Subsequently, the non-emission image conversion coefficient calculation unit 110 calculates the sum of the errors ErrH and ErrH ′ for all the small blocks belonging to the strobe non-irradiation region (step S407). Then, the non-light emitting image conversion coefficient calculation unit 110 sets the sums of the errors ErrH and ErrH ′ as sums ΣErrH and ΣErrH ′, respectively.

  Further, the non-light emission image conversion coefficient calculation unit 110 compares the magnitudes of the sums ΣErrH and ΣErrH ′ to determine whether or not the sum ΣErrH> ΣErrH ′ (step S408). The smaller the error value, the higher the accuracy of the conversion coefficient. Therefore, if the sum ΣErrH ≦ the sum ΣErrH ′ (NO in step S408), the non-light emitting image conversion coefficient calculation unit 110 uses the conversion coefficient H. Determine (step S409).

  Subsequently, the non-luminous image conversion coefficient calculation unit 110 determines whether or not conversion coefficients have been calculated for all the comparative images (step S410). When the calculation of conversion coefficients for all the comparative images is completed (YES in step S410), the non-luminous image conversion coefficient calculation unit 110 ends the projective conversion coefficient calculation process.

  On the other hand, calculation of conversion coefficients for all comparison images has not been completed (YES in step S410), the non-light emission image conversion coefficient calculation unit 110 returns to the process of step S402 and calculates conversion coefficients for the next comparison image. I do.

  If total ΣErrH> total ΣErrH ′ (YES in step S408), non-luminous image conversion coefficient calculation unit 110 determines to use conversion coefficient H ′ (step S411S409). Then, the non-light emitting image conversion coefficient calculation unit 110 proceeds to the process of step S410.

  In this way, the non-light emission image conversion coefficient calculation unit 110 determines the conversion coefficient indicating the positional deviation between the reference image and the comparison image. In the following description, the conversion coefficients of the non-light emitting images 2 to 4 (comparison image) with respect to the non-light emitting image 1 (reference image) are H12, H13, and H14, respectively.

  The light emission image conversion coefficient calculation unit 111 calculates a conversion coefficient (projection conversion coefficient) representing a positional deviation of the light emission image from a specific non-light emission image that is the reference image in the non-light emission image conversion coefficient calculation unit 110. Here, a conversion coefficient between the light emission image 0 and the non-light emission image 1 which is the reference image is calculated.

  When the synthesized image is synthesized, only the subject image irradiated with the strobe is obtained from the light emission image. For this reason, when performing alignment between the light emitting image and the non-light emitting image, it is desirable to calculate the conversion coefficient from the motion vector detected in the strobe irradiation region.

  However, the subject image irradiated with the strobe is crushed black due to insufficient light in the non-light emitting image, and it may be difficult to detect the motion vector. Accordingly, the brightness of the strobe irradiation area in the non-light-emitting image is detected, and when the brightness value is larger than a predetermined brightness value (threshold level), the conversion coefficient is calculated in the strobe irradiation area. On the other hand, when the luminance value is equal to or lower than a predetermined threshold level, the conversion coefficient is calculated in the strobe non-irradiation region.

  FIG. 6 is a flowchart for explaining processing performed by the light emission image conversion coefficient calculation unit 111 shown in FIG.

  When the light emission image conversion coefficient calculation process is started, first, the light emission image conversion coefficient calculation unit 111 calculates the average luminance value of the strobe irradiation area in the non-light emission image (step S601). Next, the light emission image conversion coefficient calculation unit 111 compares the average luminance value with a predetermined threshold value, and determines whether or not the average luminance value> the predetermined threshold value (luminance threshold value) (step S602).

  If average luminance value> predetermined threshold value (YES in step S602), the light emission image conversion coefficient calculation unit 111 determines that there is no black crushing in the strobe irradiation area in the non-light emission image, and converts in the strobe irradiation area. Calculate the coefficient.

  At this time, the light emission image conversion coefficient calculation unit 111 first adjusts the luminance level of the strobe irradiation area (step S603). For example, the light emission image conversion coefficient calculation unit 111 obtains an average luminance for each small block of the strobe irradiation area in the light emission image and the non-light emission image.

  Then, the light emission image conversion coefficient calculation unit 111 applies a gain so that the average luminance values in the light emission image and the non-light emission image are equal to each other. As a result, the luminance levels of the small blocks are equal in the light emitting image and the non-light emitting image.

  Subsequently, the light emission image conversion coefficient calculation unit 111 detects the motion vectors of the light emission image and the non-light emission image for each small block in the strobe irradiation region (step S604). Then, the light emission image conversion coefficient calculation unit 111 obtains a conversion coefficient (light emission image conversion coefficient) according to the detected motion vector, and ends the light emission image conversion coefficient calculation process.

  Note that the detection of the motion vector in step S604 and the calculation of the conversion coefficient in step S605 are performed by the same process as the non-light emitting image conversion coefficient calculation unit 110 described in FIG.

  On the other hand, when the average luminance value is equal to or less than the predetermined luminance threshold (average luminance value ≦ luminance threshold) (NO in step S602), the light emission image conversion coefficient calculation unit 111 causes black crushing in the strobe irradiation area in the non-light emission image. Therefore, the conversion coefficient is calculated in the non-flash area.

  In this case, the light emission image conversion coefficient calculation unit 111 first adjusts the luminance level of the strobe non-irradiation area (step S606). Here, the luminance level may be adjusted using the average luminance value in the same manner as in step S603. However, since the non-stroboscopic area is not affected by the strobe, similarly to step S301 described in FIG. Gain adjustment may be performed by comparing exposure conditions.

  Subsequently, the light emission image conversion coefficient calculation unit 111 detects a motion vector of the light emission image and the non-light emission image for each small block in the strobe non-irradiation region (step S607). Thereafter, the light emission image conversion coefficient calculation unit 111 performs the process of step S605.

  In addition, the detection of the motion vector in step S607 is performed by the same processing as the non-light emitting image conversion coefficient calculation unit 110 described in FIG. In the following description, the conversion coefficient of the non-light-emitting image 1 with respect to the light-emitting image 0 is H01.

  In this way, the light emission image conversion coefficient calculation unit 111 calculates a conversion coefficient (second alignment conversion coefficient) indicating the positional deviation between the light emission image and the non-light emission image group and the reference image.

  As shown in FIG. 2 described above, here, the luminescent image 0 is first taken in the shooting order. In the final composite image, it is desirable that the angle of view is as much as possible when the user releases the shutter.

  Therefore, it is desirable to use the light emission image 0 as a reference image when the image is deformed for alignment.

  However, in the illustrated example, conversion coefficients (H02, H03, H04) representing the positional deviation between the light emitting image 0 and the non-light emitting images 2 to 4 are not calculated. In addition, when first aligning non-light emitting images and then aligning the reference non-light emitting image and the light emitting image, coordinate conversion of the non-light emitting image is performed twice for alignment. become.

  In general, when transforming an image using a transform coefficient, both input and output coordinate values do not have integer precision, so it is necessary to generate pixel values by interpolation processing. For this reason, there is a problem that high-frequency components of the image are lost.

  Therefore, the alignment conversion coefficient calculation unit 112 determines that the emission image 0 and the non-emission image 0 from the conversion coefficients (hereinafter also referred to as coordinate conversion coefficients) H12, H13, H14, and H01 calculated by the processing up to the emission image conversion coefficient calculation unit 111. Conversion coefficients (H02, H03, H04) representing the positional deviation of the light emission images 2 to 4 are calculated. Then, using the conversion coefficients H01, H02, H03, and H04, the image deforming unit 113 performs alignment after deforming all the non-light-emitting images with respect to the light-emitting image 0.

  FIG. 7 is a diagram for explaining calculation of a conversion coefficient (projection conversion coefficient) by the alignment conversion coefficient calculation unit 112 shown in FIG.

  FIG. 8 is a diagram for explaining conversion of image conversion coefficients by the alignment conversion coefficient calculation unit 112 shown in FIG.

  First, referring to FIG. 7, from the conversion coefficient H01 of the light-emitting pixel 0 and the non-light-emitting pixel 1 and the conversion coefficient H12 of the non-light-emitting image 1 and the non-light-emitting image 2, the conversion coefficient H02 of the light-emitting image 0 and the non-light-emitting image 2 is obtained. The case of calculating will be described.

  Similarly, the conversion coefficient H03 of the luminescent image 0 and the non-luminescent image 3 is obtained from the conversion coefficient H02 of the luminescent pixel 0 and non-luminescent pixel 2 and the conversion coefficient H23 of the non-luminescent image 2 and non-luminescent image 3. The conversion coefficient H04 of the luminescent image 0 and the non-luminescent image 4 is obtained from the conversion coefficient H03 of the luminescent pixel 0 and non-luminescent pixel 3 and the conversion coefficient H34 of the non-luminescent image 3 and non-luminescent image 4.

  Now, assume that the coordinate value of the light-emitting image 0 is (x0, y0), the coordinate value of the non-light-emitting image 1 is (x1, y1), and the coordinate value of the non-light-emitting image 2 is (x2, y2). Also, the conversion coefficients H01 and H12 are represented by the following equations (2) and (3), respectively.

Therefore, the formulas used in the oblique conversion are expressed by the following formulas (4) to (7).

By substituting Equation (6) and Equation (7) into (x1, y1) of Equation (4) and Equation (5), respectively, Equation (8) and Equation (9) are obtained.

The coefficients a02, b02, c02, d02, e02, f02, g02, and h02 are obtained by substituting the expressions (6) and (7) into (x1, y1) of the expressions (4) and (5), respectively. Represents the coefficient after organizing.

  Therefore, the conversion coefficient H02 of the luminescent image 0 and the non-luminescent image 2 can be expressed by Expression (10).

  In this way, as shown in FIG. 7, the alignment conversion coefficient calculation unit 112 uses the conversion coefficient H01 of the luminescent pixel 0 and the non-luminescent pixel 1 and the conversion coefficient H12 of the non-luminescent image 1 and the non-luminescent image 2. Thus, the conversion coefficient H02 of the luminescent image 0 and the non-luminescent image 2 is calculated.

  By repeating the same processing, as shown in FIG. 8, the alignment conversion coefficient calculation unit 112 converts the conversion coefficient H02 of the light emitting pixel 0 and the non-light emitting pixel 2 and the conversion coefficient H23 of the non-light emitting image 2 and the non-light emitting image 3. Based on the above, the conversion coefficient H03 of the luminescent image 0 and the non-luminescent image 3 is obtained.

  The alignment conversion coefficient calculation unit 112 converts the light emission image 0 and the non-light emission image 4 from the conversion coefficient H03 of the light emission pixel 0 and the non-light emission pixel 3 and the conversion coefficient H34 of the non-light emission image 3 and the non-light emission image 4. The coefficient H04 is obtained.

  Using these conversion coefficients H01 to H04, the image transformation unit 113 aligns the non-light emitting images 1 to 4 with the light emitting image 0. Then, the image deformation unit 113 outputs the non-light-emitting images 1 to 4 subjected to the alignment processing with the light-emitting image 0 as a reference together with the light-emitting image 0 to the image composition unit 114.

  The image composition unit 114 receives the light emission image and the non-light emission image group subjected to the alignment processing by the alignment unit 107. Then, as will be described later, the image composition unit 114 outputs a region of the light emission image with respect to the region where the strobe is irradiated in the light emission image, and outputs a region of the non-light emission image to a region where the strobe is not irradiated. Further, when there are a plurality of non-light-emitting images, the image composition unit 114 performs averaging of the non-light-emitting images to reduce noise.

  FIG. 9 is a flowchart for explaining processing performed by the image composition unit 114 shown in FIG.

  When the image composition process is started, the image composition unit 114 first adjusts the luminance levels of the light-emitting image 0 and the non-light-emitting image 1 (step S901). Here, similarly to the processing in the strobe irradiation area calculation unit 109, the image composition unit 114 compares the exposure conditions of the light emission image 0 and the non-light emission image 1, and adjusts the gain so that the luminance levels are aligned according to the comparison result. I do.

  Subsequently, the image composition unit 114 calculates the absolute difference between the light-emitting image 0 and the non-light-emitting image 1 for which the luminance level has been adjusted (step S902). In the illustrated example, the absolute difference value is calculated in units of pixels. Here, since the luminance levels of the background are uniform, the absolute value of the difference is large in the region irradiated with the strobe and is small in the region not irradiated with the strobe.

  Similarly, the processes in steps S901 and S902 are performed on the light emitting image 0 and the non-light emitting images 2 to 4.

  On the other hand, since there is a concern that the difference absolute value becomes large regardless of the strobe irradiation due to the influence of local noise, the image may be low-pass filtered before calculating the difference absolute value. .

  Next, the image composition unit 114 calculates the image composition ratio of the light emission image 0 and the non-light emission image 1 based on the absolute difference value. Then, the image composition unit 114 performs image composition for each pixel in accordance with the image composition ratio (step S904).

  Thereafter, the image composition unit 114 determines whether or not all the images obtained as a result of photographing have been synthesized (step S905). If not all of the images have been combined (NO in step S905), the image combining unit 114 returns to the process of step S904.

  On the other hand, if not all of the images have been combined (NO in step S905), the image combining unit 114 ends the image combining process. At the time of image composition, the image composition unit 114 performs composition of the light emission image and the non-light emission image group.

  FIG. 10 is a diagram showing the relationship between the absolute difference value and the image composition ratio used in the image composition unit 114 shown in FIG.

  In FIG. 10, when the difference absolute value is small (if it is equal to or less than the threshold value TH <b> 1), the emission image synthesis ratio is 0 and the non-emission image synthesis ratio is 1. A 100% non-luminous image group is output as an image output from the image composition unit 114.

  When the difference absolute value exceeds the threshold value TH1 and is less than the threshold value TH2, the light emission image composition ratio decreases linearly, and the non-light emission image group increases linearly. As a result, if the difference absolute value exceeds the threshold TH1 and is less than the threshold TH2, the ratio (ratio) of the non-light emitting image group increases in the image output from the image composition unit 114 as the difference absolute value increases. Will do.

  When the absolute difference value is equal to or greater than the threshold value TH2, the emission image synthesis ratio is 1, the non-emission image synthesis ratio is 0, and a 100% emission image is output as an image output from the image synthesis unit 114.

  As described above, the luminescent image is synthesized according to the luminescent image synthesis ratio. However, since there are a plurality of non-luminescent images (here, four), the non-luminescent image synthesis ratio is set to the non-luminescent image. Divided by the number of images is the composition ratio per non-light-emitting image.

  In this manner, the light emitting image and the non-light emitting image group are combined by the image combining unit 114, and the combined image is sent to the image display unit 115 and the image recording unit 116.

  As described above, in the camera shown in FIG. 1, when aligning the strobe light emission image and the plurality of non-light emission images, the alignment of the plurality of non-light emission images is performed with reference to the predetermined non-light emission image. Thereafter, alignment between the light emitting image and a predetermined non-light emitting image is performed.

  This makes it possible to use the strobe irradiation region for alignment between the non-light emitting images, and the alignment accuracy is improved as compared with the prior art. As a result, the image quality of the composite image is improved.

  Furthermore, in the alignment of the luminescent image and the non-luminescent image, it is selected whether the alignment is performed on the main subject or the background portion according to the luminance in the non-luminescent image. It is possible to reduce deterioration in the image quality of the composite image due to the failure in alignment in the subject.

  As is clear from the above description, in the example shown in FIG. 1, the image processing unit 1006 functions as an image processing apparatus. Further, the non-light emission image conversion coefficient calculation unit 110 functions as a first calculation unit, and the light emission image conversion coefficient calculation unit 111 and the alignment conversion coefficient calculation unit 112 function as a second calculation unit. The alignment conversion coefficient calculation unit 112 and the image transformation unit 113 function as alignment means, and the image composition unit 114 functions as image composition means.

  The strobe irradiation area calculation unit 109 functions as an irradiation area detection unit. The imaging unit 101, the strobe light emission control unit 102, the exposure control unit 103, the A / D conversion unit 104, and the signal processing unit 105 function as an imaging unit.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the control method and the control program has at least a first calculation step, a second calculation step, an alignment step, and an image composition step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 106 Image processing part 107 Positioning part 108 Memory part 109 Strobe irradiation area calculation part 110 Non-light emission image conversion coefficient calculation part 111 Light emission image conversion coefficient calculation part 112 Positioning conversion coefficient calculation part 113 Image deformation part 114 Image composition part

Claims (12)

An image processing apparatus for aligning and combining a light emitting image obtained by emitting light when photographing a subject and a non-light emitting image group obtained continuously without emitting light,
A first alignment conversion coefficient used when aligning a non-light-emitting image excluding the reference non-light-emitting image with the reference non-light-emitting image with a specific non-light-emitting image as a reference non-light-emitting image in the non-light-emitting image group. A first calculating means to obtain;
Second calculation means for obtaining a second alignment conversion coefficient used when the reference non-light-emitting image is aligned with the light-emitting image on the basis of the light-emitting image;
Alignment means for aligning the light emitting image and the non-light emitting image group in accordance with the first alignment conversion coefficient and the second alignment conversion coefficient;
Image combining means for combining the light-emitting image and the non-light-emitting image group aligned by the alignment means into a composite image;
An image processing apparatus comprising: Comparing the luminance levels of the light emitting image and the reference non-light emitting image, the light emitting image includes an irradiation region detection means for specifying an irradiation region in which light emission by the light emission is performed in the light emitting image,
The first calculating means detects a motion vector between the reference non-light-emitting image and a non-light-emitting image excluding the reference non-light-emitting image, and according to the motion vector detected in the non-irradiation region excluding the irradiation region. To obtain a first conversion coefficient indicating a deviation of the image, further obtain a second conversion coefficient according to a motion vector detected in the entire image, and calculate the first conversion coefficient and the second conversion coefficient. Depending on the error between the coordinate of the predetermined point of interest converted in response to the motion vector, one of the first conversion coefficient and the second conversion coefficient is selected as the first alignment conversion coefficient The image processing apparatus according to claim 1, wherein:   The irradiation area detection unit divides each of the light emission image and the reference non-light emission image into predetermined blocks, obtains a difference in the luminance level for each block, and an absolute value of the difference is a predetermined first value. The image processing apparatus according to claim 2, wherein when the threshold value is larger than 1, the block is set as the irradiation region.   When the absolute value of the difference is equal to or less than a predetermined first threshold, the irradiation area detection unit obtains a light emission influence degree indicating the light emission influence degree according to the luminance level for each block, and the light emission The image processing apparatus according to claim 3, wherein when the influence degree is greater than a predetermined second threshold, the block is set as the irradiation area.   The second calculation means obtains an average luminance in the irradiation area in the reference non-light-emitting image, and when the average luminance is larger than a predetermined luminance threshold, the light-emitting image for each block in the irradiation area. And a motion vector in the reference non-light-emitting image is obtained, and the second alignment conversion coefficient for aligning the light-emitting image and the reference non-light-emitting image is obtained according to the motion vector. The image processing apparatus according to claim 3 or 4.   The second calculating means obtains a motion vector in the light-emitting image and the reference non-light-emitting image for each of the blocks in the non-irradiation area excluding the irradiation area when the average luminance is equal to or less than the luminance threshold, The image processing apparatus according to claim 5, wherein the second alignment conversion coefficient for performing alignment between the light emitting image and the reference non-light emitting image is obtained according to a motion vector.   The second calculation means is configured to output the light emission image and the second alignment conversion coefficient according to the first alignment conversion coefficient and the second alignment conversion coefficient for performing alignment between the light emission image and the reference non-light emission image. The image processing apparatus according to claim 6, wherein the second alignment conversion coefficient for performing alignment with a non-light-emitting image excluding a reference non-light-emitting image is obtained.   The image synthesizing unit is preset according to an absolute value of a luminance difference between the light emitting image and the non-light emitting image when the light emitting image and the non-light emitting image group aligned by the position aligning unit are combined. The light-emitting image and the non-light-emitting image group are combined based on a light-emitting image composition ratio and a non-light-emitting image composition ratio indicating a composite ratio of the light-emitting image and the non-light-emitting image. 8. The image processing device according to any one of items 7.   The image combining unit outputs the light-emitting image for the irradiation region and combines the light-emitting image and the non-light-emitting image group aligned by the position alignment unit, and the non-irradiation region excluding the irradiation region. The image processing apparatus according to claim 3, wherein an image obtained by adding and averaging the non-light emitting image group is output. The image processing apparatus according to any one of claims 1 to 9,
An imaging apparatus comprising: an imaging unit that captures a subject by capturing light to obtain a light-emitting image, and continuously captures the subject without performing the light emission to obtain a non-light-emitting image group. A control method of an image processing apparatus for aligning and combining a light emitting image obtained by emitting light when photographing a subject and a non-light emitting image group obtained continuously without emitting light,
In the non-light emitting image group, a first non-light emitting image is used as a reference non-light emitting image, and a first alignment conversion coefficient used when aligning a non-light emitting image excluding the reference non-light emitting image with the reference non-light emitting image is obtained. A first calculation step;
A second calculation step of obtaining a second alignment conversion coefficient used when aligning the reference non-light-emitting image with the light-emitting image on the basis of the light-emitting image;
An alignment step of aligning the light emitting image and the non-light emitting image group according to the first alignment conversion coefficient and the second alignment conversion coefficient;
An image combining step of combining the light-emitting image and the non-light-emitting image group aligned in the alignment step into a combined image;
A control method characterized by comprising: A control program used in an image processing apparatus for aligning and synthesizing a light-emitting image obtained by emitting light when photographing a subject and a non-light-emitting image group obtained continuously without light emission. ,
A computer included in the image processing apparatus,
In the non-light emitting image group, a first non-light emitting image is used as a reference non-light emitting image, and a first alignment conversion coefficient used when aligning a non-light emitting image excluding the reference non-light emitting image with the reference non-light emitting image is obtained. A first calculation step;
A second calculation step of obtaining a second alignment conversion coefficient used when aligning the reference non-light-emitting image with the light-emitting image on the basis of the light-emitting image;
An alignment step of aligning the light emitting image and the non-light emitting image group according to the first alignment conversion coefficient and the second alignment conversion coefficient;
An image combining step of combining the light-emitting image and the non-light-emitting image group aligned in the alignment step into a combined image;
A control program characterized by causing