JP2011199786A - Apparatus, method and program for processing image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To combine images using a plurality of images photographed time-sequentially.SOLUTION: An image processing apparatus includes an image capturing means (frame memory) 102 for capturing a plurality of images from an imaging device 101 wherein photoelectric transducers corresponding to respective color components are disposed in a predetermined pattern; and a combination processing part 106 for producing a combination image from the plurality of images. The combination processing part 106 is characterized by combining images based on a combination ratio of images determined in accordance with the color components. Furthermore, the combination processing part 106 includes a correlation calculating part in which one of a plurality of images is determined as a reference image and a correlation of pixels between the reference image and any other image than the reference image among the plurality of images or correlation between predetermined regions comprised of a plurality of pixels is calculated. when the correlation is not more than a threshold determined in accordance with the color components of pixels to be combined, as the correlation becomes small, the combination ratio of any other image than the reference image is reduced.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  It is effective to secure a sufficient exposure amount in order to obtain an image with little noise when taking a still image with an imaging device such as a digital camera. However, if the exposure time is increased in order to secure the exposure amount, there is a problem that the camera movement due to camera shake or the movement of the subject causes blurring in the image, and as a method for dealing with such blurring, Electronic blur correction methods have been proposed.

JP-A-9-261526 JP 2006-157568 A JP 2005-286678 A

  For example, in Patent Document 1 described above, after performing a plurality of continuous shootings with a short exposure time with less blur, and performing alignment processing so that the movement between the obtained images is canceled, A method of obtaining a good image without blurring by performing the composition processing is disclosed. Further, in Patent Document 2, there is a method for efficiently applying electronic blur correction with a low calculation amount to an image picked up using an image pickup device having a Bayer array widely used in digital cameras and the like. It is disclosed. Normally, the captured Bayer array image is converted into an RGB image of three plates, an image of YCbCr format, or the like by development processing. The method disclosed in Patent Document 2 directly combines images with a Bayer array image instead of a three-plate image, and reduces the number of pixels to be processed, thereby enabling synthesis with a low calculation amount. Yes.

  On the other hand, in the noise reduction method, there is a demand for saving the resolution with respect to the luminance signal rather than the noise reduction performance with respect to the luminance signal, and with regard to the noise reduction performance with respect to the saving of the resolution with respect to the color signal. In order to respond to such a request, for example, in Patent Document 3 described above, an image is separated into a luminance signal and a color signal, and parameters are changed for each, or a separate noise reduction method is applied. Thus, a technique for realizing noise reduction performance with different strengths is disclosed. Even in the electronic blur correction method, when applying synthesis to an image in YCbCr format, noise reduction for luminance signals and color signals is performed by setting the number of synthesized images for Y images and the number of synthesized images for CbCr images to be different. It is possible to change the performance.

  However, when electronic blur correction is directly applied to an image with a Bayer array, there is a problem that the noise reduction performance cannot be changed for the luminance signal and the color signal because they are not separated. Further, for example, if the YCbCr signal is created from the Bayer array image and a different synthesis method is applied, the above problem can be solved. However, in this case, an operation to convert the Bayer array into three plates and an operation to convert to the YCbCr format are required, and the amount of calculation reduced by applying electronic blur correction directly on the Bayer array image can be reduced. There was a problem of diminishing the effect.

  The present invention has been made in view of the above-described circumstances, and is an image processing apparatus capable of correcting an image with different noise reduction performance with respect to a luminance signal and a color signal while reducing a calculation amount. And a method and a program.

  In order to achieve the above object, the present invention provides the following means.

  According to a first aspect of the present invention, image acquisition means for acquiring a plurality of images from an image sensor in which photoelectric conversion elements corresponding to each color component are arranged in a predetermined pattern, and synthesis for generating a composite image from the plurality of images Processing means, and the composition processing means is an image processing apparatus for synthesizing images based on a composition ratio of images determined according to the color components.

  According to such a configuration, it is possible to reduce the amount of calculation by combining a plurality of images in the Bayer array based on a combination ratio determined according to the color component, and maintaining the Bayer array. It is possible to control the effect of noise reduction on the luminance signal and the color signal while maintaining the above. Thereby, it is possible to increase the noise reduction effect in the color signal while suppressing artifacts in the luminance signal due to the composition failure.

  In the first aspect of the present invention, the synthesis processing means determines one image from the plurality of images as a reference image, and the reference image and an image other than the reference image among the plurality of images. A correlation calculating means for calculating a correlation amount between pixels or a correlation amount between predetermined areas composed of a plurality of pixels, and the correlation amount is determined according to a color component of the synthesis target pixel In the following cases, as the correlation amount decreases, the composition ratio of images other than the reference image may be decreased.

  In the first aspect of the present invention, the synthesis processing means is configured to estimate a noise level for each pixel or for each predetermined region including a plurality of pixels in at least one of the plurality of images. An estimation means may be provided, and the threshold value may be calculated based on the noise level estimated by the noise level estimation means.

  In the first aspect of the present invention, when the motion information acquisition means for acquiring the motion information between the reference image and the target image, and the shift amount of the pixel based on the motion information is other than an even number of pixels, Image correction means for interpolating pixels of the target image and correcting the interpolated target image by shifting according to the shift amount, and the synthesis processing means includes the plurality of corrections corrected by the image correction means. Images may be combined.

  According to a second aspect of the present invention, a plurality of images are acquired from an image sensor in which photoelectric conversion elements corresponding to the respective color components are arranged in a predetermined pattern, and based on a composite ratio of images determined according to the color components. An image processing method for generating a composite image by combining the plurality of images.

  According to a third aspect of the present invention, there is provided a first process for acquiring a plurality of images from an image sensor in which photoelectric conversion elements corresponding to each color component are arranged in a predetermined pattern, and an image determined according to the color component. An image processing program for causing a computer to execute a second process of generating a combined image by combining the plurality of images based on a combining ratio.

  According to the present invention, it is possible to perform image correction with different noise reduction performances for luminance signals and color signals while reducing the amount of calculation.

1 is a block diagram illustrating a configuration example of an image processing apparatus according to a first embodiment of the present invention. It is the figure which showed the pixel of Bayer arrangement. It is a figure showing an example of composition of a composition processing part concerning a 1st embodiment of the present invention. It is the figure which showed the operation | movement flow of the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the method of synthesize | combining one image from several images. It is the figure which showed the structural example of the synthetic | combination process part which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the determination method of the synthetic | combination ratio in a synthetic | combination ratio determination part (Gr, Gb pixel). It is a figure for demonstrating the determination method of the composite ratio in a composite ratio determination part (R, B pixel). It is the figure which showed the structural example of the synthetic | combination process part which concerns on the 3rd Embodiment of this invention. It is a figure for demonstrating the typical relationship between the pixel value which an image sensor outputs, and noise amount. It is the figure which showed the structural example of the noise level estimation part. It is a figure for demonstrating the determination method of the synthetic | combination ratio in a synthetic | combination ratio determination part (Gr, Gb pixel). It is a figure for demonstrating the determination method of the composite ratio in a composite ratio determination part (R, B pixel).

  Embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 according to the present embodiment is, for example, a digital camera.
As shown in FIG. 1, the image processing apparatus 1 includes an optical system 100, an image sensor 101, a frame memory (image acquisition unit) 102, an image processing unit 103, a motion information acquisition unit (motion information acquisition unit) 104, and image correction. Section (image correction means) 105 and a composition processing section (composition processing means) 106. In the present embodiment, description will be made assuming that two images, a reference image and a target image, are to be processed.

  The frame memory 102 acquires a plurality of images from the image sensor 101 in which photoelectric conversion elements corresponding to each color component are arranged in a predetermined pattern. The acquired plurality of images are output to the motion information acquisition unit 104 and the image correction unit 105.

  The motion information acquisition unit 104 acquires motion information that is a motion between the reference image and the target image stored in the frame memory 102. In the motion information, for example, the motion between the reference image and the target image is indicated as one piece of motion vector information (horizontal movement amount and vertical movement amount).

  In addition to the motion information being one motion vector information per image, a method of dividing an image into a plurality of regions and calculating a motion vector in each region may be used. Further, the motion vector is not made up of the horizontal direction movement amount and the vertical direction movement amount, but can also be in a format that can express changes due to rotation or enlargement / reduction. Furthermore, a method of acquiring motion information using a sensor such as a gyro provided in the apparatus may be used.

  The image correction unit 105 corrects the target image stored in the frame memory 102 based on the motion information acquired by the motion information acquisition unit 104. More specifically, the position of the target image is adjusted by shifting the position of the target image based on the motion vector information including the horizontal direction movement amount and the vertical direction movement amount. For example, when shifting the position of the target image, in the case of a shift by an even number of pixels, each color (R, Gr, Gb, B) in the Bayer array (details will be described later) of the reference image and the shifted target image Therefore, the image correction unit 105 need only shift the target image.

  On the other hand, when a shift for an odd number of pixels is included, or when a shift for a fractional number of pixels is included, in other words, when the shift amount is other than an even number of pixels, if the shift is performed by that amount, Since the color arrangement and the color arrangement of the target image after the shift do not match, the image correction unit 105 performs an interpolation process and a shift process in order to match the color correspondence between the two images after the shift. .

More specifically, the processing of the image correction unit 105 when a shift of an odd number of pixels (for example, one pixel) is included will be described with reference to FIG.
FIG. 2A is a diagram illustrating pixels in a Bayer array. In the Bayer array, G (green) filters are arranged in a checkered pattern, and R (red) filters and B (blue) filters are alternately arranged for each line. Therefore, the image signal output from the image sensor 101 is a signal having a pixel value of any one color of R (red) component, G (green) component, and B (blue) component per pixel. Such image signals are input to the frame memory 102 in the order of colors such as G, R, G, R... Or B, G, B, G, for example.

In such a Bayer array pixel, when each pixel of the target image is shifted (shifted) by one pixel in the horizontal direction, the result is as shown in FIG. 2B ', and the Bayer array shown in FIG. And the sequence is different. Therefore, in this state, since it cannot be combined with the basic image, an image in which the same color array as that of the basic image is obtained by shifting one pixel, that is, in this case, an array as shown in FIG. An image is created by performing an interpolation process.
In the case of 1 pixel shift, in order to obtain the image of FIGS. 2A to 2B, for example, an R (red) pixel must be created at the position of the pixel G33 (green).

  In this case, interpolation processing is performed using R (red) pixels present around G33, for example, R (red) components arranged on both sides in the horizontal direction, that is, R32 and R34, and the position of G33 is set. An R (red) pixel is created. In this case, the R (red) pixel created in G33 is the average of the pixel values of R32 and R34. Then, by performing the same interpolation process on all other pixels, an image having the color array shown in FIG. 2B is created from the color array shown in FIG.

Then, after the image having the color arrangement shown in FIG. 2B is created, the image shown in FIG. 2C is obtained by shifting the image by one pixel in the horizontal direction. Thereby, the color arrangement of the image after the shift can be matched with the same color arrangement as that of the basic image.
As described above, when a shift for an odd number of pixels is included, an interpolation process is performed in advance to create an image in which the color arrangement after the shift is the same as the basic pixel. Accordingly, the Bayer array in the reference image can be matched with the Bayer array in the target image after the shift.

In the present embodiment, for the sake of convenience of explanation, the case of shifting by one pixel (integer pixel) in the horizontal direction has been described. However, when a shift other than even pixels is performed, the same method can be used.
For example, when the shift amount includes a decimal point, it can be dealt with by performing a weighted average process in the interpolation process. In this case, since the shift can be performed only in units of one pixel, an image having a color arrangement that enables even a shift after the decimal point is created by devising an interpolation method.

  For example, when it is necessary to shift 0.1 pixel leftward on the paper surface of FIG. 2, an average value obtained by weighting R34 pixels with 0.1 weight and R32 pixels with 0.9 weighting. And the calculated average value is set as the R component for the G33 pixel. Then, by performing the same weighted average processing on the remaining pixels, first, an image is generated by shifting the image shown in FIG. 2A toward the right side of the paper by 0.9 pixels. Then, by shifting this image by one pixel toward the left side of the paper, it is possible to finally obtain an image shifted by 0.1 pixel leftward toward the paper surface of FIG. In the above description, the interpolation processing is performed using both adjacent pixels, but the present invention is not limited to this example. For example, the pixel value of the target pixel is calculated by using pixels of the same color from a predetermined pixel area centered on the target pixel, and further giving a weighting value corresponding to the shift amount to each of these pixels. It is also good.

  For example, by using six pixels R12, R14, R32, R34, R52, and R54 surrounding the pixel G33 and assigning weights according to the shift amount to these pixels, an R component pixel is created in G33. Also good.

Thereby, the correspondence relationship of each color (R, Gr, Gb, B) in the Bayer array of the reference image and the target image is made to match.
For example, in the case of a configuration in which motion information includes information related to rotation and enlargement / reduction, the image correction unit 105 performs correction processing corresponding to rotation and enlargement / reduction to align the reference image and the target image. It becomes.

  The composition processing unit 106 outputs a composite image from the target image that has been subjected to the alignment processing by the image correction unit 105 and the reference image. Specifically, the synthesis processing unit 106 includes a weighted averaging processing unit 200 and a synthesis ratio determination unit 201. FIG. 3 is a block diagram illustrating a configuration example of the synthesis processing unit 106.

  The composition ratio determination unit 201 determines a composition ratio between the reference image and the target image. As for the composition ratio, the composition ratio of the target image when the reference image is 1.0 is represented by 0.0 to 1.0. The combination ratio determination unit 201 outputs a preset combination ratio to the weighted average processing unit 200 based on color identification information that identifies whether the processing target pixel is R, Gr, Gb, or B. When the processing target pixels are Gr and Gb, the contribution ratio of the G component to the luminance signal is larger than that of the R and B components. Therefore, the preservation of the resolution is more important than the noise reduction performance, and the composition ratio is set low.

  When the processing target pixels are R and B, since the contribution ratio of the R and B components to the color signal is larger than that of the G component, the noise reduction performance is prioritized over the resolution storage, and the synthesis ratio is increased. The combination ratio corresponding to each color component determined as described above is set in advance in the combination ratio determination unit 201, and the combination ratio determination unit 201 outputs the combination ratio to the weighted averaging processing unit 200 according to the color identification information. To do.

  The weighted average processing unit 200 performs weighted average processing of the pixels of the reference image and the target image in accordance with the combination ratio output from the combination ratio determination unit 201 to obtain the pixel of the combined image. The image processing unit 103 performs image processing such as three-plate processing, color processing, and gradation conversion processing on the composite image of the Bayer array that has undergone the composite processing, and generates an output image.

  Next, the operation of the image processing apparatus having the above configuration will be described with reference to FIG. In the image processing apparatus 1, an image is picked up by the image pickup device 101 through the optical system 100 including a lens or the like, and a plurality of picked-up images (for example, frame 1 and frame 2) are stored in the frame memory 102 (step SA1). The image sensor 101 has, for example, a Bayer array, and an image of the Bayer array is stored in the frame memory 102. By repeating such imaging processing for a predetermined number of sheets, a plurality of Bayer array images are stored in the frame memory 102.

  In the motion information acquisition unit 104, frame 1 is a reference image and frame 2 is a target image, and motion vector information is acquired (detected) based on frame 1 (reference image) and frame 2 (target image). (Step SA2). In the image correction unit 105, the frame 2 (target image) is corrected based on the motion vector information acquired from the motion information acquisition unit 104 (step SA3). The composition ratio determining unit 201 determines the composition ratio based on the color identification information (step SA4). In the weighted averaging processing unit 200, the weighted average processing is performed on the frame 1 (reference image) and the frame 2 (target image) based on the composition ratio, and a composite image is generated (step SA5). The composite image generated in this way is output to the frame memory 102 (step SA6). The processes in step SA4 and step SA5 are repeatedly performed on all images captured by the image sensor 101.

More specifically, an example in which one image is synthesized from four images is shown. The basic process of combining four images from frame 1 to frame 4 and synthesizing the two images to obtain one composite image is performed three times, so that one composite image is finally formed from the four images. The case of obtaining is described with reference to FIG.
As shown in FIG. 5, frame 1 and frame 2 are combined to generate a combined image 1. At the time of synthesis, one is defined as a reference image and the other as a target image. Subsequently, the frame 3 is combined with the frame 4 using the frame 3 as a reference image and the frame 4 as a target image, thereby generating a combined image 2. Further, the synthesized image 1 is used as a reference image and the synthesized image 2 is set as a target image, and the synthesized image 1 and the synthesized image 2 are synthesized to generate a synthesized image 3.

  Note that the method of combining a plurality of sheets is not limited to this, and for example, a method of combining the combined image 1 and the frame 3 in FIG. In addition, instead of combining the basic process of combining two reference images and one target image, a total of four reference images and three target images, the motion information acquisition unit 104, the image This can be easily achieved by expanding the correction unit 105 and the composition processing unit 106.

  In addition to the method of determining the reference image, in addition to the image picked up first, the image was picked up at an intermediate time by changing the method of making the image picked up later or each time basic processing is performed. It is also possible to use the image as a reference.

In the above-described embodiment, processing by hardware is assumed as the image processing apparatus, but it is not necessary to be limited to such a configuration. For example, a configuration in which software is separately processed based on output signals from each sensor is also possible. In this case, the image processing apparatus includes a main storage device such as a CPU and a RAM, and a computer-readable recording medium on which a program for realizing all or part of the above processing is recorded. Then, the CPU reads out the program recorded in the storage medium and executes information processing / calculation processing, thereby realizing processing similar to that of the above-described image processing apparatus.
Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  As described above, according to the image processing device, the image processing method, and the program according to the present invention, by synthesizing a plurality of Bayer array images based on the composition ratio determined according to the color components. In addition, it is possible to control the effect of reducing noise with respect to the luminance signal and the color signal while maintaining the effect of reducing the amount of calculation by synthesizing while maintaining the Bayer arrangement. As a result, artifacts in the luminance signal due to the composition failure can be suppressed, and the noise reduction effect in the color signal can be improved.

  In the present embodiment, the composition ratio is set from 0.0 to 1.0. However, the present invention is not limited to this. For example, the composition ratio may be a binary value of 0.0 and 1.0, and the weighted averaging processing unit 200 may be replaced with a simple averaging process to reduce the calculation amount. Accordingly, for example, it is possible to adopt a configuration in which two images are combined for Gr and Gb pixels, and eight images are combined for R and B pixels, and the above-described effects can be obtained. .

[Second Embodiment]
Next, an image processing apparatus according to a second embodiment of the present invention will be described with reference to FIG.
As shown in FIG. 6, the image processing apparatus according to the second embodiment is further provided with a correlation calculation unit (correlation calculation means) 300 in which the configuration of the synthesis processing unit 106 in the first embodiment is changed. It is a configuration. As shown in FIG. 6, the reference image and the target image in the Bayer array are input to the composition processing unit 106 ′ in the second embodiment. The correlation of these images is calculated by the correlation calculation unit 300.
Hereinafter, the image processing apparatus according to the present embodiment will not be described with respect to the points common to the first embodiment, and different points will be mainly described.

  The correlation calculation unit 300 calculates an absolute difference value for each pixel as a correlation value between images. Generally, when the alignment is successful, the absolute difference value is small, and when the alignment is unsuccessful, the absolute difference value is large. It is used for control to suppress artifacts such as blur and double image caused by alignment failure.

  In this embodiment, the absolute value of the difference between pixels is used as the correlation value. In addition, in order to calculate a more stable correlation value, the difference between the blocks is determined for the block including the peripheral pixels of the target pixel. The absolute value sum (SAD) may be used as the correlation value. Further, in order to reduce the amount of calculation, a correlation value may not be calculated for each pixel, but one correlation value may be calculated for a region composed of a plurality of pixels. Further, for example, instead of the correlation value between pixels in the R pixel, a correlation value of neighboring Gr or Gb pixels may be used.

  The combination ratio determination unit 301 determines the reference image and the target based on the difference absolute value calculated by the correlation calculation unit 300 and the color identification information that identifies whether the processing target pixel is R, Gr, Gb, or B. Determine the composition ratio of the images. As for the composition ratio, the composition ratio of the target image when the reference image is 1.0 is represented by 0.0 to 1.0. The composition ratio is controlled by the magnitude of the absolute difference value of the pixels. When the difference absolute value is small, it is highly likely that the alignment has succeeded, so the composition ratio is increased.

  When the difference absolute value is large, there is a high possibility that the alignment has failed, so the synthesis ratio is decreased to suppress artifacts. Furthermore, using the color identification information, for Gr and Gb pixels, the synthesis ratio is controlled so that priority is given to maintaining resolution over noise reduction performance, and for R and B pixels, noise reduction performance is prioritized over resolution maintenance. .

  FIG. 7 is a diagram illustrating an example of the relationship between the pixel difference absolute value and the composition ratio in the Gr and Gb pixels, and FIG. 8 illustrates an example of the relationship between the pixel difference absolute value and the composition ratio in the R and B pixels. FIG. In the example of FIG. 7, when the difference absolute value is smaller than the threshold value 1, the composition ratio is set to 1.0, and when the difference absolute value is greater than the threshold value 2, the composition ratio is set to 0.0. Is a linear transition of the composition ratio. Threshold values 1 and 2 are set to be smaller values, and when the difference absolute value is large, the composition ratio is quickly reduced to suppress artifacts due to composition failure.

  In the example of FIG. 8, when the difference absolute value is smaller than the threshold value 3, the composition ratio is set to 1.0, and when the difference absolute value is greater than the threshold value 4, the composition ratio is set to 0.5. Is a linear transition of the composition ratio. The thresholds 3 and 4 are set to larger values, and even when the absolute difference value is large to some extent, priority is given to maintaining the noise reduction capability so that the synthesis ratio does not become too small.

  As described above, according to the image processing apparatus, the image processing method, and the program according to the present embodiment, a plurality of images in a Bayer array are combined based on a combination ratio determined according to color components. As a result, it is possible to control the noise reduction effect on the luminance signal and the color signal while maintaining the effect of reducing the amount of calculation by synthesizing while maintaining the Bayer array. Furthermore, by setting the relationship between the correlation value between images and the composition ratio according to the color component, it is possible to further enhance the noise reduction effect in the color signal while suppressing artifacts in the luminance signal due to composition failure. .

  In this embodiment, the composition ratio is continuously changed between 0.0 and 1.0. However, this is a binary value of 0.0 and 1.0, and the weighted averaging processing unit 200 is used. It is also possible to reduce the amount of calculation by replacing with a simple averaging process.

[Third Embodiment]
The image processing apparatus according to the third embodiment of the present invention has a configuration obtained by changing the configuration of the synthesis processing unit 106 according to the second embodiment. FIG. 9 shows a configuration diagram of the composition processing unit 106 ″ in the third embodiment. In the third embodiment, the synthesis processing unit 106 ′ in the second embodiment is further provided with a noise level estimation unit (noise level estimation means) 400, and the synthesis ratio in the synthesis ratio determination unit 401 is determined based on the noise level and between the images. By determining by correlation, artifacts such as blur and double image can be suppressed more accurately and reliably, and by controlling the combination ratio using color identification information, the effect of noise reduction on the luminance signal and color signal can be improved. It is a form to control.
Hereinafter, the image processing apparatus according to this embodiment will not be described for points that are the same as those in the first and second embodiments, and different points will be mainly described.

  The reference image and target image in the Bayer array are input to the composition processing unit 106 ″. First, a correlation value between images is calculated by the correlation calculation unit 300 from the reference image and the target image. Details are the same as in the second embodiment. Further, the noise level estimation unit 400 estimates the noise intensity (noise level) included in the processing target pixel from the reference image and the target image.

  In general, it is known that there is a certain relationship between the amount of noise contained in a pixel output from an image sensor and the pixel value, and the amount of noise can be estimated from the pixel value. FIG. 10 shows a typical relationship between the pixel value output from the image sensor and the amount of noise. In FIG. 10, the horizontal axis represents the pixel value of the pixel output from the image sensor, and the vertical axis represents the amount of noise (such as the standard deviation of noise) contained in the pixel. Typically, the amount of noise included tends to increase as the pixel value output from the image sensor increases.

  The noise level estimation unit 400 estimates the noise intensity (noise level) included in the processing target pixel using the relationship between the noise amount and the pixel value as described above. FIG. 11 shows a configuration diagram of the noise level estimation unit 400. The noise level calculation units 500 and 501 prepare the relationship between the pixel value and the amount of noise as shown in FIG. 10 in advance by a method such as polygonal line approximation or table formation, and the reference image pixel or target image. The noise level of each pixel is calculated.

  Note that the noise level estimated by the noise level estimation unit 400 does not need to completely match the amount of noise output from the image sensor, and also considers noise amplified by image processing such as gradation conversion processing performed later. It may be set. Typically, since the noise in the dark portion is amplified by the gradation conversion process, a value larger than the amount of noise output from the image sensor in the dark portion can be set as the noise level.

  For pixels in which the alignment of the reference image and the target image is successful, there is no significant difference between the pixel value of the reference image and the pixel value of the target image, and no significant difference occurs in the calculated noise level. However, there is a high possibility that a difference will occur in a pixel for which alignment has failed, and this is set as the noise level by selecting the maximum value in the maximum value calculation unit 502 as the noise level of the pixel. .

  In the present embodiment, the maximum value is calculated by the maximum value calculation unit 502 and set as the noise level. However, in addition to this, a weighted average value of the noise level of the reference image and the noise level of the target image is used. For example, it is possible to estimate the noise level by weighting the pixels of the reference image. If it is not desirable to operate a plurality of noise level calculation units 500 and 501 from the viewpoint of the amount of calculation, only the noise level calculation unit 500 may be operated and this output may be used as the noise level.

Next, the operation of the composition ratio determination unit 401 will be described.
The synthesis ratio determination unit 401 determines whether the noise level calculated by the noise level estimation unit 400, the difference absolute value calculated by the correlation calculation unit 300, and the processing target pixel is R, Gr, Gb, or B. Based on the color identification information to be identified, the composition ratio of the reference image and the target image is determined.

  As in the second embodiment, the composition ratio represents the composition ratio of the target image from 0.0 to 1.0 when the reference image is 1.0. The composition ratio is controlled by the magnitude of the absolute difference value of the pixels. When the difference absolute value is small, it is highly likely that the alignment has succeeded, so the composition ratio is increased. When the difference absolute value is large, there is a high possibility that the alignment has failed, so the synthesis ratio is reduced to suppress artifacts. Furthermore, using the color identification information, for Gr and Gb pixels, the synthesis ratio is controlled so that the preservation of resolution is more important than the noise reduction performance, and for R and B pixels, the noise reduction performance is prioritized over the preservation of resolution. .

  FIG. 12 shows an example of the relationship between the pixel difference absolute value and the composition ratio in the Gr and Gb pixels, and FIG. 13 shows an example of the relationship between the pixel difference absolute value and the composition ratio in the R and B pixels. In the example of FIG. 12, as in the second embodiment, when the difference absolute value is smaller than the threshold value 1, the combination ratio is 1.0, and when the difference absolute value is greater than the threshold value 2, the combination ratio is 0. The composition ratio is shifted linearly from threshold value 1 to threshold value 2. In addition, by controlling the threshold value 1 and the threshold value 2 according to the noise level, the synthesis ratio is controlled according to the noise level.

  In a pixel with a high noise level, the necessity for noise reduction by synthesis is high, so that threshold 1 and threshold 2 are increased in order to increase the synthesis ratio. Specifically, a value obtained by multiplying the noise level by a predetermined constant may be added to threshold 1 and threshold 2, respectively. In a pixel with a low noise level, since the necessity for noise reduction by synthesis is low, the threshold value 1 and the threshold value 2 are reduced in order to reduce the synthesis ratio. Specifically, a value obtained by multiplying the noise level by a predetermined constant may be subtracted from the threshold value 1 and the threshold value 2, respectively. The composition ratio determining unit 503 may prepare such a relationship by a method such as tabulation or may calculate the relationship by mathematical formula calculation.

  In the example of FIG. 13, when the difference absolute value is smaller than the threshold value 3, the composition ratio is set to 1.0, and when the difference absolute value is larger than the threshold value 4, the composition ratio is set to 0.5. Is a linear transition of the composition ratio. The thresholds 3 and 4 are set to larger values, and even when the absolute difference value is large to some extent, priority is given to maintaining the noise reduction capability so that the synthesis ratio does not become too small. As in the case of the Gr and Gb pixels, the threshold value 3 and the threshold value 4 are controlled according to the noise level, thereby controlling the synthesis ratio according to the noise level.

  As described above, according to the image processing apparatus, the image processing method, and the program according to the present embodiment, a plurality of images in a Bayer array are combined based on a combination ratio determined according to color components. As a result, it is possible to control the noise reduction effect on the luminance signal and the color signal while maintaining the effect of reducing the amount of calculation by synthesizing while maintaining the Bayer array. Furthermore, the relationship between the correlation value between images and the composition ratio is set according to the color component, and these are controlled according to the noise level of the pixel to be processed, thereby accurately suppressing artifacts in the luminance signal due to composition failure. However, the noise reduction effect in the color signal can be further enhanced.

  In the present embodiment, the composition ratio is continuously changed between 0.0 and 1.0. However, this is a binary value of 0.0 and 1.0, and the weighted averaging processing unit 200 It is also possible to reduce the amount of calculation by replacing with a simple averaging process.

104 motion information acquisition unit 105 image correction unit 106 synthesis processing unit 200 weighted averaging processing unit 201, 301, 401 synthesis ratio determination unit 300 correlation calculation unit 400 noise level estimation unit 500, 501 noise level calculation unit 502 maximum value calculation unit

Claims (6)

Image acquisition means for acquiring a plurality of images from an image sensor in which photoelectric conversion elements corresponding to the respective color components are arranged in a predetermined pattern;
Combining processing means for generating a composite image from the plurality of images,
The image processing apparatus, wherein the composition processing unit synthesizes an image based on an image composition ratio determined according to the color component. The synthesis processing means includes
One image is defined as a reference image from the plurality of images, and a correlation amount between pixels between the reference image and an image other than the reference image among the plurality of images, or a predetermined number of pixels. A correlation calculation means for calculating a correlation amount between the regions of
The composition ratio of images other than the reference image is reduced as the correlation amount is reduced when the correlation amount is equal to or less than a threshold value determined according to the color component of the synthesis target pixel. The image processing apparatus according to 1. The synthesis processing means includes
Noise level estimation means for estimating a noise level for each pixel or for each predetermined region including a plurality of pixels in at least one of the plurality of images,
The image processing apparatus according to claim 2, wherein the threshold value is calculated based on a noise level estimated by the noise level estimation unit. Motion information acquisition means for acquiring motion information between the reference image and the target image;
Image correction means for interpolating pixels of the target image when the pixel shift amount based on the motion information is other than an even number of pixels and correcting the interpolated target image according to the shift amount; Prepared,
The synthesis processing means includes
The image processing apparatus according to claim 1, wherein the plurality of images corrected by the image correction unit are combined. A plurality of images are acquired from an image sensor in which photoelectric conversion elements corresponding to each color component are arranged in a predetermined pattern,
An image processing method for generating a composite image by combining the plurality of images based on a composite ratio of images determined according to the color components. A first process of acquiring a plurality of images from an image sensor in which photoelectric conversion elements corresponding to each color component are arranged in a predetermined pattern;
An image processing program for causing a computer to execute a second process of generating a composite image by combining the plurality of images based on a composite ratio of images determined according to the color components.

Images