JP2011055259A - Image processing apparatus, image processing method, image processing program and program storage medium stored with image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce an artifact such as a blur or a double image while maintaining a noise reduction effect in an electronic shake correction technology for reducing a shake by performing combination processing after performing the alignment processing of a plurality sheets of images. <P>SOLUTION: Frequency decomposing means 200 and 201 decompose a plurality of images into a plurality of frequency bands to generate band images for the plurality of frequency bands. When one of the plurality of images is defined as a reference image and other image are defined as target images, a combination ratio determining means determines, for each frequency band, a combination ratio of the band images of the target images relative to the band images of the reference image. Band image combining means 202, 203 and 204 combine, for each frequency band, the band images based on the combination ratio to generate, for each frequency band, a combination band image. An image combining means 205 combines a plurality of combination band images to generate a combination image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a program storage medium storing the image processing program, and in particular, an image processing apparatus that synthesizes an image using a plurality of images taken in time series, and an image The present invention relates to a program storage medium storing a processing program and an image processing program.

  It is effective to secure a sufficient exposure time in order to obtain an image with less noise when taking a still image with an imaging device such as a digital camera. However, if the exposure time is lengthened, there is a problem that the image is blurred due to camera movement due to camera shake or subject movement, resulting in blurring.

  As a method for dealing with such blurring, an electronic blur correction method has been proposed. For example, in Japanese Patent Application Laid-Open No. 9-261526 (Patent Document 1), shooting with a short exposure time with little blurring is continuously performed a plurality of times so that the movement between the obtained images is canceled. An invention is disclosed in which a good image without blurring is obtained by performing a composition process after the alignment process.

  However, in the invention disclosed in Patent Document 1, since the composition process is a simple addition process of a plurality of images, in an area where the alignment process of the plurality of images has failed, an image blur or 2 There arises a problem that artifacts such as multiple images occur.

  In order to reduce this artifact, Japanese Patent Laid-Open No. 2002-290817 (Patent Document 2) calculates a difference value between corresponding pixels before addition processing (averaging processing) by combining processing, and this difference value is calculated. An invention is disclosed in which it is determined that the alignment process has failed when the threshold value is exceeded, and the synthesis process is not performed. Japanese Patent Laid-Open No. 2008-99260 (Patent Document 3) discloses an invention in which the weight of the weighted average process in the composition process is adjusted based on the difference value between corresponding pixels.

JP-A-9-261526 JP 2002-290817 A JP 2008-99260 A

  However, in the inventions described in Patent Document 2 and Patent Document 3 described above, if an error occurs in the alignment processing of a plurality of images or a shift occurs even in several pixels due to movement of the subject, noise reduction can be achieved when a composite image is generated. There was a problem that the effect was reduced.

  The present invention has been made in view of the above-described problem, and maintains a noise reduction effect in an electronic shake correction technique that reduces shake by performing a composition process after aligning a plurality of images. An object of the present invention is to provide an image processing apparatus that can reduce artifacts such as blurring and double images.

In order to solve the above problems, the present invention employs the following means.
The present invention is an image processing apparatus that acquires a plurality of images, generates a composite image by combining the plurality of images, and decomposes each of the images into a plurality of frequency bands. A frequency resolving means for generating a band image; and when any one of the plurality of images is a reference image and another image is a target image, the band image of the target image with respect to the band image of the reference image A synthesis ratio determining unit that determines a synthesis ratio for each frequency band; and a band image of the target image is synthesized for each frequency band based on the synthesis ratio with respect to the band image of the reference image. There is provided an image processing apparatus comprising: a band image synthesizing unit for generating a synthesized band image; and an image synthesizing unit for synthesizing the plurality of synthesized band images to generate a synthesized image.

  Further, the present invention is an image processing method for acquiring a plurality of images and synthesizing the plurality of images to generate a composite image, wherein each of the images is decomposed into a plurality of frequency bands, and the frequency bands A frequency resolving step for generating a band image for each, and a band of the target image with respect to the band image of the reference image when any one of the plurality of images is a reference image and another image is a target image A synthesis ratio determining step for determining an image synthesis ratio for each frequency band; and a band image of the target image is synthesized with respect to the band image of the reference image for each frequency band based on the synthesis ratio. An image processing method comprising: a band image synthesizing step for generating a synthesized band image for each, and an image synthesizing step for synthesizing the plurality of synthesized band images to generate a synthesized image. To provide.

  Furthermore, the present invention is an image processing program for acquiring a plurality of images and generating a composite image by combining the plurality of images, wherein each of the images is decomposed into a plurality of frequency bands. A frequency resolving step for generating a band image for each, and a band of the target image with respect to the band image of the reference image when any one of the plurality of images is a reference image and another image is a target image A synthesis ratio determining step for determining an image synthesis ratio for each frequency band; and a band image of the target image is synthesized with respect to the band image of the reference image for each frequency band based on the synthesis ratio. An image processing comprising: a band image synthesizing step for generating a synthesized band image for each, and an image synthesizing step for synthesizing the plurality of synthesized band images to generate a synthesized image. It is executed to provide an image processing program characterized.

  Thus, according to the present invention, the image is frequency-decomposed into a plurality of frequency bands, the band image in each frequency band is determined for each frequency band, and the composition processing is performed for each frequency band. It is possible to perform synthesis suitable for each frequency band, and to obtain a sufficient noise reduction effect by synthesis while suppressing the occurrence of artifacts due to synthesis failure compared to synthesis without frequency decomposition. There is an effect that can be done.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. FIG. 5 is a reference diagram illustrating an example of a procedure when images are combined in the image processing apparatus according to the first embodiment of the present invention. It is a block diagram which shows schematic structure of the synthetic | combination process part in the image processing apparatus concerning the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the frequency decomposition part in the image processing apparatus concerning the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the high frequency band image synthetic | combination part in the image processing apparatus concerning the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the frequency synthetic | combination part in the image processing apparatus concerning the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the high frequency band image synthetic | combination part in the image processing apparatus concerning the 2nd Embodiment of this invention. It is a reference figure which shows the relationship between the synthetic | combination ratio determined by the synthetic | combination ratio determination part in the image processing apparatus concerning the 2nd Embodiment of this invention, and the difference absolute value of an image. It is a block diagram which shows schematic structure of the high frequency band image synthetic | combination part in the image processing apparatus concerning the 3rd Embodiment of this invention. FIG. 6 is a reference diagram illustrating a typical relationship between an output pixel value of an image sensor and a noise amount. FIG. 6 is a reference diagram illustrating a typical relationship between a pixel value and a noise amount after gradation conversion processing. It is a block diagram which shows schematic structure of the noise level estimation part in the image processing apparatus concerning the 3rd Embodiment of this invention. It is a reference figure which shows the relationship between the composition ratio determined by the composition ratio determination part in the image processing apparatus concerning the 3rd Embodiment of this invention, and the difference absolute value of an image. It is a block diagram which shows schematic structure of the high frequency band image synthetic | combination part in the image processing apparatus concerning the 4th Embodiment of this invention. It is a block diagram which shows schematic structure of the high frequency band image synthetic | combination part in the image processing apparatus concerning the 5th Embodiment of this invention.

Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the image processing apparatus according to the present embodiment includes an optical system 100, an image sensor 101, an image processing unit 102, a frame memory 103, a motion information acquisition unit 104, an image correction unit 105, and a synthesis processing unit 106. I have.

  The optical system 100 is composed of a lens or the like and forms an image of a subject, and is disposed on the image sensor 101 so as to form an image. The imaging element 101 generates an imaging signal that is electrical image information based on the subject image formed by the optical system 100, and outputs the imaging signal to the image processing unit 102. The image processing unit 102 performs image processing such as color processing and gradation conversion processing on the imaging signal input from the imaging element 101. The frame memory 103 stores an image that has been subjected to predetermined processing by the image processing unit 102.

  FIG. 2 shows an example in which one image is synthesized from four images. The four images are changed from the frame F1 to the frame F4, and the composition is performed three times by performing a basic process of combining one image from two images, thereby finally obtaining one composite image from the four images. First, the composite image 1 is generated by combining the frame F1 and the frame F2. At the time of synthesis, one is defined as a reference image and the other as a target image. Next, a composite image 2 is generated by combining the frame F3 as a reference image and the frame F4 as a target image. Finally, the synthesized image 1 is synthesized as a reference image, and the synthesized image 2 is synthesized as a target image to generate a synthesized image 3 as a final result. Note that the method of combining a plurality of sheets is not limited to this, and for example, a method of combining the combined image 1 and the frame F3 in FIG. 2 and combining the combined result with the frame F4 may be used.

  In addition, instead of combining the basic process of combining two reference images and one target image, a total of four reference images and three target images may be used. The image correction unit 105 and the composition processing unit 106 can be easily expanded. In addition to the method of determining the reference image, in addition to the image picked up first, the image was picked up at an intermediate time by changing the method of making the image picked up later or each time basic processing is performed. It is also possible to use the image as a reference.

  The motion information acquisition unit 104 outputs motion between a plurality of images stored in the frame memory 103 as motion information. That is, the motion information acquisition unit 104 uses any one of the plurality of images stored in the frame memory 103 as a reference image when performing image composition processing, and compares the image with the reference image. The target image is defined as a target image to be subjected to image synthesis processing. Then, the motion of the target image with respect to the reference image is represented by one vector information composed of the horizontal direction moving amount and the vertical direction moving amount, and this vector information is output as the motion information of the target image.

  Note that the motion information is not only one vector information per image, but for example, an image can be divided into a plurality of areas, and vector information in each divided area can be calculated for each pixel. Vector information can also be calculated. Further, the movement amount by rotation and the change amount by enlargement / reduction may be used as the motion information. Furthermore, the motion information is not only calculated by calculation, but a sensor such as a gyro may be provided in the image processing apparatus, and the motion information may be acquired by this sensor.

  The image correction unit 105 corrects the target image stored in the frame memory 103 based on the motion information acquired by the motion information acquisition unit 104. In the present embodiment, the position of the target image is adjusted by shifting the position of the target image based on the vector information output from the motion information acquisition unit 104. In the configuration example in which the motion information includes, for example, information about rotation and enlargement / reduction, the image correction unit 105 performs correction processing corresponding to rotation and enlargement / reduction, and aligns the reference image and the target image.

  The synthesis processing unit 106 generates a synthesized image by synthesizing the target image that has been subjected to the alignment processing by the image correcting unit 105 and the reference image, and outputs the generated synthesized image. As illustrated in FIG. , Frequency resolving units 200 and 201, a high frequency band image synthesizing unit 202, a medium frequency band image synthesizing unit 203, a low frequency band image synthesizing unit 204, and a frequency synthesizing unit 205.

  The frequency resolving units 200 and 201 divide the reference image and the target image into images of predetermined bands (frequency decomposition step). In this embodiment, the high-frequency band image, the medium-frequency band image, and the low-frequency band image It is assumed that the image is divided into three band images. For the decomposition in the frequency decomposition units 200 and 201, a known method may be used, and a filter bank, a wavelet transform, a Laplacian pyramid, or the like can be used. In the present embodiment, frequency decomposition is performed using a Laplacian pyramid.

  FIG. 4 is a block diagram showing the configuration of the frequency resolving units 200 and 201 using the Laplacian pyramid. As shown in FIG. 4, the frequency resolving units 200 and 201 include filter processing / reduction processing units 301 and 304, and an enlargement. Processing units 302 and 305 and subtracters 303 and 306 are provided. Hereinafter, for convenience of description, the reference image or target image input to the frequency resolution units 200 and 201 will be described as the input image 300. The input image 300 is reduced by the filter processing / reduction processing unit 301 after the low-pass filter processing. By this processing, the image size of the input image 300 is halved in both the horizontal direction and the vertical direction. The reduced image is enlarged to the original image size by the enlargement processing unit 302, and then a high frequency band image 307 is created by the subtractor 303. The band image signal of the high-frequency band image 307 corresponds to a high-frequency component blocked by the low-pass filter characteristic used in the filter processing / reduction processing unit 301.

  The output of the filter processing / reduction processing unit 301 is supplied to the next-stage filter processing / reduction processing unit 304 and further reduced. The reduced image is enlarged by the enlargement processing unit 305, and the intermediate frequency band image 308 is generated by the subtractor 306. The output of the filter processing / reduction processing unit 304 is output as a low frequency band image 309. Through the above processing, the input image 300 is decomposed into a high frequency band image 307, a medium frequency band image 308, and a low frequency band image 309. The high frequency band image 307, the medium frequency band image 308, and the low frequency band image 309 that have been decomposed into the respective bands are transmitted to the high frequency band image composition unit 202, the medium frequency band image composition unit 203, and the low frequency band image composition unit 204, respectively. The

  The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 synthesize the reference image and the target image decomposed into the frequency bands by the frequency decomposing units 200 and 201 in the respective bands. To do. That is, the high frequency band image synthesis unit 202 synthesizes the high frequency band image of the reference image and the high frequency band image of the target image, and similarly, the medium frequency band image synthesis unit 203 performs the synthesis of the medium frequency band image of the reference image and the target image. The low frequency band image composition unit 204 synthesizes the low frequency band image of the reference image and the low frequency band image of the target image, and the combined high frequency band image 310 and the combined medium frequency band image 311. And a synthetic low frequency band image 312 is generated.

  Hereinafter, the high frequency band image composition unit 202, the medium frequency band image composition unit 203, and the low frequency band image composition unit 204 will be described. Since the configurations of the high-frequency band image composition unit 202, the medium-frequency band image composition unit 203, and the low-frequency band image composition unit 204 are the same, for convenience of explanation, the high-frequency band image composition unit 202, the medium-frequency band image composition unit 203, and the low-frequency band image composition unit 203 As a representative of the band image synthesis unit 204, only the high frequency band image synthesis unit will be described. As shown in FIG. 5, the high-frequency band image composition unit 202 includes a weighted averaging processing unit 404 and a composition ratio determination unit 410.

  The high-frequency band image synthesis unit 202 receives the high-frequency band image 400 of the reference image band-divided by the frequency decomposition units 200 and 201 and the high-frequency band image 401 of the target image that has been band-divided. The composition ratio determination unit 410 determines a composition ratio between the high frequency band image 400 of the reference image and the high frequency band image 401 of the target image (composition ratio determination step). The composition ratio represents the composition ratio of the high frequency band image 401 of the target image from 0.0 to 1.0 when the high frequency band image 400 of the reference image is 1.0.

  The combination ratio determined by the combination ratio determination unit 410 may be determined in advance as a fixed value in each band, or may be dynamically changed according to the result of the motion information acquisition unit 104. For example, if the captured image has a high ISO sensitivity and includes a lot of noise, the accuracy of alignment is expected to be low, so there is a high possibility that accurate alignment is not performed in the high frequency band. In such a case, the composition ratio is reduced in the high frequency band to suppress artifacts due to alignment errors, and the noise reduction effect is maintained by increasing the synthesis ratio in the low frequency band. Is possible.

  Alternatively, when the motion vector obtained by the motion information acquisition unit 104 is large, the alignment error is likely to be large, so the synthesis ratio is reduced in the high frequency band, and when the motion vector is small, the alignment error is Since it can be expected to be small, it is possible to increase the synthesis ratio even in the high frequency band. Furthermore, the correlation between the reference image and the target image is calculated for each pixel or for each predetermined area, and it is determined that the alignment accuracy is low for pixels or areas having a small correlation, and the composition ratio in the high frequency band is reduced. It is also possible to do.

  The weighted average processing unit 404 performs weighted average processing of the pixels of the high-frequency band image 400 of the reference image and the pixels of the high-frequency band image 401 of the target image according to the combination ratio output from the combination ratio determination unit 410, and performs the combined high-frequency processing. A band image 310 is generated. Also in the medium frequency band image composition unit 203 and the low frequency band image composition unit 204, a composite medium frequency band image 311 and a composite low frequency band image 312 are generated by the above procedure (band image composition step). The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 respectively generate the synthesized high frequency band image 310, the synthesized middle frequency band image 311 and the synthesized low frequency band image 312. To 205.

  The frequency synthesizing unit 205 further synthesizes the synthesized images of the respective bands transmitted from the high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 (image synthesizing step). For the frequency synthesis in the frequency synthesis unit 205, a known method may be used similarly to the frequency resolution units 200 and 201, and a filter bank, a wavelet transform, a Laplacian pyramid, or the like can be used. In the present embodiment, frequency synthesis is performed using a Laplacian pyramid. FIG. 6 is a block diagram showing a schematic configuration of the frequency synthesis unit 205 using the Laplacian pyramid, and the frequency synthesis unit 205 includes enlargement processing units 314 and 315 and adders 313 and 315.

  The high frequency band image 310, the medium frequency band image 311, and the low frequency band image 312 are input to the frequency synthesis unit 205 from the high frequency band image synthesis unit 202, the middle frequency band image synthesis unit 203, and the low frequency band image synthesis unit 204. . The low frequency band image 312 is enlarged to the same size as the middle frequency band image 311 by the enlargement processing unit 316 and then added by the adder 315. Next, the image is enlarged to the same size as the high frequency band image 310 by the enlargement processing unit 314 and then added by the adder 313 to become an output image 317. As a result, the output image 317 is synthesized as an image including all the frequency components of the high-frequency band image 310, the medium-frequency band image 311 and the low-frequency band image 312.

  In this way, by frequency-decomposing the image and controlling the synthesis process in each band, it is possible to perform a suitable synthesis in each band, reducing the occurrence of artifacts due to synthesis failure, and the effect of noise reduction by synthesis Can be obtained.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. The image processing apparatus according to this embodiment is different from the first embodiment in that the high frequency band image composition unit 202, the medium frequency band image composition unit 203, and the low frequency band image composition unit 204 further include a correlation calculation unit 402. Is a point. Hereinafter, regarding the image processing apparatus according to the present embodiment, description of points that are the same as those of the first embodiment will be omitted, and different points will be described.

  The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 synthesize the reference image and the target image decomposed into the frequency bands by the frequency decomposing units 200 and 201 in the respective bands. Since the configuration is common, the high frequency band image composition unit 202 will be described for convenience of explanation. FIG. 7 is a block diagram illustrating a schematic configuration of the high frequency band image synthesis unit 202. As shown in FIG. 7, the high frequency band image synthesis unit 202 includes a correlation calculation unit 402, a synthesis ratio determination unit 403, and a weighted averaging processing unit 404.

  The high-frequency band image synthesis unit 202 receives the high-frequency band image 400 of the reference image that has been band-divided by the frequency decomposition units 200 and 201 and the high-frequency band image 401 of the target image that has been band-divided. The correlation calculation unit 402 calculates a difference absolute value for each pixel as a correlation value between the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image. In general, the absolute difference value is small when the alignment between the reference image and the target image is successful, and the absolute difference value is large when the alignment is unsuccessful. Therefore, this result is used to control the composition ratio in the composition ratio determination unit 403 to suppress artifacts such as blurring and double images due to alignment failure.

  In this embodiment, the absolute value of the difference between pixels is used as the correlation value. In addition to this, in order to calculate a more stable correlation value, the difference between the blocks is determined for the block including the peripheral pixels of the target pixel. The absolute value sum (SAD) may be used as the correlation value. Also, in order to reduce the amount of calculation, a correlation value may not be calculated for each pixel, but one correlation value may be calculated for a region composed of a plurality of pixels.

  The composition ratio determination unit 403 determines the composition ratio of the high frequency band image 400 of the reference image and the high frequency band image 401 of the target image based on the absolute difference value calculated by the correlation calculation unit 402. The composition ratio represents the composition ratio of the high frequency band image 401 of the target image from 0.0 to 1.0 when the high frequency band image 400 of the reference image is 1.0. The composition ratio is controlled by the magnitude of the absolute difference value of the pixels. That is, for example, as shown in FIG. 8, when the difference absolute value is small, it is highly likely that the alignment has succeeded, so the composition ratio is increased, and when the difference absolute value is large, the alignment is performed. Since the possibility of failure is high, the synthesis ratio is reduced to suppress artifacts. In the example shown in FIG. 8, when the absolute difference value is smaller than the threshold value 1, the composition ratio is set to 1.0. When the absolute difference value is larger than the threshold value 2, the composition ratio is set to 0.0. The composition ratio is shifted linearly to.

  The weighted averaging processing unit 404 performs weighted average processing of the high frequency band image 400 and the high frequency band image 401 in accordance with the composition ratio output from the composition ratio determination unit 403 to generate a composite high frequency band image 310. The intermediate frequency band image composition unit 203 and the low frequency band image composition unit 204 also generate the composite middle frequency band image 311 and the composite low frequency band image 312 according to the above procedure.

  In this way, by frequency-decomposing the image and controlling the synthesis process in each band, it is possible to prevent a process that weakens the synthesis effect in all bands due to a slight misalignment, and to generate artifacts. It is possible to apply processing that weakens the effect of synthesis only in the minimum band necessary for suppression. As a result, it is possible to obtain an effect of noise reduction by synthesis while suppressing generation of artifacts. In addition, by changing the composition ratio in each band based on the correlation value for each image in each band, the composition effect on the image can be compared to changing the composition ratio without dividing the band. It is possible to obtain an effect of further suppressing the fluctuation of the image quality and improve the image quality of the composite image.

  In this embodiment, the composition ratio is continuously changed between 0.0 and 1.0. However, this is a binary value of 0.0 and 1.0, and the weighted averaging processing unit 404 is used. It is also possible to reduce the amount of calculation by replacing with a simple averaging process.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIGS. The image processing apparatus of this embodiment is different from that of the second embodiment in that the high frequency band image composition unit 202, the medium frequency band image composition unit 203, and the low frequency band image composition unit 204 further include a noise level estimation unit 502. In other words, the synthesis ratio in the synthesis ratio determination unit 503 is determined based on the correlation between the noise level estimated by the noise level estimation unit 502 and the image. Hereinafter, with respect to the image processing apparatus of the present embodiment, description of points that are common to the first embodiment and the second embodiment will be omitted, and different points will be described.

  The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 synthesize the reference image and the target image decomposed into the frequency bands by the frequency decomposing units 200 and 201 in the respective bands. Since the configuration is common, the high frequency band image composition unit 202 will be described for convenience of explanation. FIG. 9 is a block diagram illustrating a schematic configuration of the high frequency band image synthesis unit 202. As shown in FIG. 9, the high frequency band image synthesis unit 202 includes a correlation calculation unit 402, a noise level estimation unit 502, a synthesis ratio determination unit 503, and a weighted averaging processing unit 404.

  In addition to the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image subjected to the band division, the high-frequency band image synthesis unit 202 includes the reference image and the target image, respectively. The noise level estimation images 500 and 501 to be created are input. Then, the correlation calculation unit 402 calculates a correlation value between the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image. Here, as the correlation value, as in the second embodiment described above, a difference absolute value or a sum of absolute differences (SAD) between blocks related to a block composed of peripheral pixels of the target pixel can be used. Further, the noise level estimation unit 502 calculates an estimated value of the amount of noise included in the processing target pixel based on the noise level estimation images 500 and 501.

  In general, it is known that there is a certain relationship between the amount of noise contained in a pixel output from an image sensor and the pixel value, and the amount of noise can be estimated from the pixel value. FIG. 10 shows a typical relationship between the pixel value output from the image sensor and the amount of noise. In FIG. 10, the horizontal axis represents the pixel value of the pixel output from the image sensor, and the vertical axis represents the amount of noise (such as the standard deviation of noise) contained in the pixel. Typically, the amount of noise included tends to increase as the pixel value output from the image sensor increases. Pixels output from the image sensor are stored in the frame memory 103 after image processing such as gradation conversion is performed by the image processing unit 102. In the gradation conversion process, a gradation conversion process with a gamma characteristic is typically performed such that the dark part is lifted and the bright part is crushed. FIG. 11 shows a typical relationship between the pixel value after the gradation conversion processing and the noise amount. In the region where the pixel value is small, the noise is amplified by the gradation conversion process, and in the region where the pixel value is large, the noise is crushed. As a result, the relationship shown in FIG. When the image is frequency-divided as in the present invention, the noise included in the image is divided into each band, and the relationship between the pixel value and the amount of noise in each band is also as shown in FIG.

  As described above, the noise amount is related to the pixel value, and in order to estimate the noise amount in each band, a pixel value including the DC component of the processing target pixel is required. The image of each frequency band (ie, the high-frequency band image 307, the middle-frequency band image 308, and the low-frequency band image 309 in FIG. 4) that is frequency-resolved from the reference image or the target image does not necessarily include a direct current component, and thus the amount of noise. Not suitable for use in estimating Therefore, the noise level estimation images 500 and 501 use an image including a direct current component among images generated from the reference image and the target image.

  In the present embodiment, it is assumed that the noise level estimation images 500 and 501 in the high frequency band use output images of the enlargement processing unit 302 in the frequency decomposition unit using the Laplacian pyramid shown in FIG. Although the input image 300 can be used because it includes a direct current component, the image after the low-pass filter processing has an advantage that stable noise level estimation can be expected. Similarly, the output images of the enlargement processing unit 305 are used as the noise level estimation images 500 and 501 in the medium frequency band. In the low frequency band, since the band image itself includes a direct current component, the low frequency band image 309 may be used as the noise level estimation images 500 and 501.

  Next, the configuration of the noise level estimation unit 502 will be described. FIG. 12 is a block diagram illustrating a schematic configuration of the noise level estimation unit 502. The noise level estimation unit 502 estimates noise intensity (noise level) included in the processing target pixel, and includes noise level calculation units 600 and 601 and a maximum value calculation unit 602 as shown in FIG. Yes. The noise level calculation units 600 and 601 prepare the relationship between the pixel value and the noise amount as shown in FIG. 11 in advance by a method such as polygonal line approximation or tabulation, and the pixels (reference image) of the high frequency band image 400 are prepared. Or the noise level of each pixel of the high frequency band image 401 (target image) is calculated.

  For pixels in which the alignment of the reference image and the target image is successful, there is no significant difference between the pixel value of the reference image and the pixel value of the target image, and no significant difference occurs in the calculated noise level. However, there is a high possibility that a difference will occur in a pixel for which alignment has failed, and this is set as the noise level by selecting the maximum value in the maximum value calculation unit 602 as the noise level of the pixel. . In the present embodiment, the maximum value is calculated by the maximum value calculation unit 602 and set as the noise level. In addition to this, a weighted average value of the noise level of the reference image and the noise level of the target image is used. It is also possible to estimate the noise level by weighting the pixels of the image. If it is not desirable to operate a plurality of noise level calculation units 600 and 601 from the viewpoint of the amount of computation, only the noise level calculation unit 600 may be operated and this output may be used as the noise level.

  Based on the noise level calculated by the noise level estimation unit 502 and the absolute difference value calculated by the correlation calculation unit 402, the synthesis ratio determination unit 503 performs the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image. Determine the composition ratio. FIG. 13 is a diagram for explaining a method for determining the composition ratio in the composition ratio determination unit 503. As for the composition ratio, the composition ratio of the target image when the reference image is 1.0 is represented by 0.0 to 1.0. First, the composition ratio is controlled by the magnitude of the absolute difference value of the pixels. When the difference absolute value is small, it is highly likely that the alignment has succeeded, so the composition ratio is increased. When the difference absolute value is large, there is a high possibility that the alignment has failed, so the synthesis ratio is reduced to suppress artifacts. In the example of FIG. 13, when the difference absolute value is smaller than the threshold value 1, the composition ratio is set to 1.0, and when the difference absolute value is larger than the threshold value 2, the composition ratio is set to 0.0. Is a linear transition of the composition ratio.

  In addition, by controlling the threshold value 1 and the threshold value 2 according to the noise level, the synthesis ratio is controlled according to the noise level. In a pixel with a high noise level, the necessity for noise reduction by synthesis is high, so that threshold 1 and threshold 2 are increased in order to increase the synthesis ratio. Specifically, a value obtained by multiplying the noise level by a predetermined constant may be added to threshold 1 and threshold 2, respectively. In a pixel with a low noise level, since the necessity for noise reduction by synthesis is low, the threshold value 1 and the threshold value 2 are reduced in order to reduce the synthesis ratio. Specifically, a value obtained by multiplying the noise level by a predetermined constant may be subtracted from the threshold value 1 and the threshold value 2, respectively. The composition ratio determination unit 503 may prepare such a relationship by a method such as tabulation, or may calculate by a mathematical formula calculation.

  Similar to the second embodiment, the weighted averaging processing unit 404 weights the pixels of the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image according to the combination ratio output by the combination ratio determination unit 503. An average process is performed to generate a composite high frequency band image 310. The intermediate frequency band image composition unit 203 and the low frequency band image composition unit 204 also generate the composite middle frequency band image 311 and the composite low frequency band image 312 according to the above procedure.

  As described above, by frequency-decomposing the image and controlling the synthesis process in each band as in this embodiment, it is possible to prevent a process that weakens the synthesis effect in all bands due to a slight positional deviation. Thus, it is possible to apply processing that weakens the effect of synthesis only in the minimum band necessary for suppressing the occurrence of artifacts. As a result, it is possible to obtain an effect of noise reduction by synthesis while suppressing generation of artifacts. Also, by changing the composition ratio in each band, compared to changing the composition ratio without dividing the band, it is possible to obtain an effect of suppressing fluctuations in the composition effect on the image. It is possible to improve the image quality. Furthermore, the noise level estimation unit estimates the noise level of the pixel to be processed, and is used to control the composition ratio in conjunction with the correlation between images, thereby reducing artifacts while suppressing artifacts due to alignment processing failure. Artifacts such as blurring and generation of double images due to excessive synthesis processing on pixels or regions can be accurately and reliably suppressed, and a satisfactory synthesis result can be obtained.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described with reference to FIG. The difference between the image processing apparatus of the present embodiment and the second embodiment is that the high-frequency band image synthesis unit 202, the medium-frequency band image synthesis unit 203, and the low-frequency band image synthesis unit 204 are the band image motion information acquisition unit 700 and A band image correction unit 702 is further provided, and the high frequency band image synthesis unit 202, the medium frequency band image synthesis unit 203, and the low frequency band image synthesis unit 204 perform alignment processing between the band images. Hereinafter, regarding the image processing apparatus according to the present embodiment, description of points that are common to the first to third embodiments will be omitted, and different points will be described.

  The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 synthesize the reference image and the target image decomposed into the frequency bands by the frequency decomposing units 200 and 201 in the respective bands. Since the configuration is common, the high frequency band image composition unit 202 will be described for convenience of explanation. FIG. 14 is a block diagram illustrating a schematic configuration of the high-frequency band image synthesis unit 202. As shown in FIG. 14, the high-frequency band image synthesis unit 202 includes a correlation calculation unit 402, a synthesis ratio determination unit 403, a weighted average processing unit 404, a band image motion information acquisition unit 700, and a band image correction unit 702. .

  The high-frequency band image synthesis unit 202 receives the high-frequency band image 400 of the reference image that has been band-divided by the frequency decomposition units 200 and 201 and the high-frequency band image 401 of the target image that has been band-divided. The band image motion information acquisition unit 700 calculates a shift amount between the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image subjected to the band division, and the band image motion information 701 is input to the band image correction unit 702. Output. The band image correction unit 702 corrects the high frequency band image 401 of the target image according to the motion information 701. Since the correlation calculation unit 402, the composition ratio determination unit 403, and the weighted averaging processing unit 404 are the same as those in the second embodiment described above, description thereof is omitted.

  Normally, if the target image is aligned with the reference image by the motion information acquisition unit 104 and the image correction unit 105, the amount of deviation between the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image is also determined. Since it is small, it is not necessary to newly acquire the band image motion information 701. However, when the performance of the alignment processing by the motion information acquisition unit 104 and the image correction unit 105 is low, or when the alignment processing itself is not performed, the amount of deviation cannot be reduced in all locations of the image. . In particular, a large misalignment occurs in places where the subject has moved. In order to deal with such a local positional shift, it is desirable to perform a local alignment process in units of pixels or regions in addition to the alignment process of the entire image.

  However, there is usually a problem that a large amount of calculation is required for the alignment processing. According to the present embodiment, it is possible to obtain an effect even with a relatively small amount of computation by performing local alignment processing in each band. That is, the motion information calculated by the band image motion information acquisition unit 700 is one motion vector (horizontal movement amount and vertical movement amount), and the maximum value of the motion vector (search range for determining the calculation amount) is small. Value. If the maximum value of the motion vector is small, there may be a case where alignment is impossible. However, in this embodiment, since alignment is performed in each band, for example, it is impossible to perform alignment in a high frequency band. However, since the search range is substantially increased in the low frequency band where the image is reduced, positioning may be possible. In such a case, the synthesis accuracy in the low frequency band is increased, so The effect of improving accuracy can be obtained.

  The effect of improving the accuracy described above can be obtained by dividing the image into bands and performing alignment processing in each band. However, as in the present embodiment, the composition processing unit divides the band and synthesizes in each band. In this case, by performing alignment in each band at the same time, it is possible to share the frequency decomposition process and the frequency synthesis process in the synthesis process and the alignment process, and to obtain the effect of reducing the amount of calculation.

  As described above, by frequency-decomposing the image and controlling the synthesis process in each band as in this embodiment, it is possible to prevent a process that weakens the synthesis effect in all bands due to a slight positional deviation. Thus, it is possible to apply processing that weakens the effect of synthesis only in the minimum band necessary for suppressing the occurrence of artifacts. As a result, it is possible to obtain an effect of noise reduction by synthesis while suppressing generation of artifacts. Also, by changing the composition ratio in each band, compared to changing the composition ratio without dividing the band, it is possible to obtain an effect of suppressing fluctuations in the composition effect on the image. It is possible to improve the image quality. Furthermore, by performing alignment processing in each decomposed band, it is possible to perform alignment processing in each band without newly performing frequency decomposition processing and frequency synthesis processing. Registration processing is possible.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described with reference to FIG. The difference between the image processing apparatus of this embodiment and the fourth embodiment is that the high-frequency band image composition unit 202, the medium-frequency band image composition unit 203, and the low-frequency band image composition unit 204 perform alignment processing between band images. When performing, the alignment result in the low frequency band is used as the initial search position in the alignment process on the high frequency band side. That is, in the present embodiment, when the band image synthesizing unit in the above-described fourth embodiment performs the alignment process between the band images, the alignment result in the low frequency band is used in the alignment process on the high frequency band side. Used as the initial search position. By performing a registration process by setting a relatively narrow motion information search range around the initial search position, a registration process having a wide substantial search range is performed with a low amount of computation even in a high frequency band. Hereinafter, regarding the image processing apparatus according to the present embodiment, description of points that are common to the first to fourth embodiments will be omitted, and different points will be described.

  The high frequency band image synthesizing unit 202, the medium frequency band image synthesizing unit 203, and the low frequency band image synthesizing unit 204 synthesize the reference image and the target image decomposed into the frequency bands by the frequency decomposing units 200 and 201 in the respective bands. Since the configuration is common, the high frequency band image composition unit 202 will be described for convenience of explanation. FIG. 15 is a block diagram illustrating a schematic configuration of the high frequency band image synthesis unit 202. As shown in FIG. 15, the high-frequency band image synthesis unit 202 includes a correlation calculation unit 402, a synthesis ratio determination unit 403, a weighted averaging processing unit 404, a band image motion information acquisition unit 700, and a band image correction unit 702. .

  The high-frequency band image synthesis unit 202 receives the high-frequency band image 400 of the reference image that has been band-divided by the frequency decomposition units 200 and 201 and the high-frequency band image 401 of the target image that has been band-divided. The band image motion information acquisition unit 700 calculates the amount of deviation between the high-frequency band image 400 of the reference image and the high-frequency band image 401 of the target image subjected to band division. At this time, the band image motion information acquisition unit 700 The band image motion information initial value 703 is input from the low frequency band, and the position search is performed using the deviation amount around the initial value as a candidate vector. However, the band image motion information initial value 703 in the lowest band is a zero vector. The band image motion information acquisition unit 700 outputs band image motion information 701 as a search result to the band image correction unit 702.

  The band image correction unit 702 corrects the high frequency band image 401 of the target image according to the motion information 701. Since the correlation calculation unit 402, the combination ratio determination unit 403, and the weighted averaging processing unit 404 are the same as those in the second and third embodiments described above, description thereof is omitted.

  As in this embodiment, alignment processing is performed in each divided band, and alignment processing in the high frequency band is performed using the result of the alignment processing in the low frequency band, that is, the initial search By performing the alignment process by setting a relatively narrow motion information search range around the position, it is possible to perform an alignment process with a large search range with a low amount of computation even in a high frequency band.

  The above-described effect can be obtained by dividing the image into bands and performing alignment processing in each band. However, when the band is divided in the combination processing unit and combined in each band as in the present embodiment, By performing alignment in each band at the same time, it is possible to share frequency resolution processing and frequency synthesis processing between synthesis processing and alignment processing, and to obtain an effect of reducing the amount of calculation.

  As described above, by frequency-decomposing the image and controlling the synthesis process in each band as in this embodiment, it is possible to prevent a process that weakens the synthesis effect in all bands due to a slight positional deviation. Thus, it is possible to apply processing that weakens the effect of synthesis only in the minimum band necessary for suppressing the occurrence of artifacts. As a result, it is possible to obtain an effect of noise reduction by synthesis while suppressing generation of artifacts. Also, by changing the composition ratio in each band, compared to changing the composition ratio without dividing the band, it is possible to obtain an effect of suppressing fluctuations in the composition effect on the image. It is possible to improve the image quality. Furthermore, by performing alignment processing in each decomposed band, and using the alignment result in the low frequency band for alignment in the high frequency band, each without performing new frequency decomposition processing and frequency synthesis processing, It is possible to perform alignment processing in the band, and it is possible to perform alignment processing with a low calculation amount and a wide search range and high accuracy.

In the above-described embodiment, processing based on hardware is assumed, but it is not necessary to be limited to such a configuration. For example, a configuration in which processing is performed separately by software is also possible. In this case, the image processing apparatus can be realized using a general purpose or dedicated computer. Such a computer includes a main storage device such as a CPU and a RAM, and a computer-readable recording medium on which a program for realizing all or part of the above processing is recorded. Then, the CPU reads out the program recorded in the storage medium and executes information processing / calculation processing, thereby realizing the same processing as the above-described failure diagnosis apparatus.
Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  In the above-described embodiment, the image processing apparatus includes the optical system and the image sensor, but it is not always necessary to include these. In this case, acquisition of an image by the image processing device means input of the image to the image processing device.

DESCRIPTION OF SYMBOLS 100 Optical system 101 Image pick-up element 102 Image processing part 103 Frame memory 104 Motion information acquisition part 105 Image correction part 106 Compositing process part 200,201 Frequency decomposition part (frequency decomposition means)
202 High-frequency band image composition unit (band image composition means)
203 Medium frequency band image synthesizing unit (band image synthesizing means)
204 Low-frequency band image composition unit (band image composition means)
205 Frequency synthesis unit (image synthesis means)
300 Input Image 301, 304 Filter Processing / Reduction Processing Unit 302, 305, 314, 316 Enlargement Processing Unit 303, 306 Subtractor 307 High Frequency Band Image 308 Medium Frequency Band Image 309 Low Frequency Band Image 310 Composite High Frequency Band Image 311 Composite Medium Frequency Band image 312 Composite low-frequency band image 313, 315 Adder 317 Output image 400 High-frequency band image 401 of reference image High-frequency band image 402 of target image Correlation calculation units 403, 410, 503 Composite ratio determination unit 404 Weighted averaging processing unit 500 Noise level estimation image 501 of reference image Noise level estimation image 502 of target image Noise level estimation unit 600, 601 Noise level calculation unit 602 Maximum value calculation unit 700 Band image motion information acquisition unit 701 Band image motion information 702 Band image correction 703 Band image motion Information initial value

Claims (10)

An image processing apparatus that acquires a plurality of images and generates a composite image by combining the plurality of images,
Frequency resolving means for decomposing each image into a plurality of frequency bands and generating a band image for each frequency band;
A composition for determining, for each frequency band, a composition ratio of a band image of the target image to a band image of the reference image when any one of the plurality of images is a reference image and another image is a target image. A ratio determining means;
A band image synthesizing unit that synthesizes the band image of the target image with respect to the band image of the reference image for each frequency band based on the synthesis ratio, and generates a synthesized band image for each frequency band;
Image combining means for combining the plurality of combined band images to generate a combined image;
An image processing apparatus comprising: A motion detecting means for detecting a motion amount between the plurality of images;
The image processing apparatus according to claim 1, wherein correction processing is performed on the plurality of images based on the amount of motion prior to generation of a composite image by the image composition unit. A correlation calculating means for calculating a correlation value indicating a correlation of the band image of the target image with respect to the band image of the reference image in units of a predetermined region composed of a pixel unit or a plurality of pixels for each frequency band;
The composition ratio determining means compares a predetermined threshold value with the correlation value, and sets the composition ratio to be smaller as the correlation value becomes smaller than the threshold value. Item 3. The image processing apparatus according to Item 2. Noise level estimation means for estimating a noise level in a predetermined region unit consisting of a pixel unit or a plurality of pixels for each frequency band for a band image of at least one image among the plurality of images,
The image processing apparatus according to claim 3, wherein the threshold is determined based on the noise level. Band image motion detection means for detecting the amount of motion between the band images for each frequency band,
The image processing apparatus according to claim 1, wherein correction processing is performed on the band image based on a motion amount of the band image prior to generation of a composite image by the image composition unit. A motion detection means for detecting a motion amount between the plurality of images, and a band image motion detection means for detecting a motion amount between the band images for each frequency band,
Prior to the generation of the band image by the frequency resolving means, the plurality of images are corrected based on the amount of motion, and the amount of motion of the band image prior to the generation of the composite image by the image combining means The image processing apparatus according to claim 1, wherein a correction process is performed on the band image based on the image.   The image processing apparatus according to claim 5, wherein the band image motion detection unit determines a motion information search range in the band image using motion information in a low frequency band. An image processing method for acquiring a plurality of images and combining the plurality of images to generate a combined image,
A frequency decomposition step of decomposing each of the images into a plurality of frequency bands, and generating a band image for each frequency band;
A composition for determining, for each frequency band, a composition ratio of a band image of the target image to a band image of the reference image when any one of the plurality of images is a reference image and another image is a target image. A ratio determination step;
A band image synthesis step of synthesizing the band image of the target image with respect to the band image of the reference image for each frequency band based on the synthesis ratio, and generating a synthesized band image for each frequency band;
An image synthesis step of generating a composite image by combining the plurality of composite band images;
An image processing method comprising: An image processing program for acquiring a plurality of images and generating a composite image by combining the plurality of images,
A frequency decomposition step of decomposing each of the images into a plurality of frequency bands, and generating a band image for each frequency band;
A composition for determining, for each frequency band, a composition ratio of a band image of the target image to a band image of the reference image when any one of the plurality of images is a reference image and another image is a target image. A ratio determination step;
A band image synthesis step of synthesizing the band image of the target image with respect to the band image of the reference image for each frequency band based on the synthesis ratio, and generating a synthesized band image for each frequency band;
An image synthesis step of generating a composite image by combining the plurality of composite band images;
An image processing program for causing a computer to execute image processing comprising:   A program storage medium in which the program according to claim 9 is stored.

Images