JP2012114735A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of accurately and speedily removing noise even from pixels at a high noise level.SOLUTION: An image processing device is provided with: a first coefficient determination part which determines the value of a first coefficient on the basis of the difference between a target image from which noise is to be removed and a reference image which is to be referenced in the noise removal; a second coefficient determination part which determines the value of a second coefficient on the basis of luminance information on the target image and/or the reference image; and a noise removal part which adds the target image and the reference image with weigh using the first coefficient and the second coefficient, and removes the noise superimposed on the target image.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques have been developed for noise removal processing for moving images and the like. For example, a moving part of a subject is detected based on a pixel value difference between frames of a moving image, a filter coefficient for noise removal is determined based on a detection result of the moving part of the subject, and motion and noise are detected from the frame. There is a technique for accurately determining and removing noise (see Patent Document 1).

Japanese Patent Laid-Open No. 7-79956

  However, in the prior art, the filter coefficient is determined based only on the difference in pixel values between frames, so that noise is accurately applied to pixels having pixel values with high noise levels, such as dark portions of a subject. It may not be removed.

  SUMMARY OF THE INVENTION In view of the above-described problems of the prior art, an object of the present invention is to provide a technique that can perform noise removal with high accuracy and high speed even for a pixel having a high noise level.

  In order to solve the above-described problem, the image processing apparatus according to the present embodiment determines a first coefficient value based on a difference between a target image to be denoised and a reference image that is a reference for noise removal processing. A determination unit; a second coefficient determination unit that determines a value of the second coefficient based on luminance information of the target image and / or the reference image; and a target image and a reference image using the values of the first coefficient and the second coefficient And a noise removing unit that removes noise superimposed on the target image.

  In addition, the luminance information may be an average value of luminance values of the target image and the reference image, or a value obtained by weighting and adding the luminance values of the target image and the reference image using only the value of the first coefficient.

  Further, the first coefficient determination unit may determine the value of the first coefficient for each color component included in the target image.

  In addition, the first coefficient determination unit may determine the value of the first coefficient for each luminance component and color difference component of the target image.

  The image processing apparatus of the present invention includes a first reduced image obtained by reducing a target image to be subjected to noise removal, and a second reduced image obtained by reducing a reference image used as a reference for noise removal processing at the same reduction ratio as the first reduced image. A reduced image generation unit that generates a set at at least one reduction ratio, a first coefficient determination unit that determines a value of a first coefficient based on a difference between the first reduced image and the second reduced image, and a first reduced image And / or a second coefficient determination unit that determines the value of the second coefficient based on the luminance level of the second reduced image, and the first reduced image and the second reduced image using the first coefficient and the second coefficient value, And a noise removing unit that extracts noise in the frequency band corresponding to the reduction ratio and removes noise in the frequency band from the pixel value of the target image.

  In addition, when a plurality of first reduced image and second reduced image sets are generated at a plurality of reduction ratios, the noise removal unit may sequentially extract frequency band noise from a set corresponding to the minimum reduction ratio. Good.

  The first coefficient determination unit determines a value of the third coefficient based on a difference between the reference image and the target image from which noise in the frequency band is removed, and the second coefficient determination unit includes the reference image and / or the frequency. The value of the fourth coefficient is determined based on the luminance information of the target image from which the band noise has been removed, and the noise removing unit removes the noise of the reference image and the frequency band using the third coefficient and the value of the fourth coefficient. The high frequency band noise superimposed on the target image may be removed by weighted addition with the target image.

  Moreover, you may provide the compression process part which performs the image compression with respect to the target image from which the noise of the frequency band was removed, and removes the noise of the high frequency band superimposed on the target image.

  In addition, the target image is one of continuously captured images, and the reference image is a continuously captured image that is captured before the target image and noise is removed. Good.

  An imaging apparatus of the present invention includes an imaging unit that captures a subject image and generates an image, and the image processing apparatus of the present invention.

  An image processing program according to the present invention is a first step of determining a value of a first coefficient based on an input procedure for reading a target image to be subjected to noise removal and a reference image to be referred to for noise removal processing, and a difference between the target image and the reference image A coefficient determination procedure, a second coefficient determination procedure for determining the value of the second coefficient based on the luminance information of the target image and / or the reference image, and the target image and the reference image using the values of the first coefficient and the second coefficient The computer executes a noise removal procedure for performing weighted addition and removing noise superimposed on the target image.

  The image processing program of the present invention includes an input procedure for reading a target image to be denoised and a reference image to be used as a reference for the denoising process, a first reduced image obtained by reducing the target image, and a reference image that is the same reduced as the first reduced image. A reduced image generation procedure for generating a pair with a second reduced image reduced at a rate at at least one reduction rate, and a first coefficient value determined based on a difference between the first reduced image and the second reduced image Coefficient determination procedure, second coefficient determination procedure for determining the value of the second coefficient based on the luminance information of the first reduced image and / or the second reduced image, the first reduction using the values of the first coefficient and the second coefficient A computer performs a noise removal procedure for weighting and adding the image and the second reduced image, extracting noise in the frequency band according to the reduction ratio, and removing the noise in the frequency band from the target image.

  According to the present invention, noise removal can be performed with high accuracy and at high speed even for a pixel having a high noise level.

The block diagram which shows an example of a structure of the digital camera 1 which concerns on one Embodiment. The figure which shows an example of the function f and the function G which determine the coefficient (alpha) and the coefficient (beta) 7 is a flowchart illustrating an example of noise removal processing by the digital camera 1 according to an embodiment. The block diagram which shows an example of a structure of the digital camera 1 'which concerns on other embodiment. The flowchart which shows an example of the noise removal process by the digital camera 1 'which concerns on other embodiment. The figure which shows an example of the flow of the image data in the noise removal process shown in FIG. The flowchart which shows an example of the noise removal process by the digital camera 1 'which concerns on the modification of other embodiment. The figure which shows an example of the flow of the image data in the noise removal process shown in FIG.

<< One Embodiment >>
FIG. 1 is a block diagram showing an example of the configuration of a digital camera 1 according to an embodiment of the present invention.

  The digital camera 1 of this embodiment includes an imaging optical system 11, an imaging element 12, a DFE 13, a CPU 14, a memory 15, an operation unit 16, a monitor 17, and a media interface (media I / F) 18. Here, the DFE 13, the memory 15, the operation unit 16, the monitor 17, and the media I / F 18 are connected to the CPU 14, respectively.

  The image sensor 12 is a device that captures a subject image formed by a light beam that has passed through the imaging optical system 11. The output of the image sensor 12 is input to the DFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scan type solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image at the time of imaging.

  The DFE 13 is a digital front-end circuit that performs signal processing such as A / D conversion of image signals input from the image sensor 12 and correction of defective pixels. In this embodiment, the DFE 13 constitutes an image pickup unit together with the image pickup element 12 and outputs an image signal input from the image pickup element 12 to the CPU 14 as image data.

  The CPU 14 is a processor that comprehensively controls each unit of the digital camera 1. For example, the CPU 14 performs autofocus (AF) control by known contrast detection, known automatic exposure (AE) calculation, and the like based on the output of the image sensor 12. Further, the CPU 14 performs digital processing such as interpolation processing, white balance processing, gradation conversion processing, contour enhancement processing, and color conversion processing on the image data from the DEF 13. In the present embodiment, it is assumed that each pixel of the image data has pixel values of all color components of RGB by this digital processing.

  Furthermore, the CPU 14 of this embodiment operates as a first coefficient determination unit 20, a second coefficient determination unit 21, a noise removal unit 22, and a compression processing unit 23 by executing the image processing program.

  The noise removing unit 22 performs a synthesis process that weights and adds a target image to be removed with a reference image that is a reference for the noise removing process, using a deformed recursive filter represented by the following equation (1). Remove noise from the image.

Z _k = α · βX _k + (1−α · β) Z _k−1 (1)
Here, X _k is a pixel value of any of the RGB color components in the pixel i of the target image, and Z _k−1 is a pixel value of the same color component in the pixel i of the reference image. Z _k is the pixel value of the target image from which noise has been removed from the pixel value X _k by Equation (1). k indicates the order (or time series) in which each image was captured, and in this embodiment, the k-1 reference image is an image that has been captured one noise prior to the capturing of the k target image and from which noise has been removed. And

The coefficient α in Expression (1) is a coefficient that determines the weighting of each pixel value in the synthesis process of the pixel value X _k and the pixel value Z _k−1 . In the present embodiment, the first coefficient determination unit 20 determines the value of the coefficient α from the function f stored in the memory 15 as shown in FIG. 2A and the difference between the pixel values (X _k −Z _k−1). ) And to decide. That is, the coefficient α is determined to be a different value for each color component.

The function f shown in FIG. 2A is a function in which the pixel value difference (X _k −Z _k−1 ) increases monotonically between 0 and the threshold Th1, and becomes 1 when the threshold Th1 is equal to or greater. However, in FIG. 2A, the function f increases monotonically linearly from 0 to the threshold Th1, but may be monotonously increased like a quadratic function or an exponential function. Further, for example, when the ISO sensitivity is 3200 and the gradation of the image is 0 to 255, the threshold Th1 is set as Th1 = 2 to 4 and the like. However, the threshold Th1 is preferably determined according to the subject to be imaged, ISO sensitivity, and the like. Furthermore, the threshold value Th1 of the present embodiment is set to the same value for the RGB color components, but may be set to a different value for each color component.

On the other hand, the coefficient β is a coefficient for adjusting the effect of weighting by the coefficient α in the equation (1) according to the luminance value (luminance information) at the pixel position of the pixel i. In the present embodiment, the second coefficient determination unit 21 determines the value of the coefficient β based on the function G stored in the memory 15 as shown in FIG. 2 (2) and the luminance value YB _k at the pixel i. . That is, in order to determine the coefficient β, the second coefficient determination unit 21 calculates the luminance value YX _{k of the} target image and the luminance value YZ _k−1 of the reference image from the RGB pixel values of the pixel i, for example, Equation (2). ) Using a known conversion method as shown in FIG.

Y = 0.299 × R + 0.587 × G + 0.114 × B
Cr = 0.701 × R−0.587 × G−0.114 × B (2)
Cb = −0.299 × R−0.587 × G + 0.886 × B
Then, the second coefficient determination unit 21 calculates the average value of the luminance value YX _k and the luminance value YZ _k−1 using the following equation (3), and sets it as the luminance value YB _k at the pixel i.

YB _k = (YX _k + YZ _k−1 ) / 2 (3)
The luminance value YB _k is YB _k = αYX _k + (calculated using a conventional recursive filter, in addition to the average value of the luminance value YX _k and the luminance value YZ _k−1 shown in Equation (3). 1−α) YZ _k−1 or an average value of luminance values in a region of 3 pixels × 3 pixels or 5 pixels × 5 pixels centered on the pixel position of the pixel i of the target image and / or the reference image. May be.

The function G shown in FIG. 2 (2), the brightness value YB _k is monotonically increasing between 0 and the threshold value Th2, a function which becomes 1 at the threshold Th2 or more, the noise removal in the dark portion luminance value is small Do it stronger. However, in FIG. 2 (2), the function G increases linearly and monotonically from 0 to the threshold Th2, but it may be monotonically increased like a quadratic function or an exponential function. Further, for example, when the gradation of the image is 0 to 255, the threshold Th2 is set to Th2 = 20 or the like regardless of the ISO sensitivity.

  The compression processing unit 23 compresses a target image from which noise has been removed in accordance with a moving image format such as H.264 or Motion JPEG, and generates moving image data.

  The memory 15 is a non-volatile semiconductor memory that stores various programs such as an image processing program executed by the CPU 14 together with the image data and the functions f and G shown in FIG.

  The operation unit 16 receives, for example, an imaging mode switching setting input, a still image, continuous shooting, or moving image imaging instruction from the user.

  The monitor 17 is a monitor such as a liquid crystal monitor, and displays various images according to control instructions from the CPU 14. For example, after capturing a moving image, the monitor 17 reproduces and displays the captured moving image in response to a control instruction from the CPU 14.

  A non-volatile storage medium 19 can be detachably connected to the media I / F 18. The media I / F 18 executes data writing / reading with respect to the storage medium 19. The storage medium 19 includes a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 19.

  Next, processing operations by the digital camera 1 according to the present embodiment will be described with reference to the flowchart of FIG.

  When the CPU 14 receives an instruction for capturing a moving image (for example, a full press operation of a release button included in the operation unit 16) from the user, the CPU 14 causes the image sensor 12 to start capturing a moving image of the subject. CPU14 starts the process from step S101.

  Step S101: The CPU 14 reads the first frame (k = 1) output from the image sensor 12 as a target image via the DEF 13. At the same time, the CPU 14 sets the target image as a reference image for noise removal processing. Note that the reference image for the target image after the second frame (k ≧ 2) is the previous frame from which the noise removal of the present embodiment has been performed.

Step S102: The first coefficient determination unit 20 calculates a difference Z ₁ -Z ₀ between pixel values in the pixel i of the read target image and reference image for each color component, and based on the function f in FIG. To determine the value of the coefficient α (first coefficient). At the same time, the second coefficient determination unit 21 calculates the luminance value YB ₁ at the pixel i using the equations (2) and (3), and calculates the coefficient β (second coefficient) based on the function G in FIG. Determine the value of.

  Step S103: The noise removing unit 22 applies the values of the coefficients α and β determined in Step S102 to Expression (1), and removes noise from the pixel values of the respective color components in the pixel i of the target image.

  Step S104: The noise removing unit 22 determines whether noise has been removed from all pixels of the target image. When noise removal has not been performed on all pixels, the noise removal unit 22 proceeds to step S102 (NO side), performs steps S102 and S103 on all pixels, and removes noise from the target image. . On the other hand, when the noise removing unit 22 removes noise for all pixels, the process proceeds to step S105.

  Step S105: The compression processing unit 24 compresses the target image from which noise has been removed in accordance with a moving image format such as H.264 or Motion JPEG, and generates moving image data. The CPU 14 temporarily records the moving image data in the memory 15. At the same time, the CPU 14 records and sets the target image from which noise has been removed in the memory 15 as a reference image for the target image of the next frame.

  Step S106: The CPU 14 determines whether or not a moving image capturing end instruction has been received, for example, when the user has fully released the release button of the operation unit 16. When the CPU 14 receives an instruction to end moving image capturing (YES side), the CPU 14 generates a moving image file from the moving image data temporarily recorded in the memory 15 and records it in the memory 15 or the storage medium 19. The CPU 14 ends a series of processes. On the other hand, if the CPU 14 has not received a moving image capturing end instruction, the CPU 14 proceeds to step S101 (NO side). CPU14 performs the process of step S101-step S105 with respect to the target image after the 2nd frame until an end instruction is received.

As described above, in the present embodiment, the coefficient α is determined from the difference between the pixel values in each pixel of the target image and the reference image, and at the same time, the coefficient β is determined in consideration of the luminance value in each pixel, that is, the luminance state in the subject. As a result, noise removal can be performed with high accuracy and at high speed even for pixels with high noise levels.
<< Other embodiments >>
FIG. 4 is a block diagram showing an example of the configuration of a digital camera 1 ′ according to another embodiment of the present invention. In FIG. 4, among the components of the digital camera 1 ′ of the present embodiment, the same components as those of the digital camera 1 of the first embodiment are denoted by the same reference numerals and detailed description thereof is omitted.

  Differences between the digital camera 1 ′ of the present embodiment and the digital camera 1 of the first embodiment are as follows.

  1) The CPU 14 also operates as the reduced image generation unit 24 by executing the image processing program. The reduced image generation unit 24 generates a reduced image of each frame of the moving image captured by the image sensor 12 using a known method. The reduced image generation unit 24 of the present embodiment generates a reduced image (first reduced image) with a reduction ratio of 1/4 with respect to the frame of the target image. The reduced image generation unit 24 also generates a reduced image (second reduced image) with a reduction ratio of 1/4 with respect to the reference image. It is assumed that the known method in the present embodiment is, for example, multiresolution analysis or Nearest Neighbor method.

  2) When removing noise from the target image, the noise removing unit 22 'first applies the deformed recursive filter of Expression (1) to the reduced image of the target image and the reduced image of the reference image. Then, the noise removing unit 22 ′ obtains a difference between the target images before and after the noise removal by the reduced image, and noise data of noise in a frequency band (hereinafter referred to as a low frequency band) corresponding to the reduction ratio ¼. To extract. The noise removing unit 22 ′ uses the low-frequency band noise data, removes the low-frequency band noise superimposed on the original target image, and applies Equation (1) to the target image to thereby apply the target image itself. Removes noise in the high frequency band that is superimposed.

  Next, processing operations performed by the digital camera 1 ′ according to the present embodiment will be described with reference to the flowchart of FIG. 5 and the flowchart of image data of FIG. 6.

  When the CPU 14 receives an instruction for capturing a moving image (for example, a full press operation of a release button included in the operation unit 16) from the user, the CPU 14 causes the image sensor 12 to start capturing a moving image of the subject. CPU14 starts the process from step S201.

  Step S201: The CPU 14 reads the first frame (k = 1) output from the image sensor 12 as the target image 30a via the DEF 13. At the same time, the CPU 14 sets the target image 30a as a reference image 30b for noise removal processing. Note that the reference image 30b for the target image 30a after the second frame (k ≧ 2) is the previous frame from which the noise removal of the present embodiment has been performed.

  Step S202: The reduced image generation unit 24 generates a reduced image with a reduction ratio of 1/4 for each of the read target image 30a and reference image 30b. In FIG. 6, a reduced image having a reduction ratio of 1/4 with respect to the target image 30a is denoted by reference numeral 31a, and a reduced image having a reduction ratio of 1/4 with respect to the reference image 30b is denoted by reference numeral 31b.

Step S203: The first coefficient determination unit 21 calculates, for each color component, a pixel value difference X ₁ -Z ₀ in the pixel i of the reduced image 31a and the reduced image 31b of the target image 30a and the reference image 30b, and FIG. The value of the coefficient α (first coefficient) is determined based on the function f in (1). At the same time, the second coefficient determination unit 21 calculates the luminance value YB ₁ at the pixel i using the equations (2) and (3), and calculates the coefficient β (second coefficient) based on the function G in FIG. Determine the value of.

  Step S204: The noise removing unit 22 'applies the values of the coefficients α and β determined in Step S203 to Expression (1), and removes noise from the pixel values of the respective color components in the pixel i of the reduced image 31a.

  Step S205: The noise removing unit 22 'determines whether or not noise has been removed from all the pixels of the reduced image 31a. When the noise removal unit 22 ′ has not removed noise for all pixels, the process proceeds to step S203 (NO side), and the processing of step S203 and step S204 is performed on all the pixels, and noise is reduced from the reduced image 31a. Remove. On the other hand, when noise is removed from all pixels, the noise removing unit 22 'generates a reduced image 32 from which noise in the low frequency band corresponding to the reduction ratio ¼ is removed. The noise removing unit 22 'obtains a difference between the reduced image 31a and the reduced image 32, and extracts noise data 33 in a low frequency band superimposed on the reduced image 31a. The noise removing unit 22 'proceeds to step S206 (YES side).

  Step S206: The noise removing unit 22 'converts the extracted noise data 33 into low frequency band noise data 34 in each pixel of the original target image 30a. The noise removing unit 22 ′ subtracts the noise of the noise data 34 from the pixel value of each pixel of the target image 30 a to generate the target image 35 from which low frequency band noise has been removed. However, the target image 35 has noise in a frequency band (hereinafter referred to as a high frequency band) corresponding to a reduction ratio of 1/1.

Step S207: The first coefficient determination unit 20 calculates, for each color component, a pixel value difference X ₁ −Z ₀ in the pixel i of the target image 35 and the reference image 30b, and based on the function f in FIG. To determine the value of the coefficient α (third coefficient). At the same time, the second coefficient determination unit 21 calculates the luminance value YB ₁ at the pixel i using equations (2) and (3), and calculates the coefficient β (fourth coefficient) based on the function G in FIG. Determine the value of.

  Step S208: The noise removing unit 22 ′ applies the values of the coefficients α and β determined in Step S207 to Expression (1), and removes high-frequency band noise from the pixel value of each color component in the pixel i of the target image 35. To do.

  Step S209: The noise removing unit 22 'determines whether or not high frequency band noise has been removed for all the pixels of the target image 35. If the noise removal unit 22 ′ has not removed noise for all the pixels, the process proceeds to step S207 (NO side), and the processing of step S207 and step S208 is performed on all the pixels, and the high frequency band is obtained from the target image 35. Remove noise. On the other hand, if the high frequency band noise is removed from all the pixels, the CPU 14 generates the target image 36 from which the low frequency band and high frequency band noise has been removed, and proceeds to step S210 (YES side).

  Step S210: The compression processing unit 23 compresses the target image 36 from which noise in the low frequency band and the high frequency band is removed according to a moving image format such as H.264 or Motion JPEG, and generates moving image data. The CPU 14 temporarily records the moving image data in the memory 15. At the same time, the CPU 14 records and sets the target image 36 in the memory 15 as a reference image 30b for the target image 30a of the next frame.

  Step S211: The CPU 14 determines whether or not a moving image capturing end instruction has been received, for example, when the user has fully released the release button of the operation unit 16. When the CPU 14 receives an instruction to end moving image capturing (YES side), the CPU 14 generates a moving image file from the moving image data temporarily recorded in the memory 15 and records it in the memory 15 or the storage medium 19. The CPU 14 ends a series of processes. On the other hand, if the CPU 14 has not received an instruction to end moving image capturing, the CPU 14 proceeds to step S201 (NO side). CPU14 performs the process of step S201-step S209 with respect to the target image 30a after the 2nd frame until an end instruction is received.

  As described above, in the present embodiment, the coefficient α is determined from the difference between the pixel values in each pixel of the reduced images 31a and 31b together with the target image 30a and the reference image 30b, and at the same time, the luminance value in each pixel, that is, in the subject. By determining the coefficient β in consideration of the luminance state, noise removal can be performed with high accuracy and high speed even for a pixel having a high noise level.

Further, the reduced images 31a and 31b are respectively generated from the target image 30a and the reference image 30b, and noise in the frequency band corresponding to the reduction rate is sequentially extracted, so that noise can be removed more efficiently.
<< Modifications of Other Embodiments >>
A digital camera according to a modification of the other embodiment of the present invention is the same as the digital camera 1 ′ according to the other embodiment shown in FIG. Therefore, the detailed description about each component which comprises digital camera 1 'of this embodiment is abbreviate | omitted.

  FIG. 7 shows a flowchart of processing operations by the digital camera 1 ′ of the present embodiment. In FIG. 7, the same processing as the processing in the other embodiment shown in FIG. 5 is given the same step number as the last two digits, and detailed description thereof is omitted.

  FIG. 8 shows the flow of image data in the processing shown in FIG. In FIG. 8, the same reference numerals are assigned to the same image data as in the other embodiments shown in FIG. 6, and detailed description thereof is omitted.

  The difference between the processing operation performed by the digital camera 1 ′ according to the present embodiment and that according to the other embodiments is that the high-frequency band noise removal processing from step S 207 to step S 209 is omitted. This is because the moving image compression processing performed in step S310 (corresponding to step S210) by the compression processing unit 23 to convert the target image 35 into moving image data has the same effect as removing high-frequency band noise. is there.

  Further, by omitting the processing from step S207 to step S209, in this embodiment, the target image 36 is not calculated, and the reduced image generation unit 24 reduces the reference image 30b after the second frame in step S302. The image 31b is not generated. Instead, in step S310, the CPU 14 according to the present embodiment records and sets the reduced image 32 in the memory 15 as the reduced image 31b having the reduction ratio ¼ of the reference image 30b. In step S301, the CPU 14 reads the reduced image 31b from the memory 15.

  As described above, in the present embodiment, the coefficient α is determined from the difference between the pixel values of the pixels of the reduced images 31a and 31b together with the target image 30a and the reference image 30b, and at the same time, the luminance value of the pixel i, that is, the subject By determining the coefficient β in consideration of the luminance state, noise removal can be performed with high accuracy and high speed even for a pixel having a high noise level.

  Further, the reduced images 31a and 31b are respectively generated from the target image 30a and the reference image 30b, and noise in the frequency band corresponding to the reduction rate is sequentially extracted, so that noise can be removed more efficiently.

  Furthermore, by replacing the noise removal in the high frequency band with the video compression process, the processing time of the noise removal process by the digital camera 1 ′ and the processing load can be reduced, and the circuit scale of the digital camera 1 ′ can be reduced. Can be small.

Further, by omitting the generation process of the reduced image of the reference image 30b by the reduced image generation unit 20 in the second and subsequent frames, the processing time of the noise removal process and the processing load can be further reduced, and the digital camera The circuit scale of 1 ′ can be further reduced.
<< Additional items of embodiment >>
(1) In the above embodiment, the CPU 14 implements the processes of the first coefficient determination unit 21, the second coefficient determination unit 22, the noise removal unit 22, the compression processing unit 23, and the reduced image generation unit 24 by software. Although an example has been described, each of these processes may be realized by hardware using an ASIC.

  (2) The image processing apparatus of the present invention is not limited to the examples of the digital cameras 1 and 1 'according to the above embodiment. For example, the computer may function as the image processing apparatus of the present invention by causing a computer to read a plurality of images captured by moving images or continuous shooting and causing the computer to execute a noise removal processing program.

  (3) In the above embodiment, the value of the threshold Th1 is set to the same value for each RGB color component, but the present invention is not limited to this. A different threshold value Th1 may be set for each component. Further, RGB may be converted to YCrCb based on the formula (2), and the threshold value Th1 may be set for each luminance component and color difference component. For example, by setting the threshold values of the color difference components Cr and Cb to be 1.5 times larger than the threshold value of the luminance component Y and increasing noise removal for the color difference components Cr and Cb, the color noise is less noticeable. can do.

  Also, the value of the threshold Th2 may be set to a different value for each color component, luminance component, and color difference component.

  (4) In the above embodiment, the target image is used as the reference image for the target image of the first frame. However, the present invention is not limited to this. For example, an image obtained by performing known noise removal processing on the target image is used. It may be a reference image.

  (5) In the above one embodiment, when the pixel value of the pixel i is an RGB component, the luminance value at the pixel i is calculated using the known method of the equation (2). However, the pixel value of any one of RGB color components such as the G component may be used as the luminance value.

  (6) In the other embodiments described above, the reduced image generation unit 24 generates the reduced images having the reduction ratio of 1/4 of the target image 30a and the reference image 30b by multi-resolution analysis. However, the present invention is not limited to this. . The reduced image generation unit 24 may generate reduced images with different reduction ratios of two or more.

  When reduced images having different reduction ratios of 2 or more are generated, the reduced image has noise in a frequency band corresponding to its own reduction ratio and noise in a lower frequency band than that of a smaller reduced image. . Therefore, it is preferable that the noise removing unit 22 ′ starts from a reduced image having a minimum reduction rate in order to efficiently extract noise in a frequency band corresponding to its own reduction rate from each reduced image.

  (7) In the other embodiments described above, the CPU 14 reads the reference image 30b together with the target image 30a output from the image sensor 12, and the reduced image generation unit 24 generates reduced images of the target image 30a and the reference image 30b. However, the present invention is not limited to this. For example, the CPU 14 may store and set the target image 36 as the reference image 30b in the memory 15, and may record and set the reduced image 32 in the memory 15 as the reference image 31b having a reduction ratio of 1/4. Thereby, the reduced image generating process of the reference image 30b by the reduced image generating unit 24 can be omitted, and the processing time of the noise removal processing and the processing load can be reduced. Furthermore, the circuit scale of the digital camera 1 'can be reduced.

  From the above detailed description, features and advantages of the embodiment will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents included in the scope disclosed in.

DESCRIPTION OF SYMBOLS 1, 1 '... Digital camera, 11 ... Imaging optical system, 12 ... Imaging element, 13 ... DEF, 14 ... CPU, 15 ... Memory, 16 ... Operation part, 17 ... Monitor, 18 ... Media I / F, 19 ... Memory Medium, 20 ... first coefficient determination unit, 21 ... second coefficient determination unit, 22, 22 '... noise removal unit, 23 ... compression processing unit, 24 ... reduced image generation unit

Claims (12)

A first coefficient determination unit that determines a value of the first coefficient based on a difference between a target image to be denoised and a reference image that is a reference for noise removal processing;
A second coefficient determination unit that determines a value of a second coefficient based on luminance information of the target image and / or the reference image;
A noise removing unit that weights and adds the target image and the reference image using values of the first coefficient and the second coefficient, and removes noise superimposed on the target image;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The luminance information is an average value of luminance values of the target image and the reference image, or a value obtained by weighting and adding the luminance values of the target image and the reference image using only the value of the first coefficient. An image processing apparatus. The image processing apparatus according to claim 1 or 2,
The image processing apparatus according to claim 1, wherein the first coefficient determination unit determines a value of the first coefficient for each color component of the target image. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the first coefficient determination unit determines a value of the first coefficient for each luminance component and color difference component of the target image. At least one reduction ratio is a set of a first reduced image obtained by reducing a target image to be subjected to noise removal and a second reduced image obtained by reducing a reference image used as a reference for noise removal processing at the same reduction ratio as the first reduced image. A reduced image generation unit to generate in
A first coefficient determination unit that determines a value of a first coefficient based on a difference between the first reduced image and the second reduced image;
A second coefficient determination unit that determines a value of a second coefficient based on a luminance level of the first reduced image and / or the second reduced image;
Using the values of the first coefficient and the second coefficient, the first reduced image and the second reduced image are weighted and added, noise in a frequency band corresponding to the reduction ratio is extracted, and pixels of the target image A noise removing unit for removing noise in the frequency band from the value;
An image processing apparatus comprising: The image processing apparatus according to claim 5.
The noise removing unit sequentially extracts noise in the frequency band from a set corresponding to a minimum reduction ratio when a plurality of sets of the first reduced image and the second reduced image are generated at a plurality of reduction ratios. An image processing apparatus. The image processing apparatus according to claim 5 or 6,
The first coefficient determination unit determines a value of a third coefficient based on a difference between the reference image and the target image from which noise in the frequency band has been removed,
The second coefficient determination unit determines a value of a fourth coefficient based on luminance information of the reference image and / or a target image from which noise in the frequency band has been removed,
The noise removing unit weights and adds the reference image and the target image from which noise in the frequency band is removed using the values of the third coefficient and the fourth coefficient, and superimposes the high frequency band on the target image. An image processing apparatus characterized by removing noise. The image processing apparatus according to claim 5 or 6,
An image processing apparatus comprising: a compression processing unit that performs image compression on the target image from which noise in the frequency band has been removed, and removes noise in a high frequency band superimposed on the target image. The image processing apparatus according to any one of claims 1 to 8,
The target image is one of continuously captured images,
The image processing apparatus according to claim 1, wherein the reference image is an image that is continuously captured and is an image that is captured before the target image and from which noise is removed. An imaging unit that captures a subject image and generates an image;
An image processing apparatus according to any one of claims 1 to 9,
An imaging apparatus comprising: Input procedure for reading the target image for noise removal and the reference image for noise removal processing,
A first coefficient determination procedure for determining a value of a first coefficient based on a difference between the target image and the reference image;
A second coefficient determination procedure for determining a value of a second coefficient based on luminance information of the target image and / or the reference image;
A noise removal procedure for performing weighted addition of the target image and the reference image using values of the first coefficient and the second coefficient, and removing noise superimposed on the target image;
An image processing program for causing a computer to execute. Input procedure for reading the target image for noise removal and the reference image for noise removal processing,
A reduced image generation procedure for generating a set of a first reduced image obtained by reducing the target image and a second reduced image obtained by reducing the reference image at the same reduction rate as the first reduced image, at least one reduction rate;
A first coefficient determination procedure for determining a value of a first coefficient based on a difference between the first reduced image and the second reduced image;
A second coefficient determination procedure for determining a value of a second coefficient based on luminance information of the first reduced image and / or the second reduced image;
The first reduced image and the second reduced image are weighted and added using the values of the first coefficient and the second coefficient, noise in a frequency band corresponding to the reduction ratio is extracted, and the target image is extracted from the target image. Noise removal procedure to remove noise in the frequency band,
An image processing program for causing a computer to execute.

Images