JP5035646B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and in particular, can remove noise components generated by an image sensor among noise components mixed in an image signal obtained by using an image sensor such as a solid-state imaging device. The present invention relates to an image processing apparatus and method, and a program.

  The process of separating a signal component and a noise component from a noisy signal is the greatest concern in signal processing, and a great variety of techniques have been developed so far. In the case of an image signal, a random noise component generated inside the image sensor particularly during imaging is a problem. There are various types of noise caused by image sensors. Among them, noise that is mixed in chroma components and recognized as a phenomenon of color unevenness or color shift when observing an image has been improved. Therefore, high image quality is expected.

  The cause of noise mixed in such chroma components includes at least the following two points.

  The first point is that random noise is generated independently between channels. That is, since the solid-state imaging device usually has a wide spectral sensitivity, when imaging a color image, a plurality (usually three types) of image sensors whose spectral sensitivity is changed by a color filter or the like is used. In this case, there are a method of giving independent and different spectral sensitivities using a three-chip solid-state imaging device, and a method of changing spectral sensitivities in units of pixels with only one chip. However, in either method, since the signals of the respective channels (colors) are acquired by independent and different image sensors, it is natural that the random noise generated by the image sensors is also generated independently between the channels. As a result, a noise component independent of the chroma component is generated.

  The second point is caused by chromatic aberration of the optical system. In other words, chromatic aberration is caused by the difference in the refractive index for each wavelength, so that the appearance of the aberration varies depending on the color. However, this chromatic aberration causes only the image of a specific channel to be blurred or positioned at the edge of the subject. The phenomenon of misalignment occurs. As a result, this phenomenon is observed as the color unevenness or color shift described above.

  Therefore, a technique for removing noise mixed in such chroma components has been proposed.

  For example, areas that are prone to color shift due to chromatic aberration, such as high-luminance areas of the image, are detected based on the luminance signal, and the detected area is appropriately expanded, and then the color difference signal in the expanded area is smoothed. (For example, refer to Patent Document 1).

  For areas other than the object edge area in the image, the color difference signal is calculated by calculating a color difference signal from which high-frequency components have been removed by averaging processing for areas other than the edge area, assuming that the color difference signal does not have a high frequency. Some devices remove noise (see, for example, Patent Document 2). Similarly, there is a technique that removes color difference noise based on the assumption that there are not many high-frequency components in the color difference signal where there is no high-frequency component in the luminance signal (see, for example, Patent Document 3).

  The above prior art can be performed after the luminance signal and the color difference signal are generated from the output of the image sensor. However, when the image sensor outputs the primary color (RGB) signal, the noise of the chroma component in the RGB signal. As a technology to remove the image, local RG correlation coefficient and BG correlation coefficient are calculated for RGB images, and correction values for R and B are calculated based on the respective correlation coefficients. Some attempt to effectively correct the color misalignment in (see, for example, Patent Document 4).

Japanese Patent Laid-Open No. 5-14925 Japanese Patent Laid-Open No. 3-290109 Japanese Patent Laid-Open No. 2003-224861 JP 2003-230159 A

  However, in the technique described above, for example, in the technique for dealing with the color shift due to the chromatic aberration in the high luminance region described in Patent Document 1, in order to obtain an appropriate effect, the magnitude of the chromatic aberration generated on the image is reduced. Accordingly, it is necessary to adjust the size of the expansion processing and propagation / diffusion processing. The magnitude of chromatic aberration depends on the number of pixels of the image sensor and the performance of the optical system, and in some cases, expansion processing or propagation / diffusion processing on the scale of several tens of pixels (or lines) may be required. Dilation processing and propagation / diffusion processing are repeated multiple times to perform uniform local processing on all pixels, so when considering implementing them in hardware, access to the frame memory in proportion to the number of repetitions The number of times or a considerable amount of delay lines must be mounted, and the circuit scale becomes too large, which is not realistic.

  In addition, the techniques described in Patent Documents 1 to 3 limit the place where the color difference signal is smoothed based on the luminance signal, and noise is generated by smoothing the color difference signal in the limited area. There is a common point of reduction. That is, in these techniques, there is a risk that the high-frequency component of the color difference signal is lost as a price for removing the noise of the chroma component.

  On the other hand, the technique described in Patent Document 4 calculates a correction value of a channel to be corrected (usually R and B) based on the strength of correlation between RGB channels. Since this correction value is calculated based on the high-frequency component of the channel to be referenced (usually G) so as to include an appropriate high-frequency component of the channel to be corrected, if this method is used, the above-described Patent Documents 1 to It is possible to prevent the high frequency component of the chroma component in the third technique from being deteriorated.

  However, with the increase in the number of pixels of image sensors in recent years, color misregistration due to chromatic aberration and random noise independent between channels have expanded to low frequency components, and local computations such as the method disclosed in Patent Document 4 have been performed. It is becoming inadequate to simply remove high-frequency chroma noise.

  The present invention has been made in view of such a situation, and in particular, a noise component mixed in a chroma component of an image signal obtained using an image sensor such as a solid-state image sensor is efficiently removed, and color unevenness or This suppresses the phenomenon of color misregistration.

An image processing apparatus according to an aspect of the present invention is a multiresolution conversion that converts, for each pixel, three or more images that can be assumed to have a linear relationship in a local region in the vicinity thereof into a plurality of layer images of a multiresolution image. And a layer image converted by the multi-resolution conversion unit, in each layer image excluding the lowest resolution layer image, one image among the three or more images is based on the correlation between the three or more images. Correcting means for correcting the image, multi-resolution inverse converting means for performing multi-resolution inverse conversion on each layer image corrected by the correcting means, and arranging the three or more images from the periphery of the target pixel at positions corresponding to the target pixels. Extracting means for extracting the pixel values of the peripheral pixels to be calculated, variance value calculating means for calculating the variance values for each of the three or more images based on the pixel values of the peripheral pixels, and 3 Selection for selecting the pixel value of the target pixel of any one specific image of the three or more images as the pixel value of the target pixel in an image other than the specific image based on the respective variance values of the above images Means .

The correcting means includes covariance value calculating means for calculating covariance values of three or more images based on pixel values of respective peripheral pixels of three or more images, and variances of each of the three or more images. Correlation calculation means for calculating the correlation of all combinations of values and covariance values can be provided , and the selection means includes three if the correlation is weak in all combinations of three or more images. The pixel value of the target pixel of any one specific image of the above images can be selected as the pixel value of the target pixel in an image other than the specific image.

The selection means specifies the pixel value of the target pixel of any one specific image of the three or more images when each of the variance values of the three or more images is within a predetermined range. It can be made to select as a pixel value of an attention pixel in an image other than an image.

  The multi-resolution image conversion means can be provided with non-linear conversion means for applying non-linear conversion to the input image before converting three or more images into a plurality of layer images of the multi-resolution image.

  5. The image processing apparatus according to claim 4, wherein the nonlinear conversion means converts three or more images into a plurality of layer images of a multi-resolution image by logarithmic conversion.

  The nonlinear conversion means can convert three or more images into a plurality of layer images of a multi-resolution image by gamma correction processing.

According to an image processing method of one aspect of the present invention, for each pixel, multi-resolution conversion that converts three or more images that can be assumed to have a linear relationship in a local region in the vicinity thereof to a plurality of layer images of the multi-resolution image. Of the layer images converted by the processing of the step and the multi-resolution conversion step, in each layer image excluding the lowest resolution layer image, one of the three or more images is correlated with the three or more images. A correction step for correcting based on the multi-resolution, a multi-resolution reverse conversion step for performing multi-resolution reverse conversion on each layer image corrected by the processing of the correction step, and corresponding target pixels of three or more images from the periphery of the target pixel An extraction step of extracting the pixel values of the peripheral pixels arranged at the position, and for each of the three or more images, based on the pixel values of the peripheral pixels, Based on the variance value calculating step for calculating the variance value and the variance value of each of the three or more images, the pixel value of the target pixel of any one of the three or more images is determined as the specific image. And a selection step of selecting as a pixel value of a target pixel in an image other than the above.

A program according to one aspect of the present invention includes a multi-resolution conversion step of converting three or more images that can be assumed to have a linear relationship in each local pixel to a plurality of layer images of the multi-resolution image. Based on the correlation between the three or more images, one of the three or more images in each layer image excluding the lowest resolution layer image among the layer images converted by the multi-resolution conversion step process. A correction step for correcting the image, a multi-resolution inverse conversion step for performing multi-resolution reverse conversion on each layer image corrected by the processing of the correction step, and positions corresponding to the target pixels of three or more images from the periphery of the target pixel The extraction step of extracting the pixel values of the peripheral pixels arranged in the image and the three or more images are divided based on the pixel values of the peripheral pixels. Based on the variance value calculating step for calculating the value and the variance value of each of the three or more images, the pixel value of the target pixel of any one of the three or more images is set to a value other than the specific image. And a selection step of selecting as a pixel value of a pixel of interest in the image of (2 ).

In one aspect of the present invention, three or more images are converted into a plurality of layer images of a multi-resolution image, and among the converted layer images, three or more images are included in each layer image excluding the lowest resolution layer image. One image in the image is corrected based on the correlation between the three or more images, and each of the corrected layer images is subjected to inverse multi-resolution conversion, so that each target pixel of the three or more images from the periphery of the target pixel The pixel values of the peripheral pixels arranged at the positions corresponding to are extracted, and for each of the three or more images, the variance value is calculated based on the pixel values of the peripheral pixels, and the variance of each of the three or more images is calculated. based on the value, the pixel value of the pixel of interest any one specific image of the three or more images, Ru is selected as the pixel value of the pixel of interest in the image other than the specific image.

  The image processing apparatus of the present invention may be an independent apparatus or a block that performs image processing.

  According to the present invention, it is possible to efficiently remove a noise component mixed in a chroma component of an image signal obtained using an image sensor such as a solid-state imaging device, and to suppress a phenomenon such as color unevenness or color misregistration. .

It is a block diagram explaining the structure of one Embodiment of the digital still camera to which this invention is applied. It is a block diagram which shows the structure of one Embodiment of the function implement | achieved by the DSP block of FIG. It is a block diagram which shows the structure of one Embodiment of the detailed function of the nonlinear transformation process part of FIG. It is a block diagram which shows the structure of one Embodiment of the detailed function of the noise removal process part of FIG. FIG. 3 is a block diagram showing a detailed configuration of an embodiment of a function of a multi-resolution conversion processing unit in FIG. 2. FIG. 5 is a block diagram illustrating a detailed configuration of an embodiment of a function of a correction processing unit in the noise removal processing unit of FIG. 4. FIG. 3 is a block diagram illustrating a detailed configuration of an embodiment of a function of a multi-resolution inverse transform processing unit in FIG. 2. It is a block diagram explaining the structure of detailed embodiment of the function of the nonlinear inverse transformation process part of FIG. It is a flowchart explaining the image processing by the DSP block of FIG. It is a flowchart explaining a noise removal process. It is a flowchart explaining a multi-resolution conversion process. It is a flowchart explaining a pixel value correction process. It is a flowchart explaining a distributed calculation process. It is a flowchart explaining a distributed calculation process. It is a flowchart explaining a covariance calculation process. It is a flowchart explaining an integrating | accumulating process. It is a flowchart explaining an integrating | accumulating process. It is a flowchart explaining an integral approximation process. It is a flowchart explaining a multi-resolution inverse transformation process. FIG. 5 is a block diagram showing a detailed configuration of an embodiment of other functions of the correction processing unit in the noise removal processing unit of FIG. 4. It is a flowchart explaining the pixel value correction process by the correction process part of FIG. FIG. 5 is a block diagram showing a detailed configuration of an embodiment of still another function of the correction processing unit in the noise removal processing unit of FIG. 4. It is a flowchart explaining the pixel value correction process by the correction process part of FIG. FIG. 5 is a block diagram illustrating a detailed configuration of an embodiment of the noise removal processing unit and other functions of FIG. 4. FIG. 25 is a block diagram illustrating a detailed configuration of an embodiment of a function of a correction processing unit in the noise removal processing unit of FIG. 24. It is a flowchart explaining the noise removal process by the noise removal process part of FIG. It is a flowchart explaining the pixel value correction process by the correction process part of FIG. It is a figure explaining a recording medium.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an embodiment of a digital still camera 1 that executes arithmetic processing to which the present invention is applied.

  As shown in FIG. 1, a digital still camera 1 includes a lens 11, an aperture 12, an image sensor 13, a correlated double sampling (CDS) circuit 14, an A / D (Analog / Digital) converter 15, a DSP ( Digital Signal Processor) block 16, timing generator 17, D / A (Digital / Analog) converter 18, video encoder 19, display unit 20, codec (CODEC: COmpression / DECompression) processing unit 21, memory 22, CPU (Central Processing Unit) ) 23 and the operation input unit 24.

  The image sensor 13 is a (photoelectric conversion) semiconductor element typified by a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) that converts optical information into an electrical signal, and a light receiving element that converts light into electricity ( A plurality of pixels) are arranged, and a change in light is converted into an electric signal independently for each pixel. The correlated double sampling circuit 14 uses the reset noise, which is the main component of the noise included in the output signal of the image sensor 13, and samples the video signal period among the output pixel signals and the reference period. It is a circuit that removes a sampled one by subtraction. The A / D converter 15 converts the supplied analog signal after noise removal into a digital signal.

  The DSP block 16 is a block having a signal processing processor and an image RAM (Random Access Memory). The signal processing processor performs pre-programmed image processing on image data stored in the image RAM, or It performs image processing configured as arithmetic processing by hardware. The timing generator 17 is a logic circuit that generates various horizontal and vertical drive pulses necessary for driving the image sensor 13 and pulses used in the analog front processing in synchronization with the reference clock. The timing clock generated by the timing generator 17 is also supplied to the codec processing unit 21, the memory 22, and the CPU 23 via the bus 25.

  The D / A converter 18 converts the supplied digital signal into an analog signal and outputs the analog signal. The video encoder 19 encodes the supplied analog signal into video data in a format that can be displayed on the display unit 20. The display unit 20 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and displays the video signal supplied from the video encoder 19.

  The codec processing unit 21 executes processing by a digital image data compression or expansion algorithm such as JPEG (Joint Picture Experts Group). The memory 22 is constituted by, for example, a semiconductor memory, a magnetic disk, a magneto-optical disk, an optical disk, or the like, and stores supplied data or outputs stored data based on control of the CPU 23. . The memory 22 may be detachable from the digital still camera 1.

  The CPU 23 controls each unit of the digital still camera 1 based on the user's operation input supplied from the operation input unit 24 via the bus 25. The operation input unit 24 includes, for example, a jog dial, a key, a lever, a button, or a touch panel as well as a button for instructing recording, and receives an operation input by a user.

  Light input through the lens 11 and the diaphragm 12 is incident on the image sensor 13, converted into an electrical signal by photoelectric conversion at the light receiving element, and supplied to the correlated double sampling circuit 14. The correlated double sampling circuit 14 removes noise by subtracting the sampled video signal period and the sampled reference period from the pixel signals output from the image sensor 13, and the A / D converter. 15 is supplied. The A / D converter 15 converts the supplied analog signal after noise removal into a digital signal and temporarily stores it in the image RAM of the DSP block 16.

  The timing generator 17 controls the image sensor 13, the correlated double sampling circuit 14, the A / D converter 15, and the DSP block 16 so as to maintain image capture at a constant frame rate in a state during imaging.

  The DSP block 16 receives supply of pixel stream data at a constant rate, temporarily stores it in the image RAM, and executes image processing, which will be described later, on the temporarily stored image data in the signal processor. After the image processing is completed, the DSP block 16 controls the CPU 23 to display the image data on the display unit 20. When the DSP block 16 stores the image data in the memory 22, the codec processing unit 21. Supply image data.

  The D / A converter 18 converts the digital image data supplied from the DSP block 16 into an analog signal and supplies it to the video encoder 19. The video encoder 19 converts the supplied analog image signal into a video signal and outputs it to the display unit 20 for display. That is, the display unit 20 serves as a camera finder in the digital still camera 1. The codec processing unit 21 performs a predetermined encoding on the image data supplied from the DSP block 16 and supplies the encoded image data to the memory 22 for storage.

  The codec processing unit 21 reads data designated by the user from among the data stored in the memory 22 based on the control of the CPU 23 that has received the user's operation input from the operation input unit 24, and performs predetermined decoding. The decoded signal is output to the DSP block 16 by the method. As a result, the decoded signal is supplied to the D / A converter 18 via the DSP block 16, converted into an analog signal, encoded by the video encoder 19, and displayed on the display unit 20.

  Next, the configuration of an embodiment of the function realized by the DSP block 16 will be described with reference to FIG.

  The demosaic processing unit 41 aligns all the intensities of the RGB (three primary colors of red, green, and blue light) image signals at each pixel position of the input mosaic image, and supplies them to the nonlinear conversion processing unit 42, respectively. Note that the mosaic image here is an image signal that is digitized by the A / D converter 15 of FIG. In the mosaic image, an intensity signal corresponding to any color of RGB is stored in each pixel, and is usually arranged according to a color arrangement called Bayer arrangement.

  The non-linear conversion processing unit 42 converts the RGB image signal using a non-linear conversion function, and supplies the converted signal to the noise removal processing unit 43. Details of the nonlinear conversion processing unit 42 will be described later.

  The noise removal processing unit 43 removes noise from the RGB image signal supplied from the nonlinear conversion processing unit 42 and supplies the RGB image signal to the nonlinear inverse conversion processing unit 44. Details of the noise removal processing unit 43 will be described later.

  The nonlinear inverse transformation processing unit 44 inversely transforms the noise-removed RGB image signal supplied from the noise removal processing unit 43 using a nonlinear transformation function used in the transformation of the nonlinear transformation processing unit 42, and performs white balance. This is supplied to the processing unit 45. Details of the nonlinear inverse transform processing unit 44 will be described later.

  The white balance processing unit 45 multiplies the RGB image signals from which noise has been removed by the non-linear transformation processing unit 42, the noise removal processing unit 43, and the non-linear inverse transformation processing unit 44, thereby multiplying the color of the subject area. Processing is performed so that the balance becomes an achromatic color, and the result is supplied to the gamma correction processing unit 46.

  The gamma correction processing unit 46 gamma-corrects each pixel intensity of the RGB image signals whose white balance has been adjusted by the white balance processing unit 45 and outputs the gamma correction to the YC conversion processing unit 47. The YC conversion processing unit 47 performs YC matrix processing on the gamma-corrected three-channel image signal, generates a Y image and a C image by applying band limitation to the chroma component, and outputs a YCbCr image signal to the codec processing unit 21. Output as.

  In FIG. 2, a non-linear transformation processing unit 42, a noise removal processing unit 43, and a non-linear inverse transformation processing unit 44 are arranged at the subsequent stage of the demosaic processing unit 41. For example, after the non-linear transformation processing unit 42, the noise removal processing unit 43, and the non-linear inverse transformation processing unit 44 are arranged in the preceding stage of the demosaic processing unit 41 and noise removal of the mosaic image is performed. The demosaic process may be executed. However, in this case, since each pixel of the mosaic image has only one of RGB values, a non-linear conversion processing unit 42, a noise removal processing unit 43, and a non-linear inverse conversion processing unit 44 are provided for each RGB. The processing for each pixel needs to be distributed according to color.

  In addition, each of the nonlinear transformation processing unit 42, the noise removal processing unit 43, and the nonlinear inverse transformation processing unit 44 can execute the noise removal processing alone. That is, for example, it is possible to eliminate the nonlinear transformation processing unit 42 and the nonlinear inverse transformation processing unit 44 from the configuration of FIG. Further, the nonlinear transformation processing unit 42 and the nonlinear inverse transformation processing unit 44 have an effect of removing noise even when combined with a configuration having noise removal processing capability other than the noise removal processing unit 43.

  Next, the configuration of an embodiment of the detailed functions of the nonlinear conversion processing unit 42 will be described with reference to FIG. The nonlinear conversion processing unit 42 has the same configuration for each RGB channel. FIG. 3 shows the configuration for one channel.

  The transformation curve LUT 62 stores the function shape used for nonlinear transformation in the LUT format. The mapping processing unit 61 performs nonlinear conversion on each pixel of the input image based on the LUT having a function shape used for nonlinear conversion from the conversion curve LUT 62 and outputs the result to the noise removing unit 43.

  The non-linear conversion processing by the non-linear conversion processing unit 42 is a conversion intended to convert the output of the image sensor 13 linear (linear) with respect to the incident light intensity into a luminance value having non-linear (non-linear) characteristics. This process improves the stability of linear regression calculation between channels used in the subsequent noise removal process. That is, for example, in the case of an image such as a high-contrast edge, the distribution of the luminance value seen from the center of gravity tends to be biased with linear luminance, and in such a situation, an appropriate nonlinear transformation is performed rather than performing linear regression calculation. By performing the linear regression calculation after converting the luminance value and correcting the distribution bias, the calculation becomes more stable. The non-linear transformation suitable for the present invention is an upward convex smooth monotonically increasing curve. For example, a power curve (exponential is less than 1) to be used for gamma correction or a logarithmic curve is effective.

  Next, a configuration of an embodiment of the function of the noise removal processing unit 43 will be described with reference to FIG.

  The configuration of the noise removal processing unit 43 shown in FIG. 4 is configured to remove R noise by inputting two channels of G and R. As an actual configuration of the noise removal processing unit 43, in addition to this, a configuration in which G and B are input and the B noise is removed with the same configuration is necessary. Since a chroma component noise is removed if a component having no correlation between R and B with respect to G is removed, a configuration for removing the G noise is not provided in this example. However, in practice, it is more appropriate to remove each of the R and B noises after removing the G noise by some configuration. Here, it is assumed that G is output to the subsequent stage without being processed.

  The multi-resolution conversion processing units 81-1 and 81-2 convert the input images of G and R into images of a plurality of multi-resolution representation formats, respectively, and convert the converted image signals of the respective layers to the corresponding layers, respectively. 0 image memories 82-1 and 82-2, layer 1 image memories 83-1 and 83-2, layer 2 image memories 84-1 and 84-2, and layer 3 image memories 85-1 and 85-2. .

  There are various formats for the multi-resolution expression in the multi-resolution conversion processing units 81-1 and 82-2. For example, a Laplacian Pyramid format or a multi-resolution format using wavelet conversion may be used. . The multi-resolution conversion processing unit 81 is assumed to be in the Laplacian Pyramid format.

  The correction processing units 86-1 to 86-3 respectively correspond to images of layers other than the lowest resolution layer, and correct each pixel value of the R channel mixed with noise from the G and R channel images of each layer. The noise is removed and stored in the layer 0 corrected image memory 87, the layer 1 corrected image memory 88, and the layer 2 corrected image memory 89, respectively.

  The multi-resolution inverse transform processing unit 90 reads out the R channel image from which noise of each layer has been removed from the layer 0 corrected image memory 87, the layer 1 corrected image memory 88, and the layer 2 corrected image memory 89, and the layer with the lowest resolution. The multi-resolution inverse transform process is executed using the R channel image, and the multi-resolution inverse transform process is integrated into an image having the same resolution as the original image, and is output to the nonlinear inverse transform processing unit 44.

  Next, the configuration of an embodiment of the function of the multiresolution conversion processing unit 81 that generates a Laplacian pyramid format multiresolution image will be described with reference to FIG.

  The reduction processing units 111-1 to 111-3 of the multi-resolution conversion processing unit 81 reduce the image size of the input image by sub-sampling to, for example, half (for example, convert the image to a half-resolution image), and Output.

  The enlargement processing units 113-1 to 113-3 enlarge the image size of the image input by supersampling, for example, by 2 times (for example, convert it to an image having a resolution of 2 times), and output it to the subsequent stage.

  The subtraction processing units 112-1 to 112-3 obtain a difference for each pixel of the image that has been reduced by the reduction processing unit 111 and then enlarged by the enlargement processing unit 113 and the original image, and includes the difference. The image is output as an image for each layer. That is, the subtraction processing units 112-1 to 112-3 generate images of layers 0 to 3, respectively, and a layer 0 image memory 82, a layer 1 image memory 83, a layer 2 image memory 84, and a layer 3 image memory 85, respectively. To store.

  Since the high-frequency component of the original image is removed from the image that has been reduced by the reduction processing unit 111 and then enlarged by the enlargement processing unit 113, only the high-frequency component is calculated by calculating the difference from the original image. The extracted image is generated.

  For this reason, the subtraction processing unit 112-1 generates an image composed of only the high-frequency component of each input image as a layer 0 image, and the subtraction processing unit 112-2 is lower than the layer 0 image of the input image. A frequency component image is generated as a layer 1 image. In addition, the subtraction processing unit 112-3 generates an image having a frequency component lower than the layer 1 image of the input image as a layer 2 image. Note that the image output from the reduction processing unit 111 is a layer 3 image having band components equal to or lower than layer 2. That is, it can be said that the multi-resolution conversion processing unit 81 performs band separation on the input image and outputs images of the layers 0 to 3.

  Next, the configuration of an embodiment of the function of the correction processing unit 86 will be described with reference to FIG.

  The sampling processing unit 131 samples (extracts) a plurality of G channel pixel values from neighboring pixels at a predetermined position set corresponding to the target pixel position from the G channel image of the relevant layer, and a multiplication processing unit 137. This is supplied to the variance calculation processing unit 133 and the covariance calculation processing unit 136. The sampling processing unit 132 samples (extracts) a plurality of R channel pixel values from neighboring pixels at a predetermined position set corresponding to the target pixel position from the R channel image of the relevant layer, and performs a covariance calculation process Supplied to the unit 136. Note that the sampling processing units 131 and 132 extract the G pixel value and the R pixel value at the same predetermined position set for the target pixel position.

  The variance calculation processing unit 133 calculates a variance value around the G pixel of interest based on the sampled G pixel value, and supplies the calculated variance value to the clipping processing unit 134.

  The covariance calculation processing unit 136 calculates the covariance values based on the sampled G and R pixel values and supplies them to the division processing unit 135. When the variance value of the G channel samples is smaller than a predetermined threshold, the clipping processing unit 134 performs clipping with the threshold and supplies the result to the division processing unit 135. The division processing unit 135 divides the covariance value supplied from the covariance processing unit 136 by the variance value supplied from the clipping processing unit 134, and the ratio of the covariance value to the variance value (covariance value / variance value). To the multiplication processing unit 137. Note that the processing of the clipping processing unit 135 is processing for avoiding 0% because the subsequent division processing unit 135 calculates (covariance value / dispersion value). The multiplication processing unit 137 multiplies (covariance value / dispersion value) by the G channel pixel value of the target pixel position to estimate the R channel pixel value from which the noise of the target pixel has been removed, and outputs it.

  Here, the details of the processing of the correction processing unit 86 will be described.

  The correction processing unit 86 corrects the pixel value by a pixel value estimation method using inter-channel correlation. More specifically, the pixel value estimation method using the correlation between channels is based on the assumption that there is a linear relationship between two channels (for example, G and R) when focusing on the local region. The estimated value of a certain pixel position R is calculated by linear regression calculation.

For example, in the local region around the target pixel in the image, a sample of pixel values of the C ₁ channel (for example, G) (pixel values of a plurality of pixels at predetermined positions corresponding to the target pixel position) {C ₁₁ , C ₁₂ , C ₁₃ ,..., C _1N } and C ₂ channel (for example, R) pixel value samples (pixel values of a plurality of pixels at predetermined positions corresponding to the position of the target pixel) {C ₁₁ , C ₁₂ , C ₁₃ ,..., C _1N } is obtained, the luminance estimated value C _2C ′ of the target pixel position of the C ₂ channel is obtained by assuming that there is a linear relationship between the _two. Can be estimated from the luminance value C _1C of the C ₁ channel by the following equation (1).

・・・（１）

... (1)

Here, M _C1 is the expected value of the C ₁ channel in the local region, M _C2 is the expected value of the C ₂ channel, V _C1C2 is the covariance value of the C ₁ and C ₂ channels, and V _C1C1 is C ₁ The variance value of the channel.

Further, the covariance value V _C1C2 and the variance value V _C1C1 are obtained by the following equations (2) and (3), respectively.

・・・（２）

... (2)

・・・（３）

... (3)

In Expressions (2) and (3), w _i is a predetermined weight coefficient.

  Therefore, noise can be removed by calculating the above-described equation (1). However, as shown in FIG. 4, the noise removal processing unit 43 is provided with a multi-resolution conversion processing unit 81, and the correction processing unit 86 performs the above-described two-channel processing on an image separated for each band. The pixel value is corrected using the correlation.

  By the way, in the image of a plurality of layers generated by the multi-resolution conversion, the direct current component of the image is concentrated on the layer of the lowest resolution, so that the expected value of the local pixel is 0 in the other layers. As a result, when using multi-resolution processing, the above-described equations (1) to (3) are replaced with the following equations (4) to (6).

・・・（４）

... (4)

・・・（５）

... (5)

・・・（６）

... (6)

  Accordingly, the variance value calculation processing unit 133 in FIG. 6 substantially calculates the variance value by Expression (6), the covariance calculation processing unit 136 calculates the covariance value by Expression (5), and further performs multiplication. The processing unit 137 corrects (corrects) the pixel value using Expression (4).

  Further, in the calculation processing of the covariance value and the variance value of the equations (5) to (6), the number of multiplication processing is reduced by using an approximation function in order to reduce the multiplication processing with a large processing load by the calculation processing by the computer. For example, it may be calculated by an approximation function such as the following formulas (7) to (8).

・・・（７）

... (7)

・・・（８）

... (8)

  Next, the configuration of an embodiment of the function of the multiresolution inverse transform processing unit 90 will be described with reference to FIG.

  The enlargement processing units 151-1 to 151-3 are the same as the enlargement processing unit 113 of the multi-resolution conversion processing unit 90, and enlarge the image size of the input image by supersampling, for example, twice (for example, 2) and output to the subsequent stage.

  The addition processing units 152-1 to 152-3 are layer 0 noise stored in the layer 0 noise removed image memory 87, the layer 1 noise removed image memory 88, the layer 2 noise removed image memory 89, and the layer 3 image memory 85, respectively. The removed image or the layer 2 noise-removed image and the layer 3 image are read out, added to the images supplied from the enlargement processing units 152-1 to 152-3, and output to the subsequent stage.

  That is, the enlargement processing unit 151-1 enlarges the layer 3 noise-removed image of the lowest frequency component, makes an image of the same size as the layer 2 noise-removed image, and supplies the image to the addition processing unit 152-1. The addition processing unit 152-1 adds the enlarged layer 3 noise-removed image and the layer 2 noise-removed image to generate an image having a band from layer 3 to layer 2, and the enlargement processing unit 151- 2 is supplied.

  Similarly, the enlargement processing unit 151-2 enlarges the noise-removed images of layers 3 and 2 to form an image having the same size as the layer 1 noise-removed image, and supplies the image to the addition processing unit 152-2. And the addition process part 152-2 produces | generates the image which has the zone | band from the layer 3 to the layer 1 by adding the noise removal image and the layer 1 noise removal image which were expanded to the layer 3 thru | or the layer 2, This is supplied to the enlargement processing unit 151-3.

  Further, the enlargement processing unit 151-3 enlarges the noise-removed images of layers 3 to 1 to form an image having the same size as the layer 0 noise-removed image and supplies the image to the addition processing unit 152-3. Then, the addition processing unit 152-3 adds the noise-removed images from layer 3 to layer 1 and the layer 0 noise-removed image to integrate the band images from layer 3 to layer 0, and finally Output an image with the same resolution and bandwidth as the original image

  Next, the configuration of an embodiment of the function of the nonlinear inverse transform processing unit 44 will be described with reference to FIG.

  The non-linear inverse transform processing unit 44 has substantially the same configuration as the non-linear transform processing unit 42 in order to perform the inverse transform of the transform in the non-linear transform processing unit 42. The conversion curve LUT 172 has the same configuration as the conversion curve LUT 62 of the non-linear conversion processing unit 42 and may be shared. The inverse mapping processing unit 171 performs nonlinear inverse transformation on each pixel of the input image with reference to the conversion curve LUT 172 and outputs the result to the white balance processing unit 45.

  Next, image processing by the DSP block 16 of FIG. 2 will be described with reference to the flowchart of FIG.

  The demosaic processing unit 41 determines whether or not a mosaic image has been input (whether or not it has been read from the frame memory) and repeats the processing until it is input. For example, when it is determined that an image has been input, in step S2, the demosaic processing unit 41 executes a demosaic process that generates an RGB three-channel image signal from the input mosaic image, and the generated RGB image signal Is supplied to the nonlinear conversion processing unit 42.

  In step S3, the mapping processing unit 61 of the non-linear conversion processing unit 44 performs non-linear conversion on each pixel of the input image based on the LUT having a function shape used for non-linear conversion from the conversion curve LUT 62, and a noise removing unit. Output to 43.

  In step S <b> 4, the noise removal unit 43 performs noise removal processing, removes noise for each pixel of the supplied one channel image, and supplies the noise to the nonlinear inverse processing unit 44. Details of the noise removal processing will be described later with reference to FIG.

  In step S5, the inverse mapping processing unit 171 of the nonlinear inverse transformation processing unit 44 refers to the transformation curve LUT 172 for each pixel of the input image so as to cancel the nonlinear characteristic applied by the nonlinear transformation processing in step S3. The intensity is converted into a white balance processing unit 45.

  In addition, about the process of step S3 thru | or S5, you may make it the nonlinear transformation process part 42, the noise removal process part 43, and the nonlinear inverse transformation process part 44 process in parallel with respect to each RGB image signal, You may make it process sequentially about each channel.

  In step S6, the white balance processing unit 45 multiplies each image signal by an appropriate coefficient to the RGB3 channel image from which noise has been removed and nonlinear characteristics have been canceled, so that the color balance of the subject region becomes achromatic. The white balance process is performed as described above, and the result is supplied to the gamma correction processing unit 46.

  In step S <b> 7, the gamma correction processing unit 46 gamma-corrects each pixel intensity of the RGB image signals whose white balance has been adjusted by the white balance processing unit 45 and outputs the gamma correction to the YC conversion processing unit 47.

  In step S8, the YC conversion processing unit 47 executes YC conversion processing for performing YC matrix processing on the RGB image signal subjected to the gamma correction processing by the gamma correction processing unit 46, thereby converting the image into a Y image and a C image. The Y image and the C image are output (written to the frame memory).

  With the above processing, a YC image from which noise has been removed is generated.

  Next, the noise removal process which is the process of step S4 in the flowchart of FIG. 9 will be described with reference to the flowchart of FIG.

  In step S21, the multi-resolution conversion processing units 81-1 and 81-2 determine whether or not images for the G and R channels subjected to the nonlinear conversion processing generated by the nonlinear conversion processing unit 42 are input, respectively. The process is repeated until it is determined that the input has been made. If it is determined that the input has been made, the process proceeds to step S22.

  In step S22, the multi-resolution conversion processing unit 81-1 performs multi-resolution conversion processing, generates a plurality of multi-resolution images from the input G image, and generates the generated images for each layer in a layer 0 image memory. 82-1, the layer 1 image memory 83-1, the layer 2 image memory 84-1, and the layer 3 image memory 85-1. The details of the multi-resolution conversion processing will be described later with reference to the flowchart of FIG.

  In step S23, the multi-resolution conversion processing unit 82-1 executes multi-resolution conversion processing, generates a plurality of multi-resolution images from the input R image, and generates the generated images for each layer in the layer 0 image memory. 82-2, the layer 1 image memory 83-2, the layer 2 image memory 84-2, and the layer 3 image memory 85-2. The multi-resolution processing is the same processing except that the signal component is R instead of G.

  In step S24, the correction processing units 86-1 to 86-3 execute pixel value correction processing using the G and R signals of layers 0 to 2, respectively, correct the R pixel value, and The layer 0 corrected image memory 87, the layer 1 corrected image memory 88, and the layer 2 corrected image memory 89 are stored. Details of the pixel value correction processing will be described later with reference to the flowchart of FIG.

  In step S <b> 25, the multi-resolution inverse transform processing unit 90 performs multi-resolution inverse transform processing, and stores the pixels stored in the layer 0 modified image memory 87, the layer 1 modified image memory 88, and the layer 2 modified image memory 89. By performing inverse transformation of multi-resolution transformation by the process of step S23 on the images of layers 0 to 2 whose values are corrected and the R image stored in the layer 3 image memory 85, a plurality of multi-resolutions are obtained. The images are integrated, and the integrated R image is output in step S26. The details of the multi-resolution inverse transform processing will be described later with reference to the flowchart of FIG.

  Through the above processing, the R pixel value is corrected based on the correlation with G and R.

  Next, the multi-resolution conversion process, which is the process of steps S22 and S23 in the flowchart of FIG. 10, will be described with reference to the flowchart of FIG.

  In step S41, the multi-resolution conversion processing unit 81-1 sets a counter N (not shown) to 0 and initializes it.

  In step S42, the reduction processing unit 111 reduces the image reduced N times. That is, in the first process, N is 0, that is, the input image is reduced by the reduction processing unit 111-1.

  In step S43, the multi-resolution conversion processing unit 81-1 increments a counter N (not shown) by 1.

  In step S44, the multi-resolution conversion processing unit 81-1 determines whether or not the counter N is the maximum value Nmax-1 of the counter N. Here, Nmax is the number of layers for converting the input image into a multi-resolution image. Therefore, the number of layers is four as shown in FIG.

  For example, when it is determined in step S44 that the counter N is not the maximum value (Nmax−1) of the counter N, in step S45, the reduction processing unit 111 reduces the image reduced N times to the subtraction processing unit 112 and The result is output to the enlargement processing unit 113. That is, in the first process, the reduction processing unit 111-1 outputs an image reduced once to the subtraction processing unit 112-2 and the enlargement processing unit 113-1.

  In step S <b> 46, the enlargement processing unit 113 enlarges the image reduced N times and outputs it to the subtraction processing unit 112. That is, in the first process, the enlargement processing unit 113-1 enlarges the image that has been reduced once supplied from the reduction processing unit 111-1, and supplies the enlarged image to the subtraction processing unit 112-1.

  In step S47, the subtraction processing unit 112 subtracts an image that has been reduced N times and enlarged once from an image that has been reduced (N-1) times. That is, in the first process, the subtraction processing unit 112-1 subtracts the image supplied from the enlargement processing unit 113-1 from the input image.

  In step S48, the subtraction processing unit 112 stores the layer (N-1) image obtained by the processing in step S47 in the layer (N-1) image memory, and the processing returns to step S42. That is, in the first process, the subtraction processing unit 112-1 stores the image obtained by the subtraction process in the layer 0 image memory 82-1.

  That is, in the process of step S44, the processes of steps S42 to S48 are repeated until it is determined that the counter N is the maximum value (Nmax-1). As a result, the reduction processing unit 111-2 sequentially reduces the image reduced once and supplies it to the enlargement processing unit 113-2 and the subtraction processing unit 113-3, and the enlargement processing unit 113-2 supplies the image. The two-time-reduced image is enlarged and supplied to the subtraction processing unit 112-2. The subtraction processing unit 112-2 has been reduced and enlarged twice from the image reduced once. The image is subtracted to generate a layer 1 image and stored in the layer 1 image memory 83-1.

  If it is determined in step S44 that the counter N is the maximum value (Nmax-1), in step S49, the reduction processing unit 111 converts the image reduced N times into the layer N image memory and the enlargement processing unit 113. Output to. That is, since Nmax is 4, the reduction processing unit 111-3 outputs an image reduced three times to the subtraction processing unit 112-3 and the layer 3 memory 85.

  In step S <b> 50, the enlargement processing unit 113 enlarges the image reduced N times and outputs it to the subtraction processing unit 112. That is, here, the enlargement processing unit 113-3 enlarges the image reduced three times supplied from the reduction processing unit 111-3, and supplies the enlarged image to the subtraction processing unit 112-3.

  In step S51, the subtraction processing unit 112 subtracts an image that has been reduced N times and enlarged once from an image that has been reduced (N-1) times. That is, here, the subtraction processing unit 112-3 subtracts the image supplied from the enlargement processing unit 113-3 from the image reduced twice.

  In step S52, the subtraction processing unit 112 stores the layer (N-1) image obtained by the processing in step S51 in the layer (N-1) image memory, and the processing ends. That is, here, the subtraction processing unit 112-3 stores the image obtained by the subtraction processing in the layer 2 image memory 83.

  With the above processing, the input images are separated by band and stored in the layer 0 image memory 82, the layer 1 image memory 83, the layer 2 image memory 84, and the layer 3 image memory 85 for each layer. Become.

  Next, the pixel value correction process that is the process of step S24 in the flowchart of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S71, the sampling processing units 131 and 132 of the correction processing units 86-1 to 86-3 are the layer 0 image memories 82-1 and 82-2, the layer 1 image memories 83-1 and 83-2, and the layer, respectively. It is determined whether or not R and G channel images have been input to the two-image memories 84-1 and 84-2, and the processing is repeated until it is determined that they have been input. For example, if it is determined that the input has been made, the process proceeds to step S72.

  In step S72, the sampling processing units 131 and 132 select the same unprocessed pixel as the target pixel.

  In step S73, the sampling processing unit 131 is arranged at a predetermined position corresponding to the target pixel position from the layer 0 image memory 82-1, the layer 1 image memory 83-1, or the layer 2 image memory 84-1 of the G channel. The plurality of pixels are sampled (extracted) and supplied to the variance calculation processing unit 133, the covariance calculation processing unit 134, and the multiplication processing unit 137.

  In step S74, the sampling processing unit 132 is arranged at a predetermined position corresponding to the target pixel position from the layer 0 image memory 82-2, the layer 1 image memory 83-2, or the layer 2 image memory 84-2 of the R channel. The plurality of pixels are sampled (extracted) and supplied to the covariance calculation processing unit 134.

  In step S <b> 75, the variance calculation processing unit 133 executes a variance calculation process from the G pixel value sample supplied from the sampling processing unit 131, calculates a variance value, and supplies the variance value to the clipping processing unit 134. Details of the distributed calculation processing will be described later with reference to the flowcharts of FIGS.

  In step S <b> 76, the covariance calculation processing unit 136 executes covariance calculation processing from the G and R pixel value samples supplied from the sampling processing units 131 and 132 to calculate a covariance value. Supply. Details of the covariance calculation processing will be described later with reference to the flowchart of FIG.

  In step S 77, the clipping processing unit 134 performs clipping processing on the G variance value supplied from the variance calculation processing unit 133, and supplies it to the division processing unit 135. More specifically, the clipping processing unit 134 compares the G variance value supplied from the variance calculation processing unit 133 with a predetermined threshold value. If the clipping value is smaller than the predetermined threshold value, the clipping processing unit 134 sets the G variance value to the predetermined threshold value. To the division processing unit 135, and when it is larger than the predetermined threshold, the G variance value is supplied to the division processing unit 135 as it is.

  In step S78, the division processing unit 135 calculates (covariance / variance) based on the variance value supplied from the clipping processing unit 134 and the covariance value supplied from the covariance calculation processing unit 136. And supplied to the multiplication processing unit 137.

  In step S79, the multiplication processing unit 137 corrects the pixel value of the R pixel at the target pixel position by multiplying (covariance / dispersion) by the G pixel value at the target pixel position. In other words, the multiplication processing unit 137 corrects the R pixel value by using the correlation between G and R by executing the above-described equation (4).

  In step S80, the sampling processing units 131 and 132 determine whether or not there is an unprocessed pixel. If it is determined that there is an unprocessed pixel, the processing returns to step S72. That is, until it is determined that there is no unprocessed pixel, the processes in steps S72 to S80 are repeated, and the R pixel value is sequentially corrected.

  If it is determined in step S80 that there is no unprocessed pixel, in step S81, the multiplication processing unit 137 outputs an image in which the R channel pixel value calculated so far is corrected.

  Through the above processing, it is possible to remove the noise of a chroma component caused by a random component or chromatic aberration by correcting the pixel value using the correlation between channels.

  Next, with reference to the flowchart of FIG. 13, the distributed calculation process, which is the process of step S75 in the flowchart of FIG. 18, will be described.

  In step S91, the variance calculation processing unit 133 initializes a counter Vt, w that is output as an operation result to 0.

  In step S <b> 92, the variance calculation processing unit 133 performs processing from among pixel values of a part of the local region (for example, a range of (n−2) × (n−2) centered on the target pixel). A value s of the first register indicating the pixel position to be executed is initialized to s = 1, and in step S93, a part of the local region (for example, (n−2) × Among the pixel values in the range (n-2)), the value t of the second register indicating the pixel position where the process is executed is initialized to t = 1.

  In step S94, the variance calculation processing unit 133 squares a value obtained by multiplying the intensity X of the pixel (s, t) by the weight wi of the pixel (s, t), and adds the value added to the counter Vt to the counter Vt. Update as. That is, the variance calculation processing unit 133 calculates Vt = Vt + (| (intensity X of pixel (s, t)) × (weight wi | of pixel (s, t)) 2.

  In step S95, the variance calculation processing unit 133 updates the value obtained by adding the weight wi of the pixel (s, t) to the counter w as the counter w. That is, the variance calculation processing unit 133 calculates w = w + (weight wi of pixel (s, t)).

  In step S96, the variance calculation processing unit 133 refers to the value t of the second register indicating the pixel position where the process is executed, and determines whether t = n−2.

  If it is determined in step S96 that t = n−2 is not satisfied, in step S147, the variance calculation processing unit 133 updates the value t of the second register to t = t + 1, and the process proceeds to step S94. Return to, and the subsequent processing is repeated.

  If it is determined in step S96 that t = n−2, in step S98, the variance calculation processing unit 133 refers to the value s of the first register indicating the pixel position where the process is performed, and s = n -2 is determined.

  If it is determined in step S98 that s = n−2 is not satisfied, in step S99, the variance calculation processing unit 133 updates the value s of the first register to s = s + 1, and the process proceeds to step S93. Return to, and the subsequent processing is repeated.

  If it is determined in step S98 that s = n−2, in step S100, the variance calculation processing unit 133 determines a part of the local area (for example, (n−2) The dispersion value Vxx of the pixel values in the range of (× (n−2)) is calculated as Vt / w, and the process ends.

For the calculation of the variance value, by definition, ((intensity X of pixel (s, t)) × (weight of pixel (s, t) wi)) ² is divided by the number of pixels used to calculate the variance value. Although it is necessary, the calculation result of the variance value is used for division of the calculation result of the covariance value by the process described later. It is possible to omit both the value calculation and the covariance value calculation.

In the calculation of the dispersion value, since weighting is performed here, as shown in the definition of the dispersion value described using Expression (6), ((intensity X of pixel (s, t)) × (Weight of pixel (s, t) wi)) The variance value is calculated by dividing ² by the sum of the weight coefficients. However, since the calculation result of the variance value is used to divide the calculation result of the covariance value by the processing described later, when weighting is also applied in the calculation of the covariance value, the division by the sum of the weighting factors is performed. The processing can be omitted in both the variance value calculation and the covariance value calculation.

  Further, in the distributed calculation processing described with reference to FIG. 13, there is a problem that the calculation processing takes time and the scale of hardware increases because there is a square calculation. Therefore, the square calculation may be omitted by approximate calculation.

  Next, the distributed calculation process executed in step S75 in the flowchart of FIG. 12 when the square calculation is omitted by the approximate calculation will be described with reference to the flowchart of FIG.

  In steps S121 to S123 in the flowchart of FIG. 14, the same processes as those in steps S91 to S93 described with reference to FIG. 13 are executed.

  In step S124, the variance calculation processing unit 133 adds (| 2 × (intensity X of pixel (s, t) X) × (weight (wi) of pixel (s, t)) |) to the current counter Vt. Update as. That is, the variance calculation processing unit 133 calculates Vt = Vt + (| 2 × (intensity X of pixel (s, t)) × (weight (wi) of pixel (s, t)) |).

  When the value | p | is normalized and 0 ≦ | P | <1, p2 can be approximated by | p | × 2.

  In steps S125 to S130, the same processing as that in steps S95 to S100 described with reference to FIG. 13 is executed, and the processing ends.

  As described above, it is possible to obtain an approximate value of the variance value by an approximation operation that omits the square calculation necessary for the calculation of the variance value.

  Next, the covariance calculation process executed in step S76 of FIG. 12 will be described with reference to the flowchart of FIG.

  In step S151, the covariance calculation processing unit 136 outputs counters Vtt, w (here, all the first signals have the G intensity and the second signal has the R intensity). ) Is initialized to 0.

  In step S152, the covariance calculation processing unit 136 performs processing among pixel values of a part of the local region (for example, a range of (n−2) × (n−2) centered on the target pixel). Initialize the value s of the first register indicating the pixel position for executing s = 1

  In step S153, the covariance calculation processing unit 136 performs processing among pixel values of a part of the local region (for example, a range of (n−2) × (n−2) centered on the target pixel). Initialize the value t of the second register indicating the pixel position for executing t = 1

  In step S154, the product-sum processing described later with reference to FIG. 16 or FIG. 17 is executed.

  In step S155, the covariance calculation processing unit 136 updates the value obtained by adding the weight wi of the pixel (s, t) to the counter w as the counter w. That is, the covariance calculation processing unit 136 calculates w = w + (weight (wi) of pixel (s, t)).

  In step S156, the covariance calculation processing unit 136 refers to the value t of the second register indicating the pixel position where the process is executed, and determines whether t = n−2.

  If it is determined in step S156 that t = n−2 is not satisfied, the covariance calculation processing unit 136 updates the value t of the second register to t = t + 1 in step S157, Returning to S154, the subsequent processing is repeated.

  If it is determined in step S156 that t = n−2, in step S158, the covariance calculation processing unit 136 refers to the value s of the first register indicating the pixel position where the process is performed, and s = It is determined whether or not n-2.

  If it is determined in step S158 that s = n−2 is not satisfied, the covariance calculation processing unit 136 updates the value s of the first register to s = s + 1 in step S159, Return to S153.

  When it is determined in step S158 that s = n−2, in step S159, the covariance calculation processing unit 136 calculates the covariance value Vxy as Vtt / w, and the process ends.

  The covariance value is calculated by calculating (the intensity X of the pixel (s, t)) × (the intensity Y of the pixel (s, t)) × (the weight wi of the pixel (s, t)). Although it is necessary to divide by the number of pixels used, the calculation result of the covariance value is divided by the calculation result of the dispersion value by the process described later. The division process can be omitted in both the variance value calculation and the covariance value calculation.

  In Equation (5), in calculating the covariance value, (intensity X of pixel (s, t)) × (intensity Y of pixel (s, t)) × (weight wi of pixel (s, t)) ) Is divided by the sum w of the weighting coefficients to calculate a covariance value. However, since the calculation result of the covariance value is divided by the calculation result of the dispersion value by a process described later, when weighting is applied also in the calculation of the dispersion value, the division process by the sum of the weighting coefficients is performed. It is possible to omit both the calculation of the variance value and the calculation of the covariance value.

  In the product-sum process executed in step S154 of the above-described covariance calculation process of FIG. 15, the covariance value is calculated using the product-sum process executed when the covariance value is calculated and the approximation operation. There are two types of processing: product-sum processing executed in the

  Next, the product-sum processing executed when the covariance value executed in step S154 in FIG. 15 is calculated will be described with reference to the flowchart in FIG.

  In step S181, the covariance calculation processing unit 136 calculates (the intensity X of the pixel (s, t) of the first signal), (the intensity Y of the pixel (s, t) of the second signal), and (pixel ( Multiply the weights wi) of s, t).

  In step S182, the covariance calculation processing unit 136 sets Vtt = Vtt + (multiplication result).

  By executing such a product-sum process, a covariance value is calculated. When the multiply-accumulate process described with reference to the flowchart of FIG. 16 is executed, the calculation is as defined, so the result of the calculation is very accurate, but instead, there are many multiply-accumulate processes. As a result, the computation takes time and the number of gates in hardware implementation increases.

  Next, the product-sum process executed in step S76 in FIG. 12 will be described with reference to the flowchart in FIG. The product-sum process described with reference to the flowchart of FIG. 17 is a process that is executed when an approximate covariance value is calculated.

  In step S201, the covariance calculation processing unit 136 sets {intensity X of the pixel (s, t) of the first signal} to p.

  In step S202, the covariance calculation processing unit 136 sets {intensity Y of the pixel (s, t) of the second signal} to q.

  In step S203, multiplication approximation processing described later with reference to FIG. 18 is executed.

  In step S204, the covariance calculation processing unit 136 sets Vtt = Vtt + (approximate value of pq) × (weight wi of pixel (s, t)), and the process ends.

  Next, the multiplication approximation process executed in step S203 in FIG. 17 will be described with reference to the flowchart in FIG. The multiplication approximation process is a process executed when an approximate covariance value is calculated using the above-described equation (7).

  In step S221, the covariance calculation processing unit 136 determines whether or not | p | ≧ | q | using the values p and q replaced in steps S201 and S202 of FIG.

  If it is determined in step S221 that | p | ≧ | q |, in step S262, the covariance calculation processing unit 136 determines whether p ≧ 0.

  If it is determined in step S222 that p ≧ 0, in step S223, the covariance calculation processing unit 136 sets the approximate value of pq to + 2q, and the process returns to step S204 in FIG.

  If it is determined in step S222 that p ≧ 0 is not satisfied, in step S224, the covariance calculation processing unit 136 sets the approximate value of pq to −2q, and the process returns to step S204 in FIG.

  When it is determined in step S221 that | p | ≧ | q | is not satisfied, in step S225, the covariance calculation processing unit 136 determines whether q ≧ 0.

  If it is determined in step S225 that q ≧ 0, in step S226, the covariance calculation processing unit 136 sets the approximate value of pq = + 2p, and the process returns to step S204 in FIG.

  If it is determined in step S225 that q ≧ 0 is not satisfied, in step S227, the covariance calculation processing unit 136 sets the approximate value of pq to −2p, and the process returns to step S204 in FIG.

  In this process, when q or p is 0, the approximate value of pq is always 0. Specifically, when q is 0, | p | ≧ | q | always holds, so that the approximate value of pq is 0 regardless of the value of p. Also, when p is 0, | p | ≧ | q | does not always hold, so the approximate value of pq is 0 regardless of the value of q.

  By executing the product-sum processing described with reference to FIGS. 17 and 18, the covariance value can be approximated using the above-described equation (7). In this case, the calculation speed is increased compared to the case where the calculation of the covariance value as defined in Expression (5) is executed, or the hardware gate implemented for the calculation is used. Advantages such as being able to reduce the number occur.

  Next, with reference to the flowchart of FIG. 19, the multi-resolution inverse transform process that is the process of step S25 in the flowchart of FIG. 10 will be described.

  In step S271, the multiresolution inverse transform processing unit 90 sets a counter N (not shown) to Nmax-1. This counter is set according to the number of multiple resolution images. In the multiple resolution inverse transform processing unit 90 of FIG.

  In step S272, the enlargement processing unit 151 enlarges the image of the lowest layer. That is, in the case of FIG. 7, the enlargement processing unit 151-1 enlarges the image stored in the layer 3 image memory 85 that is the lowest layer, and supplies it to the addition processing unit 152-1.

  In step S273, the multi-resolution inverse transform processing unit 90 decrements the counter N by 1.

  In step S274, the multiresolution inverse transform processing unit 90 determines whether or not the counter N is zero. For example, when the counter N is not 0, in step S275, the addition processing unit 152 adds the layer N corrected image to the image enlarged by the enlargement processing unit 151 and outputs the image to the enlargement processing unit 151 on one layer. . That is, in the first process, the addition processing unit 152-1 adds the layer 2 corrected image stored in the layer 2 corrected image memory 89 to the image supplied from the enlargement processing unit 151-1. 151-2.

  In step S275, the enlargement processing unit 151 enlarges the image to which the layer N corrected image supplied from the addition processing unit 152 is added, supplies the image to the addition processing unit 152 on one layer, and returns to the process of step S274. That is, in the case of the first process, the enlargement processing unit 151-2 enlarges the image supplied from the addition processing unit 152-1, and supplies the enlarged image to the addition processing unit 152-2.

  That is, the processes in steps S273 to S276 are repeated until it is determined in step S274 that the counter N is 0. In the case of FIG. 7, the addition processing unit 152-2 further adds the layer 1 corrected image stored in the layer 1 corrected image memory 88 to the image supplied from the expansion processing unit 151-2, 151-3.

  Further, the enlargement processing unit 151-3 enlarges the image supplied from the addition processing unit 152-2 and supplies the enlarged image to the addition processing unit 152-3. After this processing, the counter N becomes 0, so that it is determined in step S274 that the counter N is 0. In step S277, the addition processing unit 152 converts the layer N, that is, the layer 0 corrected image into the layer corrected image. The image read from the memory 87 and enlarged by the enlargement processing unit 151 is added. That is, here, the addition processing unit 152-3 adds the layer 0 corrected image that is the highest layer to the image enlarged by the enlargement processing unit 151-3, and outputs the image in step S278.

  Through the above processing, the layer 0 modified image, the layer 1 modified image, the layer 2 modified image, and the layer 3 image can be integrated into the original resolution image by inversely converting the multiplexed resolution image.

  According to the above, by using the correlation between channels, by correcting (estimating) the pixel values of other channel signals so that the correlation with the reference channel (for example, G) signal is as high as possible, It is possible to reduce uncorrelated components between channels. For this reason, it is possible to remove noise mixed in the chroma component that is an uncorrelated component between channels, and as a result, it is possible to generate an image signal in which the noise of the chroma component is reduced.

  In the above description, the pixel value is estimated using the R variance value and the G and R covariance values. However, there are two types of chroma component noise, random components and chromatic aberration, both of which have the property of weakening the inter-channel correlation due to the mixture of both. Therefore, if the inter-channel correlation is sufficiently strong, the pixel value is corrected by the above-described linear regression calculation. However, if the inter-channel correlation is not sufficiently strong, the signal of one channel is cut off and replaced with the other channel signal. It may be.

  In FIG. 20, when the correlation between channels is sufficiently strong, the pixel value is corrected by the above-described linear regression calculation. However, when the correlation between channels is not sufficiently strong, the signal of one channel is blocked and the signal of the other channel is blocked. The structure of one Embodiment of the function of the correction process part 86 made to replace is shown.

  In FIG. 20, the sampling processing units 201-1 and 201-2, the variance calculation processing unit 202-1, the covariance calculation processing unit 203, the clipping processing unit 204, the division processing unit 205, and the multiplication processing unit 206 are the same as those in FIG. Since the sampling processing units 131 and 132, the variance calculation processing unit 133, the covariance calculation processing unit 136, the clipping processing unit 134, the division processing unit 135, and the multiplication processing unit 137 are executed, the description thereof will be given. Omitted.

  20 differs from the correction processing unit 86 in FIG. 6 in that a variance calculation processing unit 202-2, a correlation coefficient calculation processing unit 207, and a switch 208 are newly provided. . Also, the variance calculation processing unit 202-1 supplies the calculated variance value to the correlation coefficient calculation processing unit 207, and the covariance calculation processing unit 203 newly calculates the calculated covariance value. It supplies to the part 207. Further, the multiplication processing unit 206 supplies the multiplication result to the switch 208.

  The variance calculation processing unit 202-2 calculates an R channel variance value from a sample of the R channel pixels near a predetermined position corresponding to the target pixel, and supplies the calculated value to the correlation coefficient calculation processing unit 207.

  The correlation coefficient calculation processing unit 207 calculates the G variance value calculated by the variance calculation processing unit 202-1, the R variance value calculated by the variance calculation processing unit 202-2, and the covariance calculation calculation processing unit 203. The correlation coefficient of R and G (= (covariance value) / (√ (G dispersion value) × √ (R dispersion value))) is calculated from the R and G covariance values and supplied to the switch 208 To do.

  The switch 208 compares the R and G correlation coefficients supplied from the correlation coefficient calculation processing unit 207 with a predetermined threshold value. If the correlation coefficient is larger than the predetermined threshold value, the switch 208 corrects the correlation coefficient. When the R channel pixel value of the target pixel is output and the correlation coefficient is smaller than the predetermined threshold, the R pixel value corrected by the multiplication processing unit 206 is blocked and the G channel pixel value of the target pixel is output as it is.

  Next, the pixel value correction processing by the correction processing unit 86 in FIG. 20 will be described with reference to the flowchart in FIG. The processes in steps S291 to S299 and steps S305 and S306 in the flowchart of FIG. 21 are the same as the processes in steps S71 to S79 and steps S80 and S81 described with reference to the flowchart of FIG. Is omitted.

  In step S300, the variance calculation processing unit 202-2 executes a variance calculation process from the R pixel value sample supplied from the sampling processing unit 201-2 to calculate a variance value, and the correlation coefficient calculation processing unit 207 receives the calculation. Supply.

  In step S301, the correlation coefficient calculation processing unit 207 outputs the G and R variance values supplied from the variance calculation processing units 202-1 and 202-2, and the G and R supplied from the covariance calculation processing unit 203, respectively. The correlation coefficient (= (GR covariance value) / (√ (G variance value) × √ (R variance value))) is calculated and supplied to the switch 208.

  In step S302, the switch 208 determines whether or not the correlation coefficient is larger than a predetermined threshold. For example, if it is determined that the correlation coefficient is larger, the correlation between G and R supplied from the multiplication processing unit 206 in step S303. Based on, the corrected R pixel value (= (covariance) / (dispersion) × (G luminance value of the target pixel)) is selected, and the process proceeds to step S305.

  On the other hand, when it is determined in step S302 that the correlation coefficient is smaller than the predetermined threshold, in step S304, the switch 208 selects the G luminance value supplied from the sampling processing unit 201-1, and the processing is performed. Advances to step S305.

  With the above processing, when the correlation between channels is weak, it is determined that the noise mixed in the chroma component is large, and the G channel is output rather than performing linear regression from a sample with large noise. As a result, it is possible to actively reduce the noise of the chroma component according to the correlation between the channels.

  In the above, the example of determining the strength of correlation between channels using the correlation coefficient has been described. However, the strength of correlation between channels may be determined by other methods, for example, two variances You may make it determine the strength of the correlation between channels by the comparison with the absolute value difference of a value, and a predetermined threshold value.

  FIG. 22 shows the configuration of an embodiment of the function of the correction processing unit 86 that determines the strength of correlation between channels based on the absolute value of the difference between two dispersion values.

  Note that the sampling processing units 221 and 221-2, the variance calculation processing units 222-1 and 222-2, the covariance calculation processing unit 223, the mapping processing unit 224, and the division processing unit 225 in the configuration of the correction processing unit 86 in FIG. , And multiplication processing unit 226, sampling processing units 201-, 201-2, variance calculation processing units 202-1, 202-2, covariance calculation processing unit 203, and mapping processing unit 204 in correction processing unit 86 of FIG. Since it is the same as the division processing unit 205 and the multiplication processing unit 206, the description thereof is omitted.

  22 differs from the configuration of the correction processing unit 86 in FIG. 20 in that a difference calculation processing unit 227 and a switch 228 are provided instead of the correlation coefficient calculation processing unit 207 and the switch 208. Is a point. The difference calculation processing unit 227 obtains an absolute difference value of the respective variance values of the G and R channels and supplies it to the switch 228. The switch 228 compares the absolute difference value with a predetermined threshold, determines the strength of the correlation according to the comparison result, and if the correlation is weak, selects the signal supplied from the sampling processing unit 221-1, and the correlation is If it is strong, the signal supplied from the multiplication processing unit 226 is selected.

  Next, the pixel value correction processing by the correction processing unit 86 in FIG. 22 will be described with reference to the flowchart in FIG. The processes in steps S321 to S330 and steps S333 to S336 in the flowchart of FIG. 22 are the same as the processes in steps S291 to S300 and steps S303 to S306 described with reference to the flowchart of FIG. Is omitted.

  In step S331, the difference calculation processing unit 227 calculates the absolute difference value (= | (G dispersion value) −) based on the G and R dispersion values supplied from the dispersion calculation processing units 222-1 and 222-2, respectively. (Variance value of R) |) is calculated and supplied to the switch 208.

  In step S332, the switch 228 determines whether or not the absolute difference value is larger than a predetermined threshold value. For example, if it is determined that the difference is larger, the correlation between G and R supplied from the multiplication processing unit 206 in step S333. Based on, the corrected R pixel value (= (covariance) / (dispersion) × (G luminance value of the target pixel)) is selected, and the process proceeds to step S335.

  On the other hand, when it is determined in step S332 that the correlation coefficient is smaller than the predetermined threshold value, in step S334, the switch 228 selects the G luminance value supplied from the sampling processing unit 221-1 and performs the processing. Advances to step S335.

  With the above processing, when the correlation between channels as seen from the absolute difference value of the variance value is weak, it is determined that the noise mixed in the chroma component is large, and the G channel is output rather than performing linear regression from a sample with large noise. As a result, it is possible to actively reduce the noise of the chroma component according to the correlation between the channels. When there is an edge color shift caused by chromatic aberration in the object outline region, the color shift is easily detected as a difference in dispersion value between channels. As a result, it is possible to effectively remove the noise caused by the color shift of the edge caused by such chromatic aberration by the above processing. In addition, since the multiplication process required when calculating the correlation coefficient is not necessary, it is possible to reduce the calculation load by the computer.

  In the above, based on the correlation between two channels such as G and R, or G and B, the corrected pixel value or the process of selecting either the G pixel value has been described. You may make it improve the noise removal effect of a chroma component more by evaluating the correlation between three or more channels simultaneously.

  FIG. 24 is a diagram illustrating a configuration of an embodiment of a function of the noise removal processing unit 43 that simultaneously evaluates the correlation between the three channels and removes the noise of the chroma component. Note that the multi-resolution conversion processing units 251-1 to 251-3, the layer 0 image memories 252-1 to 252-2, the layer 1 image memories 253-1 to 253-3, the layer 2 in the noise removal processing unit 43 of FIG. Image memories 254-1 to 254-3, layer 3 image memories 255-1 to 255-3, layer 0 modified image memories 257-1 and 257-2, layer 1 modified image memories 255-1 and 255-2, layer 2 The modified image memories 259-1 and 259-2 and the multi-resolution inverse conversion processing units 260-1 and 260-2 are the multi-resolution conversion processing units 81-1 and 81-2 and the layer 0 image memory 82-1 shown in FIG. 4. , 82-2, layer 1 image memories 83-1, 83-2, layer 2 image memories 84-1, 84-2, layer 3 image memories 85-1, 85-2, layer 0 corrected image Memory 87, the layer 1 modified image memory 88, the layer 2 modified image memory 89, and because they are similar to the multi-resolution inverse-conversion processing unit 90, and a description thereof will be omitted. However, the multi-resolution inverse transform processing units 260-1 and 260-2 output the corrected R and B pixel values, respectively. Further, even in the case of a multi-channel camera system having more than three channels, it is possible to cope with such a configuration by directly adding a portion corresponding to the R or B channel signal to the configuration shown in FIG. Here, a case of a three-channel configuration will be described.

  In the noise removal processing unit 43 of FIG. 24, a new configuration is that correction processing units 256-1 to 256-3 are provided. The correction processing units 256-1 to 256-3 correct the R and B pixel values so as to remove the chroma component noise in the R and B image signals based on the RGB three-channel correlation, respectively. The data is stored in the layer 0 corrected image memories 257-1 and 257-2, the layer 1 corrected image memories 258-1 and 258-2, and the layer 2 corrected image memories 259-1 and 259-2.

  Next, the configuration of an embodiment of the function of the correction processing unit 256 will be described with reference to FIG.

  The sampling processing units 281-1 to 281-3 sample (extract) a plurality of GRB channel pixel values from neighboring pixels at a predetermined position set corresponding to the target pixel position from each GRB channel image of the corresponding layer. And supplied to the variance-covariance correlation coefficient calculation processing unit 282.

  The variance-covariance correlation coefficient calculation processing unit 282 obtains a variance value around the pixel of interest of the GRB based on the sampled GRB pixel value, and supplies the G variance value to the clipping processing unit 283. Also, the variance-covariance correlation coefficient calculation processing unit 282 obtains covariance values around the target pixel of GR and GB based on the sampled GRB pixel values, and respectively supplies them to the division processing units 284-1 and 284-2. Supply. Further, the variance-covariance correlation coefficient calculation processing unit 282 supplies the GRB variance value, the G-R and G-B peripheral pixel covariance values, and the G-R and G-B correlation coefficients to the determination processing unit 286.

  When the variance value of the G channel samples is smaller than a predetermined threshold, the clipping processing unit 283 performs clipping with the threshold and supplies the result to the division processing units 284-1 and 284-2. The division processing units 284-1 and 284-2 divide the GR and GB covariance values supplied from the variance / covariance correlation coefficient calculation processing unit 282 by the variance values supplied from the clipping processing unit 283, respectively. The ratios of the covariance values to the values (covariance values / dispersion values) are supplied to the multiplication processing units 285-1 and 285-2, respectively. Multiplication processing units 285-1 and 285-2 correct the noise-removed R and B channel luminance values of the pixel of interest by multiplying (covariance value / variance value) by the G channel luminance value of the pixel of interest, These are supplied to switches 287 and 288, respectively.

  The determination processing unit 286 uses the GRB variance value of the GRB channel calculated by the variance-covariance correlation coefficient calculation processing unit 282, the covariance value around the target pixel of GR and GB, and the correlation coefficient of GR and GB. Based on this, it is determined whether or not the correction value of the R channel pixel and the correction value of the B channel pixel of the target pixel calculated by the multiplication processing units 281-1 and 285-2 can be used, and the determination result is sent to the switches 287 and 288. Supply.

  The switches 287 and 288 correspond to the RB channels, respectively, and correct the R channel pixel of the target pixel calculated by the multiplication processing units 285-1 and 285-2 based on the determination result supplied from the determination processing unit 286. Either the value and the correction value of the B channel pixel are selected, or the G channel pixel value of the target pixel is selected.

  Next, the noise removal processing by the noise removal processing unit 43 in FIG. 24 will be described with reference to the flowchart in FIG.

  In step S351, the multi-resolution conversion processing units 251-1 to 251-3 determine whether or not the images for the GRB channels subjected to the nonlinear conversion processing generated by the nonlinear conversion processing unit 42 are input. The process is repeated until it is determined that the input has been made, and if it is determined that the input has been made, the process proceeds to step S352.

  In steps S352 to S354, the multi-resolution conversion processing units 251-1 to 251-3 each execute multi-resolution conversion processing, generate a plurality of multi-resolution images from the input GRB images, and generate the generated images. For each layer, layer 0 image memories 252-1 to 252-2, layer 1 image memories 253-1 to 253-3, layer 2 image memories 254-1 to 254-3, and layer 3 image memories 255-1 to 255, respectively. -3. The multi-resolution conversion process for each GRB is the same as the process described with reference to the flowchart of FIG.

  In step S355, the correction processing units 256-1 to 256-3 execute the pixel value correction processing using the GRB signals of the layers 0 to 2, respectively, and correct the R and B pixel values. The data is stored in the layer 0 corrected image memories 257-1 and 257-2, the layer 1 corrected image memories 258-1 and 258-2, and the layer 2 corrected image memories 259-1 and 259-2. Details of the pixel value correction processing will be described later with reference to the flowchart of FIG.

  In steps S356 and S357, the multi-resolution inverse transform processing units 260-1 and 260-2 execute multi-resolution inverse transform processing, respectively, and perform the layer 0 modified image memory 257-1 and 257-2 and the layer 1 modified image memory 258. -1,258-2 and layer 2 corrected image memories 259-1 and 259-2, the images of layers 0 to 2 in which the R and B pixel values are corrected, and the layer 3 image memory 255 The R and B images stored in -2, 255-3 are subjected to inverse conversion of the multi-resolution conversion by the processes of steps S353 and S354, thereby integrating the plurality of R and B multi-resolution images, In step S358, the combined R and B images are output. Note that the multi-resolution inverse conversion process is the same as the process described with reference to the flowchart of FIG.

  Next, pixel value correction processing by the correction processing unit 256 of FIG. 25 will be described with reference to the flowchart of FIG.

  In step S381, the sampling processing units 281-1 to 281-3 of the correction processing units 256-1 to 256-3 are the layer 0 image memories 252-1 to 252-2 and the layer 1 image memories 253-1 to 253-, respectively. 3 and the layer 2 image memories 254-1 to 254-3, it is determined whether or not an RGB channel image has been input, and the processing is repeated until it is determined that the image has been input. For example, if it is determined that the input has been made, the process proceeds to step S382.

  In step S382, the sampling processing units 281-1 to 281-3 select the same unprocessed pixel as the target pixel.

  In steps S383 to S385, the sampling processing units 281-1 to 281-3 respectively perform layer 0 image memories 252-1 to 252-2, layer 1 image memories 253-1 to 253-3, or layer 2 images of the GBR channel. A plurality of pixels arranged at predetermined positions corresponding to the target pixel position are sampled (extracted) from the memories 254-1 to 254-3, and supplied to the variance-covariance correlation coefficient calculation processing unit 282.

  In step S386, the variance-covariance correlation coefficient calculation processing unit 282 executes a variance calculation process from the GRB pixel value samples supplied from the sampling processing units 281-1 to 281-3, and calculates an RGB variance value. Then, the G variance value is supplied to the clipping processing unit 283, and the GRB variance value is supplied to the determination processing unit 286. Since the distributed calculation process is the process described with reference to the flowcharts of FIGS. 13 and 14, the description thereof will be omitted.

  In step S387, the variance-covariance correlation coefficient calculation processing unit 282 executes covariance calculation processing from the GRB pixel value samples supplied from the sampling processing units 281-1 to 281-3, and performs GR, GB, and BR. Are respectively supplied to the division processing units 284-1 and 284-2 and the determination processing unit 286. The covariance calculation process is the same as the process described with reference to the flowchart of FIG.

  In step S388, the variance-covariance correlation coefficient calculation processing unit 282 executes correlation coefficient calculation processing, and based on the variance value and the covariance value calculated from the GRB pixel value sample, GR, GB, The correlation coefficient of BR is calculated and supplied to the determination processing unit 286. The correlation coefficient calculation process is the same as the process of step S301 in the flowchart of FIG.

  In step S389, the clipping processing unit 283 performs clipping processing on the G variance value supplied from the variance-covariance correlation coefficient calculation processing unit 282, and supplies the result to the division processing units 284-1 and 284-2.

  In steps S390 and S391, the division processing units 284-1 and 284-2 are respectively supplied with the variance value supplied from the clipping processing unit 283 and the GR and GB supplied from the variance covariance correlation coefficient calculation processing unit 282. (Covariance / dispersion) for each of R and B is calculated based on the covariance values of the two and supplied to the multiplication processing units 285-1 and 285-2, respectively.

  In steps S392 and S393, the multiplication processing units 285-1 and 285-2 multiply the R and B (covariance / dispersion) by the G pixel value of the target pixel position, respectively, thereby obtaining the R and B pixels of the target pixel position. The pixel value of is corrected.

  In step S394, the determination processing unit 286 is based on the GRB variance value, the GR and GB covariance values, and the GR, GB, and RB correlation coefficients supplied from the variance covariance correlation coefficient calculation processing unit 282. Thus, the value of the judgment formula shown by the following formula (9) is calculated and supplied to the switches 287 and 288, respectively.

・・・（９）

... (9)

Here, “switch” is a determination result, and when the condition is satisfied, it becomes “passing”, and the corrected pixel value supplied from the multiplication processing units 285-1 and 285-2 is selected. The obtained G pixel value is used as it is. V _GG , V _RR , and V _BB are RGB dispersion values, Cov _RG , Cov _GB , and Cov _BR are correlation coefficients of RG, GB, and BR, respectively, and TH is a predetermined threshold value. .

  In step S395, the switches 287 and 288 determine whether or not the determination result is passing. For example, if it is determined that the determination is passing, the multiplication processing units 285-1 and 285-2 are respectively performed in steps S396 and S397. The R and B pixel values corrected based on the correlation between GR and GB supplied from the above are selected, and the process proceeds to step S400.

  On the other hand, if it is determined in step S395 that it is not passing, in steps S398 and S399, the switches 287 and 288 select the G luminance value supplied from the sampling processing unit 281-1, and the processing is as follows. Proceed to step S400.

  In step S400, the sampling processing units 281-1 to 281-3 determine whether there is an unprocessed pixel. If it is determined that there is an unprocessed pixel, the processing returns to step S381. . That is, until it is determined that there are no unprocessed pixels, the processing of steps S381 to S400 is repeated, and the R and B pixel values are sequentially corrected.

  If it is determined in step S400 that there are no unprocessed pixels, in step S401, the switches 287 and 288 output images obtained by correcting the R and B channel pixel values calculated so far, respectively.

  Through the above processing, it is possible to remove the noise of a chroma component caused by a random component or chromatic aberration by correcting the pixel value using the correlation between channels. In addition, in this way of correcting the configuration that handles signals for 3 channels, not only the relationship between each channel (R, B) to be denoised and the reference channel (G), but also between the channels to be denoised Therefore, it is possible to cope with more characteristic noise components. In particular, according to the above-described determination formula (9), for example, a color flare phenomenon that is a kind of chromatic aberration appearing near a high-contrast edge often appears as blue to purple coloring (purple fringe) of the edge. It is possible to discriminate a case where the image is conspicuous in a specific hue and selectively remove it.

  By changing the conditions of the judgment formula, it is possible to remove noise generated by other phenomena.

  According to the above, it is possible to efficiently remove the noise component mixed in the chroma component of the image signal obtained by using an image sensor such as a solid-state image sensor, and to eliminate the phenomenon of color unevenness and color misregistration.

  In addition, according to the above, non-correlated components between channels can be reduced by estimating (correcting) so that the correlation between other channel signals is as high as possible with respect to a reference channel (for example, G) signal. Therefore, it is possible to reduce noise mixed in chroma components.

  Furthermore, when it is determined that the correlation between channels is not strong enough, the signal of one channel is cut off and replaced with the other channel signal, so that only when the correlation between channels is sufficiently high, By using a pixel value correction method using inter-correlation, noise can be effectively removed.

  In addition to processing based on the correlation between two channels such as G and R, or G and B, as well as evaluating the correlation between three or more channels at the same time, the noise of chroma components can be further effectively reduced. It can be removed.

  Furthermore, after separating the input image into a plurality of bands by multi-resolution processing, noise removal processing based on inter-channel correlation is performed, so that noise components of low frequency components that could not be removed conventionally by multi-resolution processing Can be effectively removed.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processes is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 28 shows a configuration of an embodiment of a personal computer when the electrical internal configuration of the DSP block 16 of FIG. 2 is realized by software. The CPU 401 of the personal computer controls the overall operation of the personal computer. Further, when a command is input from the input unit 406 such as a keyboard or a mouse from the user via the bus 404 and the input / output interface 405, the CPU 401 stores the instruction in a ROM (Read Only Memory) 402 correspondingly. Run the program. Alternatively, the CPU 401 reads a program read from the magnetic disk 421, the optical disk 422, the magneto-optical disk 423, or the semiconductor memory 424 connected to the drive 410 and installed in the storage unit 408 into a RAM (Random Access Memory) 403. To load and execute. Thereby, the function of the DSP block 16 of FIG. 2 described above is realized by software. Furthermore, the CPU 401 controls the communication unit 409 to communicate with the outside and exchange data.

  As shown in FIG. 26, the recording medium on which the program is recorded is distributed to provide the program to the user separately from the computer. The magnetic disk 421 (including the flexible disk) on which the program is recorded is distributed. By a package medium composed of an optical disk 422 (including compact disc-read only memory (CD-ROM), DVD (digital versatile disk)), a magneto-optical disk 423 (including MD (mini-disc)), or a semiconductor memory 424 In addition to being configured, it is configured by a ROM 402 in which a program is recorded and a hard disk included in the storage unit 408 provided to the user in a state of being pre-installed in a computer.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

  11 lens, 12 aperture, 13 image sensor, 14 correlated double sampling circuit, 15 A / D converter, 16 DSP block, 17 timing generator, 18 D / A converter, 19 video encoder, 20 display unit, 21 codec processing unit, 22 memory, 23 CPU, 24 operation input unit, 25 bus, 41 demosaic processing unit, 42 nonlinear transformation processing unit, 43 noise removal processing unit, 44 nonlinear inverse transformation processing unit, 45 white balance processing unit, 46 gamma correction processing unit, 47 YC conversion processing unit, 61 mapping processing unit, 62 conversion curve LUT, 81, 81-1, 81-2 multi-resolution conversion processing unit, 82 layer 0 image memory, 83 layer 1 image memory, 84 layer 2 image memory, 85 Layer 3 image memory, 6, 86-1 to 86-3 correction processing unit, 87 layer 0 noise-removed image, 88 layer 1 noise-removed image, 89 layer 2 noise-removed image, 90 multi-resolution inverse transform processing unit, 111, 111-1 to 111- 3 reduction processing unit, 112, 112-1 to 112-3 subtraction processing unit, 113, 113-1 to 113-3, enlargement processing unit, 131, 132 sampling processing unit, 133 variance calculation processing unit, 134 clipping processing unit, 135 division processing unit, 136 covariance calculation processing unit, 137 multiplication processing unit, 151, 151-1 to 151-3 enlargement processing unit, 152, 152-1 to 152-3 addition processing unit, 171 inverse mapping processing unit, 172 Conversion curve LUT, 201, 201-1, 201-2 sampling processing unit, 202, 02-1, 202-2 variance calculation processing unit, 203 covariance calculation processing unit, 204 clipping processing unit, 205 division processing unit, 206 multiplication processing unit, 207 correlation coefficient calculation processing unit, 208 switch, 221, 221-1 , 221-2 sampling processing unit, 222, 222-1, 222-2 variance calculation processing unit, 223 covariance calculation processing unit, 224 clipping processing unit, 225 division processing unit, 226 multiplication processing unit, 227 difference calculation processing unit, 228 switch, 251, 251-1, 251-2 multi-resolution conversion processing unit, 252, 252-1 to 252-3 layer 0 image memory, 253, 253-1 to 253-3 layer 1 image memory, 254, 254 1 to 254-3 Layer 2 image memory, 255, 255-1 to 55-3 Layer 3 image memory, 256, 256-1 to 256-3 Correction processing unit, 257, 257-1, 257-2 Layer 0 noise-removed image, 258, 255-1, 255-2 Layer 1 noise-removed image 259, 259-1, 259-2 layer 2 noise-removed image, 260, 260-1, 260-2 multi-resolution inverse transform processing unit, 281, 281-1 to 281-3 sampling processing unit, 282 dispersive covariance phase Relation number calculation processing unit, 283 clipping processing unit, 284, 284-1, 284-2 division processing unit, 285, 285-1, 285-2 multiplication processing unit, 286 determination processing unit, 287, 288 switch

Claims (8)

For each pixel, multi-resolution conversion means for converting three or more images that can be assumed to have a linear relationship in a local region in the vicinity thereof to a plurality of layer images of the multi-resolution image;
Among the layer images converted by the multi-resolution conversion means, in each layer image excluding the lowest resolution layer image, one of the three or more images is determined based on the correlation between the three or more images. Correction means to correct,
Multi-resolution inverse transforming means for multi-resolution inverse transforming each layer image modified by the modifying means ;
Extraction means for extracting pixel values of peripheral pixels arranged at positions corresponding to the respective target pixels of the three or more images from the periphery of the target pixel;
For each of the three or more images, a variance value calculating means for calculating a variance value based on the pixel values of the surrounding pixels;
Based on the variance value of each of the three or more images, the pixel value of the target pixel of any one specific image of the three or more images is used as the pixel of the target pixel in an image other than the specific image. An image processing apparatus comprising: selection means for selecting as a value . The correcting means is
Based on the pixel value of each of the peripheral pixels of the three or more images, and covariance value calculation means for calculating a covariance value of said three or more images,
Each of the dispersion values of the three or more images, and a correlation calculating means for calculating a correlation of all combinations of the covariance value,
When the correlation is weak in all combinations of the three or more images , the selection unit determines the pixel value of the pixel of interest of any one specific image of the three or more images other than the specific image. It is selected as the pixel value of the pixel of interest in the image
The image processing apparatus according to claim 1, characterized in that. The selection means determines the pixel value of the target pixel of any one specific image of the three or more images when the variance values of the three or more images are all within a predetermined range. It is selected as the pixel value of the pixel of interest in the image other than the specific image
The image processing apparatus according to claim 1, characterized in that. The multi-resolution image conversion means includes:
The image processing apparatus according to claim 1, further comprising non-linear conversion means that applies non-linear conversion to an input image before converting the three or more images into a plurality of layer images of a multi-resolution image. The image processing apparatus according to claim 4, wherein the non-linear conversion unit converts the three or more images into a plurality of layer images of a multi-resolution image by logarithmic conversion. The image processing apparatus according to claim 4, wherein the non-linear conversion unit converts the three or more images into a plurality of layer images of a multi-resolution image by gamma correction processing. For each pixel, a multi-resolution conversion step of converting three or more images that can be assumed to have a linear relationship in a local region in the vicinity thereof to a plurality of layer images of the multi-resolution image;
Among the layer images converted by the multi-resolution conversion step, in each layer image excluding the lowest resolution layer image, one of the three or more images is correlated with the three or more images. A correction step to correct based on
A multi-resolution inverse transform step of performing multi-resolution inverse transform on each layer image modified by the modification step ;
An extraction step of extracting pixel values of peripheral pixels arranged at positions corresponding to the target pixels of the three or more images from the periphery of the target pixel;
For each of the three or more images, a variance value calculating step for calculating a variance value based on a pixel value of the surrounding pixels;
Based on the variance value of each of the three or more images, the pixel value of the target pixel of any one specific image of the three or more images is used as the pixel of the target pixel in an image other than the specific image. And a selection step of selecting as a value . For each pixel, a multi-resolution conversion step of converting three or more images that can be assumed to have a linear relationship in a local region in the vicinity thereof to a plurality of layer images of the multi-resolution image;
Among the layer images converted by the multi-resolution conversion step, in each layer image excluding the lowest resolution layer image, one of the three or more images is correlated with the three or more images. A correction step to correct based on
A multi-resolution inverse transform step of performing multi-resolution inverse transform on each layer image modified by the modification step ;
An extraction step of extracting pixel values of peripheral pixels arranged at positions corresponding to the target pixels of the three or more images from the periphery of the target pixel;
For each of the three or more images, a variance value calculating step for calculating a variance value based on a pixel value of the surrounding pixels;
Based on the variance value of each of the three or more images, the pixel value of the target pixel of any one specific image of the three or more images is used as the pixel of the target pixel in an image other than the specific image. A program for causing a computer to execute processing including a selection step of selecting as a value .

Images