JP5935649B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  An imaging apparatus such as a digital camera includes a predetermined imaging sensor, and acquires a captured image by imaging with the imaging sensor. In general, a captured image (image signal) includes a noise component specific to an imaging sensor, for example, due to dark current or the like. It is required to be removed with high accuracy.

  In view of such a situation, in a conventional image processing apparatus, an input image is separated into a plurality of frequency bands, edge strength is detected from low frequency components in frequency components of each frequency band, and based on the detected edge strength. After calculating the degree of preservation of edge components, preserving edge components for high frequency components and removing noise components, and then synthesizing low frequency components again, noise removal and edge component preservation are performed simultaneously. In some cases, a high-quality image is obtained (for example, Patent Document 1).

JP 2008-15741 A

  However, in the image processing apparatus described in Patent Document 1, in the case of a color image, it is convenient to perform noise removal and edge component storage by the above-described processing for each RGB channel. If the distribution status of the noise differs depending on the channel, there is a possibility that the noise is erroneously detected as an edge in the channel having a lot of noise, and the noise is erroneously detected and stored. Therefore, a color image has a problem that noise removal accuracy is not good.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of improving noise removal accuracy while maintaining good storage accuracy of edge components in a color image.

In order to solve the above-described problems, the invention described in claim 1 is an image processing apparatus.
A frequency separation unit that separates an input image including a plurality of wavelength regions into frequency components in a plurality of frequency bands;
A wavelength region selection unit for selecting one of the plurality of wavelength regions;
For each of the plurality of frequency bands, an edge detection unit that detects edge intensity from a low frequency component in a frequency component of the wavelength region selected by the wavelength region selection unit;
Noise that removes noise components from high-frequency components in the frequency components of each frequency band separated by the frequency separation unit and that stores edge components for the high-frequency components based on the edge strength detected by the edge detection unit A removal section;
In each of the plurality of frequency bands, a frequency synthesizer that synthesizes the high frequency component in which the noise component is removed and the edge component is preserved by the noise removing unit;
Equipped with a,
The wavelength region selector is
For each of the plurality of wavelength regions, a noise amount calculation unit that calculates a noise amount from a high frequency component in a frequency component of each frequency band separated by the frequency separation unit;
A wavelength region determination unit that determines a wavelength region of a high-frequency component that is the smallest amount of noise from the amount of noise of the high-frequency component for each of the plurality of wavelength regions obtained by the noise amount calculation unit;
With
The wavelength region determined by the wavelength region determination unit is selected from the plurality of wavelength regions .

The invention according to claim 2 is the image processing apparatus according to claim 1 ,
The plurality of wavelength regions include a red region, a green region, and a blue region of visible wavelengths.

The invention according to claim 3 is the image processing apparatus according to claim 2 ,
The plurality of wavelength regions include luminance information obtained from a red region, a green region, and a blue region of the visible wavelength.

The invention according to claim 4 is the image processing apparatus according to any one of claims 1 to 3 ,
The plurality of wavelength regions include an infrared wavelength region.

The invention according to claim 5 is an image processing method,
A frequency separation step of separating an input image including a plurality of wavelength regions into frequency components in a plurality of frequency bands;
A wavelength region selection step of selecting any one of the plurality of wavelength regions;
For each of the plurality of frequency bands, an edge detection step of detecting edge intensity from a low frequency component in a frequency component of the wavelength region selected in the wavelength region selection step;
Noise that removes noise components from high-frequency components in the frequency components of each frequency band separated in the frequency separation step and that stores edge components for the high-frequency components based on the edge strength detected in the edge detection step A removal step;
In each of the plurality of frequency bands, a frequency synthesizing step of synthesizing the high frequency component in which the noise component is removed and the edge component is stored in the noise removing step;
Only including,
The wavelength region selection step includes
For each of the plurality of wavelength regions, a noise amount calculation step of calculating a noise amount from a high frequency component in a frequency component of each frequency band separated in the frequency separation step;
A wavelength region determination step of determining a wavelength region of a high-frequency component that is the least amount of noise from the amount of noise of the high-frequency component for each of the plurality of wavelength regions obtained in the noise amount calculation step;
Including
In the wavelength region selection step, the wavelength region determined in the wavelength region determination step is selected from the plurality of wavelength regions .

  According to the present invention, noise removal accuracy can be improved in a color image while maintaining good preservation accuracy of edge components.

It is a block diagram which shows the functional structure of a digital camera. It is a functional block diagram which shows the circuit structural example regarding the noise removal process of an image process part. It is a graph explaining the edge preservation | save coefficient E calculated in an edge detection part. It is a graph explaining the coring characteristic in the coring process of a noise removal part. It is a flowchart explaining the operation | movement regarding a noise removal process. It is a functional block diagram which shows the circuit structural example regarding the noise removal process of the image process part in 2nd Embodiment. It is a functional block diagram which shows the circuit structural example regarding the noise removal process of the image process part in 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The scope of the invention is not limited to the illustrated example.

(First embodiment)
FIG. 1 is a schematic functional block diagram of a digital camera 1 that is an example of an imaging apparatus to which the image processing apparatus according to the first embodiment is applied. As shown in FIG. 1, the digital camera 1 includes a lens unit 2, an image sensor 3, an amplifier 4, an A / D conversion unit 5, an image processing unit 6, an image memory 7, a control unit 8, a display unit 9, and an operation unit 10. It has.

  The lens unit 2 functions as a lens window for capturing subject light (light image), and constitutes an optical lens system for guiding the subject light to the image sensor 3 disposed inside the camera body. The lens unit 2 is a lens group including, for example, a zoom lens, a focus lens, and other fixed lens blocks arranged in series along the optical axis L of the subject light. The lens unit 2 includes a diaphragm and a shutter for adjusting the amount of light transmitted through the lens, and these are driven and controlled by the control unit 8.

  The imaging sensor 3 photoelectrically converts the image signals of R, G, and B components to output to the amplifier 4 in accordance with the amount of light of the subject light image formed in the lens unit 2. In the present embodiment, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a VMIS (threshold voltage modulation image sensor), a CCD (Charge Coupled Device) image sensor, or the like can be employed as the imaging sensor 3.

  The amplifier 4 amplifies the image signal output from the image sensor 3 and includes, for example, an AGC (Auto Gain Control) circuit, and adjusts the gain of the output signal. The amplifier 4 may include a CDS (Correlated Double Sampling) circuit that reduces sampling noise of an image signal as an analog value in addition to the AGC circuit. The AGC circuit also has a function of compensating for an insufficient level of a captured image when a proper exposure is not obtained, for example, when photographing a very low-luminance subject. The gain value for the AGC circuit is set by the control unit 8.

  The A / D conversion unit 5 performs A / D conversion processing for converting an analog image signal (analog signal) amplified by the amplifier 4 into a digital image signal (digital signal). Each pixel signal obtained by receiving light at each pixel 3 is converted into, for example, 12-bit pixel data.

  The image processing unit 6 performs various image processing on the image signal obtained by the A / D conversion process described above. Specifically, for example, color processing such as color interpolation processing, color correction processing, and color space conversion processing, white balance correction processing, noise removal processing, dynamic range compression processing, and the like are performed. In the present embodiment, there is a main feature point regarding noise removal processing in the image processing, which will be described later.

  The image memory 7 includes a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The image memory 7 is RAW image data before image processing by the image processing unit 6, or the image processing unit 6 or the control unit 8. Data such as image data at the time of various processing in or after processing is stored.

  The control unit 8 includes a ROM that stores each control program, a RAM that temporarily stores various data, a CPU (Central Processing Unit) that reads and executes the control program from the ROM, and the like. It governs the operation control. The control unit 8 calculates control parameters required by each unit of the apparatus based on various signals from the respective units of the apparatus such as the imaging sensor 3 and the operation unit 10, and controls the operation of each unit by transmitting the calculated control parameters. The control parameters include, for example, an exposure amount control parameter for setting an optimal exposure amount at the time of shooting, a dynamic range control parameter for setting an optimal dynamic range, and the like.

  The display unit 9 includes, for example, a liquid crystal display (LCD) composed of a color liquid crystal display element disposed on the back of the camera, and displays an image captured by the image sensor 3.

  The operation unit 10 instructs the digital camera 1 to be operated by the user, and includes, for example, various operation switch groups such as a power switch, a release switch, a mode setting switch for setting various shooting modes, and a menu selection switch. For example, when the release switch is pressed, the subject light is imaged by the imaging sensor 3, and the captured image obtained thereby is subjected to necessary image processing and then recorded in the image memory 7 or the like. A series of shooting operations is executed.

  By the way, a wide DR (Dynamic Range) image (input image) obtained by photographing by the image sensor 3 is input to the image processing unit 6. The noise removal processing in the image processing unit 6 for this input image will be described below. explain. FIG. 2 is a functional block diagram illustrating a circuit configuration example mainly relating to noise removal processing in the image processing unit 6. As shown in FIG. 2, the image processing unit 6 includes a frequency separation unit 601, an HPF unit 604, a noise amount calculation unit 605, a channel determination unit 606, an edge detection unit 607, a noise removal unit 608, and a frequency synthesis unit 611. Yes.

  Describing these functional units schematically, the input image 6200 input to the image processing unit 6 is separated for each RGB channel. That is, the input image 6200 includes a plurality of wavelength regions having a red (R) region, a green (G) region, and a blue (B) region of visible wavelengths, and is input to the image processing unit 6. In addition, these wavelength regions included in the input image 6200 are separated as an R channel, a G channel, and a B channel, respectively. Then, the frequency separation unit 601 performs an LPF process and a downsampling process on a plurality of hierarchical processing steps on the input image 6200 separated into the RGB channels, thereby converting the input image 6200 into the RGB image. Separate into multiple frequency components for each channel. By the processing for separating the frequency components (frequency separation processing), an image having a low frequency component (low frequency image) is generated for each processing step.

  In each processing step, the HPF unit 604 subtracts a low-frequency image, which is an image after LPF processing, from an image before LPF processing that is performed by frequency separation processing by the frequency separation unit 601, thereby high-frequency signals for each RGB channel. Generate an image.

  In each processing step, the noise amount calculation unit 605 obtains how much noise is contained in each RGB channel from the high-frequency image generated by the HPF unit 604.

  In each processing step, the channel determination unit 606 determines which channel's image noise amount is the smallest from the RGB channel noise amounts obtained by the noise amount calculation unit 605.

  In each processing step, the edge detection unit 607 performs detection processing (edge detection processing) related to edge components for the low-frequency image after the LPF processing. Here, the target of the edge detection process is a low-frequency image of a channel that is determined by the channel determination unit 606 to have the least amount of noise.

  The noise removal unit 608 performs noise component removal processing (noise removal processing) on the high-frequency image generated by the HPF unit 604 for each of the RGB channels.

  The frequency synthesizer 611 synthesizes a high-frequency image after noise removal processing in each processing step with respect to the low-frequency image in the final processing step among the processing steps generated by the frequency separation processing in the frequency separation unit 601. Then, an upsampling process is performed to generate an output image 6210. When the output image 6210 is output from the image processing unit 6, the images of the RGB channels are integrated.

  Details of the configuration and operation of each unit described above will be described. The frequency separation unit 601 includes an LPF unit 602 that performs the above-described LPF processing and a DS unit 603 that performs the above-described downsampling (DS) processing. The LPF unit 602 includes an LPF unit 6021 for processing step 1, an LPF unit 6022 for processing step 2,..., An LPF unit 6023 for processing step (n−1), and an LPF unit 6024 for processing step n. Each of the LPF units 6021 to 6024 performs LPF processing using a low-pass filter in each processing step on each of the input images input to the image processing unit 6 and separated for each RGB channel. For example, in the processing step (n-1), LPF processing is performed by the LPF unit 6023 on the input images of the R channel, G channel, and B channel, respectively, so that the low frequency of the R channel, G channel, and B channel is obtained. An image is extracted. The “input image” here is used to mean an image input to each processing step. This input image is the image data of the captured image itself transmitted from the imaging sensor 3 that has not been subjected to the LPF process for the processing step 1, but the low frequency image extracted by the lower processing from the processing step 2 is It can also be said that this is an image for edge detection (for edge conservation coefficient calculation) described later.

  The DS unit 603 includes a DS unit 6031 corresponding to the processing step 1, a DS unit 6032... Corresponding to the processing step 2, and a DS unit 6033 corresponding to the processing step (n−1). However, since the processing step n is the final processing step, the DS unit corresponding to the processing step n is not provided. In each DS unit 6031 to 6033, the size (number of pixels) of the low frequency image extracted by the LPF process in each processing step is halved in both vertical and horizontal directions, for example, DS processing by pixel thinning is performed. The DS-processed image is output to the LPF unit in the next processing step. For example, in the processing step (n−1), the DS unit 6033 performs DS processing on the low-frequency image after the LPF processing by the LPF unit 6023, and an input image for the processing step n is generated. In this embodiment, DS processing is performed to reduce the image size to ½. However, the present invention is not limited to this, and DS processing at any magnification such as 1/3 or 1/4 can be adopted. It is.

  The number of processing steps described above can be arbitrarily set, but in the present embodiment, for example, it is assumed that “4” (n = 4) is set. In the present embodiment, when the number of processing steps is n = 4, for example, 7 × 7 LPF processing is applied as the LPF processing of each LPF unit. The 7 × 7 LPF process is a process using an LPF having a filter size of “7” taps. Regarding the relationship between the number of processing steps and the number of taps, when performing 5 × 5 LPF processing, which is LPF processing when the number of taps is smaller than “7”, for example, “5”, the number of processing steps n is It is preferable to set a number larger than “4” (for example, “8”). This is because as the number of taps decreases, the degree of smoothing of the image each time the LPF process is performed decreases, and accordingly, the number of times the LPF process is performed, that is, the number of processing steps must be increased. It depends. However, it is not always necessary to set the number of processing steps and the number of taps so that this relationship is established, and a process of extracting a low frequency image that eliminates residual noise from the low frequency image obtained by the LPF process in the LPF unit. The number of steps and the number of LPF taps may be set.

  As described above, the frequency separation unit 601 performs a plurality of LPF processing and DS processing by a plurality of processing steps from the uppermost processing step 1 to the lowermost processing step 4, thereby generating a plurality of input images 6200. Is decomposed into frequency images. It should be noted that repeatedly performing the LPF process and the DS process is equivalent to performing the LPF process while increasing the number of taps in order.

  The HPF unit 604 includes HPF units 6041 to 6044 corresponding to the processing steps 1 to n (1 to 4). The HPF unit 604 performs high-pass filter (HPF) processing for subtracting the low-frequency image after the LPF processing by the LPF unit 602 from the input image before being input to the LPF unit 602 for each of the RGB channels, and performs RGB processing. Extract a high-frequency image of the channel.

  For example, the processing step (n−1), that is, the HPF unit 6043 in the processing step 3 is output from the DS unit 6032 in the processing step (n−2) (processing step 2) preceding the processing step (n−1). The input image and the low-frequency image output from the LPF unit 6023 are input, and the low-frequency image is subtracted from the input image. The HPF unit 6043 outputs a high-frequency image obtained as a result of the subtraction. As described above, the high-frequency images generated by the HPF units 6041 to 6044 are obtained by separating high-frequency components including noise components from the input images of the respective processing steps, and noise removal is performed from the high-frequency images in the processing described later. Will do. As described above, by performing the HPF processing a plurality of times by a plurality of processing steps, it becomes possible to separate noise components that could not be separated by one HPF processing.

  The noise amount calculation unit 605 includes noise amount calculation units 6051 to 6054 corresponding to the processing steps 1 to n (1 to 4). The noise amount calculation unit 605 calculates the amount of noise included in the high-frequency image generated by the HPF unit 604 for each of the RGB channels. The noise amount calculation unit 605 calculates the noise amount by, for example, obtaining an average of pixel values of each pixel included in the high frequency image. At this time, it is preferable that the pixel to be calculated when obtaining the average of the pixel values is a pixel value exceeding a predetermined threshold value. As long as the amount of noise can be calculated, it may be calculated by other methods.

  The channel determination unit 606 as a wavelength region determination unit includes channel determination units 6061 to 6064 corresponding to processing steps 1 to n (1 to 4). The channel determination unit 606 determines the channel with the smallest noise amount among the RGB channels from the noise amount of the high-frequency image for each RGB channel calculated by the noise amount calculation unit 605. The channel determination unit 606 outputs a channel selection signal to the edge detection unit so that the low-frequency image of the determined channel is selected as an object of edge detection processing.

  Thus, in the present embodiment, the noise amount calculation unit 605 and the channel determination unit 606 constitute a wavelength region selection unit.

  The edge detection unit 607 includes edge detection units 6071 to 6074 corresponding to the processing steps 1 to n (1 to 4). The noise removing unit 608 includes noise removing units 6081 to 6084 corresponding to the processing steps 1 to n (1 to 4). The edge detection unit 607 and the noise removal unit 608 remove noise components while preserving the edge components in the high-frequency image with respect to the high-frequency image including the noise component transmitted from the HPF unit 604. A specific method of edge detection processing and noise removal processing in each of the edge detection unit 607 and the noise removal unit 608 will be described.

  The edge detection units 6071 to 6074 receive from the LPF units 6021 to 6024 the low frequency images of the channels determined by the channel determination units 6061 to 6064 as having the least amount of noise among the RGB channels. The edge detection units 6071 to 6074 perform filter processing (edge strength detection filter processing) using an edge strength detection filter such as a Sobel filter on the low frequency image input in each processing step as edge detection processing. To calculate the edge strength of the low-frequency image, and the edge preservation coefficient E is calculated from the calculated edge strength information. Information on the calculated edge preserving coefficient E is output to each of the noise removing units 6081 to 6084. For example, in processing step 3, when the channel determination unit 6063 determines that the G channel is the channel with the least amount of noise, the edge detection unit 6073 inputs a low frequency image of the G channel from the LPF unit 6023. The edge detection unit 6073 performs edge strength detection filter processing on the low-frequency image to obtain edge strength, calculates an edge storage coefficient E from the edge strength, and calculates information on the calculated edge storage coefficient E, It outputs to the noise removal part 6083 as edge preservation | save coefficient information. The Sobel filter described above detects an inclination between pixel values in an image based on differential calculation, and can thereby detect the intensity of an edge in the image.

Regarding the calculation of the edge strength and the edge preserving coefficient E, specifically, each of the edge detection units 6071 to 6074 uses a 3 × 3 Sobel filter having a tap number “3” for each low-frequency image. The edge strength e is calculated by performing (3 × 3 Sobel filter processing), and the edge preserving coefficient E for the calculated edge strength e is calculated by the following equation (1).
When e <e1, E = 0
When e1 <e <e2 E = (e−e1) / (e2−e1)
When e> e2 E = 1.0 (1)
Here, e1 and e2 are preset threshold values for edge preservation coefficient calculation (edge preservation threshold values), and satisfy the relationship of e1 <e2.

  As shown in FIG. 3, when the vertical axis is the edge preserving coefficient E and the horizontal axis is the edge strength e, as shown in FIG. The storage coefficient E = 0. In the range where the edge strength e indicated by reference numeral 657 is e> e2, the edge preserving coefficient E = 1.0. In the range where the edge strength e indicated by reference numeral 656 is e1 <e <e2, the edge preserving coefficient E changes linearly with the slope being 1 / (e2-e1) in the range of 0 to 1.0. It will be a thing.

  The edge storage thresholds e1 and e2 are values related to the texture of the image, such as how much an edge should be stored, and are set in advance as values that can suitably perform edge storage in each processing step. The edge storage thresholds e1 and e2 may be set in advance for each processing step, or may be uniformly set as the same value in all processing steps. Further, a value may be set according to the selected channel. Information on the edge storage threshold values e1 and e2 is stored in each of the edge detection units 6071 to 6074. In this embodiment, the 3 × 3 Sobel filter is used as the edge intensity detection filter as described above. However, a Sobel filter having a tap number other than “3”, for example, greater than 3, may be used. . In addition, other known filters such as a Prewitt filter may be used instead of the Sobel filter.

  The noise removing units 6081 to 6084 perform coring processing on the high-frequency images of the RGB channels in the processing steps, respectively, as noise removal processing. At this time, by using the information of the edge preserving coefficient E calculated by the edge detection process, for the pixel having a large edge strength e, the edge in which the coring strength in the coring process is weakened and the edge component is preserved. Perform the save process. On the other hand, for a pixel having a small edge strength e, the coring strength is increased and the degree of edge preservation is decreased.

  Regarding the coring processing described above, specifically, each of the noise removal units 6081 to 6084 performs a high frequency image according to a coring characteristic 665 as illustrated in FIG. 4 with respect to the high frequency image input from the HPF units 6041 to 6044. The noise component is removed by performing the conversion process. “Th” shown in FIG. 4 is a coring coefficient for removing a noise component of the image signal. In the coring process according to the coring characteristic 665, when the input is larger than -th and smaller than th, the output becomes zero (that is, the data in the range where the input is larger than -th and smaller than th is noise). When the input is -th or less, the output is a value obtained by adding th to the input. When the input is greater than or equal to th, the output is a value obtained by subtracting th from the input. Such a conversion process is performed.

  However, the high-frequency image includes not only a noise component but also a detail component. Therefore, when the coring process uniformly cuts the image component in which the absolute value of the pixel of interest is smaller than the coring coefficient, that is, the range of the image component that is larger than -th and smaller than th, the detail component is lost. Will end up. Therefore, instead of the coring characteristic 665, a coring characteristic 666 having a relationship of “y = kx” in which the slope of the graph is k (0 ≦ k ≦ 1) is introduced in a range larger than −th and smaller than th. However, by adjusting the kn value, a slight noise that is not conspicuous in the image is allowed, and a processing result that suppresses the loss of the detail component may be obtained. When the coring characteristic 666 is used, the detail component is left as y = kx, for example, the value of the input x is closer to the coring coefficient th (−th) as indicated by reference numerals 6661 and 6662. become. Here, the detail component is an image other than the noise component in the high-frequency image, and is an original image of the high-frequency image that expresses the texture with respect to the low-frequency image that is a blurred image.

  The coring coefficient th (-th) is set to a value corresponding to the amount of noise present in the high-frequency image at each processing step, thereby enabling accurate and accurate noise removal. Actually, the imaging sensor 3 has noise such as dark current, and the noise amount of the high-frequency image is determined according to the noise peculiar to the sensor, and the high-frequency image of each processing step has a different noise amount. Therefore, the coring coefficient th (−th) is set to a suitable value corresponding to the amount of noise to be removed for each imaging sensor 3 used in each digital camera 1 or for each processing step. Is done. The coring coefficient th is stored in each noise removing unit 6081-6084 as a predetermined value. Note that the coring coefficient th may not be a predetermined value, and may be configured to be calculated and set each time based on the noise amount calculated by the above-described noise amount calculation units 6051 to 6054, for example.

Further, regarding the edge preserving process, specifically, each of the noise removing units 6081 to 6084 uses the information of the edge preserving coefficient E obtained in the above-described edge detecting process to obtain a noise removing unit 6081 according to the following equation (2). ˜6084 and a weighted average process of the image obtained as a result of the coring process (hereinafter also referred to as a coring image). This weighted average process is performed for each of the RGB channels. For example, in processing step 3, when the edge detection coefficient E is obtained from the G channel low frequency image in the edge detection unit 6073, the noise removal unit 6083 performs the edge storage coefficient for each of the RGB high frequency images. Using the information of E, an image (edge-preserved image) in which the edge component is stored is acquired, a coring image is acquired, and a weighted average process is performed on these images. As a result, the edge preservation is performed with the edge preservation amount according to the edge strength e, that is, the edge preservation image in which the edge preservation processing in which the coring strength changes according to the edge strength e is generated. it can. By performing such edge storage processing, as shown in FIG. 3, as the edge storage coefficient E is larger (closer to a value of 1.0), the pixel value of the high-frequency image before coring processing is set. The edge component is stored with a closer value, and the smaller the edge storage coefficient E (the closer the value is to 0), the closer to the value of the coring image pixel.
Img_output = (1.0−E) * Img_core + E * Img_input (2)
Where Img_output: edge-preserved image
Img_core: Coring image
Img_input: a high-frequency image input to each noise removing unit

  Thus, for example, in processing step 3, the noise removal unit 6083 performs the coring process and the edge storage process on the high-frequency image from the HPF unit 6043 based on the edge storage coefficient information from the edge detection unit 6073. And output a high-frequency image from which noise components have been removed while preserving edge components.

  In the present embodiment, as described above, the channel with the least amount of noise is selected from the RGB channels, and the edge detection process is performed on the low-frequency image of the selected channel. The edge component is stored and the noise component is removed from the high-frequency image of the channel. As a result, it is possible to reduce the influence of the accuracy of edge detection by the low-frequency image of the channel with a lot of noise. That is, the possibility of erroneous detection of noise as an edge can be reduced, and the noise removal accuracy can be improved while maintaining the preservation accuracy of the edge component.

  The frequency synthesis unit 611 includes a synthesis unit 609 that performs the above-described synthesis process and a US unit 610 that performs an upsampling (US) process. The synthesis unit 609 includes synthesis units 6091 to 6094 corresponding to the processing steps 1 to n (1 to 4).

  In each synthesis unit 6091 to 6094, the image (frequency synthesized image) that has been subjected to the upsampling process in the previous processing step and the high-frequency image that has been subjected to noise removal by each noise removal unit 6081 to 6084 are synthesized, and each processing step A frequency composite image is generated for each RGB channel. For example, in processing step 3, a high frequency image from the noise removal unit 6083 and a frequency composite image from the US unit 6103 are combined to generate a frequency composite image, and this frequency composite image is the upper processing step 2. To the US unit 6102 on the side. In the processing step 4 which is the lowest processing step, the low frequency image itself output from the LPF unit 6024 is used instead of the frequency synthesized image described above. That is, in the synthesis unit 6094, a synthesis process of the low frequency image and the high frequency image from the noise removal unit 6084 is performed.

  Note that the composite image processing in each of the synthesis units 6091 to 6094 includes a high-frequency image from the noise removal unit 608 and a low-frequency image from the lower processing step (the low-frequency image at the same stage as described above in the processing step 4). ) And output as a low-frequency image to the upper processing step. This is because, when viewed from the synthesis unit in each processing step, for example, the synthesis unit 6093 side, the frequency synthesized image generated in the lower processing step is a low-frequency image with respect to the high-frequency image.

  The US unit 610 includes a US unit 6101 corresponding to processing step 1, a US unit 6102 corresponding to processing step 2, and a US unit 6103 corresponding to processing step 3. Each of the US units 6101 to 6103 doubles the image size (number of pixels) both vertically and horizontally with respect to the frequency synthesized image generated in the synthesizing unit in the previous processing step, for example, by pixel interpolation such as linear interpolation. The US process is performed, and the image after the US process is output to the synthesis unit in the next processing step. Note that the magnification in the US processing here is set to double in order to restore the original size corresponding to the half magnification in the DS processing in the DS section described above. If it is 1/3, the magnification according to the DS processing magnification can be set such that the US processing magnification is tripled. However, it is not always necessary to set the magnification according to the magnification of the DS processing, and any magnification may be used.

  In such a configuration, the frequency synthesizing unit 611 performs a plurality of frequency synthesis processes and a US process by a plurality of process steps from the lowermost process step 4 to the uppermost process step 1, and as a result, the image An output image 6210 is output as an image for the input image 6200 to the processing unit 6. At this time, as described above, the images of the RGB channels are integrated when output from the image processing unit 6.

  Next, operations related to the noise removal processing according to the first embodiment will be described with reference to FIG.

  First, an input image 6200 is acquired by imaging or the like by the imaging sensor 3, and is input to the frequency separation unit 601 of the image processing unit 6 (step S10). Next, the LPF unit 602 and the DS unit 603 of the frequency separation unit 601 perform a plurality of times of LPF processing and DS in a plurality of processing steps from the uppermost processing step to the lowermost processing step on the input image 6200. Is processed and separated into a plurality of frequency components. That is, a low frequency image is generated for each processing step (step S20). As the low-frequency image is generated for each processing step, the HPF unit 604 subtracts the low-frequency image after the LPF processing from the input image for each processing step to generate a high-frequency image for each RGB channel (step S30). ).

  In each processing step, the noise amount calculation unit 605 and the channel determination unit 606 determine the high-frequency image of the channel with the least amount of noise among the RGB channels. That is, the noise amount calculation unit 605 calculates the noise amount for each of the RGB high-frequency images generated by the HPF unit 604. Then, the channel determination unit 606 determines which channel has the smallest noise amount from the noise amount of the high-frequency image of each of the RGB channels calculated by the noise amount calculation unit 605 (step S40).

  In each processing step, edge detection processing and noise removal processing are performed by the edge detection unit 607 and the noise removal unit 608. That is, when a low frequency image of a channel determined to have the least amount of noise by the channel determination unit 606 is input to the edge detection unit 607, the edge detection unit 607 applies an edge strength detection filter to the low frequency image. Is used to calculate the edge intensity of the low-frequency image, and the edge preserving coefficient E is calculated from the calculated edge intensity information. Further, the noise removing unit 608 generates a coring image from which noise components have been removed by performing coring processing on the high-frequency images of the RGB channels input to the noise removing unit 608 to detect edges. By using the information of the edge preserving coefficient E calculated in the unit 607, the coring image and the high-frequency image are weighted to remove the noise component, and the edge component is changed according to the edge strength e. A stored edge storage image is generated for each of the RGB channels (step S50). Then, the frequency synthesis unit 611 adds the low frequency image of the lowest processing step and the high frequency image of each processing step in order from the lowest processing step to the uppermost processing step while repeating the frequency synthesis processing and the US processing. As a result, a composite image is obtained in which the noise component is removed and the edge component is stored. This composite image is output from the image processing unit 6 as an output image 6210. At this time, the RGB channel images are integrated (step S60).

(Second Embodiment)
Next, a digital camera 1 that is an example of an imaging apparatus to which the image processing apparatus according to the second embodiment is applied will be described. In the second embodiment, the image processing unit is the same as that of the first embodiment except that the image processing unit is different from the first embodiment. Therefore, here, the image processing unit will be described and other configurations will be described. Description of is omitted. In the image processing unit, the description of the same configuration as that of the first embodiment is also omitted.

FIG. 6 is a functional block diagram illustrating a circuit configuration example mainly related to noise removal processing in the image processing unit 6a. In FIG. 6, only step 1 is illustrated, but the other steps have the same configuration. In the second embodiment, the image processing unit 6a receives the input image 6200 input to the image processing unit 6a and separated into RGB channels, calculates a luminance signal (Y), and outputs the luminance signal (Y). A luminance calculation unit 612 is provided. The luminance calculation unit 621 obtains the luminance signal (Y) from the RGB pixel values according to the following equation (3).
Y = 0.299 * R + 0.587 * G + 0.144 * B (3)

  The luminance signal obtained as described above is added to each RGB channel as a Y channel. For each YRGB channel, frequency separation processing by the frequency separation unit 601, generation of a high-frequency image by the HPF 604, and noise amount calculation unit 605 are performed. Calculation of the amount of noise, channel determination by the channel determination unit 606, and edge detection processing by the edge detection unit 607 are performed. The noise removal unit 608 performs noise removal processing on the high-frequency image of each of the RGB channels generated by the HPF unit 604.

  For example, when the noise amount calculation unit 605 calculates the noise amount of the high-frequency image of each YRGB channel, and the channel determination unit 606 determines that the noise amount of the Y-channel high-frequency image is the smallest, the LPF unit 602. The Y-channel low-frequency image output from is input to the edge detection unit 607, and edge detection processing and noise removal processing are performed based on the low-frequency image.

  As described above, when noise removal processing is performed in each of the lowermost processing step n to the uppermost processing step 1 to obtain a high-frequency image of each channel of RGB, frequency synthesis is performed as described above. A plurality of frequency synthesis processes and US processes are performed by the unit 611 to generate an output image 6210. In the lowest processing step n, the YRGB low-frequency image is generated in the LPF unit. However, in the frequency synthesis process, the low-frequency image and noise removal processing of each RGB channel excluding the Y channel are performed. The high-frequency images of the respective RGB channels are synthesized.

  As described above, in the second embodiment, as in the first embodiment, it is possible to reduce the possibility that noise is erroneously detected as an edge, and the noise removal accuracy is improved while maintaining the preservation accuracy of the edge component. Can be improved. Further, edge detection can be performed with high accuracy from luminance information with relatively little noise, and noise removal accuracy can be improved.

(Third embodiment)
Next, a digital camera 1 that is an example of an imaging apparatus to which the image processing apparatus according to the third embodiment is applied will be described. In the third embodiment, the image processing unit is the same as that of the first embodiment except that the image processing unit is different from the first embodiment. Therefore, here, the image processing unit will be described and other configurations will be described. Description of is omitted. In the image processing unit, the description of the same configuration as that of the first embodiment is also omitted.

  FIG. 7 is a functional block diagram illustrating a circuit configuration example mainly relating to noise removal processing in the image processing unit 6b. In FIG. 7, only step 1 is illustrated, but the other steps have the same configuration. In the third embodiment, the image processing unit 6b receives the luminance calculation unit 612 described in the second embodiment and the channel designation signal from the control unit 8, and corresponds to the received channel designation signal. A channel selection unit 613 that outputs a channel selection signal to the edge detection unit 607 is provided so that the channel is subjected to edge detection processing. In the present embodiment, a configuration in which the luminance calculation unit 612 is not provided may be employed. The channel selection unit 613 is provided corresponding to each of processing steps 1 to n (1 to 4). For example, in step 1, a channel selection unit 6131 is provided. The channel designation signal is output from the control unit 8 according to, for example, a time zone such as temperature and humidity, day and night, ambient brightness, the type of the image sensor 3, and the like. Note that the channel designation signal corresponding to which channel is output under which condition can be appropriately set according to the state of occurrence of noise in the image of each RGB channel. Further, for example, it may be configured such that the channel designation signal to be output can be set in advance in accordance with the operation of the operation unit 10. In addition, a different channel designation signal may be output from the control unit 8 for each step, or the same channel designation signal may be output from the control unit 8 in all steps.

  The edge detection unit 607 performs an edge detection process by inputting a low frequency image of a channel corresponding to a channel selection signal from the channel selection unit 613 from each YGRB channel.

  As described above, in the third embodiment, as in the first embodiment, it is possible to reduce the possibility of erroneous detection of noise as an edge, and the noise removal accuracy can be improved while maintaining the preservation accuracy of the edge component. Can be improved. In addition, since it is not necessary to switch the channel each time an image is taken, the amount of processing in real time at the time of image taking can be reduced.

  As described above, according to the first to third embodiments, the frequency separation unit 601 separates an input image including a plurality of wavelength regions into frequency components in a plurality of frequency bands. The channel selection unit 613 (noise amount calculation unit 605, channel determination unit 606) selects one from a plurality of wavelength regions. The edge detection unit 607 detects the edge intensity from the low frequency components in the frequency components of the wavelength region selected by the channel selection unit 613 (noise amount calculation unit 605, channel determination unit 606) for each of the plurality of frequency bands. The noise removing unit 608 removes the noise component from the high frequency component in the frequency component of each frequency band separated by the frequency separating unit 601, and based on the edge strength detected by the edge detecting unit 607, the edge component for the high frequency component Save the file. The frequency synthesizing unit 611 synthesizes the high frequency components in which the noise components are removed by the noise removing unit 608 and the edge components are stored in each of a plurality of frequency bands. As a result, in a color image including a plurality of wavelength regions, for example, a low-frequency image in a wavelength region with a small amount of noise is selected, and edge detection is performed on the low-frequency image. Since the components are stored, the noise removal accuracy can be improved in the color image while maintaining the storage accuracy of the edge components in good quality.

  Further, according to the first and second embodiments, the noise amount calculation unit 605 is configured to reduce the noise amount from the high frequency component in the frequency component of each frequency band separated by the frequency separation unit 601 for each of the plurality of wavelength regions. Is calculated. The channel determination unit 606 determines the wavelength region of the high frequency component that is the smallest noise amount from the noise amount of the high frequency component for each of the plurality of wavelength regions obtained by the noise amount calculation unit 605. The wavelength region determined by the channel determination unit 606 is selected from a plurality of wavelength regions. As a result, edge detection is performed on low-frequency images in the wavelength region with the least amount of noise, and noise removal and edge component preservation are performed based on this detection, eliminating noise while maintaining good edge component preservation accuracy. The accuracy can be further improved.

  Further, according to the first to third embodiments, the plurality of wavelength regions include the red region, the green region, and the blue region of visible wavelengths, so that the edge detection in the color channel can be accurately performed. Therefore, the noise removal accuracy can be further improved.

  In addition, according to the second and third embodiments, the plurality of wavelength regions include luminance information obtained from the red region, the green region, and the blue region of the visible wavelength, so that luminance information with less noise is used. Thus, edge detection can be performed with high accuracy, and noise removal accuracy can be further improved.

  Note that the description in this embodiment is an example of the digital camera according to the present invention, and the present invention is not limited to this. The detailed configuration and detailed operation of each functional unit constituting the digital camera can be changed as appropriate.

In the present embodiment, a visible light sensor that acquires signals in the three wavelength regions of RGB, such as a COMS image sensor and a CCD image sensor, is applied as the imaging sensor 3, and this is subjected to A / D conversion for image processing. In addition to the visible light sensor described above, for example, a complementary color sensor that acquires signals in each wavelength region of W (White), Y (Yellow), I (infrared wavelength region), and R (Red) May be applied. In this case, RGB channel images can be generated from signals in the WYIR wavelength regions, and edge detection processing and noise removal processing can be performed on them. Here, any one of the images of each channel of WYIR may be selected, and the edge detection process may be performed on the low frequency image of the selected channel.
That is, according to the present embodiment, since the plurality of wavelength regions include the infrared wavelength region, it is possible to correspond to an image including a signal in the infrared wavelength region.

  In the present embodiment, the timing for selecting the low-frequency image of the channel to be subjected to edge detection processing by the edge detection unit 607 from each of the RGB (YRGB) channels can be set as appropriate. For example, it may be performed every time one frame is captured by the image sensor 3, or may be performed every predetermined time, or may be performed only once at the start of imaging. .

  In the present embodiment, an example in which a hard disk, a semiconductor nonvolatile memory, or the like is used as a computer-readable medium for the program according to the present invention is disclosed, but the present invention is not limited to this example. As another computer-readable medium, a portable recording medium such as a CD-ROM can be applied. A carrier wave is also used as a medium for providing program data according to the present invention via a communication line.

DESCRIPTION OF SYMBOLS 1 Digital camera 6 Image processing part 601 Frequency separation part 602 LPF part 604 HPF part 605 Noise amount calculation part (wavelength area selection part)
606 Channel determination unit (wavelength region selection unit / wavelength region determination unit)
607 Edge detection unit 608 Noise removal unit 611 Frequency synthesis unit

Claims (5)

A frequency separation unit that separates an input image including a plurality of wavelength regions into frequency components in a plurality of frequency bands;
A wavelength region selection unit for selecting one of the plurality of wavelength regions;
For each of the plurality of frequency bands, an edge detection unit that detects edge intensity from a low frequency component in a frequency component of the wavelength region selected by the wavelength region selection unit;
Noise that removes noise components from high-frequency components in the frequency components of each frequency band separated by the frequency separation unit and that stores edge components for the high-frequency components based on the edge strength detected by the edge detection unit A removal section;
In each of the plurality of frequency bands, a frequency synthesizer that synthesizes the high frequency component in which the noise component is removed and the edge component is preserved by the noise removing unit;
Equipped with a,
The wavelength region selector is
For each of the plurality of wavelength regions, a noise amount calculation unit that calculates a noise amount from a high frequency component in a frequency component of each frequency band separated by the frequency separation unit;
A wavelength region determination unit that determines a wavelength region of a high-frequency component that is the smallest amount of noise from the amount of noise of the high-frequency component for each of the plurality of wavelength regions obtained by the noise amount calculation unit;
With
An image processing apparatus, wherein a wavelength region determined by the wavelength region determination unit is selected from the plurality of wavelength regions . The image processing apparatus according to claim 1, wherein the plurality of wavelength regions include a red region, a green region, and a blue region of visible wavelengths. The image processing apparatus according to claim 2 , wherein the plurality of wavelength regions include luminance information obtained from a red region, a green region, and a blue region of the visible wavelength. Wherein the plurality of wavelength regions, an image processing apparatus according to any one of claim 1 to 3, characterized in that it comprises an infrared wavelength region. A frequency separation step of separating an input image including a plurality of wavelength regions into frequency components in a plurality of frequency bands;
A wavelength region selection step of selecting any one of the plurality of wavelength regions;
For each of the plurality of frequency bands, an edge detection step of detecting edge intensity from a low frequency component in a frequency component of the wavelength region selected in the wavelength region selection step;
Noise that removes noise components from high-frequency components in the frequency components of each frequency band separated in the frequency separation step and that stores edge components for the high-frequency components based on the edge strength detected in the edge detection step A removal step;
In each of the plurality of frequency bands, a frequency synthesizing step of synthesizing the high frequency component in which the noise component is removed and the edge component is stored in the noise removing step;
Only including,
The wavelength region selection step includes
For each of the plurality of wavelength regions, a noise amount calculation step of calculating a noise amount from a high frequency component in a frequency component of each frequency band separated in the frequency separation step;
A wavelength region determination step of determining a wavelength region of a high-frequency component that is the least amount of noise from the amount of noise of the high-frequency component for each of the plurality of wavelength regions obtained in the noise amount calculation step;
Including
In the wavelength region selection step, the wavelength region determined in the wavelength region determination step is selected from the plurality of wavelength regions .

Images