JP2004159176A - Noise elimination method, imaging apparatus, and noise elimination program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To effectively reduce the noise of an image constituted of multiple pixels. <P>SOLUTION: A single CCD image pickup device 1 subjects a G signal out of output signals of a single CCD imaging sensor 20 to special median processing and subjects an R signal and a B signal to averaging processing or processing with a median filter. Consequently, information of details of the image are allowed to remain while an effect of noise elimination is kept equivalent to that in images subjected to processing with the median filter throughout, and isolate points in the image are maintained though noise elimination is performed. That is, proper noise elimination can be performed while maintaining the quality of the image. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a noise elimination method, an imaging device, and a noise elimination program for reducing noise of an image captured by an image sensor.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a process for removing noise from a captured image is performed in an image pickup apparatus or the like including a single-plate type color image sensor including a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. ing.
[0003]
As a method of removing noise from a captured image, for example, a method of processing an image (a completed image subjected to color interpolation or the like) generated from an output signal of a single-chip image sensor using an LPF (Low Pass Filter), A processing method using a median filter is known.
For example, Japanese Patent Laying-Open No. 4-235472 discloses a technique for removing noise from digital image data using a median filter.
[0004]
Also, Japanese Patent Application Laid-Open No. 2001-144964 discloses a technique for removing noise by approximating a change in an image by a function and replacing the approximate data with the original image data.
[0005]
[Patent Document 1]
JP-A-4-235472 [Patent Document 2]
JP 2001-144964 A
[Problems to be solved by the invention]
However, the technique described in Japanese Patent Application Laid-Open No. 4-235472 has a problem that an isolated point in an image is easily lost as a result of the filter processing because a median filter is used.
In the technique described in Japanese Patent Application Laid-Open No. 2001-144964, it is necessary to obtain a function for approximating a change in an image. However, the process of calculating the function is complicated, and the information is mosaic-like. Does not correspond to an image in which are arranged (for example, an image of each color in the RGB system).
[0007]
An object of the present invention is to effectively reduce noise of an image formed by a plurality of pixels.
[0008]
[Means for Solving the Problems]
In order to solve the above problems, the present invention provides
This is a method for removing noise from an image composed of a plurality of pixels, wherein a median of signal values of predetermined surrounding pixels (for example, a 3 × 3 small matrix shown in FIG. 2) including a processing target pixel and a processing target pixel (for example, 2, a difference (for example, “diff” in the embodiment of the present invention) from a signal value of a central G signal of interest in the small matrix of FIG. The threshold value “th” in the embodiment), and when the difference is larger than the threshold value, the signal value of the processing target pixel is used. When the difference is equal to or smaller than the threshold value, the median is processed. It is characterized in that a special median process used as a signal value of a pixel is performed on a pixel constituting the image. An image pickup apparatus that removes noise of an image composed of a plurality of pixels, comprising: a difference between a median of a signal value of a predetermined surrounding pixel including a pixel to be processed and a signal value of the pixel to be processed; If the difference is larger than the threshold value, the signal value of the pixel to be processed is used. If the difference is smaller than the threshold value, the special median using the median as the signal value of the pixel to be processed. It is characterized in that the processing is performed on pixels constituting the image.
[0009]
Further, a noise removal program for removing noise of an image composed of a plurality of pixels, the difference between the median of the signal value of the predetermined surrounding pixels including the processing target pixel and the signal value of the processing target pixel, Compare with a predetermined threshold, if the difference is greater than the threshold, use the signal value of the processing target pixel, if the difference is less than or equal to the threshold, the median as the signal value of the processing target pixel It is characterized in that a computer realizes a function of performing a special median process to be used on pixels constituting the image.
[0010]
ADVANTAGE OF THE INVENTION According to this invention, compared with the case where a median filter is applied uniformly to the whole image, the information of the detail of an image can be left, making the effect of noise removal equal, and even if noise removal is performed, Isolated points can be maintained. That is, it is possible to perform appropriate noise removal while maintaining the information amount.
Further, among the signal components constituting the pixel, a signal component having a higher relationship with the luminance of an image (for example, a G signal in an RGB system) is subjected to the special median processing.
ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to reduce the processing load in a filtering process, leaving comparatively detailed information of an image.
Further, the image is formed by an image sensor that outputs color signals of different components (for example, an R signal, a G signal, a B signal, or the like in the RGB system) based on a predetermined system (for example, an RGB system or a CyYeMgGr system). And the special median process is performed on the color signal of at least one of the color signals of the different components.
[0011]
ADVANTAGE OF THE INVENTION According to this invention, compared with the case where the whole image is processed by the median filter with respect to the image image | photographed by the single-plate type image sensor, it can leave the information of the detail of an image, making the effect of a noise removal equal. Even when noise is removed, isolated points in an image can be maintained. That is, it is possible to perform appropriate noise removal while maintaining the information amount.
Further, the image sensor outputs color signals of different components based on the RGB system, and performs the special median processing on the G signal in the RGB system.
[0012]
According to the present invention, in an RGB image captured by a single-plate image sensor, a special median process is performed on a G signal having a large effect on luminance, so that noise can be effectively removed.
Further, of the color signals of the different components, the color signal of the component not subjected to the special median processing is subjected to either an averaging process or a process by a median filter.
[0013]
According to the present invention, since a process using a conventionally used filter is used for a color signal of a component having a relatively small relationship with the luminance of an image, it is possible to improve image quality while improving practicality.
Further, it is characterized in that the threshold value is changed based on information relating to the luminance of the image.
[0014]
Here, the "information related to the luminance of the image" refers to a signal component representing the luminance of the image, a signal component having a constant correlation with the luminance of the image, or a signal component for adjusting the luminance. It includes the gain. Further, since the luminance of an image has a considerable relationship with the amount of noise included in the image, the luminance of the image includes a result obtained by measuring noise included in the image.
According to the present invention, it is possible to perform appropriate noise removal according to an image by dynamically changing a threshold based on information relating to luminance.
[0015]
A method for removing noise of an image captured by a single-panel imaging sensor in which an image sensor that outputs color signals of different components based on a predetermined method is arranged in a predetermined pattern, wherein the image signal includes It is characterized in that a median filter is applied to the color signal of a component having a higher relationship with the luminance of the component, and an averaging process is applied to the color signals of other components.
An image pickup device that includes an image sensor that outputs color signals of different components based on a predetermined method and includes a single-plate image sensor arranged in a predetermined pattern, and removes noise of an image captured by the single-plate image sensor. Among the color signals of the different components, a color signal of a component having a higher relationship with the luminance of an image is subjected to processing by a median filter, and an averaging process is performed to color signals of other components.
[0016]
A noise removal program that removes noise of an image captured by a single-panel imaging sensor in which an image sensor that outputs color signals of different components based on a predetermined method is arranged in a predetermined pattern. Among them, the feature is that a computer realizes a function of performing a process using a median filter on a color signal of a component having a higher relationship with the luminance of an image and performing an averaging process on color signals of other components.
According to the present invention, it is possible to reduce the processing load as compared with the conventional case where only the processing by the median filter is performed, while relatively maintaining the information of the details of the image.
Further, the image pickup device outputs color signals of different components based on the RGB system, performs a process by a median filter on the G signal, and performs an averaging process on the R and B signals.
According to the present invention, in an image based on the RGB system captured by a single-panel imaging sensor, a G signal before color interpolation is subjected to processing by a median filter, thereby leaving relatively detailed information of the image, By performing the averaging process on the B signal and the B signal, a certain noise removing effect can be obtained.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of a single-panel imaging device according to the present invention will be described with reference to the drawings.
First, a noise removal method according to the present invention will be described.
In the noise elimination method according to the present invention, a predetermined noise elimination process is performed on an output signal of a single-chip image sensor provided in the single-chip image sensor before performing color interpolation or the like. In this way, by performing the noise removal processing on the output signal of the single-plate image sensor, the memory used for the filter processing is reduced compared to the case where the filter processing is performed on the completed image after performing the color interpolation or the like. Can be reduced.
[0018]
Hereinafter, a specific method will be described.
The single-plate image sensor has a configuration in which image sensors that output respective color signals are arranged in a mosaic pattern. That is, the single-plate image sensor has a row (hereinafter referred to as an “RG line”) in which a predetermined number of image sensors that output R signals and image sensors that output G signals in a primary color filter system (RGB system) are alternately arranged. Similarly, rows in which a predetermined number of image pickup elements that output G signals and image pickup elements that output B signals are alternately arranged (hereinafter, referred to as “GB lines”) are repeated in the column direction. It is constituted by being arranged.
[0019]
The present invention is applicable to a single-plate image sensor that outputs color signals of the RGB system and also outputs other color signals such as a complementary color filter system (CyYeMgGr system). At this time, the lines corresponding to the above-described RG line and GB line have an arrangement of color signals corresponding to the respective systems. For example, in the case of the CyYeMgGr system, an imaging element that outputs a Cy (cyan) signal and a Ye (yellow) A CyYe line in which imaging elements for outputting signals are alternately arranged, and an MgGr line in which imaging elements for outputting Mg (magenta) signals and imaging elements for outputting Gr (green) signals are alternately arranged. The following description is based on the assumption that the single-plate image sensor outputs RGB color signals.
[0020]
At this time, an array (hereinafter, referred to as a “sensor output matrix”) including output signal values of the image sensor on each of the RG line and the GB line is obtained as an output of the RGB single-panel imaging sensor. FIG. 1 is a conceptual diagram showing a sensor output matrix.
In FIG. 1, each matrix element (hereinafter, appropriately referred to as “imaging pixel”) includes only one of R, G, and B signal values. Since the G signal has a spectral distribution characteristic close to that of the luminance signal, and the human eye has a good spatial frequency characteristic with respect to luminance, a high-resolution image can be restored by appropriately capturing a spatial change of the G signal. Therefore, the G signal is configured to output twice as many pixels as the R and B signals.
[0021]
Therefore, for the G signal, it is desirable that a certain level of noise removal be performed while the information remains as much as possible by performing the filtering process.
Therefore, in this method, noise removal is performed on the G signal using a process based on a median filter (hereinafter, referred to as “special median process”) in which detailed information is relatively likely to remain. .
[0022]
When a median filter is used, there is a feature that edge information and the like are left, but information of an isolated point is easily lost. Therefore, in the special median processing, in order to prevent such an adverse effect, it is determined whether the isolated point is noise or the original image information, and when it is determined that the isolated point is noise, the isolated point is determined. Is processed by the median filter, and when it is determined that the image information is the original image information, the isolated point is held as it is.
[0023]
It should be noted that an isolated point due to a pixel defect or the like of the image sensor is processed by a conventionally performed pixel defect correction.
Subsequently, a method of determining whether an isolated point is noise or original image information in the special median processing will be described.
In this determination method, in a matrix including only G signals among the sensor output matrices (hereinafter referred to as a “G matrix”), a 3 × 3 matrix as shown in FIG. Extract the small matrix of. Then, when the difference between the median of the five G signals included in the small matrix and the G signal of interest is larger than the threshold value, it is determined that the image information is the original image information. , Noise.
[0024]
For example, it is assumed that the signal values of the five G signals in FIG. 2 are (d1, d2, d3, d4, d5) and the signal value of the G signal of interest is d3. At this time, if the output value of the median filter is dm, the difference diff = | d3-dm | is satisfied. If the difference diff at this time is larger than the threshold value th (diff> th), d3 is filtered. If the difference diff is equal to or smaller than the threshold th (diff ≦ th), the median dm is used as the output value after the filtering process instead of d3.
[0025]
Here, the threshold value th is determined based on the following examination result.
Assuming that the distribution of noise included in the image information can be approximated by a Gaussian distribution, the Gaussian distribution can be uniquely determined by a standard deviation σ as a function of luminance.
Therefore, the threshold th is determined using the Gaussian distribution, and the threshold th is determined by obtaining an appropriate value of the coefficient a with respect to the standard deviation σ. Decreasing the threshold th decreases the noise removal performance, and increasing the threshold th tends to cause loss of image information. Therefore, it is important to determine an appropriate threshold th.
[0026]
FIG. 3 is a diagram showing the relationship between various thresholds th (a · σ) when the noise distribution is a Gaussian distribution and the ratio of noise that is removed from the entire noise (actually, Many of the distributions are close to Gaussian). FIG. 3 also shows the number of error pixels and the frequency of occurrence of errors in the case of a CIF (Common Intermediate Format) image.
[0027]
In general, a value of 3σ is often used as a criterion for determining whether a certain value is an error. However, as shown in FIG. 3, when the threshold th is set to 3σ in the image, a noise error of 282 occurs in an image of 352 × 288 pixels of the CIF size, and the noise removal performance is insufficient. There is (ie, an image in which noise errors are visually noticeable).
[0028]
Therefore, here, 4σ is adopted as the threshold th.
However, the amount of noise included in the image information often changes depending on the value of the data, depending on the configuration of the image sensor and the system used. FIG. 4 is a diagram illustrating an example in which the noise amount increases due to the sensor output. Therefore, it is possible to perform appropriate noise removal by reading the luminance value and changing the threshold value according to the luminance value. That is, it is possible to dynamically change the threshold according to the luminance value. Furthermore, the threshold value th can be determined by referring to the gain of an analog amplifier connected for the purpose of amplifying the output signal of the single-chip image sensor to a value within a certain range as a parameter.
[0029]
Next, in the noise elimination method according to the present invention, a noise elimination method for the R signal and the B signal will be described.
For the R signal and the B signal, averaging processing (processing by a low-pass filter) is performed with emphasis on practicality.
Here, the averaging process is characterized in that the burden on hardware is smaller than that of the process using a median filter, and the processing time is shorter in terms of software.
[0030]
On the other hand, the averaging process has a feature in that although the effect of noise reduction is great, information on details of an image is easily lost.
However, the R signal and the B signal have half the number of pixels as compared to the G signal, and have little effect on luminance information. Has no significant effect.
[0031]
The R signal and the B signal can be subjected to the same processing as the processing for the G signal described above or the processing using a median filter.
When the same processing or filtering processing as the above-described processing for G is performed on the R signal and the B signal, a matrix including only the R signal in the sensor output matrix (hereinafter, referred to as an “R matrix”). And a matrix having only B signals as elements (hereinafter, referred to as “B matrix”), a 1 × 5 small matrix as shown in FIG. Then, an average of the three R signals or B signals included in the small matrix is obtained to obtain a filter output value for the target pixel.
[0032]
As described above, in the filter removal method according to the present invention, the G matrix of the sensor output matrix is subjected to the special median processing based on the median filter, and the R matrix and the B matrix of the sensor output matrix are processed. , Processing using a low-pass filter (averaging processing) or processing using a median filter is performed.
Here, assuming that the image quality is represented by the variation (standard deviation) from the pixels of the reference image, the reference image (for example, the second image of the Macbeth chart) can be obtained by using the noise removal method according to the present invention. The standard deviation for an image of a bright patch is as follows.
[0033]
That is, (1) “1.97” before intentional noise was added, (2) “2.83” when noise generated by simulation was added, and (3) noise removal by a median filter. In this case, the result is "2.12", when (4) the averaging process is performed, it is "1.94", and when (5) the special median process is performed, it is "2.13".
However, the standard deviation shown here is an example in a case where the filtering process is performed on a small matrix of 3 × 3 G signals.
[0034]
The standard deviation shown here means that the smaller the value is, the smaller the noise is. In the above results, (3) the case where noise is removed by the median filter and (5) the special median processing is performed In the case, almost the same noise removal effect is obtained. On the other hand, special median processing has the advantage that isolated points in an image are less likely to be lost than noise removal processing using a median filter. It can be said that can be performed.
[0035]
Next, the configuration of the single-panel imaging device 1 using the present method will be described.
FIG. 6 is a block diagram illustrating a configuration of the single-panel imaging device 1 according to the present embodiment. In FIG. 6, a single-panel imaging device 1 includes a lens 10 that converges light from a subject, a single-panel imaging sensor 20 that detects light converged by the lens 10, and a color signal output by the single-panel imaging sensor 20. (RGB signal) and a noise removal unit 30 that performs noise removal processing. The single-panel imaging device 1 includes other functional units for performing color interpolation, color space conversion, and the like as appropriate. However, these functional units are the same as in the related art, and a description thereof will be omitted.
[0036]
The single-plate image sensor 20 has a configuration in which RG lines and GB lines are repeatedly arranged in the column direction.
The noise removing unit 30 holds the G matrix, the R matrix, and the B matrix among the signals output from the single-plate imaging sensor 20, performs a special median process on the G matrix, and averages the R matrix and the B matrix. Or a median filter.
[0037]
In addition, the noise removing unit 30 requires the line memory 40 to perform such processing. For this processing, when the G signal is processed by a small matrix of 3 × 3, and the R and B signals are processed by a small matrix of 1 × 5, the line memory 40 stores the output signal of the single-panel image sensor 20 in two. What is necessary is just to have the capacity to store for the line.
Next, the operation will be described.
[0038]
The single-plate image sensor 20 sequentially outputs image pixels by a raster scan method. That is, the single-plate imaging sensor 20 sequentially outputs the imaging pixels in each row of the sensor output matrix in a certain direction.
Then, the noise removing unit 30 averages the R matrix and the B matrix using the data of the previous line and the previous line stored in the line memory 40 and the output signal of the single-plate image sensor 20. Processing (or processing by a median filter) is performed to generate noise-processed R and B signals. The G matrix is subjected to special median processing to generate noise-processed G signal pixels. Then, the output signal of the single-plate image sensor 20 is stored in the line memory 40 instead of the unnecessary data on the line memory 40 after the processing.
[0039]
Next, using the RGB signals subjected to the noise removal processing, image processing such as color interpolation is performed in the next-stage functional unit, and a completed image is generated.
As described above, the single-panel imaging device 1 according to the present embodiment performs the special median processing on the G signal among the output signals of the single-panel imaging sensor 20, and performs the processing on the R signal and the B signal. Performs an averaging process or a process using a median filter.
[0040]
Therefore, as compared with the case where the processing by the median filter is performed on the entire image, it is possible to retain the information of the details of the image while maintaining the same effect of the noise removal, and to maintain the isolated points in the image even after the noise removal. be able to. That is, it is possible to perform appropriate noise removal while maintaining the information amount.
Further, the single-panel imaging apparatus 1 performs a filtering process on an output signal of the single-panel imaging sensor 20, and thus performs a filtering process on a completed image after performing color interpolation or the like (line memory for each of the R, G, and B signals). The required memory can be greatly reduced as compared with the case where the same capacity is required.
[0041]
Although the present embodiment has been described mainly with respect to color signals of the RGB system, it is also possible to target color signals of other systems such as a complementary color filter system. The effects of the present invention can be achieved by performing special median processing on signal components having a correlation and performing averaging processing or processing by a median filter on other signal components.
Further, although an example in which the reference range of 3 × 3 for the G signal and the reference range of 1 × 5 for the R and B signals is described, the present invention is not limited to this reference range, and a wider reference range can be used, for example.
[0042]
Although an example in which processing is performed on the pixel array of the single-plate sensor has been described, the same effect can be obtained by applying the above processing to general image data.
Further, in the present embodiment, the case has been described where the averaging process is performed on the R signal and the B signal, and the special median process is performed on the G signal, but instead of performing the special median process on the G signal, In addition, the effect of reducing the processing load can be achieved only by performing the processing using the normal median filter. That is, conventionally, a noise removal method of performing processing by a median filter on each of R, G, and B signal component images (three primary color images) on which color interpolation has been performed has been used. For the output signal of the single-plate image sensor 20, the G signal can be subjected to processing by a median filter, and the R and B signals can be averaged. In other words, only the G signal, which easily affects the luminance, is processed by the median filter having a relatively large processing load, and the R signal and the B signal are subjected to the averaging processing with the small processing load, so that the detail of the image can be reduced. It is possible to reduce the processing load as compared with the case where only the processing by the median filter is performed while relatively retaining information.
[Brief description of the drawings]
FIG. 1 is a conceptual diagram showing a sensor output matrix.
FIG. 2 is a diagram showing a small matrix of a G signal.
FIG. 3 is a diagram illustrating a relationship between various thresholds th and a ratio of noise that is removed from the entire noise.
FIG. 4 is a diagram illustrating a relationship between a luminance value and an amount of noise included in image information.
FIG. 5 is a diagram showing a small matrix of a B signal.
FIG. 6 is a block diagram showing a configuration of a single-panel imaging device 1 according to the present embodiment.
[Explanation of symbols]
Reference Signs List 1 single-chip imaging device, 10 lens, 20 single-chip imaging sensor, 30 noise removing unit, 40 line memory

Claims (12)

A noise removal method for an image constituted by a plurality of pixels,
The difference between the median of the signal value of the predetermined surrounding pixel including the pixel to be processed and the signal value of the pixel to be processed is compared with a predetermined threshold. If the difference is larger than the threshold, the signal of the pixel to be processed is A noise removal method, wherein, when the difference is equal to or less than the threshold, a special median process using the median as a signal value of a pixel to be processed is performed on pixels constituting the image. . 2. The noise removing method according to claim 1, wherein the special median processing is performed on a signal component having a higher relationship with the luminance of an image among signal components constituting the pixel. The image is obtained by photographing an image sensor that outputs color signals of different components based on a predetermined method by a single-plate image sensor arranged in a predetermined pattern, and at least one component among the color signals of the different components. 3. The noise removal method according to claim 1, wherein the special median processing is performed on the color signal. The image sensor outputs color signals of different components based on the RGB system,
The noise removal method according to any one of claims 1 to 3, wherein the special median processing is performed on a G signal in the RGB system. The color signal of the component not subjected to the special median processing among the color signals of the different components is subjected to one of an averaging process and a process by a median filter. Noise removal method. The method according to any one of claims 1 to 5, wherein the threshold value is changed based on information related to luminance of an image. An image sensor that outputs color signals of different components based on a predetermined method, a noise removal method for an image captured by a single-plate image sensor arranged in a predetermined pattern,
Noise removal, wherein a color signal of a component having a higher relationship with the luminance of an image among the color signals of the different components is processed by a median filter, and an averaging process is performed on color signals of other components. Method. 8. The imaging device according to claim 7, wherein the image sensor outputs color signals of different components based on the RGB system, performs a G signal processing by a median filter, and performs an averaging process on the R signal and the B signal. Noise removal method. An image pickup apparatus that removes noise of an image formed by a plurality of pixels,
The difference between the median of the signal value of the predetermined surrounding pixel including the pixel to be processed and the signal value of the pixel to be processed is compared with a predetermined threshold. If the difference is larger than the threshold, the signal of the pixel to be processed is compared. An imaging apparatus, wherein a special median process using the median as a signal value of a pixel to be processed is performed on pixels constituting the image when the difference is equal to or less than the threshold value. An image pickup device that outputs a color signal of a different component based on a predetermined method, comprising a single-plate image sensor arranged in a predetermined pattern, and an image pickup device that removes noise of an image captured by the single-plate image sensor.
An image pickup apparatus, wherein, among the color signals of the different components, a color signal of a component having a higher relationship with the luminance of an image is processed by a median filter, and an averaging process is performed on color signals of other components. . A noise removal program for removing noise from an image composed of a plurality of pixels,
The difference between the median of the signal value of the predetermined surrounding pixel including the pixel to be processed and the signal value of the pixel to be processed is compared with a predetermined threshold. If the difference is larger than the threshold, the signal of the pixel to be processed is compared. Value, and when the difference is equal to or less than the threshold, causing a computer to realize a function of performing a special median process using the median as a signal value of a pixel to be processed on pixels constituting the image. Characteristic noise removal program. An image sensor that outputs color signals of different components based on a predetermined method, a noise removal program that removes noise of an image captured by a single-plate image sensor arranged in a predetermined pattern,
Among the color signals of the different components, a function of performing a process by a median filter on a color signal of a component having a higher relationship with the luminance of an image and performing an averaging process on color signals of other components is realized by a computer. A noise removal program characterized by the following.

Images