JP7423033B2 - Image signal processing method - Google Patents

Description

The present invention relates to an image signal processing method for reducing color noise.

In recent years, image signal processing methods have been proposed for reducing color noise, which is color-related noise, in image data.

Color noise is noise that occurs when color information that should not originally exist is included in image data for some reason.

Therefore, when image data containing this color noise is output on the rear LCD screen of a digital camera, a monitor screen, or a paper medium, color information that does not originally exist is displayed as an image signal, giving the photographer a sense of discomfort.

Patent Document 1 discloses that after white balance correction, color difference data is calculated for each pixel of image data, and for pixel data whose color difference data is within a certain range and whose brightness is within a specified brightness range, A technique is disclosed in which color noise reduction processing is performed so that the color difference data of the pixel is reduced.

An object of the present invention is to be able to remove high frequency components that can cause color noise, and even if processing such as filtering with a large filter size is performed on an image signal from which color noise of high frequency components has been removed, color noise can be removed. It is an object of the present invention to provide an image signal processing method capable of suppressing spread and maintaining high frequency information more than when noise processing is applied to all high frequency components of each color.
Publication No. 5977565

The image signal processing method disclosed in Patent Document 1 has a problem in that the smaller the color difference data, the more effective the color noise reduction process can be obtained, but it is difficult to obtain the effect of the color noise reduction process in areas where the color difference data is large. has.

Therefore, it becomes difficult to obtain a color noise reduction effect particularly for isolated strong noise components, and there is a problem that isolated color noise remains.

On the other hand, if an attempt is made to obtain the effect of color noise reduction processing even in areas with large color difference data in order to eliminate isolated color noise, the result will be multicolor bleeding and a significant loss of color difference detail.

The present invention was made in view of this situation, and by performing image signal processing using multicolor high frequency information, it is possible to remove high frequency components that can cause color noise, and it is possible to remove high frequency components that can cause color noise. Even if processing such as filtering with a large filter size is performed on the image signal from which component color noise has been removed, spread noise is less likely to occur. An object of the present invention is to provide an image signal processing method that can maintain information on images.

A first invention, which is a means for solving the above problems, is an image signal processing method for processing a plurality of color image signals, in which the color image signal of one color, which is a color to be corrected, is selected as a signal value to be corrected. a weighted average calculation unit that calculates a weighted average signal value of image signals of a plurality of colors other than the correction target signal value, and a weighted average calculation unit that calculates a first a first subtraction unit that calculates a subtracted signal value of the first subtracted signal value; a first high-frequency component reduction unit that calculates a first high-frequency component reduced signal value by applying a low-pass filter to the first subtracted signal value; a second subtraction unit that calculates a second subtraction signal value by subtracting the first high frequency component reduction signal value from the subtraction signal value; a second high-frequency component reduction unit that calculates a high-frequency component reduction signal value; and a third subtraction unit that calculates a third subtraction signal value by subtracting the second high-frequency component reduction signal value from the second subtraction signal value. and a fourth subtraction unit that calculates a fourth subtraction signal value by subtracting the third subtraction signal value from the correction target signal value, and a fourth subtraction unit that calculates a fourth subtraction signal value by subtracting the third subtraction signal value from the correction target signal value. An image signal processing method characterized in that the signal value to be corrected is selected in .

Further, a second invention, which is a means for solving the above-mentioned problem, is the image signal processing method according to the first invention, wherein the first high frequency component reduction unit reduces the first subtracted signal value. An image signal processing method comprising: resizing and reducing the image, and then resizing and enlarging it again to return to the original size to calculate a first high frequency component reduction signal value.

Further, a third invention, which is a means for solving the above-mentioned problem, is the image signal processing method according to the first or second invention, wherein the second subtraction unit An image signal processing method, comprising: a first threshold section that applies a process of replacing the second subtracted signal value with a threshold value when the value exceeds the threshold value.

Further, a fourth invention, which is a means for solving the above-mentioned problem, is the image signal processing method according to any one of the first invention to the third invention, wherein the third subtraction unit includes: An image signal processing method, comprising: a second threshold unit that applies a process of replacing the third subtraction signal value with a threshold value when the third subtraction signal value exceeds a threshold value.

Further, a fifth invention, which is a means for solving the above-mentioned problem, is the image signal processing method according to any one of the first invention to the fourth invention, wherein the plurality of color image signals are An image signal processing method characterized in that the image signal is an image signal converted by a stacked solid-state image sensor.

According to the present invention, by performing image signal processing using multicolor high frequency information, it is possible to remove high frequency components that can cause color noise, and to produce an image signal from which color noise of high frequency components has been removed. Even when processing such as filtering with a large filter size is performed, spreading noise is less likely to occur, and furthermore, it is possible to maintain high-frequency information better than when noise processing is applied to all color signals of each color. A signal processing method can be provided.

A block diagram showing the configuration of a solid-state imaging device according to an embodiment of the present invention Flowchart for performing noise reduction according to an embodiment of the present invention

DESCRIPTION OF THE PREFERRED EMBODIMENTS The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment.

In this embodiment, 100 is a lens barrel, 101 is a solid-state image sensor, 1011 is a microlens, 1012 is a photoelectric conversion unit, 1013 is a wiring unit, 102 is a CPU, 1021 is an image signal determination unit, and 1022 is a weighted average calculation unit. 1023 is a first subtraction unit, 1024 is a first high frequency reduction unit, 1025 is a second subtraction unit, 1026 is a second high frequency reduction unit, 1027 is a third subtraction unit, 1028 is a fourth subtraction unit, and 1029 is a first subtraction unit. The threshold section 1030 is a second threshold section.

FIG. 1 is a configuration diagram showing the configuration of an imaging device according to this embodiment.

The imaging device 10 includes a lens barrel 100 and a camera body (not shown).

The lens barrel 100 is removably attached to the camera body, and the mount portion of the lens barrel 100 and the mount portion of the camera body are connected by a bayonet mechanism or the like. Note that the lens barrel 100 may be fixed to the camera body.

The solid-state image sensor 101 includes a microlens 1011, a photoelectric conversion section 1012, a wiring section 1013, and the like.

The microlens 1011 is a lens that condenses the light flux incident on the lens barrel 100 so that each photoelectric conversion unit 1012 is suitable for receiving the light.

The photoelectric conversion unit 1012 converts the light beam that has passed through the microlens 1011 into an image signal.

The image signal converted by the photoelectric conversion unit 1012 will be explained. A photoelectric conversion unit 1012 that converts a luminous flux into an image signal is arranged at each pixel. When the solid-state image sensor 101 is a Bayer type, the image signal is converted from the photoelectric conversion unit 1012 arranged in each pixel into an image signal of the same color as the microlens 1011 and the color filter arranged in the photoelectric conversion unit 1012. Three colors, red (R), green (G), and blue (B), are generally used for color filters, and at this time, there are three types of image signals to be converted.

On the other hand, when the solid-state image sensor 101 is a stacked type, image signals corresponding to the number of stacked photoelectric conversion units are converted from one pixel. In the embodiment of the present application, the photoelectric conversion section 1012 is laminated in three layers in order to obtain image signals of three colors, red (R), green (G), and blue (B) in one pixel.

The wiring unit 1013 transfers the image signal converted by the photoelectric conversion unit 1012 from each photoelectric conversion unit 1012 to the CPU 102.

The CPU 102 performs various processes on the image signals converted by the photoelectric conversion unit 1012 and transferred through the wiring unit 1013.

The image signal selection unit 1021 determines an image signal to be corrected from the image signal converted by the photoelectric conversion unit 1012 and transferred through the wiring unit 1013.

The weighted average calculation unit 1022 calculates a weighted average from the image signals other than the image signal to be corrected determined by the image signal selection unit 1021. The specific calculation method will be described later.

The first subtraction unit 1023 subtracts the weighted average of image signals other than the correction target calculated by the weighted average calculation unit 1022 from the correction target image signal determined by the image signal selection unit 1021.

The first high-frequency component reduction unit 1024 applies a low-pass filter to the weighted average calculated by the weighted average calculation unit 1022, thereby reducing high-frequency components and extracting low-frequency components as a result. .

The second subtraction unit 1025 subtracts the image signal extracted by the first high frequency component reduction unit 1024 from the calculation result of the first subtraction unit 1023.

The second high-frequency component reduction unit 1026 applies a low-pass filter to the calculation result of the second subtraction unit 1025.

The third subtraction unit 1027 subtracts the image signal extracted by the second high frequency component reduction unit 1026 from the result calculated by the second subtraction unit 1026.

The fourth subtraction unit 1028 subtracts the image signal calculated by the third subtraction unit 1027 from the image signal determined by the image signal selection unit 1021. As a result, the image signal to be corrected becomes an image signal from which color noise has been removed.

The first threshold unit 1029 controls the correction amount by applying the calculation result of the second subtraction unit 1024 and replacing it with the threshold value when the calculation result exceeds the threshold range.

The second threshold unit 1030 controls the correction amount by applying the calculation result of the third subtraction unit 1026 and replacing it with the threshold value when the calculation result exceeds the threshold range.

Next, an embodiment of the image signal processing method according to the present invention will be described using the flowchart of FIG.

As shown in FIG. 1, the light flux that has passed through the lens barrel 100 reaches the solid-state image sensor 101, is guided to the photoelectric conversion unit 1012 via the microlens 1011, and is then converted into an image signal. In the solid-state image sensor 101 used in the embodiment of the present application, three layers of photoelectric conversion sections 1012 are stacked in one pixel. Therefore, three types of image signals are converted from one pixel.

In the image signal determination unit in step #1, the image signal selection unit 1021 determines an image signal to be corrected from among the three types of image signals converted by the photoelectric conversion unit 1012 of the solid-state image sensor 101.

In step #2, the weighted average calculation unit 1022 calculates a weighted average using two types of image signals other than the image signal to be corrected determined by the image signal selection unit 1021 in step #1.

The weighted average coefficient is determined from the difference between the spectral sensitivity of the image signal to be corrected determined in step #1 and the spectral sensitivity of the image signal other than the image signal to be corrected.

For example, increase the weighted average coefficient for image signal components whose spectral sensitivity shape and peak are close to the image signal to be corrected, and increase the weighted average coefficient for image signals with large differences in spectral sensitivity shape and peak. Make it smaller.

In step #3, the first subtraction unit 1023 subtracts the weighted average calculated by the weighted average calculation unit 1022 in step #2 from the image signal to be corrected determined by the image signal determination unit 1021 in step #1. . This subtraction result is defined as the first subtraction result.

In step #4, the first high frequency component reduction unit 1024 extracts a low frequency component from the first subtraction result calculated by the first subtraction unit 1023 in step #3 by applying a low pass filter.

In step #4, a relatively large filter size of the low-pass filter is applied.

The reason for increasing the filter size of the low-pass filter is to remove color noise having a wide range, and to determine a signal that takes into account a wider range as the correct color.

This is because if the filter size of the low-pass filter were to be made small, the reliability of colors that are determined to be correct would decrease in cases where the signal is influenced by extreme signals or when there is noise with a wide range.

Note that as long as low frequency components can be extracted, there is no problem with resizing or the like in addition to the method of applying a low-pass filter.

Specifically, there is no problem with a method in which high frequency components are reduced by reducing and resizing at a specific ratio, and then enlarging and resizing at the reciprocal of the ratio.

In step #5, the second subtraction unit 1025 subtracts the low frequency component extracted in step #4 from the first subtraction result in step #3. This result is defined as a second subtraction result, and the second subtraction result is one in which a high frequency component is extracted.

This high frequency component has a color blurring component as well as a noise component. This color blur component occurs around areas where there is a significant color change, such as edges, due to the effect of increasing the filter size of the low-pass filter used in step #4.

For example, if a difference signal is calculated by subtracting the signal after applying a low-pass filter from the original signal for a portion where there is an edge that changes from the small side of the signal to the large side, then in the edge portion from the small side of the signal, the difference signal The signal has a characteristic in which the negative amount increases little by little as it approaches the edge, and the positive amount of the difference signal increases little by little as it approaches the edge from the side where the signal is large.

The range in which the difference signal increases in negative and positive amounts near the edge increases as the filter size of the low-pass filter increases, and becomes a component of color blur.

Since the color blurring component included in this high frequency component has a characteristic that it increases as it approaches the edge, the amount of color blurring and noise component near the edge may be adjusted by providing a threshold value.

In step #6, the second high frequency component reduction unit 1026 applies a low-pass filter to the second subtraction result calculated by the second subtraction unit 1025 in step #5.

By making the filter size of the low-pass filter of the second high-frequency component reduction unit 1026 smaller than that of the low-pass filter used in step #4, the image signal becomes an image signal including the color blurring component of the edge portion. In addition, the reason for reducing the filter size will be described below.

For example, the high frequency component obtained in step #5 includes a color blur component as described above, and this color blur component occurs as a positive side component and a negative side component with the edge portion as a boundary. The second high-frequency component reduction unit in step #6 particularly wants to extract only color blurring components, so it is necessary to apply a low-pass filter to each positive side component bordering on the edge portion and to each negative side component bordering on the edge portion as much as possible. is desired.

If a low-pass filter of the same size as in step #4 is applied, the positive side component bordering on the edge part and the negative side component bordering on the edge part will be added together, making it impossible to exclude color blurring components. Put it away.

In step #7, the third subtraction unit 1027 subtracts the second subtraction result obtained by applying a low-pass filter in step #6 from the second subtraction result calculated in step #5. This subtraction result is defined as the third subtraction result.

The third subtraction result becomes an image signal mainly composed of noise components of the weighted average difference signal between the image signal correction target color and the color other than the image signal target.

Since the color blurring component remaining in this high frequency component has a characteristic that it increases as it approaches the edge, the amount of color blurring and noise component near the edge may be adjusted by providing a threshold value.

In step #8, the fourth subtraction unit 1026 subtracts the image signal to be corrected determined by the image signal determination unit 1021 in step #1 from the third subtraction result calculated in step #7. This subtraction result is defined as the fourth subtraction result.

This fourth subtraction result is used as the corrected image signal of the correction target image signal determined by the image signal determination unit 1021.

Compared to the image signal before processing, the correction target image signal becomes an image signal in which color noise is reduced while maintaining resolution compared to when noise processing is performed on a single color.

Each of the above steps will be specifically explained below.

In this embodiment, a correction method will be specifically described using green (G) among the image signals converted by the photoelectric conversion unit 1012 as an image signal to be corrected.

Since the image signal to be corrected is green (G), in step #1, the image signal selection unit 1021 sets green (G) as the image signal to be corrected.

Next, in step #2, the weighted average calculation unit 1022 calculates a weighted average using red (R) and blue (B), which are image signals other than green (G), which is the image signal to be corrected.

The image signal to be corrected determined in step #1 and the two types of image signals on which the weighted average is calculated in step #2 are image signals obtained from the same pixel. This is because the solid-state image sensor 101 used in the embodiment of the present application is a stacked solid-state image sensor that can obtain three types of image signals from one pixel, as described above.

A specific calculation method performed by the weighted average calculation unit 1022 of the embodiment will be described.

If the red (R) and blue (B) image signals are R and B, respectively, and the coefficients are α and β, respectively, the weighted average can be expressed as follows.
Weighted average: (α*R+β*B) (α+β=1)

Next, the calculation of the first subtraction unit 1023 will be explained in step #3. When the green (G) image signal converted by the photoelectric conversion unit 1012 is G and the green (G) after calculation is G1, the following equation is obtained.
G1=G-(α*R+β*B) (α+β=1)

In step #4, a low frequency component is extracted from G1 calculated in step #3.
G2=LPF(G1)

In step #5, the frequency component extracted in step #4 is subtracted from G1 calculated in step #3. By subtracting the low frequency component from G1, it is possible to extract only the high frequency component.
G3=G1-G2

This high-frequency component becomes the difference between the green image signal and the weighted average image signal using red and blue, and can become a green high-frequency color noise component, but at the same time, this high-frequency component causes color blurring at the edges. have elements.

In step #6, a low-pass filter is applied to the calculation result of the second subtraction unit 1025 as in step #4, thereby extracting a low frequency component from the calculation result of the second subtraction unit 1025.
G4=LPF(G3)
At this time, the size of the low-pass filter to be applied has a smaller filter size (or reduction rate) than the low-pass filter used in step #4.

The reduction ratio represents the image size change ratio when resizing processing is used instead of a low-pass filter. The larger the reduction ratio value, the smaller the image created by resizing.

In step #7, the image signal is obtained by applying a low-pass filter in the second high frequency reduction unit 1026 in step #6 based on the calculation obtained in the second subtraction unit 1025 in step #5.
G5=G3-G4

Further, as an additional embodiment, a threshold value may be used instead of the calculation result.

Specifically, step #5.5 is added between steps #5 and #6. In step #5.5, the first threshold unit 1029 compares the calculation result calculated by the second subtraction unit 1025 in step #5 with the threshold, and if the calculation result is outside the threshold range, it is used in the subsequent steps. The calculation result of the second subtraction unit 1025 is used as a threshold value.

Similarly, step #7.5 is added between steps #7 and #8. In step #7.5, the second threshold unit 1030 compares the calculation result calculated by the third subtraction unit 1027 in step #6 with the second threshold, and if the calculation result is outside the second threshold range, the subsequent steps are performed. The calculation result of the third subtraction unit 1027 used in the step is set as the second threshold value.

Furthermore, in the method of using the first threshold section 1029 and the second threshold section 1030 in addition to the above-described embodiment, it is possible to add only one threshold section or both threshold sections. good.

In step #8, the image signal to be corrected determined by the image signal determination unit 1021 in step #1 from the calculation result calculated in the previous step, step #7 or step #7.5, is corrected. image signal.

Further, in the embodiments described above, the solid-state image sensor 101 is assumed to be a stacked type, but a Bayer type solid-state image sensor having one type of image signal per pixel may be used.

In the case of a stacked solid-state image sensor, since signals at the same position can be acquired, loss of high-frequency information can be easily prevented and color noise suppressing effects can be easily obtained. Furthermore, in the case of a sensor in which colors are mixed, high frequency information of the target signal is included in other color signals, so it is easy to prevent loss of high frequency information and obtain a color noise suppressing effect.

Furthermore, with a Bayer type solid-state image sensor, only one type of image signal can be obtained from one pixel. Therefore, the other two types of image signals can be obtained by interpolating from surrounding pixels. In the present invention, when a Bayer type solid-state image sensor is used, the image signal of one pixel is changed into three types of states through interpolation processing, and then the image signal to be corrected in step #1 is determined. The processing can be similar to that when using a stacked solid-state image sensor.

10 Imaging device 100 Lens barrel 101 Solid-state imaging device 1011 Microlens 1012 Photoelectric conversion unit 1013 Wiring layer 102 CPU
1020 Subtraction unit 1021 Image signal determination unit 1022 Linear sum calculation unit 1023 First low-pass filter unit 1024 First subtraction unit 1025 Second low-pass filter unit 1026 Second subtraction unit 1027 Third subtraction unit 1028 First threshold unit 1029 Second threshold unit

Claims (5)

In an image signal processing method for processing multiple color image signals,
an image signal selection unit that selects the color image signal of one color that is a color to be corrected as a signal value to be corrected;
a weighted average calculation unit that calculates a weighted average signal value of the plurality of color image signals other than the correction target signal value;
a first subtraction unit that obtains a first subtraction signal value by subtracting the weighted average signal value from the correction target signal value;
a first high-frequency component reduction unit that calculates a first high-frequency component reduction signal value by applying a low-pass filter to the first subtraction signal value;
a second subtraction unit that calculates a second subtraction signal value by subtracting the first high frequency component reduction signal value from the first subtraction signal value;
a second high-frequency component reduction unit that calculates a second high-frequency component reduction signal value by applying a low-pass filter to the second subtraction signal value;
a third subtraction unit that calculates a third subtraction signal value by subtracting the second high frequency component reduction signal value from the second subtraction signal value;
a fourth subtraction unit that calculates a fourth subtraction signal value by subtracting the third subtraction signal value from the correction target signal value;
Consisting of
An image signal processing method characterized in that the fourth subtraction signal value is the correction target signal value selected by the image signal selection section . Regarding the image signal processing method according to claim 1,
The first high-frequency component reduction unit is characterized in that after resizing and reducing the first subtraction signal value, the first subtraction signal value is resized and enlarged again to return to the original size, thereby calculating a first high-frequency component reduction signal value. Image signal processing method. Regarding the image signal processing method according to any one of claims 1 to 2,
The second subtraction unit includes a first threshold unit that applies a process of replacing the second subtraction signal value with a threshold value when the second subtraction signal value exceeds a threshold value. Processing method. Regarding the image signal processing method according to any one of claims 1 to 3,
The third subtraction unit includes a second threshold unit that applies a process of replacing the third subtraction signal value with a threshold value when the third subtraction signal value exceeds a threshold value. Processing method. Regarding the image signal processing method according to any one of claims 1 to 4 ,
An image signal processing method, wherein the plurality of color image signals are image signals converted by a stacked solid-state image sensor.

Images