JP2020198540A - Image signal processing method - Google Patents

Abstract

To provide an image signal processing method capable of removing a high-frequency component that may cause color noises.SOLUTION: An image signal processing method includes: selecting a signal value of one color that is a correction target color; calculating a weighted average signal value of image signals of a plurality of colors other than the correction target; subtracting the weighted average signal value from the correction target signal value to calculate a first subtraction signal value; applying a low-pass filter to the first subtraction signal value to calculate a first high-frequency component reduction signal value; subtracting the first high-frequency component reduction signal value from the first subtraction signal value to calculate a second subtraction signal value; applying a low-pass filter to the second subtraction signal value to calculate a second high-frequency component reduction signal value; subtracting the second high-frequency component reduction signal value from the second subtraction signal value to calculate a third subtraction signal value; subtracting the third subtraction signal value from the correction target signal value to calculate a fourth subtraction signal value; and setting the fourth subtraction signal value as an image signal value for the correction target color.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image signal processing method for reducing color noise.

In recent years, an image signal processing method for reducing color noise, which is noise related to color in image data, has been proposed.

Color noise is noise generated when color information that should not originally exist in image data is included for some reason.

Therefore, when the image data including the color noise is output on the rear liquid crystal screen of the digital camera, the monitor screen, the paper medium, or the like, the color information that does not originally exist is displayed as an image signal, which gives the photographer a sense of discomfort.

In Patent Document 1, after white balance correction, color difference data is calculated for each pixel of image data, and for pixel data in which the color difference data is within a certain range and the brightness is within the specified brightness range, A technique for performing color noise reduction processing so that the color difference data of the pixel becomes small is disclosed.

An object of the present invention is that it is possible to remove high-frequency components that can cause color noise, and even if processing such as filtering with a large filter size is performed on the image signal from which the color noise of the high-frequency components has been removed, the color noise can be removed. It is an object of the present invention to provide an image signal processing method capable of maintaining high frequency information as compared with the case where noise processing is applied to all high frequency components of each color while suppressing the spread.
Gazette No. 5977565

In the image signal processing method disclosed in Patent Document 1, the smaller the color difference data, the more effective the color noise reduction processing can be obtained, but the problem is that it is difficult to obtain the effect of the color noise reduction processing in the portion where the color difference data is large. Has.

Therefore, it is difficult to obtain the effect of reducing color noise especially in the isolated strong noise component, and there is a problem that the isolated color noise remains.

On the other hand, if an attempt is made to obtain the effect of the color noise reduction processing even in a portion where the color difference data is large in order to eliminate the isolated color noise, the result is that the bleeding of multiple colors and the details of the color difference are significantly impaired.

The present invention has been made in view of such a situation, and by performing image signal processing using multicolor high frequency information, it is possible to remove high frequency components that may cause color noise, and high frequencies can be removed. Even if processing such as filtering with a large filter size is performed on the image signal from which the color noise of the component has been removed, widening noise is less likely to occur, and the frequency is higher than when noise processing is applied to the color signals of all colors. It is an object of the present invention to provide an image signal processing method capable of maintaining the information of.

The first invention, which is a means for solving the above problems, is an image signal selection unit that selects a correction target signal value of one color to be a correction target color in an image signal processing method for processing a plurality of color image signals. A weighted average calculation unit that calculates a weighted average signal value of a plurality of colors of image signals other than the correction target, and a first subtraction signal value obtained by subtracting the weighted average signal value from the correction target signal value. The subtraction unit, the first high frequency component reduction unit that calculates the first high frequency component reduction signal value by applying a low pass filter to the first subtraction signal value, and the first from the first subtraction signal value. A second subtraction unit that subtracts the high-frequency component reduction signal value to calculate the second subtraction signal value, and a low-pass filter is applied to the second subtraction signal value to calculate the second high-frequency component reduction signal value. From the second high frequency component reduction unit, the third subtraction unit that calculates the third subtraction signal value by subtracting the second high frequency component reduction signal value from the second subtraction signal value, and the correction target signal value. It comprises a fourth subtraction unit that subtracts the third subtraction signal value and calculates a fourth subtraction signal value, and the fourth subtraction signal value is used as an image signal value of the correction target color. Image signal processing method.

Further, the second invention, which is a means for solving the above-mentioned problems, is the image signal processing method according to the first invention, and the first high frequency component reducing unit is the first subtracted signal value. An image processing method characterized in that the first high-frequency component reduction signal value is calculated by resizing and reducing the size of the image and then resizing and returning the image to the original size.

Further, the third invention, which is a means for solving the above-mentioned problems, is the image signal processing method according to the first or second invention, and the second subtraction unit is the second subtraction signal. An image signal processing method comprising a first threshold value portion to which a process of replacing the second subtraction signal value with a threshold value is applied when the value exceeds the threshold value.

Further, the fourth invention, which is a means for solving the above-mentioned problems, is the image signal processing method according to any one of the first invention to the third invention, and the third subtraction unit is a method. An image signal processing method comprising a second threshold value portion to which a process of replacing the third subtraction signal value with a threshold value is applied when the third subtraction signal value exceeds a threshold value.

The fifth invention, which is a means for solving the above-mentioned problems, is the image signal processing method according to any one of the first invention to the fourth invention, and is the second high-frequency component reduction unit. The image signal processing method, characterized in that the filter size of the low-pass filter used in the above is smaller than the filter size or reduction ratio of the low-pass filter used in the first high-frequency component reduction unit.

The sixth invention, which is a means for solving the above-mentioned problems, is the image signal processing method according to any one of the first to fifth inventions, wherein the plurality of color image signals are , An image signal processing method characterized in that the spectral sensitivities are mixed.

According to the present invention, it is possible to remove high frequency components that can cause color noise by performing image signal processing using multicolor high frequency information, and to obtain an image signal from which the color noise of the high frequency components has been removed. Even if processing such as filtering with a large filter size is performed, widespread noise is unlikely to occur, and an image that can maintain high-frequency information compared to the case where noise processing is applied to the color signals of all colors. A signal processing method can be provided.

A block diagram showing a configuration of a solid-state image sensor according to an embodiment of the present invention. Flowchart for performing noise reduction according to an embodiment of the present invention

Hereinafter, the best mode for carrying out the present invention will be described with reference to the accompanying drawings. The present invention is not limited to this embodiment.

In this embodiment, 100 is a lens barrel, 101 is a solid-state imaging device, 1011 is a microlens, 1012 is a photoelectric conversion unit, 1013 is a wiring unit, 102 is a CPU, 1021 is an image signal determination unit, and 1022 is a weighted average calculation. 1023 is the first subtraction part, 1024 is the first high frequency reduction part, 1025 is the second subtraction part, 1026 is the second high frequency reduction part, 1027 is the third subtraction part, 1028 is the fourth subtraction part, and 1029 is the first. The threshold unit and 1030 are the second threshold unit.

FIG. 1 is a configuration diagram showing a configuration of an imaging device of this embodiment.

The image pickup apparatus 10 includes a lens barrel 100 and a camera body (not shown).

The lens barrel 100 is removable from the camera body, and the mount portion of the lens barrel 100 and the mount portion of the camera body are connected by a bayonet mechanism or the like. The lens barrel 100 may be fixed to the camera body.

The solid-state image sensor 101 includes a microlens 1011, a photoelectric conversion unit 1012, a wiring unit 1013, and the like.

The microlens 1011 is a lens that collects the light beam incident on the lens barrel 100 so that each photoelectric conversion unit 1012 is suitable for receiving light.

The photoelectric conversion unit 1012 converts the luminous flux that has passed through the microlens 1011 into an image signal.

The image signal converted by the photoelectric conversion unit 1012 will be described. The photoelectric conversion unit 1012 that converts the luminous flux into an image signal is arranged in each pixel. When the solid-state image sensor 101 is a Bayer type, the photoelectric conversion unit 1012 arranged in each pixel is converted into an image signal having the same color as the color filter arranged in the microlens 1011 and the photoelectric conversion unit 1012. As the color filter, three colors of red (R), green (G), and blue (B) are generally used, and at this time, three types of image signals are converted.

On the other hand, when the solid-state image sensor 101 is a laminated type, image signals corresponding to the number of stacked photoelectric conversion units are converted from one pixel. In the embodiment of the present application, the photoelectric conversion unit 1012 is laminated in three layers in order to obtain image signals of three colors of red (R), green (G), and blue (B) in one pixel.

The wiring unit 1013 transfers the image signal converted by the photoelectric conversion unit 1012 from each photoelectric conversion unit 1012 to the CPU 102.

The CPU 102 performs each process on the image signal converted by the photoelectric conversion unit 1012 and transferred through the wiring unit 1013.

The image signal selection unit 1021 determines the image signal to be corrected from the image signal converted by the photoelectric conversion unit 1012 and transferred through the wiring unit 1013.

The weighted average calculation unit 1022 calculates the weighted average from the image signals in the image signals other than the image signal to be corrected determined by the image signal selection unit 1021. The specific calculation method will be described later.

The first subtraction unit 1023 subtracts the weighted average of the image signals other than the correction target calculated by the weighted average calculation unit 1022 from the correction target image signal determined by the image signal selection unit 1021.

By applying a low-pass filter to the weighted average, which is the result calculated by the weighted average calculation unit 1022, the first high frequency component reduction unit 1024 can reduce the high frequency component and extract the low frequency component as a result. ..

The second subtraction unit 1025 subtracts the image signal extracted by the first high frequency component reduction unit 1024 from the calculation result of the first subtraction unit 1023.

The second high frequency component reduction unit 1026 applies a low-pass filter to the calculation result of the second subtraction unit 1025.

The third subtraction unit 1027 subtracts the image signal extracted by the second high frequency component reduction unit 1026 from the result calculated by the second subtraction unit 1026.

The fourth subtraction unit 1028 subtracts the image signal calculated by the third subtraction unit 1027 from the image signal determined by the image signal selection unit 1021. As a result, the image signal to be corrected becomes an image signal from which color noise has been removed.

The first threshold value unit 1029 is applied to the calculation result of the second subtraction unit 1024, and when the calculation result exceeds the threshold value range, the correction amount is controlled by replacing the calculation result with a threshold value.

The second threshold value unit 1030 is applied to the calculation result of the third subtraction unit 1026, and when the calculation result exceeds the threshold value range, it is replaced with a threshold value to control the correction amount.

Next, an example of the image signal processing method in the present invention will be described with reference to the flowchart of FIG.

As shown in FIG. 1, the luminous flux that has passed through the lens barrel 100 reaches the solid-state image sensor 101, is guided to the photoelectric conversion unit 1012 via the microlens 1011 and then converted into an image signal. In the solid-state image sensor 101 used in the embodiment of the present application, three layers of photoelectric conversion units 1012 are laminated on one pixel. Therefore, one pixel to three types of image signals are converted.

In the image signal determination unit of step # 1, the image signal selection unit 1021 determines the image signal to be corrected from the three types of image signals converted by the photoelectric conversion unit 1012 of the solid-state image sensor 101.

In step # 2, the weighted average calculation unit 1022 calculates the weighted average using two types of image signals other than the image signal to be corrected determined by the image signal selection unit 1021 in step # 1.

The coefficient of the weighted average is obtained from the difference between the spectral sensitivity of the image signal to be corrected determined in step # 1 and the spectral sensitivity of the image signal other than the image signal to be corrected.

For example, increase the weighted average coefficient of the image signal component whose spectral sensitivity shape and peak are close to those of the image signal to be corrected, and if the image signal has a large difference in spectral sensitivity shape and peak, use the weighted average coefficient. Make it smaller.

In step # 3, the first subtraction unit 1023 subtracts the weighted average calculated by the weighted average calculation unit 1022 in step # 2 from the image signal to be corrected determined by the image signal determination unit 1021 in step # 1. .. This subtraction result is used as the first subtraction result.

In step # 4, the first high-frequency component reduction unit 1024 extracts the low-frequency component from the first subtraction result calculated by the first subtraction unit 1023 in step # 3 by applying a low-pass filter.

In step # 4, the filter size of the low-pass filter is relatively large and applied.

The reason for increasing the filter size of the low-pass filter is to remove color noise having a width, and to judge a signal considering a wider range as a correct color.

This is because if the filter size of the low-pass filter is reduced, the reliability of the color judged to be the correct color is lowered in the case where it is pulled by an extreme signal or when there is noise having a width.

If the low frequency component can be extracted, there is no problem in resizing other than the method of applying the low-pass filter.

Specifically, there is no problem in a method of reducing high frequency components by performing reduction and resizing at a specific ratio and then enlarging and resizing at the reciprocal of the ratio.

In step # 5, the second subtraction unit 1025 subtracts the low frequency component extracted in step # 4 from the first subtraction result in step # 3. This result is used as the second subtraction result, and the second subtraction result is obtained by extracting the high frequency component.

This high frequency component has a color bleeding component as well as a noise component. This color bleeding component is caused by increasing the filter size of the low-pass filter used in step # 4, and is generated around a portion such as an edge where a significant color change has occurred.

For example, when the difference signal is calculated by subtracting the signal after applying the low-pass filter from the original signal for the part having an edge that changes from the small side to the large side of the signal, the difference is calculated from the small side of the signal to the edge part. The signal has a characteristic that the minus amount gradually increases as it approaches the edge, and the plus amount gradually increases as the difference signal approaches the edge from the side where the signal is large to the edge portion.

The width at which the negative amount and the positive amount increase near the edge of the difference signal becomes wider as the filter size of the low-pass filter increases, and becomes a component of color bleeding.

Since the color bleeding component contained in this high frequency component has a characteristic that it becomes larger as it is closer to the edge, the amount of color bleeding near the edge and the noise component may be adjusted by setting a threshold value.

In step # 6, the second high-frequency component reduction unit 1026 applies a low-pass filter to the second subtraction result calculated by the second subtraction unit 1025 in step # 5.

The low-pass filter of the second high-frequency component reduction unit 1026 becomes an image signal including the color bleeding component of the edge portion by making the filter size smaller than the filter size of the low-pass filter used in step # 4. The reason for reducing the filter size will be described below.

For example, the high frequency component obtained in step # 5 contains a color bleeding component as described above, and this color bleeding component is generated in the plus side component and the minus side component with the edge portion as a boundary. Since the second high-frequency component reduction section in step # 6 particularly wants to extract only the color bleeding component, apply a low-pass filter to each positive component with the edge as a boundary and for each negative component with the edge as a boundary. Is desired.

If a low-pass filter of the same size as step # 4 is applied, the positive side component bordered by the edge portion and the negative side component bordered by the edge portion are added together, and the color bleeding component cannot be excluded. It ends up.

In step # 7, the third subtraction unit 1027 subtracts from the second subtraction result calculated in step # 5 by applying a low-pass filter to the second subtraction result in step # 6. This subtraction result is used as the third subtraction result.

The third subtraction result is an image signal mainly composed of the image signal correction target color and the noise component of the weighted average difference signal other than the image signal target.

Since the color bleeding component remaining in the high frequency component has a characteristic that it becomes larger as it is closer to the edge, the amount of color bleeding near the edge and the noise component may be adjusted by setting a threshold value.

In step # 8, the fourth subtraction unit 1026 subtracts the image signal to be corrected determined by the image signal determination unit 1021 in step # 1 from the third subtraction result calculated in step # 7. This subtraction result is used as the fourth subtraction result.

The fourth subtraction result is used as the corrected image signal of the correction target image signal determined by the image signal determination unit 1021.

The image signal to be corrected is an image signal in which color noise is reduced while maintaining a sense of resolution as compared with the image signal before processing, as compared with the case where noise processing is performed with a single color.

Each of the above steps will be specifically described below.

In this embodiment, among the image signals converted by the photoelectric conversion unit 1012, the green (G) is set as the image signal to be corrected, and the correction method will be specifically described.

Since the image signal to be corrected is green (G), the image signal selection unit 1021 in step # 1 sets green (G) as the image signal to be corrected.

Next, in step # 2, the weighted average calculation unit 1022 calculates the weighted average using the red (R) and blue (B) image signals other than the green (G) image signal to be corrected.

The image signal to be corrected determined in step # 1 and the two types of image signals on which the weighted average is calculated in step # 2 are image signals obtained from the same pixel. This is because the solid-state image sensor 101 used in the embodiment of the present application is a stacked solid-state image sensor capable of obtaining three types of image signals from one pixel as described above.

A specific calculation method performed by the weighted average calculation unit 1022 of the embodiment is shown.

Assuming that the red (R) and blue (B) image signals are R and B, respectively, and the coefficients are α and β, respectively, the weighted average can be expressed as follows.
Weighted average: (α * R + β * B) (α + β = 1)

Next, the calculation of the first subtraction unit 1023 will be described in step # 3. Assuming that the converted green (G) image signal of the photoelectric conversion unit 1012 is G and the calculated green (G) is G1, the following equation is obtained.
G1 = G- (α * R + β * B) (α + β = 1)

In step # 4, the low frequency component is extracted from G1 calculated in step # 3.
G2 = LPF (G1)

In step # 5, the frequency component extracted in step # 4 is subtracted from G1 calculated in step # 3. By subtracting the low frequency component from G1, it is possible to extract only the high frequency component.
G3 = G1-G2

This high-frequency component is the difference between the green image signal and the weighted average image signal using red and blue, and can be a green high-frequency color noise component, but at the same time, this high-frequency component causes color bleeding at the edge portion. Has an element.

In step # 6, a low-pass filter is applied to the calculation result of the second subtraction unit 1025 in the same manner as in step # 4, so that a low frequency component is extracted from the calculation result of the second subtraction unit 1025.
G4 = LPF (G3)
At this time, the size of the low-pass filter to be applied is a filter size (or reduction ratio) smaller than that of the low-pass filter used in step # 4.

The reduction ratio represents the image size change ratio when the resizing process is used instead of the low-pass filter. The larger the reduction ratio value, the smaller the image created by resizing.

In step # 7, it is an image signal to which a low-pass filter is applied by the second high frequency reduction unit 1026 of step # 6 from the calculation obtained by the second subtraction unit 1025 of step # 5.
G5 = G3-G4

Further, as an additional embodiment, a threshold value may be used instead of the calculation result.

Specifically, step # 5.5 is added between steps # 5 and # 6. In step # 5.5, the first threshold unit 1029 compares the calculation result calculated by the second subtraction unit 1025 in step # 5 with the threshold value, and if the calculation result is out of the threshold range, it is used in the subsequent steps. The calculation result of the second subtraction unit 1025 is a threshold value.

Similarly, step # 7.5 is added between steps # 7 and # 8. In step # 7.5, the second threshold unit 1030 compares the calculation result calculated by the third subtraction unit 1027 in step # 6 with the second threshold value, and if the calculation result is outside the second threshold range, the subsequent steps. The calculation result of the third subtraction unit 1027 used in the above is set to the second threshold value.

Further, in the method using the first threshold value portion 1029 or the second threshold value portion 1030 in addition to the above-described embodiment, only one of the threshold value portions may be added, or both threshold values may be added. Good.

Step # 8 is after the correction of the image signal to be corrected determined by the image signal determination unit 1021 of step # 1 from the calculation result calculated by step # 7 or step # 7.5, which is the previous step. It becomes the image signal of.

Further, in the examples so far, the solid-state image sensor 101 is assumed to be a laminated type, but a Bayer type solid-state image sensor having one type of image signal per pixel may be used.

In the case of a stacked solid-state image sensor, since signals at the same position can be acquired, it is easy to prevent loss of high-frequency information and to obtain a color noise suppression effect. Further, in the case of a sensor in which colors are mixed, since high frequency information of the target signal is also included in other color signals, it is easy to prevent loss of high frequency information and to obtain a color noise suppression effect.

Further, in the Bayer type solid-state image sensor, only one type of image signal can be obtained from one pixel. Therefore, the other two types of image signals can be obtained by interpolating from the peripheral pixels. In the present invention, when the Bayer type solid-state image sensor is used, the image signal of one pixel is made into three kinds of states by interpolation processing, and then the image signal to be corrected in step # 1 is determined. The process can be the same as when a laminated solid-state image sensor is used.

10 Image pickup device 100 Lens lens barrel 101 Solid-state image sensor 1011 Micro lens 1012 Photoelectric conversion unit 1013 Wiring layer 102 CPU
1020 Subtraction unit 1021 Image signal determination unit 1022 Linear sum calculation unit 1023 1st low-pass filter unit 1024 1st subtraction unit 1025 2nd low-pass filter unit 1026 2nd subtraction unit 1027 3rd subtraction unit 1028 1st threshold unit 1029 2nd threshold unit

Claims (6)

In an image signal processing method for processing a plurality of color image signals,
An image signal selection unit that selects the correction target signal value of one color that is the correction target color, and
A weighted average calculation unit that calculates a weighted average signal value of image signals of a plurality of colors other than the correction target,
A first subtraction unit that obtains a first subtraction signal value by subtracting the weighted average signal value from the correction target signal value,
A first high-frequency component reduction unit that calculates a first high-frequency component reduction signal value by applying a low-pass filter to the first subtraction signal value, and a first high-frequency component reduction unit.
A second subtraction unit that calculates the second subtraction signal value by subtracting the first high-frequency component reduction signal value from the first subtraction signal value, and
A second high-frequency component reduction unit that calculates a second high-frequency component reduction signal value by applying a low-pass filter to the second subtraction signal value, and a second high-frequency component reduction unit.
A third subtraction unit that calculates a third subtraction signal value by subtracting the second high-frequency component reduction signal value from the second subtraction signal value, and a third subtraction unit.
A fourth subtraction unit that calculates a fourth subtraction signal value by subtracting the third subtraction signal value from the correction target signal value, and
Consists of
An image signal processing method characterized in that the fourth subtraction signal value is used as the image signal value of the color to be corrected. The image signal processing method according to claim 1.
The first high-frequency component reduction unit is characterized in that the first high-frequency component reduction signal value is calculated by resizing and reducing the first subtraction signal value and then resizing and returning to the original size. Image processing method. The image signal processing method according to any one of claims 1 and 2.
The second subtraction unit has an image signal having a first threshold value unit to which a process of replacing the second subtraction signal value with a threshold value is applied when the second subtraction signal value exceeds a threshold value. Processing method. The image signal processing method according to any one of claims 1 to 3.
The image signal is characterized in that the third subtraction unit has a second threshold value unit to which a process of replacing the third subtraction signal value with a threshold value is applied when the third subtraction signal value exceeds the threshold value. Processing method. The image signal processing method according to any one of claims 1 to 4.
An image signal processing method characterized in that the filter size of the low-pass filter used in the second high-frequency component reduction unit is smaller than the filter size or reduction ratio of the low-pass filter used in the first high-frequency component reduction unit. The image signal processing method according to any one of claims 1 to 5.
An image signal processing method characterized in that the plurality of color image signals have mixed spectral sensitivities.

Images