JP7263018B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to image processing techniques such as noise reduction.

2. Description of the Related Art Conventionally, an image pickup apparatus such as a digital camera is known to convert an image pickup signal having a small amount of information captured at low illuminance into an output signal having a large amount of information by amplifying the image signal with a digital gain. However, when the imaging signal is amplified, the amount of information increases, and at the same time, the random noise included in the imaging signal is also amplified. In particular, it is known that there is more random noise included in low-luminance areas than in high-luminance areas.
On the other hand, for example, a method disclosed in Patent Document 1 is known. Patent Literature 1 discloses a method of estimating the amount of noise contained in a signal and executing noise reduction processing according to the estimated amount of noise. In this way, by executing noise reduction processing according to the amount of noise contained in the signal, noise reduction processing can be executed even when different amounts of random noise are included between the high luminance portion and the low luminance portion. it becomes possible to

JP-A-2004-72422

However, in the method of executing noise reduction processing according to the estimated noise amount disclosed in Patent Document 1, the red (R) signal, green (G) signal, and blue (B) signal that constitute the image signal When there is a deviation in the distribution, a phenomenon of bias toward a specific color may occur. Such a phenomenon of bias toward a specific color is called "color cast", and in particular, the phenomenon of strange hues in low-luminance areas is emphasized.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to enable noise reduction processing that suppresses color cast.

The present invention comprises a color value calculating means for calculating a color value from an image signal, an average value calculating means for calculating an average value of the image signal, and an evaluation value including the average value and the color value. The image processing apparatus further comprises determining means for determining an intensity coefficient indicating intensity of reduction processing, and processing means for performing the noise reduction processing on the image signal according to the intensity coefficient.

According to the present invention, it is possible to reduce noise while suppressing color cast.

1 is a diagram showing a schematic configuration example of an imaging system; FIG. It is a figure which shows the internal structural example of an imaging device. 3 is a diagram showing the internal configuration of an image processing unit; FIG. It is a figure which shows the example of a chart for demonstrating the subject of a general technique. FIG. 4 is a diagram showing a histogram for explaining a general technical problem; FIG. 4 is a diagram showing a histogram for explaining a general technical problem; It is a figure which shows the internal structural example of the NR process part of 1st Embodiment. 4 is a flowchart of NR processing according to the first embodiment; It is a figure which shows the internal structural example of the NR intensity|strength determination part of 1st Embodiment. 4 is a flowchart of NR intensity determination processing of the first embodiment; FIG. 5 is a diagram showing another internal configuration example of the NR processing unit of the first embodiment; FIG. 9 is a diagram showing still another internal configuration example of the NR processing unit of the first embodiment; It is a figure which shows the internal structural example of the NR process part of 2nd Embodiment. 9 is a flowchart of NR processing according to the second embodiment;

Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram showing a schematic configuration example of an imaging system according to this embodiment.
The imaging system shown in FIG. 1 is composed of a monitoring camera 101 as a device for capturing and processing moving images, and a client device 102 connected in a mutually communicable state via an IP network. . Here, an example in which the monitoring camera 101 has the functions of the image processing apparatus of this embodiment will be described. Note that the client device 102 may have the functions of the image processing device of this embodiment. A moving image captured by the monitoring camera 101 is sent to the client device 102 for display, and is recorded as necessary. In the present embodiment, the monitoring camera 101 is used as an example, but the present invention is not limited to this. For example, digital cameras, digital video cameras, various mobile terminals such as smartphones and tablet terminals equipped with a camera function, industrial cameras, and vehicle-mounted cameras. , a medical camera, and the like.

FIG. 2 is a block diagram showing a schematic internal configuration example of the monitoring camera 101 of this embodiment shown in FIG.
The imaging optical system 201 includes a zoom lens, a focus lens, a blur correction lens, an aperture, a shutter, and the like, and forms (images) an optical image of an object or the like on an imaging surface of the imaging element unit 202 . The image pickup device unit 202 is a color image sensor including an image pickup device that converts light incident on an image pickup surface into an electric signal, and color filters corresponding to pixels of the image pickup device. The imaging device unit 202 receives light that has passed through the color filter and is incident on the imaging surface with the imaging device, converts the light into an electrical signal, and outputs the electrical signal. Assume that the image sensor unit 202 is an image sensor capable of setting an arbitrary exposure time for all pixels. In the case of the monitoring camera 101 of this embodiment, the imaging element unit 202 captures moving images, and the imaging element unit 202 outputs image signals of successive frames on the time axis of the moving images.

The CPU 203 executes processing related to all components of the surveillance camera 101 of this embodiment. The CPU 203 sequentially reads program instructions stored in a ROM (Read Only Memory) 204 and a RAM (Random Access Memory) 205 and executes processing according to the interpreted results. The image pickup system control unit 206 performs various controls such as focusing, shutter opening/closing, and aperture adjustment of the image pickup optical system 201 based on instructions supplied from the CPU 203, such as focus control instructions, shutter control instructions, and aperture control instructions. . A control unit 207 controls each unit via the CPU 203 based on an instruction from the client device 102 . A ROM 204 stores programs executed by the CPU 203 and various setting values. The RAM 205 expands the programs stored in the ROM 204, and temporarily stores data during processing in each configuration.

The A/D conversion unit 208 converts an analog electric signal (analog imaging signal) obtained by photoelectric conversion by the imaging element unit 202 into a digital signal value. A digital signal value obtained by analog-to-digital conversion by the A/D conversion unit 208 is sent to the image processing unit 209 as an image signal for each frame of the moving image. The image processing unit 209 is a part that performs image processing on the image signal, and the details of the image processing will be described later. The encoder unit 210 uses the image signal after image processing by the image processing unit 209 to encode, for example, JPEG or H.264. 264 or other predetermined file format, that is, encoding. The image data after being encoded by the encoder unit 210 is sent to the client device 102 .

<First embodiment>
FIG. 3 is a block diagram showing a schematic internal configuration example of the image processing unit 209 included in the monitoring camera 101 of this embodiment.
The image input unit 301 receives image signals of each frame of a moving image captured by the image sensor unit 202 and analog-to-digital converted by the A/D conversion unit 208 as described above. A development processing unit 302 performs processing such as demosaicing, white balance, gamma, and sharpness on the image signal from the image input unit 301 . An NR (noise reduction) processing unit 303 performs NR processing for reducing random noise generated in the spatial direction and the temporal direction by applying spatial filtering processing or temporal filtering processing to the image processed by the development processing unit 302. . The detailed configuration and operation of the NR processing unit 303 in this embodiment will be described later. The image output unit 304 outputs the image signal processed by the NR processing unit 303 to the encoder unit 210 in FIG. Note that the image input unit 301 and the image output unit 304 are not essential, and the image signal output from the A/D conversion unit 208 is directly input to the development processing unit 302, and the image signal output from the NR processing unit 303 is directly input to the development processing unit 302. It may be configured to output to the encoder section 210 .

Before describing the details of the image processing according to this embodiment, the principle and problems of general NR processing will be described below.
First, the principle of NR processing will be explained.
One of the common NR processes is to compare the pixel values of a pixel of interest and a plurality of reference pixels in the vicinity thereof, and calculate the value obtained by adding and averaging the highly correlated pixels to obtain the pixel value of the pixel of interest after noise reduction processing. I know how to output as At this time, the strength of the NR processing can be controlled by changing the range in which the reference pixels are referred to and the parameters used when calculating the correlation between the reference pixels and the pixel of interest.

Further, as an example of a method for determining the NR processing strength, there is a method for determining the NR processing strength disclosed in Japanese Patent Application Laid-Open No. 2004-72422. In this method, the factors that affect noise are obtained, the noise level on the sensor is estimated, and the NR processing intensity is set according to the estimated noise level. The intensity of NR processing can be set appropriately.

Next, problems of general NR processing known as prior art will be described.
In images with a low SN ratio taken in dark environments such as at night, the R (red) component, G (green) component, and B (blue) component, which are the three primary colors of the image, are biased towards specific colors. phenomenon may occur. For example, as shown in FIG. 4, assume that a chart 400 in which three achromatic regions are arranged in order from the top: a white region 401, a gray region 402, and a black region 403 is photographed. When the chart 400 shown in FIG. 4 is photographed in a low-illuminance environment or a high-temperature environment, for example, random noise that is greatly amplified by the digital gain and dark current noise that occurs due to a temperature rise may affect a particular color. A bias, that is, a color cast occurs. For example, as shown in FIG. 5, a magenta-tinged color cast occurs due to the deviation of the average value among the R, G, and B components (that is, the deviation of each distribution of each color component). . Such color cast is particularly likely to occur in low-luminance areas containing a large amount of random noise. If the NR processing strength determination method described in Japanese Patent Application Laid-Open No. 2004-72422 is applied in the case where there is a deviation in the average value in a low luminance area with a lot of noise, it is possible that a lot of random noise is included in the low luminance area. A strong NR processing strength is set for the estimated region. Therefore, compared to before the NR processing in FIG. 5, as shown in FIG. 6, there is no overlap between the colors of the R, G, and B components, and the difference widens, giving the image a magenta tint. This results in an image in which the color cast is further emphasized.

Therefore, in the present embodiment, as will be described later, not only the noise level obtained from the image signal, but also the average value of each color component of RGB and the color change before and after image processing including NR processing are performed. Determine treatment intensity. As a result, in the present embodiment, it is possible to realize NR processing that suppresses color cast even when the distribution of each color component is uneven.

FIG. 7 is a block diagram showing an example of the internal configuration of the NR processing unit 303 of this embodiment, which performs NR processing on a moving image signal and outputs it.
The NR processing unit 303 of this embodiment includes an input signal acquisition unit 701, a noise level calculation unit 702, an average value calculation unit 703, a color value calculation unit 704, an NR intensity determination unit 705, an NR processing execution unit 706, and an output signal holding unit. 707.

The input signal acquisition unit 701 selects, for example, a frame of interest at a certain time among frames that constitute a moving image and are continuous on the time axis, and obtains the R, G, and B components of the frame of interest. Image signals of three primary colors (hereinafter referred to as RGB input signals) are acquired.

The noise level calculator 702 performs noise level calculation processing to calculate the noise level (noise amount) for each RGB component from the RGB input signal. The details of the noise level calculation process for each RGB component will be described later. Noise level calculation section 702 then outputs a signal representing the noise level calculated for each RGB component to NR intensity determination section 705 .

The average value calculation unit 703 performs a process of calculating an average value for each RGB component from the RGB input signal. The details of the average value calculation process for each RGB component will be described later. Then, average value calculation section 703 outputs a signal representing the average value calculated for each RGB component to NR intensity determination section 705 .

A color value calculation unit 704 performs processing for calculating color values of each signal from an RGB input signal and a signal after NR processing by an NR processing execution unit 706 described later (hereinafter referred to as an RGB output signal). The details of the process of calculating color values from the RGB input signal and the RGB output signal will be described later. Then, the color value calculation unit 704 outputs signals representing the colors of the RGB input signal and the RGB output signal to the NR intensity determination unit 705 .

The NR intensity determination unit 705 calculates the noise level for each RGB component calculated by the noise level calculation unit 702, the average value for each component calculated by the average value calculation unit 703, and the color value calculation unit 704. An intensity coefficient for NR processing is determined based on the signal representing the calculated color value. The details of how to determine the intensity coefficient will be described later. NR strength determination section 705 then outputs a signal representing the strength coefficient to NR processing execution section 706 .

The NR processing execution unit 706 executes NR processing on the RGB input signals acquired by the input signal acquisition unit 701 based on the NR processing intensity determined by the NR intensity determination unit 705 . The RGB output signals after this NR processing are sent to the output signal holding unit 707 and the image output unit 304 in FIG.

The output signal holding unit 707 holds the RGB output signals on which NR processing has been performed by the NR processing execution unit 706 . The RGB output signals held in the output signal holding unit 707 are input as RGB output signals after the above-described NR processing when the color value calculation unit 704 calculates colors.

Finally, the RGB output signal output from the NR processing execution unit 706 is sent to the image output unit 304 in FIG. 3 as the current frame image signal after the NR processing by the NR processing execution unit 706 .

FIG. 8 is a flow chart showing the flow of processing performed in each block of the NR processing unit 303 shown in FIG. In the description of the flowchart in FIG. 8, steps S801 to S807 are abbreviated as S801 to S807, respectively. The configuration of FIG. 7 and the processing of the flowchart of FIG. 8 may be executed by a hardware configuration, or may be partially implemented by a software configuration and the rest by a hardware configuration. When executed by software, the processing of the flowchart of FIG. 8 is realized by the CPU executing a program written in, for example, a ROM. The same applies to other flowcharts to be described later.

In S801 of FIG. 8, the input signal acquisition unit 701 acquires the above-described RGB input signal. In the following description, the RGB components of the RGB input signal are expressed as R _IN (v, h), G _IN (v, h), and B _IN (v, h). Note that (v, h) represents two-dimensional coordinate positions in the horizontal direction (x-axis direction) and vertical direction (y-axis direction) in the image. Here, in this embodiment, processing for RGB input signals will be described, but there are no particular restrictions on the data of the input signals processed by the NR processing unit 303, and the input signal acquisition unit 701 can freely convert the data. . For example, NR processing, which will be described later, may be performed on YUV signals, L ^* a ^* b ^* components, and visible/invisible components.

Next _, in S802, the noise level calculator _{702 calculates} the noise level (noise amount ) is calculated. For example, the noise level calculation unit 702 calculates the variance of noise using equations (1) and (2) as an evaluation value representing the noise level contained in the image.

Equations (1) and (2) are equations for obtaining the noise variance σ ² _R(v,h) for the R component. Although the description of the equations is omitted, the noise level calculation unit 702 calculates the noise variance σ ² _G(v,h) and σ ² _B(v,h) are also calculated. Here, by setting the values of s1 and s2 in equations (1) and (2) large, it is possible to accurately calculate the noise variance contained in the corresponding signal.

Note that the evaluation value of the noise level is not limited to the variance obtained by the equations (1) and (2), and various evaluation values such as the noise standard deviation and the noise characteristics of the image sensor can be used. may be used. Also, the evaluation value of the noise level may be a combination of at least two of the variance, the standard deviation, the noise characteristics of the image sensor, and the like.

In S803, the average value calculation unit 703 performs processing for calculating an average value for each RGB component as an evaluation value from the RGB input signal. For example, the average value calculation unit 703 calculates the average value of the R component by using Equation (2) described above. Although the description of the formula is omitted, the average value calculation unit 703 calculates the average value of the G component and the B component by using the formula (2) in which the R component is replaced with the G component or the B component. do.

In S804, the color value calculation unit 704 calculates the color of each signal from the RGB input signal and the RGB output signal read from the output signal holding unit 707 holding the signal after the NR processing, which will be described later. A color value is calculated as an evaluation value to represent. The RGB output signals read out from the output signal holding unit 707 are NR-processed signals of frames preceding the frame to be NR-processed, preferably NR-processed signals of the immediately preceding frame. Although description of formulas is omitted, the color value calculation unit 704 calculates the color values of the RGB input signal as a ^* _IN(v,h) and b ^* _IN(v,h) , and calculates the color value of the RGB output signal. are calculated as a ^* _OUT(v,h) and b ^* _OUT(v,h) . Note that the evaluation values representing colors are not limited to the values of a ^* and b ^* in the Lab space. , S value may be used.

Next, in S805, the NR intensity determination unit 705 determines the intensity of NR processing based on the evaluation values obtained as described above. That is, the NR intensity determination unit 705 determines the noise level for each RGB component from the noise level calculation unit 702, the average value for each RGB component from the average value calculation unit 703, and the color value from the color value calculation unit 704. Based on this, the strength of the NR processing is determined.

FIG. 9 is a block diagram showing an example of the configuration of NR intensity determination section 705. As shown in FIG.
As shown in FIG. 9 , the NR intensity determination unit 705 includes a color value change amount calculation unit 901 , an average value difference calculation unit 902 , a clip processing unit 903 and an NR intensity coefficient calculation unit 904 .

A color value change amount calculation unit 901 calculates a color value change amount based on the color value calculated by the color value calculation unit 704 . Details of the color value change amount calculation processing will be described later. Then, the color value change amount calculation unit 901 outputs a signal representing the color value change amount to the clip processing unit 903 .

The average value difference calculation unit 902 calculates a difference value of average values (hereinafter referred to as an average value difference) using the average values of the RGB input signals calculated by the average value calculation unit 703 in FIG. Details of the average value difference calculation process will be described later. Then, the average value difference calculation unit 902 outputs a signal representing the average value difference to the clip processing unit 903 .

The clip processing unit 903 calculates the noise variance of the RGB input signal input from the noise level calculation unit 702, the color value change amount input from the color value change amount calculation unit 901, and the average value difference calculation unit 902. Clip processing is performed on the average value difference obtained. Details of clip processing will be described later. Clip processing section 903 outputs the clipped signal to NR intensity coefficient calculation section 904 .

The NR strength coefficient calculation unit 904 calculates the strength coefficient of the NR processing executed by the NR processing execution unit 706 based on the signal after the clip processing by the clip processing unit 903 . Details of the NR intensity coefficient calculation process will be described later. The strength coefficient of the NR processing calculated by the NR strength coefficient calculator 904 is output to the NR processing execution unit 706 in FIG.

FIG. 10 is a flow chart showing the flow of processing performed in each block of the NR intensity determination section 705 shown in FIG.
In S1001 of FIG. 10, the color value change amount calculation unit 901 calculates the color values a ^* _IN(v,h) and b ^* _IN(v,h) of the RGB input signals input from the color value calculation unit 704 and the RGB output. Calculate the amount of change between the signal color values a ^* _OUT(v,h) and b ^* _OUT(v,h) . That is, the color value change amount calculator 901 calculates the amount of color value change before and after NR processing. For example, the color value change amount calculation unit 901 calculates the color value change amount Δa ^* b ^* (v, h) before and after the NR processing using Equation (3). Then, the color value change amount Δa ^* b ^* (v, h) calculated by the color value change amount calculation unit 901 is sent to the clip processing unit 903 . Note that if there is a change in the subject between the frame to be subjected to NR processing and the frame read from the output signal holding unit 707, the amount of change in the color value will change regardless of the result of NR processing. growing. Therefore, a motion vector is calculated for each region between frames, and in a region where the motion vector is large and it is highly likely that the subject has changed, the color value change amount Δa ^* b ^* (v,h) may be fixed to 0 or a default value.

In S<b>1002 , the average value difference calculation unit 902 uses the average value of the RGB input signals input from the average value calculation unit 703 to calculate the average value difference ΔAve(v, h). For example, average value difference calculation section 902 calculates average value difference ΔAve(v, h) using equation (4). The average value difference ΔAve(v, h) calculated by the average value difference calculation unit 902 is sent to the clip processing unit 903 .

Next, in S1003, the clip processing unit 903 calculates the noise level (noise variance σ ² _R(v,h) , σ ² _G(v,h) , σ ² _B(v,h) ) and the color value of the RGB input signal. Clip processing is performed on the amount of change Δa ^* b ^* (v, h) and the average value difference ΔAve(v, h). For example, the clip processing unit 903 performs clip processing on each signal using equations (5), (6), and (7).

Here, equation (5) is an equation representing clip processing for the noise variance σ ² _R(v,h) for the R component. In equation (5), σ ² _MAX is a parameter that defines the upper limit for the R component noise variance σ ² _R(v,h) , and σ ² _MIN is the lower limit for the R component noise variance σ ² _R(v,h). is a parameter that determines Although the description of the equation is omitted, the clip processing unit 903 calculates the noise variance σ ² _{G(v, h )} and σ ² _B(v,h) are also clipped.

Equation (6) is an equation that expresses the clip processing for the color value change amount Δa ^* b ^* (v, h). In equation (6), Δa ^* b ^* (v, h) _MAX is a parameter that defines the upper limit for the amount of change in color value Δa ^* b ^* (v, h), and Δa ^* b ^* (v, h) _MIN is This is a parameter that defines the lower limit for the color value change amount Δa ^* b ^* (v, h).

Equation (7) is an equation representing clip processing for the average value difference ΔAve(v, h). In equation (7), ΔAve _MAX is a parameter that defines the upper limit for the average difference ΔAve(v,h), and ΔAve _MIN is a parameter that defines the lower limit for the average difference ΔAve(v,h).

Next, in S1004, the NR strength coefficient calculation unit 904 calculates the strength coefficient NRST of the NR processing based on the noise variance after the clip processing, the change amount of the color value, and the mean value difference. For example, the NR intensity coefficient calculation unit 904 calculates noise variances σ ² _R(v,h) , σ ² _G(v,h) , σ ² _B(v,h) and variation amounts Δa ^* b ^* (v,h) and The intensity coefficient NRST of the NR processing is calculated by calculating the ratio as in Equation (8) using the average value difference ΔAve(v, h).

Formula (8) is a formula for calculating the intensity coefficient NRST of the NR processing for the R component. Although the description of the formula is omitted, the NR intensity coefficient calculation unit 904 also calculates the intensity coefficient of the NR processing for the G component and the B component by using the equation in which the R component in Equation (8) is replaced with the G component or the B component. calculate. NR _MAX in equation (8) is a parameter that defines the range between the upper and lower limits of the NR intensity NR _ST (v,h). α in equation (8) is a parameter that controls the effect of the noise variance on the NR intensity coefficient; similarly, β is a parameter that controls the effect of the amount of change in color value on the NR intensity coefficient; is a parameter that controls the effect of on the NR intensity coefficient. These α, β, and γ parameters control the ratio of the noise variance, color value variation, and mean value difference to the NR intensity coefficient. In this embodiment, the range of NR _MAX is defined as a range from 0 to 128, but it is not limited to this. σ ² _MAX , σ ² _MIN , Δa ^* b ^* (v, h) _MAX , Δa ^* b ^* (v, h) _MIN , ΔAve _MAX , ΔAve _MIN are obtained from the above equations (5) and (6) , which are similar to the parameters used in equation (7).

Here, the value of the average difference is a parameter that indicates that the smaller the value, the more achromatic the area is, while the larger the value, the more chromatic the area is. That is, the NR intensity NR _ST (v, h) defined by Equation (8) is a value calculated by considering not only the amount of noise but also the average value difference of each component of RGB. Therefore, when NR processing is performed using the NR intensity NR _ST (v, h), the NR processing intensity is controlled for an achromatic region in which the color cast phenomenon is conspicuous, and the color cast phenomenon is suppressed. It will be possible. In addition, in the case of the present embodiment, the NR intensity NR _ST (v, h) is calculated in consideration of the amount of color change before and after the NR processing. Therefore, when NR processing is performed using the NR intensity NR _ST (v, h), it is possible to suppress the phenomenon in which the color cast caused by the NR processing as shown in FIG. will be possible.

Note that each parameter such as the upper limit, the lower limit, α, β, γ, and the range of NR intensity is not limited to a predetermined fixed value. Each of these parameters is, for example, a value changed according to the amount of noise, a value dynamically changed by the user, a value changed depending on the sensor temperature, etc., a value changed according to the brightness change of the entire image, a sensor, etc. may be changed due to various factors, such as a value changed in consideration of aging deterioration. For example, the temperature of the imaging sensor may be detected, and the parameter may be changed according to the temperature change. When changing a parameter according to a change in temperature, for example, the parameter value is set large when the temperature is high, and conversely, the parameter value is set small when the temperature is low. Also, the parameter may be changed according to the luminance distribution of the entire image. For example, when there is a large distribution on the low luminance side, the entire image may become purplish. Therefore, the parameter value is set large when there are many distributions on the low luminance side, and conversely, the parameter values are set small when there are many distributions on the high luminance side. Also, the weights of α, β, and γ may be changed according to the characteristics of the imaging sensor and the noise variance in the OB (Optical Black) area. If, for example, the characteristic of the image sensor is such that the G component tends to be small, the weights are set so that the values of β and γ are relatively large compared to α. Further, for example, when the dynamic range of the camera is set to WDR (wide dynamic range), the parameter may be set large. In the case of the WDR setting, for example, the larger the difference in brightness, the more conspicuous color bleeding may occur, so the parameter is set large so that the NR processing is strongly applied. It should be noted that, in the case of night scene photography, the parameters related to this embodiment may be set to zero so that conventional NR processing can be performed. In addition, for example, it is possible to set an area corresponding to the sky as a subject for the image, and for areas that are bluish in fine weather and areas that are reddish in dusk, it is possible to maintain those bluish and reddish colors. good. In this case, for example, the weights for α, β, and γ may be changed according to the photographing time zone or the like by learning processing.

Returning to the flowchart of FIG.
In S<b>806 , the NR processing execution unit 706 executes NR processing on the RGB input signals from the input signal acquisition unit 701 based on the NR processing intensity determined by the NR intensity determination unit 705 as described above. For example, if the NR processing implemented in the NR processing execution unit 706 is NR processing using an epsilon filter, the NR processing execution unit 706 performs the NR processing represented by the calculations of equations (9) and (10). will be

Here, when the NR intensity coefficient is input with a large value, for example, the values of s1 and s2 in equation (9) are increased, and the threshold ε in equation (10) is set to be increased, and NR processing is performed. A process that strengthens the effect of is executed. Although description of the formula is omitted, the NR processing execution unit 706 also executes NR processing for the G component and the B component by using the formula (9) in which the R component is replaced with the G component or the B component. Note that the implementation example of the NR processing is not limited to the implementation using the arithmetic expressions of Expressions (9) and (10). For example, the NR processing execution unit 706 uses a bilateral spatial NR filter, an NR filter that refers to edge information, a filtering process that removes isolated points, and a cyclic NR processing filter that reduces random noise in the time direction. may A combination of processing of at least two of these filters may also be used.

Next, in step S<b>807 , the output signal holding unit 707 holds the RGB output signals after the NR processing is executed by the NR processing execution unit 706 .
Next, in step S808, the NR processing unit 303 determines whether or not the processing of steps S801 to S807 has been performed for all pixels of the input current frame. Then, if it is determined in S808 that there is an unprocessed pixel, the processing of the NR processing unit 303 returns to S801, and the processing of S801 to S807 is performed on the unprocessed pixel. On the other hand, if it is determined in S808 that the processing of S801 to S807 has been performed for all pixels of the current frame input, the NR processing unit 303 ends the flowchart of FIG. 8 for that frame. After that, the NR processing unit 303 starts the processing of the flow chart of FIG. 8 for the next input frame.

As described above, in the present embodiment, the NR processing intensity is determined not only based on the noise level obtained from the RGB input signal, but also based on the average value of each component of RGB and the color change before and after the NR processing. NR processing is performed. As a result, according to this embodiment, it is possible to suitably reduce noise while reducing the phenomenon of color cast that occurs due to low-luminance areas, temperature rise, and the like. That is, according to the present embodiment, for example, in a surveillance camera, noise reduction processing with high color reproducibility can be performed while effectively suppressing color cast caused by changes in the shooting environment such as changes in illuminance and temperature. Furthermore, in the image processing of this embodiment, not only color cast caused by changes in the shooting environment such as changes in illuminance and temperature, but also color cast caused by various factors such as deterioration of each component inside the surveillance camera over time. can be suppressed, and noise can be preferably reduced.

Note that in this embodiment, the color value calculation unit 704 calculates the color values of the image signal of the frame to be subjected to NR processing and the image signal of the previous frame after NR processing, and the NR intensity determination unit 705 sent to, but not limited to.
A color value calculation unit 704 calculates color values of image signals before and after NR processing in the same frame. It is good also as a structure which calculates an amount. Specifically, as shown in FIG. 11, the NR processing unit 303 includes an input signal acquisition unit 1101, a noise amount calculation unit 1102, an average value calculation unit 1103, a color value calculation unit 1106, an NR intensity determination unit 1107, and an NR processing unit. In addition to the execution unit 1108 , it is composed of a temporary NR intensity determination unit 1104 and a temporary NR processing execution unit 1105 .

Temporary NR strength determination section 1104 determines strength coefficient NR' of provisional NR processing based on the noise level calculated by noise amount calculation section 1102 and the average value of the components calculated by average value calculation section 1103. Determine ST. That is, in Equation (8), the term relating to the amount of change in color value is excluded to determine the intensity coefficient NR'ST of the provisional NR processing. The provisional NR processing execution unit 1105 executes NR processing based on this provisional NR processing intensity, and outputs the image signal after the execution of the NR processing to the color value calculation unit 1106 . Then, the color value calculation unit 1106 calculates the color values of the image signal after the NR processing, which is calculated by the provisional NR processing execution unit 1105, and the image signal before the NR processing of the same frame. The same processing as after S805 is executed.

As another configuration, it is also possible to combine image signals before and after NR processing according to the amount of change in color values in the same frame. Specifically, as shown in FIG. 12, in addition to an input signal acquisition unit 1201, a noise amount calculation unit 1202, an average value calculation unit 1203, an NR intensity determination unit 1204, an NR processing execution unit 1205, and a color value calculation unit 1206, , a synthesis ratio calculation unit 1207 and a synthesis processing unit 1208 . A synthesis ratio calculation unit 1207 calculates a synthesis ratio based on the color values before and after the NR processing calculated by the color value calculation unit 1206, the noise level, and the average value. More specifically, the combination ratio is calculated as in Equation (11) in the same manner as Equation (8).

Formula (11) is a formula for calculating the composition ratio for the R component, and although description of the equation is omitted, the composition ratio for the G component and the B component is also calculated. Here, Blend _MAX in Equation (11) is a parameter that defines the range between the upper limit and the lower limit of the synthesis ratio Blend(v, h), and other parameters are the same as in Equation (8). In this embodiment, the range of Blend _MAX is defined as a range from 0.0 to 1.0, but it is not limited to this.

Based on the synthesis ratio calculated by the synthesis ratio calculation unit 1207, the synthesis processing unit 1208 calculates the signal R _IN (v, h) before NR processing and the signal R ' Synthesize _OUT (v, h) to calculate the final output signal R _OUT (v, h).

Formula (12) is a formula for calculating the final output signal for the R component, and although description of the formula is omitted, the G component and the B component are also calculated. Here, the synthesis ratio of equation (11) is calculated by taking into consideration the average value of the RGB components of the input signal, the noise level, and also the change in the saturation value before and after the NR processing. Therefore, the final output signal calculated by the equation (12) is a combined signal that achieves both the effect of suppressing the phenomenon of color cast and the effect of noise reduction.
By configuring as shown in FIGS. 11 and 12 described above, the strength and effect of NR processing are determined based on the amount of change in color value before and after execution of NR processing of the image signal of the same frame, and NR processing is performed. becomes possible.

<Second embodiment>
A second embodiment will be described below.
The internal configurations of the imaging system and surveillance camera 101 of the second embodiment are generally the same as those of the first embodiment described above, so illustration and description thereof will be omitted.
FIG. 11 is a block diagram showing an example internal configuration of the NR processing unit 303 according to the second embodiment. The NR processing unit 303 of the second embodiment includes an input signal acquisition unit 1301, an average value calculation unit 1302, a color value calculation unit 1303, an NR intensity determination unit 1304, an NR processing execution unit 1305, an output signal storage unit 1306, and a correction It has a value calculator 1307 and a correction processor 1308 . The input signal acquisition section 1301 is similar to the input signal acquisition section 701 in FIG. Hereinafter, the average value calculation unit 1302 is the average value calculation unit 703, the color value calculation unit 1303 is the color value calculation unit 704, the NR processing execution unit 1305 is the NR processing execution unit 706, and the output signal holding unit 1306 is the output signal holding unit. Since it is the same as the part 707, detailed description thereof will be omitted.

Here, in the case of the first embodiment described above, the NR processing execution unit 706 of the NR processing unit 303 executes NR processing on the RGB input signal as in S806 of FIG. On the other hand, the NR processing unit 303 of the second embodiment corrects the RGB input signal based on the average value calculated by the average value calculation unit 1302 and the color value calculated by the color value calculation unit 1303. NR processing is performed on the corrected RGB input signals.

Also, the NR intensity determination unit 705 of the first embodiment determines the NR intensity based on the noise level, average value, and color value of the RGB input signal, as in S805 of FIG. On the other hand, the NR intensity determination unit 1304 of the second embodiment determines the intensity of the NR processing according to the correction value calculated based on the average value calculated by the average value calculation unit 1302 and the color value calculated by the color value calculation unit 1303. do. As described in the first embodiment, the color values calculated by the color value calculator 1303 are the color values a ^* _IN(v,h) and b ^* _IN(v,h) of the RGB input signal and the NR processing. Color values a ^* _OUT(v,h) and b ^* _OUT(v,h) of subsequent RGB output signals.

FIG. 14 is a flow chart showing the flow of processing performed in each block of the NR processing unit 303 shown in FIG. 13 of the second embodiment. 12 and FIG. 8, S1401 is the same process as S801, and hereinafter, S1402 is the same process as S803, S1403 is the same process as S804, S1408 is the same process as S807, and S1409 is the same process as S808. are omitted. Processing different from that in FIG. 8 will be described below with reference to the flowchart of FIG. 14 .

In S1404, the correction value calculation unit 1307 generates a correction value based on the average value calculated in the same manner as described above by the average value calculation unit 1302 and the color value calculated in the same manner as described above by the color value calculation unit 1303. . For example, the correction value calculation unit 1307 calculates the color values a ^* _IN(v,h) and b ^* _IN(v,h) of the RGB input signals from the color value calculation unit 704 and the color values a ^* _OUT(v) of the RGB output signals. _,h) and b ^* _OUT(v,h) are used to calculate the amount of change .DELTA.a ^* b ^* (v,h) according to the equation (3). This amount of change Δa ^* b ^* (v, h) is a value representing the color change before and after execution of the NR processing, as described in the first embodiment. Further, the correction value calculation unit 1307 calculates the average value difference ΔAve(v, h) using the above-described formula (4).

Then, the correction value calculation unit 1307 determines a correction value based on the color value change amount Δa ^* b ^* (v, h) and the average value difference ΔAve(v, h). Specifically, the correction value calculation unit 1307 sets a larger correction value as the average difference is smaller and the color change amount is larger. At this time, the correction value is set to a value such that the average values of the respective RGB components approach equal values. Here, an area with a small difference in mean value difference and a large amount of color change is originally considered an achromatic area, but it is an area that is highly likely to cause color cast due to NR processing. Therefore, the correction value calculation unit 1307 performs the processing of S1404. For such an area, a correction value is calculated that approaches an achromatic color with respect to the average value of each component of RGB.

Next, in S1405, the correction processing unit 1308 applies the correction calculated by the correction value calculation unit 1307 to the RGB input signals R _IN (v, h), G _IN (v, h), and B _IN (v, h). Perform correction processing according to the value. Then, the corrected RGB input signal is sent to the NR processing execution unit 1305 .

Also, in S<b>1406 , the NR intensity determination unit 1304 determines the intensity of NR processing according to the correction value calculated by the correction value calculation unit 1307 . The processing in the NR intensity determination unit 1304 is generally the same as the processing in the first embodiment described above, but the noise level as in the first embodiment is not used, and the NR intensity coefficient is determined according to the correction value. is made. For example, when the correction value is large, it is assumed that gain is applied to each component of RGB to amplify random noise. is set to a larger value as the value of NR increases.

Next, in S1407, the NR processing execution unit 1305 executes NR processing according to the NR intensity determined in S1406 and S1405 on the RGB input signal corrected in S1405.

As described above, in the second embodiment, the RGB input signal is corrected using a correction value based on the average value of each RGB component obtained from the RGB input signal and the color change before and after execution of the NR processing, and , the NR intensity is determined according to the correction value. Then, in the second embodiment, NR processing corresponding to the NR intensity determined according to the correction value is performed on the RGB input signal corrected by the correction value. Thus, according to the second embodiment, it is possible to preferably reduce noise while reducing the phenomenon of color cast that occurs in an achromatic region.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.
All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical spirit or main features.

101: surveillance camera, 102: client device, 303: NR processing unit, 701: input signal acquisition unit, 702: noise level calculation unit, 703: average value calculation unit, 704: color value calculation unit, 705: NR intensity determination unit , 706: NR processing execution unit, 707: output signal holding unit

Claims (17)

a color value calculation means for calculating a color value from an image signal;
average value calculation means for calculating an average value of the image signal;
determining means for determining an intensity coefficient indicating intensity of noise reduction processing according to the evaluation value including the average value and the color value;
a processing means for performing the noise reduction processing according to the intensity coefficient on the image signal;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said average value calculating means calculates an average value for each of a plurality of color components included in said image signal. The color value calculation means calculates the color value of the signal before noise reduction processing of the image signal is executed and the color value of the signal after the noise reduction processing is executed. 3. The image processing apparatus according to claim 2. The determining means calculates the amount of change in the color values before and after the noise reduction processing is executed, calculates the difference value between the average values for each of the color components, and calculates the difference value between the average values and the 4. The image processing apparatus according to claim 3, wherein the intensity coefficient is calculated based on the amount of change in color value. The determination means performs clip processing based on comparison with at least two set values for each of the difference value of the average value and the amount of change in the color value, and after the clip processing is performed, the 5. The image processing apparatus according to claim 4, wherein the intensity coefficient is calculated based on the difference value of the average values and the amount of change in the color value. 6. The image processing apparatus according to claim 4, wherein the determining means sets the intensity coefficient to be larger as the difference between the average values is larger. 7. The image processing apparatus according to any one of claims 4 to 6, wherein the determining means sets the intensity coefficient larger as the change amount of the color value is smaller. 8. The image processing apparatus according to claim 7, wherein said determining means calculates said intensity coefficient according to a ratio between said difference value of said average value and said amount of change in said color value . 9. The image processing apparatus according to claim 2, further comprising noise level calculation means for calculating a noise level of said image signal, wherein said evaluation value includes said noise level. The determining means calculates the amount of change in the color value before and after the noise reduction processing is executed, calculates the difference value between the average values for each of the color components, and calculates the difference between the noise level and the average value. 10. The image processing apparatus according to claim 9, wherein the intensity coefficient is calculated based on the difference value and the change amount of the color value. 11. The image processing apparatus according to claim 10, wherein said determining means sets said intensity coefficient to be larger as the value of noise level increases. 11. The image processing apparatus according to claim 10, wherein said determining means calculates said intensity coefficient in accordance with a ratio of a difference value between said noise level and said average value and an amount of change in said color value . correction means for correcting the image signal based on the average value and the color value;
3. The image processing apparatus according to claim 2, wherein said processing means performs said noise reduction processing on the image signal corrected by said correction means. 14. The image processing apparatus according to claim 13, wherein said determining means calculates a difference value between said average values for each of said color components, and sets said intensity coefficient larger as said difference value decreases. The determining means calculates a change amount between the color value of the signal before the noise reduction processing is executed and the color value of the signal after the noise reduction processing is executed, and the larger the change amount, the 15. The image processing apparatus according to claim 13, wherein the intensity coefficient is set large. An image processing method executed by an image processing device,
a step of calculating a color value from the image signal;
calculating an average value of the image signal;
determining an intensity coefficient indicating the intensity of noise reduction processing according to the evaluation value including the average value and the color value;
performing the noise reduction process on the image signal according to the intensity coefficient;
An image processing method characterized by comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 15.

Images