JP4465002B2 - Noise reduction system, noise reduction program, and imaging system. - Google Patents

Description

  The present invention relates to a noise reduction process for reducing randomness and impulsive noise of a video signal caused by an imaging system.

  A video signal obtained from an imaging system including an imaging device and an accompanying analog circuit and an A / D converter generally contains a noise component. This noise component can be roughly divided into random and impulsive noise. Random noise is generated in an image sensor and an analog circuit, and has characteristics close to white noise characteristics. On the other hand, the impulsive noise is noise mainly caused by the image sensor represented by a defective pixel or the like.

  Regarding the reduction processing of random and impulsive noise, for example, as disclosed in Patent Document 1, a video signal is shot a plurality of times, recorded in a plurality of frame memories, and a maximum value for a video signal on the same coordinate. An example in which the minimum value is removed and averaged or the median is taken is disclosed. As a result, random noise and extrinsic impulsive noise that is not fixed, such as defective pixels, can be removed, and a high-quality video signal can be obtained.

  In addition, as disclosed in Patent Document 2, an example is disclosed in which a random noise amount and an impulsive noise amount are estimated, and noise reduction is performed by one reduction process common to both noises. Yes. As a result, random noise impulsive noise can be removed, and a high-quality video signal can be obtained. In addition, since noise reduction is performed by one reduction process, it is possible to perform a noise reduction process with less discontinuity and artifacts due to the noise reduction process.

Furthermore, regarding the reduction processing of defective pixels, for example, as shown in Patent Document 3, after performing correction or generation processing on a defective pixel that has been measured in advance, a plurality of noise reduction processing such as a low-pass filter and a median filter are performed. An example of weighted addition of results is disclosed. As a result, the occurrence of discontinuity and artifacts due to correction or generation processing of defective pixels can be suppressed, and a high-quality video signal can be obtained.
JP 2007-110445 JP JP 2005-318126 A JP2003-69901

  In Patent Document 1, since a plurality of video signals are used, a still area can be processed satisfactorily, but there is a problem that side effects such as afterimages occur in a moving area. In addition, there is a problem that fixed impulsive noise such as defective pixels cannot be removed. Further, a plurality of frame memories are required, and there is a problem that the cost of the system is increased.

  In Patent Document 2, the occurrence of discontinuity can be suppressed because random and impulsive noise reduction processing is performed with a single reduction processing, but there is a problem that both noises having different properties cannot be optimally removed. is there.

  In Patent Document 3, there is a problem that random noise generated in pixels other than defective pixels cannot be dealt with.

  An object of the present invention is to obtain a high-definition video signal by paying attention to the above problems and optimally removing both random and impulse noises. Another object of the present invention is to suppress the occurrence of discontinuity and artifacts by synthesizing video signals after reduction processing for both noises.

  According to the present invention, in a noise reduction system that performs noise reduction processing on a video signal captured from an imaging system, local region extraction means that sequentially extracts a local region containing a target pixel that performs noise reduction processing from the video signal. First noise reduction means for performing random noise reduction processing on the local region, second noise reduction means for performing impulsive noise reduction processing on the local region, and the first Combining means for synthesizing the video signal noise-reduced by the noise reducing means and the video signal noise-reduced by the second noise reducing means.

  Since the reduction processing is independently performed on random noise and impulsive noise having different properties, optimal noise reduction processing is possible, and a high-quality video signal is obtained. Further, by synthesizing the video signals after the processing of both, discontinuity and artifacts due to different noise reduction processes can be suppressed, and a high-quality video signal can be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

First Embodiment [Configuration]
FIG. 1 is a configuration diagram of the first embodiment. A video signal photographed through the lens system 100, the aperture 101, and the CCD 102 is amplified by an amplifier (hereinafter “Gain”) 104, and is converted into a digital signal by an A / D converter (hereinafter “A / D”) 105. Converted to The video signal from the A / D 105 is transferred to the color signal separation / extraction unit 111 via the buffer 106.

  The buffer 106 is also connected to a pre-white balance adjustment unit (hereinafter “PreWB unit”) 107, a photometric evaluation unit 108, and a focus detection unit 109. The PreWB unit 107 is connected to the Gain 104, the photometric evaluation unit 108 is connected to the aperture 101, the CCD 102, and the Gain 104, and the in-focus detection unit 109 is connected to the AF motor 110. The color signal separation / extraction unit 111 is connected to the first noise reduction unit 112 and the second noise reduction unit 113. The first noise reduction unit 112 and the second noise reduction unit 113 are connected to an output unit 116 such as a memory card via a synthesis unit 114 and a signal processing unit 115.

  A control unit 117 such as a microcomputer includes Gain 104, A / D 105, PreWB unit 107, photometric evaluation unit 108, in-focus detection unit 109, color signal separation and extraction unit 111, first noise reduction unit 112, and second noise reduction unit 113. , Bi-directionally connected to the combining unit 114, the signal processing unit 115, and the output unit 116. An external I / F unit 118 having a power switch, a shutter button, and an interface for setting various modes at the time of shooting is also connected to the control unit 117 in both directions. Further, a signal from the temperature sensor 103 arranged in the vicinity of the CCD 102 is connected to the control unit 117.

[Action]
In FIG. 1, the flow of the video signal will be described. After setting shooting conditions such as ISO sensitivity via the external I / F unit 118, the pre-shooting mode is entered by pressing the shutter button halfway. A video signal photographed through the lens system 100, the diaphragm 101, and the CCD 102 is output as an analog signal. In the first embodiment, a single plate CCD having a Bayer type primary color filter disposed on the front surface is assumed as the CCD 102.

  FIG. 2A shows the configuration of the Bayer-type primary color filter. The Bayer type has 2 × 2 pixels as a basic unit, in which red (R) and blue (B) filters are arranged one pixel at a time, and two green (Gr, Gb) filters are arranged. The green filter has the same characteristics, but in the first embodiment, this is distinguished from Gr and Gb for convenience of processing.

  The analog signal is amplified by a predetermined amount at Gain 104, converted to a digital signal by A / D 105, and transferred to buffer 106. The video signal in the buffer 106 is transferred to the PreWB unit 107, the photometric evaluation unit 108, and the in-focus detection unit 109 based on the control of the control unit 117.

  The PreWB unit 107 calculates a simple white balance coefficient by integrating signals of a predetermined level for each color signal corresponding to the color filter. The above coefficients are transferred to Gain 104, and white balance is performed by multiplying a different gain for each color signal.

  The photometric evaluation unit 108 controls the electronic shutter speed of the aperture 101 and the CCD 102, the gain of the gain 104, and the like so as to achieve proper exposure, taking into account the set ISO sensitivity, shutter speed at the limit of camera shake, and the like.

  Further, the focus detection unit 109 detects the edge intensity in the video signal and controls the AF motor 110 so as to maximize the edge intensity, thereby obtaining a focus signal.

  Next, full shooting is performed by fully pressing the shutter button via the external I / F unit 118, and the video signal is transferred to the buffer 106 in the same manner as the pre-shooting. The actual photographing is performed based on the simple white balance coefficient obtained by the PreWB unit 107, the exposure condition obtained by the photometric evaluation unit 108, and the focusing condition obtained by the in-focus detection unit 109. The time condition is transferred to the control unit 117. The video signal in the buffer 106 is transferred to the color signal separation / extraction unit 111.

Based on the control of the control unit 117, the color signal separation / extraction unit 111 sequentially extracts a local region including a target pixel to be subjected to subsequent noise reduction processing and neighboring pixels located in the vicinity of the target pixel for each color signal. In the first embodiment, for example, 10 × 10 pixels shown in FIG. 2A are extracted from the video signal as a basic unit. In this case, the target pixel to be subjected to the noise reduction process is four pixels R ₂₂ , Gr ₂₂ , Gb ₂₂ , and B ₂₂ .

Next, as shown in FIG. 2B, each R, Gr, Gb, B color filter is separated as a local region of 5 × 5 pixels. Thereafter, the pixels in the local area are represented by C _ij (C is a color signal, C = R, Gr, Gb, B, i is an X coordinate, i = 0-4, and j is a Y coordinate, j = 0-4) write. The 5 × 5 cases of local region of pixels, the target pixel becomes C _22. In order to extract the pixel of interest from the video signal, the basic unit of 10 × 10 pixels shown in FIG. 2 (a) is sequentially extracted by overlapping four rows and four columns. The extracted local region is transferred to the first noise reduction unit 112 and the second noise reduction unit 113.

Based on the control of the control unit 117, the first noise reduction unit 112 estimates the random noise amount N _{22 related} to the target pixel C ₂₂ from the low frequency components in the local region. Thereafter, a random noise reduction process is performed by performing a coring process on the target pixel C ₂₂ using the low frequency component of the local region and the estimated noise amount N ₂₂ . Hereinafter, the ^target pixel on which random noise reduction processing has been performed by the first noise reduction unit 112 is described as C ^N1 ₂₂ . The target pixel C ^N1 ₂₂ that has been subjected to random noise reduction processing is transferred to the synthesis unit 114.

On the other hand, based on the control of the control unit 117, the second noise reduction unit 113 sets the index coefficient IC indicating the degree of impulsive noise from the local region to the target pixel C ₂₂ and its neighboring eight pixels C ₁₁ , C ₂₁ , C _31. , C ₁₂ , C ₃₂ , C ₁₃ , C ₂₃ , C ₃₃ are calculated. Hereinafter, the index coefficient of the pixel of interest C ₂₂ is _denoted by IC ₀ , the index coefficients of the neighboring 8 pixels are denoted by IC ₁ to IC ₈ , and the entire index coefficient is denoted by IC _k (k = 0 to 8). Further, if necessary, the pixel of interest C ₂₂ is simply expressed as C ₀ , the neighboring 8 pixels as C _{1 to} C ₈ , and all 9 pixels as C _k .

Thereafter, the weight coefficient of the pixel of interest and its neighboring eight pixels is obtained using the index coefficient IC _k and the impulsive noise reduction process is performed by performing the weighting filter process. Hereinafter, the pixel of interest that has been subjected to the impulsive noise reduction processing by the second noise reduction unit 113 is described as C ^N2 ₂₂ . C ^N2 ₂₂ is transferred to the synthesizing unit 114 as an index coefficient IC ₀ of the pixel of interest C _{22 and} the pixel of interest subjected to the impulsive noise reduction process.

Based on the control of the control unit 117, the synthesis unit 114 uses the index coefficient IC ₀ transferred from the second noise reduction unit 113 to select a target pixel that has been subjected to random noise reduction processing by the first noise reduction unit 112. C ^N1 ₂₂ and the target pixel C ^N2 ₂₂ that has been subjected to the impulsive noise reduction process are synthesized, and the synthesized target pixel C ^N ₂₂ is obtained.

ここで、Th1,Th2は所定の閾値を、w=0〜１は合成のための合成係数を意味する。合成後の注目画素C^N ₂₂は、信号処理部115へ転送される。上記色信号分離抽出部111,第１ノイズ低減部112,第２ノイズ低減部113,合成部114における処理は、制御部117の制御に基づき局所領域単位で同期して行われる。

Here, Th1 and Th2 are predetermined threshold values, and w = 0 to 1 are synthesis coefficients for synthesis. The combined target pixel C ^N ₂₂ is transferred to the signal processing unit 115. The processes in the color signal separation / extraction unit 111, the first noise reduction unit 112, the second noise reduction unit 113, and the synthesis unit 114 are performed synchronously in units of local regions based on the control of the control unit 117.

  Based on the control of the control unit 117, the signal processing unit 115 performs known interpolation processing, enhancement processing, compression processing, and the like on the noise-reduced video signal that has been subjected to the synthesis processing, and transfers it to the output unit 116.

  The output unit 116 records and stores the video signal in a recording medium such as a magnetic disk or a memory card.

  FIG. 3 shows an example of the configuration of the first noise reduction unit 112. The buffer 200, the average value calculation unit 201, the gain calculation unit 202, the standard value assignment unit 203, the parameter ROM 204, the parameter selection unit 205, and the noise interpolation unit 206 and a coring unit 207.

  The color signal separation / extraction unit 111 is connected to the average value calculation unit 201 via the buffer 200. Average value calculating section 201 is connected to parameter selecting section 205 and coring section 207. The gain calculation unit 202, the standard value assigning unit 203, and the parameter ROM 204 are connected to the parameter selection unit 205. The parameter selection unit 205 is connected to the coring unit 207 via the noise interpolation unit 206. The coring unit 207 is connected to the combining unit 114. The control unit 117 is bidirectionally connected to the average value calculation unit 201, the gain calculation unit 202, the standard value assigning unit 203, the parameter selection unit 205, the noise interpolation unit 206, and the coring unit 207.

  As shown in FIG. 2B, the 5 × 5 pixel local area is sequentially transferred from the color signal separation / extraction unit 111 to the buffer 200 for each of the R, Gr, Gb, and B color filters.

  Based on the control of the control unit 117, the average value calculation unit 201 calculates the average value C_AV (C = R, Gr, Gb, B) of the local region as shown in the equation (2).

算出された平均値C_AVはパラメータ選択部205へ、平均値C_AVおよび注目画素C₂₂はコアリング部207へ転送される。

Calculated average value C_AV is the parameter selection unit 205, the average value C_AV and the target pixel C ₂₂ are transferred to the coring unit 207.

  The gain calculation unit 202 obtains the amplification amount in the Gain 104 based on the information regarding the ISO sensitivity and the exposure condition transferred from the control unit 117, and transfers the gain to the parameter selection unit 205. Further, the control unit 117 obtains the temperature information of the CCD 102 from the temperature sensor 103, and transfers this to the parameter selection unit 205.

Parameter selection unit 205, the average value of the local area from the average value calculator 201, the gain information from the gain calculation unit 202, a noise amount of randomness about the target pixel C ₂₂ on the basis of the temperature information from the control unit 117 N ₂₂ Is estimated.

  FIG. 4 is an explanatory diagram regarding estimation of random noise amount. FIG. 4A is a plot of the random noise amount N against the signal level L, which increases in a quadratic curve with respect to the signal level. When FIG. 4A is modeled by a quadratic function, equation (3) is obtained.

ここで、α,β,γは定数項である。しかしながら、ノイズ量は信号レベルだけではなく、撮像素子の温度やゲインによっても変化する。図４(a)は、一例としてある温度下においてゲインに関連する３種類のISO感度100,200,400に対するノイズ量をプロットしている。個々の曲線は(3)式に示される形態をしているが、その係数はゲインに関連するISO感度により異なる。温度をt、ゲインをgとし、上記を考慮した形でモデルの定式化を行うと、

Here, α, β, and γ are constant terms. However, the amount of noise varies not only with the signal level but also with the temperature and gain of the image sensor. FIG. 4A plots the amount of noise with respect to three types of ISO sensitivities 100, 200, and 400 related to gain at a certain temperature as an example. Each curve has the form shown in equation (3), but its coefficient depends on the ISO sensitivity associated with the gain. When the temperature is t, the gain is g, and the model is formulated in consideration of the above,

となる。

It becomes.

Here, α _gt , β _gt , and γ _gt are constant terms. However, it is cumbersome in terms of processing to record a plurality of functions of the equation (4) and calculate the noise amount by calculation each time. For this reason, the model as shown in FIG. 4B is simplified. In FIG. 4B, a model that gives the maximum amount of noise is selected as a reference noise model, and this is approximated by a predetermined number of broken lines. The inflection point of the broken line is represented by coordinate data (L _n , N _n ) composed of the signal level L and the noise amount N. Here, n indicates the number of inflection points.

A correction coefficient k _gt for deriving another noise model from the reference noise model is also prepared. The correction coefficient k _gt is calculated by the least square method from between each noise model and the reference noise model. In order to derive another noise model from the reference noise model, the correction coefficient k _gt is multiplied. The coordinate data (L _n , N _n ) and the correction coefficient k _gt of the reference noise model can be easily obtained by measuring the characteristics of the imaging system in advance. The coordinate data (L _n , N _n ) and the correction coefficient k _gt of the reference noise model are recorded in the parameter ROM 204.

FIG. 4C shows a method for calculating the noise amount from the simplified noise model shown in FIG. For example, assume that a noise amount N corresponding to a given signal level l, gain g, and temperature t is obtained. First, it is searched to which section of the reference noise model the signal level l belongs. Here, it is assumed that it belongs to the section between (L _n , N _n ) and (L _{n + 1} , N _{n + 1} ). The reference noise amount N _l in the reference noise model is obtained by linear interpolation.

次に補正係数k_gtを乗算することで、ノイズ量Nを求める。

Next, the noise amount N is obtained by multiplying the correction coefficient k _gt .

パラメータ選択部205は、平均値算出部201からの局所領域の平均値C_AVから信号レベルlを、ゲイン算出部202からのゲイン情報からゲインgを、制御部117からの温度情報から温度tを設定する。次に、信号レベルlが属する区間の座標データ(L_n, N_n)と(L_n+1, N_n+1)をパラメータ用ROM204から探索し、これをノイズ補間部206へ転送する。さらに、補正係数k_gtをパラメータ用ROM204から探索し、これをノイズ補間部206へ転送する。

The parameter selection unit 205 sets the signal level l from the average value C_AV of the local area from the average value calculation unit 201, the gain g from the gain information from the gain calculation unit 202, and the temperature t from the temperature information from the control unit 117. To do. Next, the coordinate data (L _n , N _n ) and (L _{n + 1} , N _{n + 1} ) of the section to which the signal level l belongs are searched from the parameter ROM 204 and transferred to the noise interpolation unit 206. Further, the correction coefficient k _gt is searched from the parameter ROM 204 and transferred to the noise interpolation unit 206.

Based on the control of the control unit 117, the noise interpolation unit 206 calculates (5) from the signal level l from the parameter selection unit 205 and the coordinate data (L _n , N _n ) and (L _{n + 1} , N _{n + 1} ) of the section. The reference noise amount N _l in the reference noise model is calculated based on the formula (1). Thereafter, the noise amount N is calculated from the correction coefficient k _g from the parameter selection unit 205 based on the equation (6). In the case of the local region shown in FIG. 2B, the calculated noise amount N is transferred to the coring unit 207 as the noise amount N ₂₂ of the target pixel C ₂₂ .

  In the process of calculating the amount of noise, it is not necessary to obtain information such as temperature t and gain g for each image. A configuration in which arbitrary information is recorded in the standard value assigning unit 203 and the calculation process is omitted is also possible. Thereby, high-speed processing and power saving can be realized.

Based on the control of the control unit 117, the coring unit 207 reads the target pixel C ₂₂ and the average value C_AV from the average value calculation unit 201 and the noise amount N ₂₂ from the noise interpolation unit 206, and performs coring processing on the target pixel C _22. C ^N1 ₂₂ is obtained for the ^target pixel that has been subjected to random noise reduction processing.

ランダム性のノイズ低減処理がなされた注目画素をC^N1 ₂₂は、合成部114へ転送される。

The pixel of interest C ^N1 _{22 that} has ^undergone random noise reduction processing is transferred to the combining unit 114.

  In the above configuration, the average value is calculated as the low frequency component of the local region and the interpolation processing is used for the noise estimation processing. However, the configuration is not limited to this configuration. For example, a configuration in which low-pass filter processing is used as a low frequency component in the local region and a lookup table is used in noise estimation processing is also possible.

  FIG. 5 shows an example of another configuration of the first noise reduction unit 112. The average value calculation unit 201, the parameter ROM 204, the parameter selection unit 205, and the noise interpolation unit 206 shown in FIG. 208 and a noise table unit 209 are added. The basic configuration is equivalent to the first noise reduction unit 112 shown in FIG. 3, and the same name and number are assigned to the same configuration. Only different parts will be described below.

  The color signal separation / extraction unit 111 is connected to the low-pass filter unit 208 via the buffer 200. The low-pass filter unit 208 is connected to the noise table unit 209 and the coring unit 207. The gain calculation unit 202 and the standard value giving unit 203 are connected to the noise table unit 209. The noise table unit 209 is connected to the coring unit 207. The control unit 117 is bidirectionally connected to the low-pass filter unit 208 and the noise table unit 209.

Based on the control of the control unit 117, the low-pass filter unit 208 performs low-pass filter processing having a predetermined frequency characteristic on the local region, and generates a low-frequency component C_LO (C = R, Gr, Gb, B) in the local region. calculate. Low-frequency component C_LO calculated in the noise table unit 209, the low-frequency component C_LO and the target pixel C ₂₂ are transferred to the coring unit 207. The low frequency component C_LO of the local region from the low pass filter unit 208, the gain information from the gain calculation unit 202, and the temperature information from the control unit 117 are transferred to the noise table unit 209.

The noise table unit 209 outputs the noise amount N ₂₂ of the target pixel C ₂₂ based on the low frequency component related to the local region from the low pass filter unit 208, the gain information from the gain calculation unit 202, and the temperature information from the control unit 117. The noise table unit 209 is a look-up table that records the relationship among temperature, signal value level, gain, and noise amount, and is constructed based on the relationship shown in equation (4). The noise amount N ₂₂ obtained by the noise table unit 209 is transferred to the coring unit 207.

Based on the control of the control unit 117, the coring unit 207 reads the target pixel C ₂₂ and the low-frequency component C_LO from the low-pass filter unit 208, reads the noise amount N ₂₂ from the noise table unit 209, and performs coring processing on the target pixel C _22. C ^N1 ₂₂ is obtained for the ^target pixel that has been subjected to random noise reduction processing. This coring process is performed by replacing the average value C_AV in the equation (7) with the low frequency component C_LO.

  FIG. 6 shows an example of the configuration of the second noise reduction unit 113. The buffer 300, the difference calculation unit 301, the buffer 302, the sort unit 303, the sum calculation unit 304, the weight coefficient table unit 305, the buffer 306, and the weighting filter unit 307.

  The color signal separation / extraction unit 111 is connected to the difference calculation unit 301 and the weighting filter unit 307 via the buffer 300. The difference calculation unit 301 is connected to the sum calculation unit 304 via the buffer 302 and the sort unit 303. The sum total calculation unit 304 is connected to the weighting coefficient table unit 305 and the synthesis unit 114. The weighting coefficient table unit 305 is connected to the weighting filter unit 307 via the buffer 306. The weighting filter unit 307 is connected to the combining unit 114. The control unit 117 is bi-directionally connected to the difference calculation unit 301, the sort unit 303, the sum calculation unit 304, the weighting coefficient table unit 305, and the weighting filter unit 307.

  As shown in FIG. 2B, the 5 × 5 pixel local area is sequentially transferred from the color signal separation / extraction unit 111 to the buffer 300 for each of R, Gr, Gb, and B color filters.

Difference calculation unit 301, based on the control of the control unit 117, a total of nine pixels of the target pixel C ₂₂ and its neighboring eight pixels _{C 11, C 21, C 31} , C 12, C 32, C 13, C 23, C 33 For, the absolute value of the difference from each of the neighboring 8 pixels is calculated.

FIG. 7A shows the arrangement of the target pixel C ₂₂ and its neighboring eight pixels C ₁₁ , C ₂₁ , C ₃₁ , C ₁₂ , C ₃₂ , C ₁₃ , C ₂₃ , and C _{33 in} the local region. The difference calculation unit 301 calculates the absolute value Δ of the difference from the neighboring 8 pixels for each of the 9 pixels. FIG. 7B shows an arrangement of pixels for calculating the absolute value Δ of the difference. For example, in the case of C ₁₁ pixels, the absolute value Δ of the difference represented by the equation (8).

以降は、算出された８個の差の絶対値をΔ_l（l=1〜8）で表記する。上記差の絶対値Δ_lは、バッファ302に転送される。

Hereinafter, the absolute values of the calculated eight differences are expressed as Δ _l (l = 1 to 8). Absolute value delta _l of the difference is transferred to the buffer 302.

Sorting unit 303, under the control of the control unit 117 sorts the absolute value of the difference between the buffer 302 the delta _l in ascending order, a predetermined number from the smallest, in the first embodiment four to total sum calculation unit 304 Forward. Thereafter, it denoted the absolute value of the sorted in ascending order by the difference SΔ _{m (m} = 1~8).

  Based on the control of the control unit 117, the total calculation unit 304 obtains the sum of the absolute values of the four differences sorted in the ascending order transferred from the sorting unit 303. This sum is the index index IC.

欠陥画素のようなインパルス性のノイズでは、近傍８画素全ての画素間で差が大きいため指標指数ICは大きな値となる。一方、平坦部は全ての画素間で差が小さいため指標指数ICは小さな値となる。また、直線状の単純なエッジ部では、近傍８画素の半分が小さな値を、半分が大きな値をとる。指標指数ICは差の小さなものを４つ選択して積算するため、単純なエッジ部では小さな値をとる。複雑なエッジ構造では大きな値となるが、インパルス性のノイズよりは小さな値をとる。以上から、指標指数ICによりインパルス性のノイズとエッジ、平坦部を分離することができる。

In the case of impulsive noise such as defective pixels, the index index IC has a large value because there is a large difference between all the neighboring eight pixels. On the other hand, since the flat portion has a small difference between all the pixels, the index index IC has a small value. In a simple straight edge, half of the neighboring 8 pixels have a small value and half has a large value. Since the index index IC is selected and integrated with four small differences, a simple edge takes a small value. A complex edge structure has a large value, but takes a smaller value than impulsive noise. From the above, the impulsive noise can be separated from the edge and the flat portion by the index index IC.

  The weighting coefficient table unit 305 is a look-up table that outputs a weighting coefficient F used for subsequent weighted filter processing based on the index coefficient IC. This is constructed based on (10).

(10)式におけるσは調整用のパラメータである。重み係数Fはバッファ306に転送される。上記指標係数ICおよび重み係数Fは、図７(c),(d)に示されるように注目画素C₂₂およびその近傍８画素C₁₁,C₂₁,C₃₁,C₁₂,C₃₂,C₁₃,C₂₃,C₃₃の計９画素に関して算出される。すなわち、差分算出部301,ソート部303,総和算出部304,重み係数テーブル部305は、制御部117の制御に基づき上記処理を９回繰り返すことになる。以降は、指標係数はIC_kで、重み係数はF_kで表記される。なお、総和算出部304は注目画素C₂₂の指標係数IC₀に関してのみ合成部114へも転送する。

In the equation (10), σ is a parameter for adjustment. The weighting factor F is transferred to the buffer 306. The index coefficient IC and the weight coefficient F are the target pixel C ₂₂ and its neighboring eight pixels C ₁₁ , C ₂₁ , C ₃₁ , C ₁₂ , C ₃₂ , C ₁₃ as shown in FIGS. , C ₂₃ and C ₃₃ are calculated for a total of 9 pixels. That is, the difference calculation unit 301, the sort unit 303, the total sum calculation unit 304, and the weight coefficient table unit 305 repeat the above process nine times based on the control of the control unit 117. Hereinafter, the index coefficient is denoted by IC _k and the weight coefficient is denoted by F _k . Note that the sum calculation unit 304 also transfers only the index coefficient IC ₀ of the pixel of interest C _{22 to} the synthesis unit 114.

Weighting filter 307, based on the control of the control unit 117, the pixel of interest and its neighboring eight pixels in the local area from the buffer 300, reads the weighting coefficient F _k from the buffer 306, performs the weighting filtering. Here, the pixel of interest C ₂₂ is simply expressed as C ₀ , the neighboring 8 pixels as C _{1 to} C ₈ , and all 9 pixels as C _k . The ^target pixel C ^N2 ₂₂ that has been subjected to the impulsive noise reduction process is obtained by the weighting filter process represented by the equation (11).

インパルス性のノイズ低減処理がなされた注目画素C^N2 ₂₂は、合成部114へ転送される。

The target pixel C ^N2 ₂₂ that has been subjected to the impulsive noise reduction processing is transferred to the synthesis unit 114.

  The second noise reduction unit 113 shown in FIG. 6 sorts the absolute values of the differences in ascending order in calculating the index index, takes a predetermined number of sums from the smallest, and performs weighting filter processing as impulsive noise reduction processing. Although it has become a configuration, it is not necessary to be limited to such a configuration. For example, as shown in FIG. 8, a configuration in which the sum of all the absolute values of the differences is calculated in calculating the index index, or a configuration in which nonlinear filter processing such as a median filter is performed as impulsive noise reduction processing is possible.

  FIG. 8 shows an example of another configuration of the second noise reduction unit 113. The second noise reduction unit 113 shown in FIG. 6 is replaced with a buffer 302, a sorting unit 303, a weighting coefficient table unit 305, a buffer 306, and a weighting filter unit 307. Is omitted, and a pixel-of-interest readout unit 308, a median filter unit 309, and a switching unit 310 are added. The basic configuration is the same as the second noise reduction unit 113 shown in FIG. 6, and the same name and number are assigned to the same configuration. Only different parts will be described below.

  The buffer 300 is connected to the difference calculation unit 301, the target pixel reading unit 308, and the median filter unit 309. The difference calculation unit 301 is connected to the sum calculation unit 304. The total calculation unit 304 is connected to the switching unit 310 and the combining unit 114. The target pixel readout unit 308 and the median filter unit 309 are connected to the switching unit 310. The switching unit 310 is connected to the combining unit 114. The control unit 117 is bi-directionally connected to the target pixel reading unit 308, the median filter unit 309, and the switching unit 310.

Difference calculation unit 301, based on the control of the control unit 117 calculates an absolute value delta _l of the difference between the neighboring eight pixels with respect to the target pixel C _22. Absolute value delta _l of the calculated differences is transferred to the total sum calculation unit 304.

Total sum calculation unit 304, based on the control of the control unit 117, obtaining the sum of the absolute values delta _l of the difference. This sum is an index index IC ₀ regarding the target pixel C ₂₂ .

指標係数IC₀は、切り換え部310および合成部114へ転送される。

The index coefficient IC ₀ is transferred to the switching unit 310 and the combining unit 114.

The target pixel reading unit 308 reads the target pixel C ₂₂ from the buffer 300 based on the control of the control unit 117 and transfers it to the switching unit 310.

Median filter unit 309 under the control of the control unit 117 reads the local area from the buffer 300, performs the known median filtering, to obtain a processing result MC ₂₂ relates to the target pixel C _22. The median filter processing result MC ₂₂ is transferred to the switching unit 310.

Based on the control of the control unit 117, the switching unit 310 uses the index index IC ₀ from the sum calculation unit 304 and the target pixel C ₂₂ from the target pixel reading unit 308 and the median filter processing result MC ₂₂ from the median filter unit 309. As a result of the switching control relating to the pixel of interest C ^N2 _{22 that} has ^undergone impulsive noise reduction processing, the pixel C ^N2 ₂₂ is obtained.

ここで、Th3は所定の閾値を意味する。(12)式に基づく指標指数IC₀は、欠陥画素のようなインパルス性のノイズでは、近傍８画素全ての画素間で差が大きいため指標指数ICは大きな値となる。一方、平坦部は全ての画素間で差が小さいため指標指数ICは小さな値となる。このため、(13)式に示されるように指標指数ICが所定の閾値Th3より大きい場合にメディアンフィルタ処理結果MC₂₂を選択することでインパルス性のノイズ低減処理が行われることになる。

Here, Th3 means a predetermined threshold value. The index index IC ₀ based on the equation (12) has a large value for the impulsive noise such as a defective pixel because the index index IC ₀ has a large difference among all the neighboring eight pixels. On the other hand, since the flat portion has a small difference between all the pixels, the index index IC has a small value. Therefore, so that the noise reduction processing of impulsive by selecting the median filtering processing result MC ₂₂ is carried out when the index index IC is greater than a predetermined threshold value Th3 as shown in equation (13).

  FIG. 9 shows an example of the configuration of the synthesis unit 114, which includes a first signal selection unit 400, a second signal selection unit 401, a synthesis coefficient table unit 402, a multiplication unit 403, a multiplication unit 404, and an addition unit 405.

  The first noise reduction unit 112 is connected to the first signal selection unit 400. The second noise reduction unit 113 is connected to the first signal selection unit 400, the second signal selection unit 401, and the synthesis coefficient table unit 402. The first signal selection unit 400 is connected to the multiplication unit 403 and the signal processing unit 115. The second signal selection unit 401 is connected to the multiplication unit 404 and the signal processing unit 115. The synthesis coefficient table unit 402 is connected to the multiplication unit 403 and the multiplication unit 404. Multiplier 403 and multiplier 404 are connected to adder 405, and adder 405 is connected to signal processor 115. The control unit 117 is bi-directionally connected to the first signal selection unit 400, the second signal selection unit 401, the synthesis coefficient table unit 402, the multiplication unit 403, the multiplication unit 404, and the addition unit 405.

Based on the control of the control unit 117, the first signal selection unit 400 is subjected to the index index IC ₀ regarding the target pixel C ₂₂ from the second noise reduction unit 113, and the random noise reduction processing from the first noise reduction unit 112. Read the target pixel C ^N1 ₂₂ . When the index index IC ₀ is equal to or smaller than the predetermined threshold Th1, the target pixel C ^N1 ₂₂ is transferred to the signal processing unit 115 as the synthesized target pixel C ^N ₂₂ . When the index index IC ₀ is larger than the predetermined threshold Th1, the target pixel C ^N1 ₂₂ is transferred to the multiplication unit 403.

Based on the control of the control unit 117, the second signal selection unit 401 reads from the second noise reduction unit 113 the index index IC ₀ related to the target pixel C ₂₂ and the target pixel C ^N2 ₂₂ that has undergone impulsive noise reduction processing. When the index index IC ₀ is equal to or greater than the predetermined threshold Th2, the target pixel C ^N2 ₂₂ is transferred to the signal processing unit 115 as the synthesized target pixel C ^N ₂₂ . When the index index IC ₀ is smaller than the predetermined threshold Th2, the target pixel C ^N2 ₂₂ is transferred to the multiplication unit 404.

The synthesis coefficient table unit 402 is a look-up table in which synthesis coefficients w = 0 to 1 and 1-w used for the synthesis process for the index index IC ₀ are recorded. FIG. 10 is an explanatory diagram relating to the composite coefficient w, where w = 0 when the index index IC ₀ ≦ Th1, w = 1 when the index index IC ₀ ≧ Th2, and w when Th1 <IC ₀ <Th2. The characteristic changes linearly from 0 to 1. The synthesis coefficient table unit 402 transfers the synthesis coefficient 1-w to the multiplication unit 403 and the synthesis coefficient w to the multiplication unit 404.

When the target pixel C ^N1 ₂₂ is transferred from the first signal selection unit 400 based on the control of the control unit 117, the multiplication unit 403 multiplies the target pixel C ^N1 ₂₂ by the synthesis coefficient 1-w, and the result ( 1-w) · C ^N1 ₂₂ is transferred to the adder 405.

When the pixel of interest C ^N2 ₂₂ is transferred from the second signal selection unit 401 based on the control of the control unit 117, the multiplication unit 404 multiplies the pixel of interest C ^N2 ₂₂ by the synthesis coefficient w, and as a result, w · C ^N2 ₂₂ is transferred to the adding unit 405.

Based on the control of the control unit 117, the addition unit 405 adds (1-w) · C ^N1 ₂₂ from the multiplication unit 403 and w · C ^N2 ₂₂ from the multiplication unit 404, and combines the target pixel C ^N after synthesis. _Ask for ₂₂ . As a result, the synthesis process shown in the equation (1) is performed.

[Action]
With the above-described configuration, it is possible to perform reduction processing independently on random noise and impulsive noise having different properties, and to combine both based on an index coefficient indicating the degree of impulsiveness. For this reason, it is possible to perform optimum noise reduction processing for both random and impulsive noise, and a high-quality video signal can be obtained.

  In addition, it is possible to suppress the occurrence of discontinuity and artifacts due to two types of noise reduction processing. In the random noise reduction process, the amount of noise is estimated for each pixel of interest and the random noise reduction process is performed. Therefore, only the noise component can be reduced with high accuracy, and a high-quality video signal can be obtained.

  The configuration using the average value for estimating the amount of noise is easy to implement, and can reduce the cost of the system and perform high-speed processing. On the other hand, the configuration using a low-pass filter for estimating the amount of noise enables stable processing because the weight of the target pixel and other pixels can be controlled. The estimation of the noise amount is dynamically adapted to different conditions for each shooting, and the noise amount can be estimated with high accuracy and stability. Further, the configuration using the interpolation calculation for the calculation of the amount of noise is easy to implement, and the cost of the system can be reduced. On the other hand, the configuration for obtaining the noise amount from the look-up table enables high-speed processing.

  In addition, since the coring process is used for the random noise reduction process, only the noise component can be reduced intensively, and continuity with pixels other than noise such as edges can be secured.

  In the impulsive noise reduction processing, in the configuration in which the degree of impulsiveness is obtained for each pixel in the local region and the weighting filter processing is performed, it is possible to reduce only the noise component with high accuracy. In addition, in the configuration in which the weight coefficient is obtained from the lookup table, high-speed processing is possible. On the other hand, in the configuration in which only the target pixel obtains the degree of impulsiveness and performs the nonlinear filter, the processing speed can be increased. Further, since the median filter is used as the non-linear filter processing, the cost of the entire system can be reduced.

  In the configuration using the sum of differences from neighboring pixels in the index coefficient indicating the degree of impulsiveness, a high-speed processing and a low-cost system can be provided. On the other hand, with the configuration that sorts the difference between neighboring pixels and uses a predetermined number of sums from small values, it is possible to discriminate between impulsive noise and edge portions with high precision, and high quality with little deterioration of edge portions Video signal can be obtained.

  Further, since noise reduction processing is performed for each color signal, noise can be reduced with high accuracy, and a high-quality video signal can be obtained. In addition, since there is no preprocessing such as interpolation processing before the noise reduction processing, the accuracy of the noise reduction processing can be improved.

  Furthermore, the Bayer type primary color filter has a high affinity with the current imaging system, and can be combined with various systems.

[Modification]
In the first embodiment, the Bayer-type primary color filter is used as the image sensor, but it is not necessary to be limited to such a configuration. For example, the color difference line sequential complementary color filter shown in FIG. 2C can be used, and a two-plate or three-plate image sensor can be used.

FIG. 2C shows the configuration of the color difference line sequential complementary color filter. The color difference line sequential method uses 2 × 2 pixels as a basic unit, and cyan (Cy), magenta (Mg), yellow (Ye), and green (G) are arranged one by one. However, the positions of Mg and G are reversed for each line. In the case of the color difference line sequential complementary color filter, the color signal separation and extraction unit 111 reads the video signal in units of 10 × 10 pixels shown in FIG. 2 (c), and reads the video signal as shown in FIG. For each color filter of G, Ye, and Cy, 5 × 5 pixels centered on the target pixel are separated as local regions. The pixel value in the local area is represented by C _ij (C is a color signal, C = Mg, G, Ye, Cy).

  Further, in the first embodiment, an imaging unit including a lens system 100, an aperture 101, a CCD 102, a temperature sensor 103, a Gain 104, an A / D 105, a PreWB unit 107, a photometric evaluation unit 108, a focus detection unit 109, and an AF motor 110, Although it is an integrated configuration, it need not be limited to such a configuration. For example, as shown in FIG. 11, the video signal captured by a separate imaging unit is recorded in raw data format, and additional information such as the color filter of the CCD 102 and exposure conditions at the time of shooting is recorded in the header part. It is also possible to process from the recorded medium.

  11 omits the lens system 100, the diaphragm 101, the CCD 102, the temperature sensor 103, the Gain 104, the A / D 105, the PreWB unit 107, the photometric evaluation unit 108, the in-focus detection unit 109, and the AF motor 110 from the configuration shown in FIG. The input unit 500 and the header information analysis unit 501 are added. The basic configuration is the same as in FIG. 1, and the same name and number are assigned to the same configuration. Only different parts will be described below.

  The input unit 500 is connected to the buffer 106 and the header information analysis unit 501. The control unit 117 is bi-directionally connected to the input unit 500 and the header information analysis unit 501.

  By starting the reproduction operation via the external I / F unit 118 such as a mouse or a keyboard, the video signal and header information stored in the recording medium are read from the input unit 500. The video signal from the input unit 500 is transferred to the buffer 106, and the header information is transferred to the header information analysis unit 501.

  The header information analysis unit 501 extracts information at the time of shooting from the header information and transfers the information to the control unit 117. The subsequent processing is the same as in FIG.

  In the first embodiment, processing based on hardware is assumed. However, it is not necessary to be limited to such a configuration. For example, the video signal from the CCD 102 can be output as unprocessed raw data, and the accompanying information such as the color filter of the CCD 102 and the exposure condition at the time of shooting is output as header information, and can be configured to be processed separately by software. . FIG. 12A shows a flow related to software processing of signal processing.

  In step S1, header information such as a video signal and exposure conditions at the time of shooting is read.

  In step S2, separation is performed for each color signal based on the color filter of the CCD 102 as shown in FIG.

  In step S3, as shown in FIG. 2B, a local area having a predetermined size, for example, a 5 × 5 pixel size, including the target pixel to be subjected to noise reduction processing is extracted.

  In step S4, a first noise reduction process, which is a random noise reduction process, is performed as described separately.

  In step S5, a second noise reduction process that is an impulsive noise reduction process is performed as described separately.

  In step S6, the signal subjected to the first noise reduction processing and the signal subjected to the second noise reduction processing are synthesized as will be described separately.

  In step S7, it is determined whether or not all the local regions are completed. If not, the process branches to step S3, and if completed, the process branches to step S8.

  In step S8, it is determined whether all the color signals are completed. If not completed, the process branches to step S2, and if completed, the process branches to step S9.

  In step S9, known interpolation processing, gradation conversion processing, edge enhancement processing, color enhancement processing, and other signal processing are performed.

  In step S10, the processed video signal is output and the process ends.

  FIG. 12B is a flow relating to the first noise reduction processing in step S4.

  In step S20, the average value of the local region is calculated as shown in equation (2).

  In step S21, information such as temperature and gain is set from the read header information. If a necessary parameter does not exist in the header information, a predetermined standard value is assigned.

  In step S22, the coordinate data and correction coefficient of the reference noise model are read.

  In step S23, the coordinate data of the section of the reference noise model to which the target pixel belongs and the corresponding correction coefficient are selected.

  In step S24, the amount of noise is obtained by interpolation processing shown in equations (5) and (6).

  In step S25, a signal that has undergone random noise reduction processing in the coring processing shown in equation (7) is obtained.

  In step S26, the signal subjected to the first noise reduction process is output and the process ends.

  FIG. 12C is a flow relating to the second noise reduction processing in step S5.

  In step S30, one of the target pixel and the neighboring eight pixels in the local region is selected.

  In step S31, the absolute value of the eight differences is calculated as shown in equation (8).

  In step S32, the absolute values of the eight differences are sorted in ascending order.

  In step S33, a sum of absolute values of a predetermined number, for example, four differences is calculated from the smaller one as shown in the equation (9) and used as an index coefficient.

  In step S34, a look-up table for outputting a weighting coefficient is input with an index coefficient constructed based on equation (10) as an input.

  In step S35, a weighting coefficient is output based on the index coefficient.

  In step S36, it is determined whether all of the target pixel and neighboring 8 pixels have been selected. If selection has not been completed, the process branches to step S30. If completed, the process branches to step S37.

  In step S37, the weighting filter process shown in the equation (11) is performed.

  In step S38, the signal obtained by the weighting filter process is output as a signal subjected to the second noise reduction process.

  In step S39, the index coefficient relating to the target pixel is output and the process ends.

  FIG. 12D is a flow relating to the composition processing in step S6.

  In step S40, an index coefficient related to the target pixel is input.

  In step S41, the index coefficient is compared with a predetermined threshold Th1, and if the index coefficient is equal to or less than the threshold Th1, the process branches to step S42, and if the index coefficient is greater than the threshold Th1, the process branches to step S43.

  In step S42, the signal subjected to the first noise reduction processing is output and the process ends.

  In step S43, the index coefficient is compared with a predetermined threshold Th2. If the index coefficient is equal to or greater than the threshold Th2, the process branches to step S44. If the index coefficient is smaller than the threshold Th2, the process branches to step S45.

  In step S44, the signal subjected to the second noise reduction process is output and the process ends.

  In step S45, a look-up table for outputting a composite coefficient with the index coefficient shown in FIG. 10 as input is input.

  In step S46, a composite coefficient is output based on the index coefficient.

  In step S47, the signal subjected to the first noise reduction process is multiplied by (1-synthesis coefficient).

  In step S48, the signal subjected to the second noise reduction process is multiplied by a synthesis coefficient.

  In step S49, a signal obtained by multiplying the signal subjected to the first noise reduction process by (1-synthesis coefficient) and a signal obtained by multiplying the signal subjected to the second noise reduction process by the synthesis coefficient are added.

  In step S50, the synthesized signal is output and the process ends.

  As described above, the signal processing may be performed by software, and the same operational effects as in the case of processing by hardware are achieved.

Second Embodiment [Configuration]
FIG. 13 is a configuration diagram of the second embodiment. In the second embodiment, the color signal separation / extraction unit 111 in the first embodiment shown in FIG. 1 is changed to the luminance / color separation / extraction unit 600, the first noise reduction unit 112 is changed to the first noise reduction unit 601, and the second noise is changed. The reduction unit 113 is replaced with the second noise reduction unit 602, the synthesis unit 114 is replaced with the synthesis unit 603, and a buffer 604 and a synchronization unit 605 are added. The basic configuration is the same as that of the first embodiment, and the same name and number are assigned to the same configuration. Only the different parts will be described below.

  The buffer 106 is connected to the PreWB unit 107, the photometric evaluation unit 108, the in-focus detection unit 109, and the luminance color separation / extraction unit 600. The luminance color separation / extraction unit 600 is connected to the first noise reduction unit 601 and the second noise reduction unit 602. The first noise reduction unit 601 and the second noise reduction unit 602 are connected to the synthesis unit 603. The combining unit 603 is connected to the signal processing unit 115 via the buffer 604 and the synchronization unit 605. The control unit 117 is bi-directionally connected to the luminance color separation / extraction unit 600, the first noise reduction unit 601, the second noise reduction unit 602, the synthesis unit 603, and the synchronization unit 605.

[Action]
This is basically the same as that of the first embodiment, and only different parts will be described. In FIG. 13, the flow of signals will be described. The imaging mode is entered by pressing the shutter button via the external I / F unit 118. Video signals photographed through the lens system 100, the diaphragm 101, and the CCD 102 are continuously output as analog signals at predetermined time intervals. In the second embodiment, the CCD 102 is assumed to be a single-plate CCD having a color difference line sequential complementary color filter arranged on the front surface.

  FIG. 14A shows a configuration of a color difference line sequential complementary color filter. The color difference line sequential method uses 2 × 2 pixels as a basic unit, and cyan (Cy), magenta (Mg), yellow (Ye), and green (G) are arranged one by one. However, the positions of Mg and G are reversed for each line. As shown in FIG. 14A, the video signal from the CCD 102 is composed of two field signals (even field signal and odd field signal) in which the upper and lower lines are added and separated into even lines and odd lines. Further, 1/60 second is assumed as the predetermined time interval. A single video signal is obtained by combining the even and odd field signals, and the single video signal is referred to as a frame signal. The frame signal is synthesized at 1/30 second intervals.

  The analog signal from the CCD 102 is amplified by a predetermined amount by the gain 104, converted into a digital signal by the A / D 105, and transferred to the buffer 106.

  The buffer 106 can record two field signals, that is, one frame signal, and is sequentially overwritten as the image is taken. The field signal in the buffer 106 is intermittently transferred to the PreWB unit 107, the photometric evaluation unit 108, and the in-focus detection unit 109 at predetermined time intervals based on the control of the control unit 117.

  On the other hand, the luminance color separation / extraction unit 600 calculates the luminance signal Y and the color difference signals Cb, Cr from the even and odd field signals based on the control of the control unit 117.

この後、ノイズ低減処理の対象となる注目画素および注目画素の近傍に位置する近傍画素からなる局所領域を順次抽出する。第２の実施形態においては、局所領域としてとして5×5画素を想定する。ただし、輝度信号Yは5×5画素全てに存在するが、色差信号Cb,Crは5×3画素または5×2画素となる。

After that, the local region including the target pixel that is the target of the noise reduction process and the neighboring pixels located in the vicinity of the target pixel is sequentially extracted. In the second embodiment, 5 × 5 pixels are assumed as the local region. However, although the luminance signal Y exists in all 5 × 5 pixels, the color difference signals Cb and Cr are 5 × 3 pixels or 5 × 2 pixels.

  FIGS. 14B and 14C show examples of local regions extracted from even and odd field signals. FIG. 14B shows an example in which the luminance signal Y and the color difference signals Cb and Cr are extracted from the even field signal. The color difference signal Cr is 5 × 3 pixels, and the color difference signal Cb is 5 × 2 pixels. In this case, the target pixel for noise reduction processing is the luminance signal Y and the color difference signal Cr, and the color difference signal Cb is not the target. Note that if the target pixel position is different, an example in which the color difference signal Cb exists and the color difference signal Cr does not exist may occur contrary to the above. FIG. 14C shows an example in which the luminance signal Y and the color difference signals Cb, Cr are extracted from the odd field signal. The color difference signal Cb is 5 × 3 pixels, and the color difference signal Cr is 5 × 2 pixels. In this case, the target pixel for noise reduction processing is the luminance signal Y and the color difference signal Cb, and the color difference signal Cr is excluded.

Note that if the target pixel position is different, an example may occur in which the color difference signal Cr exists and the color difference signal Cb does not exist, contrary to the above. Thereafter, the pixels in the local area are represented by C _ij (C is a luminance or chrominance signal and C = Y, Cb, Cr, i is an X coordinate, i = 0 to 4, j is a Y coordinate, and j is an even field signal. = 0,2,4,6,8, and in case of odd field signal, it is expressed as j = 1,3,5,7,9). Regarding the color difference signal, pixels that are missing in the 5 × 5 pixel local region are not processed.

For the target pixel, the luminance signal is Y ₂₄ , the color difference signal is Cr ₂₄ or Cb _{24 for} the even field signal, the luminance signal is Y ₂₅ , and the color difference signal is Cr ₂₅ or Cb ₂₅ for the odd field signal. The following description will be made with respect to the even field signal and the target pixel Y ₂₄ and Cr ₂₄ as shown in FIG. 14B. However, the even field signal and the target pixel are local to Y ₂₄ , Cb ₂₄ and the odd field signal. The same holds true only for the region configuration. The extracted local region is transferred to the first noise reduction unit 601 and the second noise reduction unit 602.

Based on the control of the control unit 117, the first noise reduction unit 601 estimates the random noise amount N _{24 related} to the target pixel C ₂₄ from the low-frequency component in the local region. Thereafter, a low-pass filter is selected using the estimated noise amount N ₂₄ , and a random noise reduction process is performed by performing a low-pass filter process on the local region. Hereinafter, the ^target pixel on which random noise reduction processing has been performed by the first noise reduction unit 112 is described as C ^N1 ₂₄ .

The target pixel C ^N1 ₂₄ that has been subjected to random noise reduction processing is transferred to the synthesis unit 603. On the other hand, based on the control of the control unit 117, the second noise reduction unit 602 uses the index coefficient IC indicating the degree of impulsive noise from the local region of the luminance signal as the target pixel Y ₂₄ and its neighboring eight pixels Y ₁₂ and Y _22. , Y ₃₂ , Y ₁₄ , Y ₄₄ , Y ₁₆ , Y ₂₆ , Y ₃₆ are calculated. For the color difference signal, the index coefficient IC is not calculated, but the index coefficient IC of the luminance signal is used. Hereinafter, the index coefficient of the pixel of interest C ₂₄ is _denoted by IC ₀ , the index coefficients of the neighboring eight pixels are denoted by IC ₁ to IC ₈ , and the entire index coefficient is denoted by IC _k (k = 0 to 8). Further, as necessary, the pixel of interest C ₂₄ is simply expressed as C ₀ , the neighboring 8 pixels as C _{1 to} C ₈ , and all 9 pixels as C _k .

Thereafter, the weight coefficient of the pixel of interest and its neighboring eight pixels is obtained using the index coefficient IC _k and the impulsive noise reduction process is performed by performing the weighting filter process. Hereinafter, the ^target pixel which has been subjected to the impulsive noise reduction processing by the second noise reduction unit 113 is described as C ^N2 ₂₄ . The index coefficient IC ₀ of the target pixel Y ₂₄ of the luminance signal and the target pixel C ^N2 ₂₄ subjected to the impulsive noise reduction process are transferred to the synthesis unit 603.

Based on the control of the control unit 117, the synthesizing unit 603 uses the index coefficient IC ₀ transferred from the second noise reduction unit 113 to select a target pixel that has been subjected to random noise reduction processing by the first noise reduction unit 112. the target pixel C ^N2 ₂₄ which noise reduction processing C ^N1 ₂₄ and impulsive was made by combining processing, obtaining the target pixel C ^N ₂₄ after combination as shown in (1). The synthesis coefficient w is obtained from the index coefficient IC _{0 of} the luminance signal, and is used in common for the synthesis of the luminance signal and the color difference signal. The synthesized pixel of interest C ^N ₂₄ is transferred to the buffer 604.

  The buffer 604 can record two field signals, that is, one frame signal, and is sequentially overwritten as the image is taken. The processes in the luminance color separation / extraction unit 600, the first noise reduction unit 601, the second noise reduction unit 602, and the synthesis unit 603 are performed synchronously in units of local regions based on the control of the control unit 117.

  The synchronizer 605 reads an even field signal and an odd field signal that have been combined from the buffer 604 under the control of the controller 117, performs a known interpolation process on the color difference signal, and then performs an even field signal and an odd field signal. A frame signal is generated by performing a known synchronization process on the signal. The generated frame signal is transferred to the signal processing unit 115.

  FIG. 15 shows an example of the configuration of the first noise reduction unit 601. The average value calculation unit 201 and the coring unit 207 are omitted from the configuration of the first noise reduction unit 112 shown in FIG. , Buffer 701, difference component table 702, coordinate table 703, weight coefficient synthesis unit 704, bilateral filter unit 705, filter ROM 706, filter selection unit 707, and frequency filter unit 708. ing. The basic configuration is equivalent to the first noise reduction unit 112 shown in FIG. 3, and the same name and number are assigned to the same configuration. Only different parts will be described below.

As shown in FIGS. 14B and 14C, the local area of 5 × 5 pixels is sequentially transferred from the luminance color separation / extraction unit 600 to the buffer 200 for each of Y, Cb, and Cr. Regarding the color difference signal, pixels that are missing in the 5 × 5 pixel local region are not processed. The following description will be made with respect to the even field signal and the target pixel Y ₂₄ and Cr ₂₄ as shown in FIG. 14B. However, the even field signal and the target pixel are local to Y ₂₄ , Cb ₂₄ and the odd field signal. The same holds true only for the region configuration.

Difference component calculation unit 700, based on the control of the control unit 117 reads the local area from the buffer 200, calculates the absolute value [delta] _ij of a difference component between the target pixel value C _24, as shown in equation (15).

差成分の絶対値δ_ijは、バッファ701へ転送される。

The absolute value δ _ij of the difference component is transferred to the buffer 701.

Based on the control of the control unit 117, the difference component table 702 reads the absolute value δ _ij of the difference component from the buffer 701, and obtains a first weighting factor w1 _ij corresponding to the absolute value δ _ij of the difference component. The first weighting coefficient w1 _ij is determined by a function expression shown in, for example, Expression (16).

(16)式におけるσ1は調整用のパラメータで、σ1=1〜10程度を用いる。差成分用テーブル702は、(16)式に示される関数式に基づき予め算出しておいた第１の重み係数w1_ijを記録したテーブルである。求められた第１の重み係数w1_ijは、重み係数合成部704へ転送される。

In the equation (16), σ1 is a parameter for adjustment, and σ1 = 1 to 10 is used. The difference component table 702 is a table in which the first weighting coefficient w1 _ij calculated in advance based on the function equation shown in the equation (16) is recorded. The obtained first weight coefficient w1 _ij is transferred to the weight coefficient synthesis unit 704.

Based on the control of the control unit 117, the weighting factor combining unit 704 performs the second weighting factor corresponding to the first weighting factor w1 _ij from the difference component table 702 and the coordinate value (i, j) from the coordinate table 703. Read w2 _ij and synthesize them. The second weighting coefficient w2 _ij is determined by a function expression shown in, for example, the expression (17).

(17)式におけるσ2は調整用のパラメータで、σ2=1〜10程度を用いる。また、TiおよびTjは注目画素の座標を意味し、第２の実施形態においてはTi=2, Tj=4となる。

In the equation (17), σ2 is an adjustment parameter, and σ2 = 1-10 is used. Ti and Tj mean the coordinates of the pixel of interest, and in the second embodiment, Ti = 2 and Tj = 4.

The coordinate table 703 is a table in which the second weighting coefficient w2 _ij calculated in advance based on the function expression shown in the expression (17) is recorded. The first weight coefficient w1 _ij and the second weight coefficient w2 _ij are combined based on the equation (18) to calculate the weight coefficient w _ij .

算出された重み係数w_ijは、バイラテラルフィルタ部705へ転送される。

The calculated weight coefficient w _ij is transferred to the bilateral filter unit 705.

Based on the control of the control unit 117, the bilateral filter unit 705 performs bilateral filter processing on the local region from the buffer 200 using the weight coefficient w _ij from the weight coefficient synthesis unit 704.

(19)式に示されるバイラテラルフィルタ処理の結果C_Biはパラメータ選択部205へ転送される。

The result C_Bi of the bilateral filter process shown in the equation (19) is transferred to the parameter selection unit 205.

The parameter selection unit 205 obtains the signal level l from the bilateral filter processing result C_Bi from the bilateral filter unit 705, the gain g from the gain information from the gain calculation unit 202, and the temperature t from the temperature information from the control unit 117. Set. Next, the coordinate data and the correction coefficient k _gt of the section (L _n , N _n ) and (L _{n + 1} , N _{n + 1} ) to which the signal level l belongs are searched from the parameter ROM 204 and transferred to the noise interpolation unit 206. To do.

Under the control of the control unit 117, the noise interpolation unit 206 calculates the reference noise amount _Nl in the reference noise model from the equation (5), and calculates the noise amount N from the correction coefficient _kg based on the equation (6). The noise amount N is transferred to the filter selection unit 707 as the noise amount N ₂₄ of the target pixel C ₂₄ . In the process of calculating the amount of noise, it is not necessary to obtain information such as temperature t and gain g for each image. A configuration in which arbitrary information is recorded in the standard value assigning unit 203 and the calculation process is omitted is also possible.

Based on the control of the control unit 117, the filter selection unit 707 selects a filter coefficient to be used for filter processing from the filter ROM 706 using the noise amount N ₂₄ from the noise interpolation unit 206.

  FIG. 16 shows an example of the filter coefficient recorded in the filter ROM 706. The size is 5 × 5 pixels, and four types of frequency characteristics from Type 1 to Type 4 are recorded. Each coefficient is multiplied by 128. Type 1 has a frequency characteristic in which high frequency components remain, and Type 4 sequentially suppresses high frequency components.

Filter selection unit 707 selects the frequency characteristics of the noise amount N ₂₄ Type1~Type4. This selection is performed based on the relationship between the noise amount N ₂₄ and the filter type Type shown in FIG. 17, for example. The frequency characteristic that suppresses the high frequency component is selected as the noise amount N ₂₄ increases. The filter selection unit 707 transfers the selected filter coefficient to the frequency filter unit 708.

Based on the control of the control unit 117, the frequency filter unit 708 performs frequency filter processing on the local region from the buffer 200 using the filter coefficient from the filter selection unit 707. In the case of the luminance signal Y, the size of the filter is the same in the local area of 5 × 5 pixels, but in the case of the color difference signal Cr, the size of the filter is not the same in the local area of 5 × 3 pixels. In this case, a pixel that does not exist is excluded from the target of the filtering process, and is normalized by normalizing based on the filter coefficient of the actually used pixel. The result of the frequency filter processing is transferred to the synthesis unit 603 as the ^target pixel C ^N1 ₂₄ that has been subjected to random noise reduction processing.

  18 shows an example of the configuration of the second noise reduction unit 602. In the configuration of the second noise reduction unit 113 shown in FIG. 6, the difference calculation unit 301 is used as the luminance difference calculation unit 800, and the weighting filter unit 307 is used as the luminance weighting. The filter unit 801 and the color difference weighting filter unit 802 are replaced. The basic configuration is the same as the second noise reduction unit 113 shown in FIG. 6, and the same name and number are assigned to the same configuration. Only different parts will be described below.

  The luminance color separation / extraction unit 600 is connected to the luminance difference calculation unit 800, the luminance weighting filter unit 801, and the color difference weighting filter unit 802 via the buffer 300. The luminance difference calculation unit 800 is connected to the buffer 302. The buffer 306 is connected to the luminance weighting filter unit 801 and the color difference weighting filter unit 802. The luminance weighting filter unit 801 and the color difference weighting filter unit 802 are connected to the synthesis unit 603. The control unit 117 is bidirectionally connected to the luminance difference calculation unit 800, the luminance weighting filter unit 801, and the color difference weighting filter unit 802.

As shown in FIGS. 14B and 14C, the luminance color separation / extraction unit 600 sequentially transfers a local area of 5 × 5 pixels to the buffer 300 for each of Y, Cb, and Cr. Regarding the color difference signal, pixels that are missing in the 5 × 5 pixel local region are not processed. The following description will be made with respect to the even field signal and the target pixel Y ₂₄ and Cr ₂₄ as shown in FIG. 14B. However, the even field signal and the target pixel are local to Y ₂₄ , Cb ₂₄ and the odd field signal. The same holds true only for the region configuration.

Luminance difference calculation unit 800, based on the control of the control unit 117, the target pixel Y ₂₄ and its neighboring eight pixels Y ₁₂ of the brightness _{signal, Y 22, Y 32, Y} 14, Y 34, Y 16, Y 26, Y 36 respect of a total of 9 pixels, and calculates the absolute value delta _l of the difference between the neighboring eight pixels each, as shown in equation (8). The absolute value of the difference delta _l is transferred to the buffer 302.

Sorting unit 303, under the control of the control unit 117 sorts the absolute value of the difference between the buffer 302 the delta _l in ascending order, a predetermined number from the smallest, in the second embodiment four to total sum calculation unit 304 Forward.

  Based on the control of the control unit 117, the sum calculation unit 304 calculates the sum of the absolute values of the four differences sorted in ascending order transferred from the sort unit 303 as shown in the equation (9). This sum is the index index IC.

The weighting coefficient table unit 305 outputs the weighting coefficient F used for the subsequent weighted filter processing based on the index coefficient IC. The weighting factor F is transferred to the buffer 306. The luminance difference calculation unit 800, the sorting unit 303, the total sum calculation unit 304, and the weight coefficient table unit 305 repeat the above process nine times based on the control of the control unit 117. Hereinafter, the finger index coefficient is represented by IC _k and the weight coefficient is represented by F _k . Note that the sum total calculation unit 304 also transfers only the index coefficient IC ₀ of the target pixel Y ₂₄ of the luminance signal to the synthesis unit 603.

Based on the control of the control unit 117, the luminance weighting filter unit 801 reads the target pixel of the luminance signal in the local area and the eight neighboring pixels from the buffer 300, and reads the weighting factor F _k from the buffer 306. The target pixel Y ^N2 ₂₄ that has been subjected to the impulsive noise reduction processing is obtained by performing the weighting filter processing. The target pixel Y ^N2 ₂₄ that has been subjected to the impulsive noise reduction processing is transferred to the synthesis unit 603.

Based on the control of the control unit 117, the chrominance weighting filter unit 802 reads the target pixel of the chrominance signal in the local area and its neighboring eight pixels from the buffer 300, and reads the weighting factor F _k from the buffer 306, and shows the equation (11). The target pixel Cr ^N2 ₂₄ that has been subjected to the impulsive noise reduction processing is obtained. That is, the color difference signal is processed with the weighting coefficient F _k obtained based on the index coefficient IC _k of the luminance signal. The degree of impulsiveness can be obtained with sufficient accuracy only from the luminance signal. By applying this result to the color difference signal, impulsive noise can be reduced with high accuracy without performing processing such as interpolation processing on the color difference signal in which missing pixels exist. The target pixel Cr ^N2 ₂₄ that has been subjected to the impulsive noise reduction processing is transferred to the synthesis unit 603.

  FIG. 19 shows an example of the configuration of the synthesis unit 603. In the configuration of the synthesis unit 114 shown in FIG. 9, the multiplication unit 403 is used as the luminance multiplication unit 900 and the color difference multiplication unit 901, and the multiplication unit 404 is used as the luminance multiplication unit 902 and The color difference multiplication unit 903 is replaced. The basic configuration is the same as that of the combining unit 114 shown in FIG. 9, and the same name and number are assigned to the same configuration. Only different parts will be described below.

The first noise reduction unit 601 is connected to the first signal selection unit 400. The second noise reduction unit 602 is connected to the first signal selection unit 400, the second signal selection unit 401, and the synthesis coefficient table unit 402. The first signal selection unit 400 is connected to the luminance multiplication unit 900, the color difference multiplication unit 901, and the buffer 604. The second signal selection unit 401 is connected to the luminance multiplication unit 902, the color difference multiplication unit 903, and the buffer 604. The synthesis coefficient table unit 402 is connected to the luminance multiplication unit 900, the color difference multiplication unit 901, the luminance multiplication unit 902, and the color difference multiplication unit 903. The luminance multiplication unit 900, the color difference multiplication unit 901, the luminance multiplication unit 902, and the color difference multiplication unit 903 are connected to the addition unit 405, and the addition unit 405 is connected to the buffer 604. The control unit 117 is bi-directionally connected to the luminance multiplier 900, the color difference multiplier 901, the luminance multiplier 902, and the color difference multiplier 903. The following description will be made with respect to the even field signal and the target pixel Y ₂₄ and Cr ₂₄ as shown in FIG. 14B. However, the even field signal and the target pixel are local to Y ₂₄ , Cb ₂₄ and the odd field signal. The same holds true only for the region configuration.

Based on the control of the control unit 117, the first signal selection unit 400 receives the index index IC ₀ related to the target pixel Y ₂₄ of the luminance signal from the second noise reduction unit 602, and the random noise reduction processing from the first noise reduction unit 112. The target pixel C ^N1 ₂₄ of the luminance and chrominance signals subjected to is read. When the index index IC ₀ is equal to or less than the predetermined threshold Th1, the target pixel C ^N1 ₂₄ is transferred to the buffer 604 as the synthesized target pixel C ^N ₂₄ . When the index index IC ₀ is larger than the predetermined threshold Th1, the target pixel Y ^N1 ₂₄ of the luminance signal is transferred to the luminance multiplier 900, and the target pixel Cr ^N1 ₂₄ of the color difference signal is transferred to the color difference multiplier 901.

Based on the control of the control unit 117, the second signal selection unit 401 receives the index index IC ₀ related to the pixel of interest Y ₂₄ of the luminance signal from the second noise reduction unit 602 and the luminance and color difference signals that have undergone impulsive noise reduction processing. ^{Read the target} pixel C ^N2 ₂₄ . When the index index IC ₀ is equal to or greater than the predetermined threshold Th2, the target pixel C ^N2 ₂₄ is transferred to the buffer 604 as the target pixel C ^N ₂₄ after synthesis. When the index index IC ₀ is smaller than the predetermined threshold Th2, the target pixel Y ^N2 ₂₄ of the luminance signal is transferred to the luminance multiplier 902, and the target pixel Cr ^N2 ₂₄ of the color difference signal is transferred to the color difference multiplier 903.

The synthesis coefficient table unit 402 is a lookup table in which synthesis coefficients w = 0 to 1 and 1-w used for the synthesis process for the index index IC ₀ are recorded as shown in FIG. The synthesis coefficient table unit 402 transfers the synthesis coefficient 1-w to the luminance multiplication unit 900 and the color difference multiplication unit 901, and transfers the synthesis coefficient w to the luminance multiplication unit 902 and the color difference multiplication unit 903.

Brightness multiplication unit 900, based on the control of the control unit 117, if the target pixel Y ^N1 ₂₄ of the brightness signal from the first signal selection unit 400 is transferred, by multiplying the target pixel Y ^N1 ₂₄ Synthesis coefficient 1-w Then, the result (1-w) · Y ^N1 ₂₄ is transferred to the adding unit 405.

Color difference multiplication unit 901, based on the control of the control unit 117, if the target pixel Cr ^N1 ₂₄ of the color difference signal from the first signal selection unit 400 is transferred, by multiplying the target pixel Cr ^N1 ₂₄ Synthesis coefficient 1-w Then, the result (1-w) · Cr ^N1 ₂₄ is transferred to the adding unit 405.

Luminance multiplying unit 902 under the control of the control unit 117 if the target pixel Y ^N2 ₂₄ of the brightness signal from the second signal selector 401 is transferred, by multiplying the target pixel Y ^N2 ₂₄ and combination coefficient w, the The result w · Y ^N2 ₂₄ is transferred to the adder 405.

Color difference multiplication unit 903, based on the control of the control unit 117, if the target pixel Cr ^N2 ₂₄ of the color difference signal from the second signal selector 401 is transferred, by multiplying the target pixel Cr ^N2 ₂₄ and combination coefficient w, the The result w · Cr ^N2 ₂₄ is transferred to the adding unit 405.

Based on the control of the control unit 117, the addition unit 405 adds (1-w) · Y ^N1 ₂₄ from the luminance multiplication unit 900 and w · Y ^N2 ₂₄ from the luminance multiplication unit 902, and performs a combined luminance signal The target pixel Y ^N ₂₄ is obtained. Similarly, (1-w) · Cr ^N1 ₂₄ from the color difference multiplying unit 901 and w · Cr ^N2 ₂₄ from the color difference multiplying unit 903 are added to obtain the target pixel Cr ^N ₂₄ of the combined color difference signal. That is, the color difference signal is synthesized using the synthesis coefficient w obtained based on the index coefficient IC _k of the luminance signal. The degree of impulsiveness can be obtained with sufficient accuracy only from the luminance signal. By applying this result to the color difference signal, high-accuracy synthesis processing can be performed. The target pixel C ^N ₂₂ of the synthesized luminance and color difference signals is transferred to the buffer 604.

[Action]
With the above-described configuration, it is possible to perform reduction processing independently on random noise and impulsive noise having different properties, and to combine both based on an index coefficient indicating the degree of impulsiveness. For this reason, it is possible to perform optimum noise reduction processing for both random and impulsive noise, and a high-quality video signal can be obtained.

  In addition, it is possible to suppress the occurrence of discontinuity and artifacts due to two types of noise reduction processing. In the random noise reduction process, the amount of noise is estimated for each pixel of interest and the random noise reduction process is performed. Therefore, only the noise component can be reduced with high accuracy, and a high-quality video signal can be obtained.

  Since the configuration using the bilateral filter processing result for estimating the amount of noise eliminates noise around the pixel of interest, edge structure, and the like and obtains a low frequency component, highly accurate and stable processing is possible. The estimation of the noise amount is dynamically adapted to different conditions for each shooting, and the noise amount can be estimated with high accuracy and stability. Further, the configuration using the interpolation calculation for the calculation of the amount of noise is easy to implement, and the cost of the system can be reduced.

  Further, since the frequency filter processing selected based on the amount of noise is used for the random noise reduction processing, only the random noise components can be reduced intensively, and a high-quality video signal can be obtained. Further, the filter processing is relatively easy to implement, and it is possible to increase the speed and cost of the entire system.

  In the impulsive noise reduction processing, the configuration in which the degree of impulsiveness is obtained for each pixel in the local region and the weighting filter processing is performed can reduce only the noise component with high accuracy. Further, since the weighting coefficient is obtained from the lookup table, high speed processing is possible. In the index coefficient indicating the degree of impulsiveness, the difference between neighboring pixels is sorted and the configuration using a predetermined number of sums from a small value can accurately identify impulsive noise and edge portions, A high-quality video signal with little deterioration of the edge portion can be obtained.

  Further, since the luminance signal and the color difference signal are separated from the video signal and noise reduction processing is performed for each of the luminance signal and the color difference signal, noise can be reduced with high accuracy, and a high-quality video signal can be obtained. Moreover, it becomes possible to adapt to various imaging systems.

  Furthermore, in the impulsive noise reduction and synthesis processing, the color difference signal is processed based on the processing result of the luminance signal, so the processing between the luminance signal and the color difference signal is unified, and a high-quality video signal with few artifacts can be obtained. .

  Furthermore, the color difference line sequential complementary color filter has high affinity with the current imaging system, and can be combined with various systems.

[Modification]
In the second embodiment, the color difference line sequential complementary color filter is used as the image sensor, but it is not necessary to be limited to such a configuration. For example, a configuration using a Bayer-type primary color filter shown in FIG. In this case, the missing RGB signal is compensated by a known interpolation process, and the luminance signal Y and the color difference signals Cb, Cr are obtained based on the equation (20).

また、この場合はフレーム信号のみでフィールド信号が存在しないことになる。さらに、第１の実施形態における図１１に示される形態と同様に、別体の撮像部で撮像された時系列的に連続する複数の映像信号を未処理のRawデータ形態で、さらにCCD102の色フィルタや撮影時の露光条件などの付随情報をヘッダ部に記録した記録媒体から処理をする構成も可能である。

In this case, only the frame signal is present and no field signal exists. Further, similarly to the embodiment shown in FIG. 11 in the first embodiment, a plurality of time-sequential video signals captured by a separate imaging unit are converted into unprocessed raw data format, and further the color of the CCD 102 A configuration is also possible in which processing is performed from a recording medium in which accompanying information such as filters and exposure conditions at the time of photographing is recorded in the header portion.

  Furthermore, although the second embodiment is premised on processing by hardware, it is not necessary to be limited to such a configuration. For example. For example, time-sequential continuous video signals from the CCD102 are output as raw data as raw data, and accompanying information such as the CCD102 color filter and exposure conditions at the time of shooting is output as header information. It is also possible to have a configuration for processing.

  FIG. 20A shows a flow relating to software processing of signal processing. Note that the same number of steps is assigned to the same processing steps as the signal processing flow in the first embodiment shown in FIG. 12A.

  In step S1, header information such as a plurality of video signals and color filters and exposure conditions at the time of photographing is read.

  In step S60, an even field signal and an odd field signal are sequentially extracted from one video signal, that is, a frame signal.

  In step S61, the video signal is separated into a luminance signal and a color signal as shown in equation (14).

  In step S62, as shown in FIGS. 14B and 14C, a local area of a predetermined size, for example, 5 × 5 pixels including the target pixel to be subjected to noise reduction processing is extracted with respect to the luminance signal and the color difference signal. Is done.

  In step S63, a first noise reduction process, which is a random noise reduction process, is performed on the luminance signal and the color difference signal as described separately.

  In step S64, as will be described separately, a second noise reduction process that is an impulsive noise reduction process is performed on the luminance signal and the color difference signal.

  In step S65, the signal subjected to the first noise reduction processing and the signal subjected to the second noise reduction processing are synthesized with respect to the luminance signal and the color difference signal as described separately.

  In step S7, it is determined whether all the local areas are completed. If not, the process branches to step S62, and if completed, the process branches to step S66.

  In step S66, a known interpolation process is performed on the color difference signal, and then a known synchronization process is performed on the even field signal and the odd field signal to generate a frame signal.

  In step S9, known tone conversion processing, edge enhancement processing, color enhancement processing, and other signal processing are performed.

  In step S67, the processed frame signal is output.

  In step S68, it is determined whether all field signals are completed. If not completed, the process branches to step S60, and if completed, the process ends.

  FIG. 20B is a flow relating to the first noise reduction processing in step S63. Note that the same number of steps is assigned to the same processing steps as the flow of the first noise reduction processing in the first embodiment shown in FIG. 12B.

  In step S70, the local area of the luminance or color difference signal is input.

  In step S71, the absolute value of the difference component from the target pixel value shown in the equation (15) is calculated.

  In step S72, a difference component table constructed based on the function equation shown in equation (16) is input.

  In step S73, a weighting coefficient for the difference component is obtained.

  In step S74, a coordinate table constructed based on the function equation shown in equation (17) is input.

  In step S75, a weighting coefficient relating to coordinates is obtained.

  In step S76, the weighting coefficient used for the bilateral filter is obtained by multiplying the weighting coefficient related to the difference component and the weighting coefficient related to the coordinates.

  In step S77, the bilateral filter process shown in the equation (19) is performed.

  In step S21, information such as temperature and gain is set from the read header information. If a necessary parameter does not exist in the header information, a predetermined standard value is assigned.

  In step S22, the coordinate data and correction coefficient of the reference noise model are read.

  In step S23, the coordinate data of the section of the reference noise model to which the target pixel belongs and the corresponding correction coefficient are selected.

  In step S24, the result of bilateral filter processing is used as a signal level to determine the amount of noise by interpolation processing shown in equations (5) and (6).

  In step S78, filter coefficients as shown in FIG. 16 are input.

  In step S79, a filter coefficient is selected from the amount of noise based on the relationship shown in FIG.

  In step S80, frequency filter processing is performed using the selected filter coefficient.

  In step S26, the signal subjected to the first noise reduction process is output.

  In step S81, it is determined whether the processing of the luminance and color difference signals is completed. If not completed, the process branches to step S70, and if completed, the process ends.

  FIG. 20C is a flow relating to the second noise reduction processing in step S64. Note that the same number of steps is assigned to the same processing steps as the flow of the second noise reduction processing in the first embodiment shown in FIG. 12C.

  In step S30, one of the target pixel and the neighboring eight pixels in the local region of the luminance signal is selected.

  In step S90, the absolute values of the eight differences are calculated as shown in equation (8).

  In step S32, the absolute values of the eight differences are sorted in ascending order.

  In step S33, a sum of absolute values of a predetermined number, for example, four differences is calculated from the smaller one as shown in the equation (9) and used as an index coefficient.

  In step S34, a look-up table for outputting a weighting coefficient is input with an index coefficient constructed based on equation (10) as an input.

  In step S35, a weighting coefficient is output based on the index coefficient.

  In step S36, it is determined whether all of the target pixel and neighboring 8 pixels have been selected. If selection has not been completed, the process branches to step S30. If completed, the process branches to step S91.

  In step S91, the weighting filter processing expressed by the equation (11) is performed on the luminance signal.

  In step S92, the weighting filter processing shown in the equation (11) is performed on the color difference signal.

  In step S38, the signal obtained by the weighting filter process for the luminance signal and the color difference signal is output as a signal subjected to the second noise reduction process.

  In step S39, the index coefficient related to the target pixel of the luminance signal is output and the process ends.

  FIG. 20D is a flow relating to the composition processing in step S65. Note that the same number of steps is assigned to the same processing steps as those in the synthesis processing flow in the first embodiment shown in FIG. 12D.

  In step S40, an index coefficient related to the target pixel of the luminance signal is input.

  In step S41, the index coefficient is compared with a predetermined threshold Th1, and if the index coefficient is equal to or less than the threshold Th1, the process branches to step S42, and if the index coefficient is greater than the threshold Th1, the process branches to step S43.

  In step S42, the signal subjected to the first noise reduction processing is output and the process ends.

  In step S43, the index coefficient is compared with a predetermined threshold Th2. If the index coefficient is equal to or greater than the threshold Th1, the process branches to step S44. If the index coefficient is smaller than the threshold Th2, the process branches to step S45.

  In step S44, the signal subjected to the second noise reduction process is output and the process ends.

  In step S45, a look-up table for outputting a composite coefficient with the index coefficient shown in FIG. 10 as input is input.

  In step S46, a composite coefficient is output based on the index coefficient.

  In step S100, the luminance signal on which the first noise reduction processing has been performed is multiplied by (1-synthesis coefficient).

  In step S101, the luminance signal that has been subjected to the second noise reduction processing is multiplied by a synthesis coefficient.

  In step S102, a signal obtained by multiplying the luminance signal subjected to the first noise reduction process by (1-synthesis coefficient) and a signal obtained by multiplying the luminance signal subjected to the second noise reduction process by the synthesis coefficient are added.

  In step S103, the color difference signal subjected to the first noise reduction processing is multiplied by (1-synthesis coefficient).

  In step S104, the color difference signal subjected to the second noise reduction processing is multiplied by a synthesis coefficient.

  In step S105, a signal obtained by multiplying the color difference signal subjected to the first noise reduction process by (1-synthesis coefficient) and a signal obtained by multiplying the color difference signal subjected to the second noise reduction process by the synthesis coefficient are added.

  In step S50, the synthesized luminance and color difference signals are output and the process ends.

  As described above, the signal processing may be performed by software, and the same operational effects as in the case of processing by hardware are achieved.

  The embodiment of the present invention has been described above, but the above embodiment merely shows an application example of the present invention, and the technical scope of the present invention is not limited to the specific configuration of the above embodiment.

It is a block diagram of 1st Embodiment. It is explanatory drawing regarding arrangement | positioning and a local area | region of a color filter. It is a block diagram of a 1st noise reduction part. It is explanatory drawing regarding estimation of noise amount. It is a block diagram of the 1st noise reduction part of another form. It is a block diagram of a 2nd noise reduction part. It is explanatory drawing regarding index coefficient calculation. It is a block diagram of the 2nd noise reduction part of another form. It is a block diagram of a synthetic | combination part. It is explanatory drawing regarding a synthetic | combination coefficient. It is a block diagram of 1st Embodiment of another form. It is a flow regarding the whole processing among the flow of signal processing of a 1st embodiment. It is the flow regarding the 1st noise reduction process among the flows of signal processing of a 1st embodiment. It is a flow regarding the 2nd noise reduction processing among the flows of signal processing of a 1st embodiment. It is a flow regarding a synthetic | combination process among the flows of the signal processing of 1st Embodiment. It is a block diagram of 2nd Embodiment. It is explanatory drawing regarding arrangement | positioning and a local area | region of a color difference line sequential type complementary color filter. It is a block diagram of a 1st noise reduction part. It is explanatory drawing regarding a frequency filter. It is explanatory drawing regarding selection of a frequency filter. It is a block diagram of a 2nd noise reduction part. It is a block diagram of a synthetic | combination part. It is a flow regarding the whole processing among the flow of signal processing of a 2nd embodiment. It is a flow regarding the 1st noise reduction process among the flows of signal processing of a 2nd embodiment. It is a flow regarding a 2nd noise reduction process among the flows of the signal processing of 2nd Embodiment. It is a flow regarding a synthetic | combination process among the flows of the signal processing of 2nd Embodiment.

Explanation of symbols

111 Color signal separation and extraction unit (local area extraction means)
600 Luminance color separation and extraction unit (local area extraction means)
112,601 1st noise reduction part (1st noise reduction means)
113,602 Second noise reduction unit (second noise reduction means)
114,603 Combining part (combining means)

Claims (26)

In a noise reduction system that performs noise reduction processing on video signals captured from the imaging system,
Local region extraction means for sequentially extracting a local region containing a target pixel for performing noise reduction processing from the video signal;
First noise reduction means for performing random noise reduction processing on the local region;
Second noise reduction means for performing impulsive noise reduction processing on the local region;
A noise reduction system comprising: synthesis means for synthesizing the video signal subjected to noise reduction processing by the first noise reduction means and the video signal subjected to noise reduction processing by the second noise reduction means. The noise reduction system of claim 1,
The first noise reduction means includes
Low frequency extraction means for extracting low frequency components from the local region;
Noise estimation means for estimating a noise amount for the target pixel based on the low frequency component;
A noise reduction system comprising smoothing means for performing a smoothing process on the target pixel based on at least one of the low frequency component and the noise amount. The noise reduction system according to claim 1 or 2,
The second noise reduction means includes
Index coefficient calculating means for calculating an index coefficient indicating the degree of impulsiveness for each pixel in the local region;
Weighting factor calculating means for calculating a weighting factor for each pixel of the local region based on the index coefficient;
A noise reduction system comprising weighting filter means for performing filter processing on the local region based on the weighting factor. The noise reduction system according to claim 1 or 2,
The second noise reduction means includes
Index coefficient calculating means for calculating an index coefficient indicating the degree of impulsiveness with respect to the pixel of interest in the local region;
A noise reduction system comprising nonlinear filter means for performing nonlinear filter processing on the local region based on the index coefficient.
The noise reduction system according to claim 3 or 4,
The synthesis means includes
First selection means for selecting a video signal subjected to noise reduction processing by the first noise reduction means when the index coefficient is equal to or less than a predetermined first threshold;
Second selection means for selecting a video signal subjected to noise reduction processing by the second noise reduction means when the index coefficient is equal to or greater than a predetermined second threshold;
When the index coefficient is larger than the first threshold value and smaller than the second threshold value, the video signal subjected to noise reduction processing by the first noise reduction unit and the noise reduction processing by the second noise reduction unit A noise reduction system comprising weighted addition means for weighted addition of video signals. The noise reduction system according to claim 2,
The low frequency extraction means includes
A noise reduction system comprising average value calculation means for calculating an average value from the local region. The noise reduction system according to claim 2,
The low frequency extraction means includes
A noise reduction system comprising low-pass filter means for applying a low-pass filter to the local region. The noise reduction system according to claim 2,
The low frequency extraction means includes
A noise reduction system comprising bilateral filter means for applying a bilateral filter to the local region. The noise reduction system according to claim 2,
The noise estimation means includes
A collecting means for collecting information on a temperature value of the imaging system and a gain for the video signal;
An assigning means for giving a standard value for information that cannot be obtained by the collecting means;
Parameter recording means for recording a parameter group related to the reference noise model;
Parameter selection means for selecting necessary parameters from the parameter group based on information from the collection means or the giving means and a low frequency component of the local region;
A noise reduction system comprising interpolation means for obtaining a noise amount of the pixel of interest by an interpolation operation based on a low frequency component of the local region and the selected parameter. The noise reduction system according to claim 2,
The noise estimation means includes
A collecting means for collecting information on a temperature value of the imaging system and a gain for the video signal;
An assigning means for giving a standard value for information that cannot be obtained by the collecting means;
A noise reduction system comprising: noise table means for receiving the information from the collecting means or the assigning means and the low frequency component of the local region and outputting the noise amount of the pixel of interest. The noise reduction system according to claim 2,
The smoothing means includes
A noise reduction system comprising coring means for performing coring processing on the target pixel based on a low frequency component of the local region and the amount of noise. The noise reduction system according to claim 2,
The smoothing means includes
Filter recording means for recording a plurality of filters having a predetermined frequency characteristic;
Filter selection means for selecting the filter based on the amount of noise;
A noise reduction system comprising frequency filter means for performing filter processing on the local region using the selected filter. The noise reduction system according to claim 3 or 4,
The index coefficient calculation means includes
Difference means for calculating an absolute value of a difference between a pixel whose index coefficient is to be calculated and a predetermined number of pixels located in the vicinity thereof;
A noise reduction system comprising sum total calculating means for calculating a sum of absolute values of the differences. The noise reduction system according to claim 3 or 4,
The index coefficient calculation means includes
Difference means for calculating an absolute value of a difference between a pixel whose index coefficient is to be calculated and a predetermined number of pixels located in the vicinity thereof;
Sorting means for rearranging the absolute values of the differences in order of magnitude;
A noise reduction system comprising a sum total calculating means for calculating a predetermined number of sums from small values with respect to the absolute values of the sorted differences. The noise reduction system according to claim 3.
The weight coefficient calculating means includes
A noise reduction system comprising weight coefficient table means for outputting a weight coefficient based on the index coefficient. The noise reduction system according to claim 4,
The nonlinear filter means includes
A noise reduction system using median filter processing as the non-linear filter processing. The noise reduction system according to any one of claims 1 to 16,
The imaging system uses an imaging device having a color filter disposed on the front surface,
Color signal separation means for separating the video signal into a plurality of color signals for each color filter used in the image sensor;
A noise reduction system comprising: signal control means for controlling the local area extraction means, the first noise reduction means, the second noise reduction means, and the synthesis means to be sequentially applied to each color signal. . The noise reduction system according to any one of claims 1 to 16,
The imaging system uses an imaging device having a color filter disposed on the front surface,
Luminance color difference separation means for separating a luminance signal and a color difference signal from the video signal;
Signal control means for controlling the local area extraction means, the first noise reduction means, the second noise reduction means, and the synthesis means to be applied sequentially for each of the luminance signal and the color difference signal;
The noise reduction system, wherein the second noise reduction unit and the synthesis unit process the color difference signal based on a processing result of the luminance signal. The noise reduction system according to claim 17 or 18,
The image sensor is
Image sensor with R (red), G (green), B (blue) Bayer type primary color filters arranged in front or Cy (cyan), Mg (magenta), Ye (yellow), G (green) color difference line sequential complementary color A noise reduction system characterized by being an image sensor having a filter disposed in front. An imaging system;
Video signal storage means for capturing a video signal captured by the imaging system;
A noise reduction system according to any one of claims 1 to 19,
An image pickup system, wherein noise reduction processing is performed on the captured video signal by the noise reduction system. In a noise reduction program that performs noise reduction processing on video signals captured from the imaging system,
A local region extraction step for sequentially extracting a local region containing a target pixel for performing noise reduction processing from the video signal;
A first noise reduction step for performing random noise reduction processing on the local region;
A second noise reduction step of performing impulsive noise reduction processing on the local region;
A noise reduction program comprising: a synthesis step of synthesizing the video signal subjected to noise reduction processing in the first noise reduction step and the video signal subjected to noise reduction processing in the second noise reduction step. The noise reduction program according to claim 21,
The first noise reduction step includes:
A low frequency extraction step for extracting a low frequency component from the local region;
A noise estimation step of estimating a noise amount for the target pixel based on the low frequency component;
A noise reduction program comprising a smoothing step of performing a smoothing process on the target pixel based on at least one of the low-frequency component and the noise amount. The noise reduction program according to claim 21 or 22,
The second noise reduction step includes:
An index coefficient calculating step of calculating an index coefficient indicating the degree of impulsiveness for each pixel in the local region;
A weighting factor calculating step for calculating a weighting factor for each pixel in the local region based on the index factor;
A noise reduction program comprising a weighting filter step for performing a filtering process on the local region based on the weighting factor. The noise reduction program according to claim 21 or 22,
The second noise reduction step includes:
An index coefficient calculating step for calculating an index coefficient indicating a degree of impulsiveness with respect to the pixel of interest in the local region;
A noise reduction program comprising a non-linear filter step for performing non-linear filter processing on the local region based on the index coefficient. In the noise reduction program according to claim 23 or 24,
The index coefficient calculating step includes:
A difference step for calculating an absolute value of a difference between a pixel whose index coefficient is to be calculated and a predetermined number of pixels located in the vicinity thereof;
A sorting step for rearranging the absolute values of the differences in order of magnitude;
A noise reduction program, comprising: a sum total calculating step of calculating a predetermined number of sums from small values with respect to the absolute values of the sorted differences. In the noise reduction program according to claim 23 or 24,
The synthesis step includes
A first selection step of selecting a video signal subjected to noise reduction processing in the first noise reduction step when the index coefficient is equal to or less than a predetermined first threshold;
A second selection step of selecting a video signal subjected to noise reduction processing in the second noise reduction step when the index coefficient is equal to or greater than a predetermined second threshold;
When the index coefficient is larger than the first threshold and smaller than the second threshold, the video signal subjected to the noise reduction processing in the first noise reduction step and the noise reduction processing in the second noise reduction step A noise reduction program comprising a weighted addition step for weighted addition of video signals.

Images