KR100767634B1 - Apparatus and method for removing flicker noise - Google Patents

Abstract

The present invention relates to an apparatus and method for eliminating flicker noise, and extracting resampled luminance data from an input Bayer pattern image, measuring a current frame rate, and dividing the frame to a current value without dividing it into 100 Hz and 120 Hz. Adjust the speed and extract frames C100 and C120 where flicker noise in the opposite phase appears. Then, the average of the first frame and (C100 + 1) frame sample is obtained and added or subtracted from the sample of the (C100 + 1) frame, and the average of the first frame and the (C120 + 1) frame sample is obtained to determine the (C120 + 1) frame. Each sample is obtained by adding or subtracting data to obtain data containing only flicker noise, and normalizing the data including only flicker noise, and obtaining 100 Hz flicker components and 120 Hz flicker components from the samples, wherein flicker components larger than a predetermined threshold value are obtained. If a frequency exists, the frequency can be determined as a flicker noise frequency and used to change the exposure time. According to the present invention, by extracting data including only flicker noise components using an image when flicker noise of an opposite phase is generated, an accurate flicker noise frequency may be measured and removed.

Flicker Noise, Reverse Phase, Exposure Time, Flicker Noise Frequency

Description

Flicker noise canceller and its method {APPARATUS AND METHOD FOR REMOVING FLICKER NOISE}

1 is a structural diagram of an embodiment of a flicker noise removing device according to the present invention;

2 is a detailed structural diagram of an embodiment of the preprocessor of FIG. 1;

3 is an exemplary view illustrating a Bayer pattern image input to the preprocessor of FIG. 1;

4 is a schematic diagram illustrating bilinear interpolation of the resampling performer of FIG. 2;

5 is a detailed structural diagram of an embodiment of the flicker noise detector of FIG. 1;

6 is a flowchart illustrating an embodiment of a method for removing flicker noise according to the present invention.

<Description of the symbols for the main parts of the drawings>

110: preprocessor 120: flicker noise detector

130: exposure time control unit 210: luminance component extractor

220: average calculator 230: resampling performer

510: frame rate adjuster 520: frame extractor

530: flicker noise detector 540: normalizer

The present invention relates to a flicker noise canceling apparatus and a method thereof, and more particularly, to a flicker noise canceling apparatus and method for use in a complementary metal oxide semiconductor (hereinafter referred to as "CMOS") image sensor, etc. will be.

In general, when photographing using a CMOS image sensor, since exposure timings differ from line to line, and a difference in exposure amount occurs in each line, striped flicker noise appears on the screen. For example, if one frame is 1/25 second, and the period of the luminance change of the fluorescent lamp is 1/120 second, flicker noise is not present at the slow time when the exposure time is n / 120 seconds (where n is a positive integer). Although not generated, flicker noises having continuity on the time axis are generated at other exposure times.

When the frame period is changed to 1/30 second only under the above conditions, when the exposure time is n / 120 seconds, flicker noise does not occur as in the above, but at other exposure times, the flicker noise streaks do not appear in each frame. Equivalent, that is to say, complete flicker noise in one frame. In conclusion, if you can detect the rate of change in brightness of a fluorescent lamp (f _p ) during flicker noise, you can eliminate the flicker noise by setting the exposure time to n / f _p (n is a positive integer). have.

As described above, various techniques for removing flicker noise have been proposed and disclosed in Korean Patent Laid-Open Publication No. 2004-35234 (Pixel Array of Image Sensor and Image Sensor and Image Flicker Noise Removal Method Including The Same) and Korea Publication No. 2005-6045 (imaging device and flicker reduction method) is a representative example.

Korean Patent Laid-Open Publication No. 2004-35234 includes unit pixels of N × M (N and M are natural numbers) and is arranged in the row direction of the pixel group to detect flicker noise and a pixel group for detecting an image signal. And a pixel column for flicker noise detection for detecting an average frequency of the pixel rows.

However, since the Patent Publication No. 2004-35234 uses only pixel value information in a single image, there is a problem in that it is likely to be affected by existing frequency components in the input image. For example, if there is a significant amount of horizontal stripe data of 100 (or 120) Hz component in the input image and not flicker noise, it is easy to mistake it for flicker noise.

The above-mentioned Patent Publication No. 2005-6045 discloses a combination of a power supply frequency and a vertical synchronization frequency of an imaging device without detecting a frequency of a power supply for driving a fluorescent lamp in an XY address scanning type imaging device such as a CMOS imaging device. Irrespective of, it is for reducing a fluorescent lamp flicker component.

However, the Patent Publication No. 2005-6045 has a limitation of a fixed frame rate, and accordingly, it requires a sufficient memory space for storing image data of a predetermined number of frames, and computation has a lot of problems. .

The present invention has been proposed to solve the above problems, and generates data in which only flicker noise components exist by using an image when flicker noise of an opposite phase occurs based on the phase of the flicker noise present in the current image. Accordingly, an object of the present invention is to provide a flicker noise canceling apparatus and method for detecting and removing an accurate flicker noise frequency.

In order to achieve the above object, according to a preferred embodiment of the present invention, the flicker noise removing device, Pre-processing unit for extracting the resampled luminance data from the Bayer pattern image input; A flicker noise extractor for extracting data having flicker noise by using a frame in which flicker noise of an opposite phase appears; And an exposure time controller for determining a flicker noise frequency from the flicker noise data obtained by the flicker noise extractor and controlling the exposure time therefrom.

The preprocessing unit may include: a luminance component extractor for extracting a luminance component from an input Bayer pattern image; An average value calculator for obtaining an average value in the horizontal direction of the luminance image obtained from the luminance component extractor; And a resampling performer for acquiring a sample at intervals of sampling intervals in a vertical direction among the average values in the horizontal direction obtained from the average value calculator, wherein the resampling performer is a bilinear interpolation when the sampling interval is a real number. Samples can be obtained by interpolation through.

The flicker noise detector may further include: a frame rate adjuster for measuring a current frame rate and adjusting a frame rate to be close to a current value without being divided into 100 Hz and 120 Hz; A frame extractor for extracting frames C100 and C120, respectively, in which flicker noise of an opposite phase appears; Obtain the average of the first frame and (C100 + 1) frame sample and subtract it from the sample of the (C100 + 1) frame, and average the first frame and the (C120 + 1) frame sample to find the sample of the (C120 + 1) frame. A flicker noise detector for acquiring and subtracting from each other to obtain data in which only flicker noise exists; And a normalizer for normalizing data received from the flicker noise detector, wherein the frame extractor multiplies '100 / frame rate' and '120 / frame rate' by any positive integer to reduce flicker noise of opposite phase. The frames C100 and C120 that appear may be extracted.

및

(이 때, n은 0 내지 (샘플수-1)의 정수임)로부터 구한 계수를 각각 이용하여 100Hz 플리커 성분과 120Hz 플리커 성분을 구하고, 그 중 소정의 임계치보다 큰 플리커 성분이 있는 주파수가 존재하면, 해당 주파수를 플리커 잡음 주파수로 결정하고 이를 이용하여 노광 시간을 변경할 수 있다.On the other hand, the exposure time control unit,

And

(Where n is an integer from 0 to (sample number-1)) to obtain a 100 Hz flicker component and a 120 Hz flicker component using the coefficients obtained from each, and if there is a frequency having a flicker component larger than a predetermined threshold, The frequency can be determined as the flicker noise frequency and used to change the exposure time.

On the other hand, according to another embodiment of the present invention, a flicker noise removing method, the method comprising the steps of: extracting the resampled luminance data from the input Bayer pattern image (a); Measuring the current frame rate and adjusting the frame rate to be close to the present value without dividing into 100 Hz and 120 Hz; (C) extracting frames C100 and C120 in which flicker noise of opposite phases appears; Obtain the average of the first frame and (C100 + 1) frame sample and subtract it from the sample of the (C100 + 1) frame, and average the first frame and the (C120 + 1) frame sample to find the sample of the (C120 + 1) frame. Subtracting at and obtaining data each having only flicker noise; (E) normalizing data in which only flicker noise is present; (F) obtaining a 00Hz flicker component and a 120Hz flicker component, and if there is a frequency having a flicker component larger than a predetermined threshold, determining the frequency as the flicker noise frequency and using the same to change the exposure time; A flicker noise canceling method is provided, and the method may further include a step (g) of repeatedly performing the steps (b) to (f).

및

(이때, n은 0 내지 (샘플수-1)의 정수임)로부터 추출한 계수를 각각 이용하여 구할 수 있다.In this case, step (c) may extract the frames C100 and C120 in which the flicker noise of the opposite phase appears by multiplying '100 / frame rate' and '120 / frame rate' by a positive integer. The 100Hz flicker component and 120Hz flicker component of step (f)

And

(N is an integer of 0 to (sample number-1)), and can be obtained using the coefficients extracted from each.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same components have the same number as much as possible even if displayed on different drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a structural diagram of an embodiment of an apparatus for removing flicker noise according to the present invention.

As shown in the figure, the removal apparatus of the present invention includes a preprocessor 110, a flicker noise detector 120 and an exposure time controller 130.

The preprocessor 110 is responsible for extracting the resampled luminance data. This will be described in detail with reference to FIG. 2.

FIG. 2 is a detailed structural diagram of an embodiment of the preprocessor of FIG. 1.

As shown in the figure, the preprocessor 110 includes a luminance component extractor 210, an average value calculator 220, and a resampling performer 230.

The luminance component extractor 210 is responsible for extracting a luminance component from an input bayer pattern image. The Bayer pattern image is as shown in FIG. 3. FIG. 3 is an exemplary view illustrating a Bayer pattern image input to the preprocessor of FIG. 1.

The equation for obtaining the luminance Y in the Bayer pattern image of FIG. 3 is as follows.

 [Equation 1]

That is, the luminance Y may be obtained using two green components G and one blue component B and a red component R in the rectangular area indicated by red at the upper left of FIG. 3.

The average value calculator 220 is responsible for calculating an average value in the horizontal direction of the luminance image obtained by the luminance component extractor 210. The resampling performer 230 is responsible for acquiring samples at intervals of sampling periods in the vertical direction among the average values in the horizontal direction. At this time, the equation for obtaining the sampling interval is as follows.

 [Equation 2]

Where fr is the frame rate and Image - height is the height of the image. For example, fr is 25, Image - height is 240, re - sampling When the rate is 480 Hz and the number of samples is 24, the sampling interval is 12.5, and 24 samples can be acquired at 12.5 intervals in the vertical direction. In this case, when the sampling interval is a real number, the sample may be acquired after interpolation through bilinear inperpolation. Bilinear interpolation will be described with reference to FIG. 4.

FIG. 4 is a schematic diagram illustrating bilinear interpolation of the resampling performer of FIG. 2.

As shown in the figure, a pixel of a real position may be obtained by referring to positions of pixels (i-th pixel and (i + 1) -th pixel of adjacent integer positions. .

 [Equation 3]

Through this process, the preprocessor 110 may extract the resampled luminance data.

Subsequently, the flicker noise detector 120 of FIG. 1 is responsible for detecting data in which flicker noise exists from the luminance data obtained by the preprocessor 110. This will be described with reference to FIG. 5.

5 is a detailed structural diagram of an embodiment of the flicker noise detector of FIG. 1.

As shown in the figure, the flicker noise detector 120 of FIG. 1 includes a frame rate adjuster 510, a frame extractor 520, a flicker noise detector 530, and a normalizer 540.

The frame rate adjuster 510 measures the current frame rate fr and adjusts the frame rate close to the current value without dividing 100 and 120 Hz. As an example of the frame rate to be used, the measured frame rate can be adjusted to 7.4 when the measured frame rate is 5 to 10, and can be adjusted to 11.4 when the measured frame rate is 10 to 15. In addition, when the measured frame rate is 15 to 20, it may be adjusted to 16, and when the measured frame rate is 20 to 25, it may be adjusted to 22. However, this is a value adjusted by the experimental value is not limited thereto.

The frame extractor 520 is responsible for extracting a frame in which flicker noise of an opposite phase appears. In other words, the frame extractor 520 multiplies 100 / fr by any positive integer, which is a constant when it becomes close to 'xx.5' (i.e., a frame in which flicker in the opposite phase appears) (called 'C100'). Find it. We can also find the constant (referred to as 'C120') when it approaches xxx (5) by multiplying 120 / fr by any positive integer.

For example, when C100 and C120 are obtained with respect to the frame rate obtained by the frame rate adjuster 510, it is as follows. That is, when fr is 7.4, C100 may be 1 and C120 may be 2, and when fr is 11.4, C100 may be 2 and C120 may be 1. In addition, when fr is 16, C100 may be 2 and C120 may be 1, and when fr is 22, C100 may be 1 and C120 may be 1.

Thereafter, the frame extractor 520 obtains C _max = max (C100, C120), which is a larger number among C100 and C120, and receives and stores sample values from the preprocessor 110 for C _max +1 frames. In charge. In this case, C _max +1 may be limited not to exceed the number of frames corresponding to 0.5 seconds.

The flicker noise detector 530 calculates an average of the samples of the first frame and the (C100 + 1) frame, and then subtracts the value obtained from the sample of (C100 + 1) to obtain data having only flicker noise. In charge. In addition, after obtaining the average of the samples of the first frame and (C120 + 1) frame, and subtracted the value obtained from the sample of (C120 + 1) again, it is responsible for acquiring data having only flicker noise.

The normalizer 540 is responsible for normalizing data in which only flicker noise components corresponding to 100 and 120 Hz received from the flicker noise detector 530 are present. In this case, a zero mean or unit variance method may be used.

In this manner, the flicker noise detector 120 of FIG. 1 may detect data in which only flicker noise components exist.

The exposure time controller 130 is responsible for controlling the exposure time by determining a flicker noise frequency.

The exposure time controller 130 is responsible for obtaining a flicker component of 100 Hz and a flicker component of 120 Hz, respectively, by using the data normalized by the normalizer 540.

That is, the data normalized by the normalizer 540 is multiplied by 100 Hz and 120 Hz coefficients, respectively, real numbers are added to real numbers, imaginary numbers are added to imaginary numbers, and then the squares of the real part and the imaginary part are added and the number of samples divide by number). At this time, the equation for obtaining the 100Hz and 120Hz coefficients are as shown in Equations 4 and 5, respectively.

 [Equation 4]

 [Equation 5]

At this time, n is an integer of 0 to (number of samples-1), and the coefficient when the resampling ratio is 480 Hz and the number of samples 24 is as follows.

At 100 Hz, the coefficients of the real part are [1, 0.26, -0.87, -0.71, 0.5, 0.97, 0, -0.97, -0.5, 0.71, 0.87, -0.26, -1, -0.26, 0.87, 0.71, -0.5 , -0.97, 0, 0.97, 0.5, -0.71, -0.87, 0.26], and the imaginary part coefficients are [0, -0.97, -0.5, 0.71, 0.87, -0.26, -1, -0.26, 0.87, 0.71, -0.5, -0.97, 0, 0.97, 0.5, -0.71, -0.87, 0.26, 1, 0.26, -0.87, -0.71, 0.5, 0.97]. In addition, at 120 Hz, the coefficient of the real part is [1, 0, -1, 0, 1, 0, -1, 0, 1, 0, -1, 0, 1, 0, -1, 0, 1, 0. , -1, 0, 1, 0, -1, 0], the imaginary part's coefficient is [0, -1, 0, 1, 0, -1, 0, 1, 0, -1, 0, 1, 0 , -1, 0, 1, 0, -1, 0, 1, 0, -1, 0, 1].

If any of the 100 Hz flicker components and 120 Hz flicker components obtained above are larger than a threshold (in the present invention, for example, 5), the frequency at which the flicker components appear is determined as the flicker noise frequency, and then removed. In order to control the exposure time. If the flicker noise frequency is f _p , as described above, the exposure time t is expressed by the following equation.

 [Equation 6]

N is a positive integer.

If neither component is larger than the threshold in both cases, the exposure time controller 130 may assume that there is no flicker noise and maintain the current exposure time.

The flicker noise canceling apparatus of the present invention can repeat this process periodically to determine the flicker noise frequency and to control the exposure time.

6 is a flowchart illustrating an exemplary method for removing flicker noise according to the present invention.

As shown in the figure, the flicker noise removing method of the present invention is responsible for extracting the resampled luminance data from the input Bayer pattern image (S600). In this case, as described above, the Bayer pattern image is as shown in FIG. 2. The luminance Y may be obtained from the input Bayer pattern image, the average value in the horizontal direction of the luminance image thus obtained may be obtained, samples may be acquired at sampling intervals in the vertical direction, and the resampled luminance data may be extracted. The equation for obtaining the sampling interval is as shown in Equation 2 above. If the sampling interval is a real number, the sample may be acquired after interpolation through bilinear interpolation.

Thereafter, the current frame rate fr is measured to adjust the frame rate close to the current value without dividing the frequency (100 or 120 Hz) at which flicker noise occurs (S602). Examples of fr to be used are 7.4 when the measured fr is 5 to 10, 11.4 when the measured fr is 10 to 15, 16 when the measured fr is 15 to 20, and the measured fr is 20 In the case of ˜25, it can be adjusted to 22 as described above.

Using the thus obtained fr , a frame in which flicker noise of the opposite phase appears can be extracted (S604). That is, 100 / fr is multiplied by any positive integer to find a constant (ie, a frame in which flicker in the opposite phase appears) (close to 'C100') when it comes close to 'xx.5', and also 120 / fr . Multiply fr by any positive integer to extract the constant (called 'C120') when it comes close to 'xx.5'.

C _max = max (C100, C120), which is a larger number of C100 and C120, is obtained, and the sample values can be received from the preprocessor 110 for C _max +1 frames.

Then, after averaging the samples of the first frame and the (C100 + 1) frame, subtracting the value obtained from the sample of (C100 + 1), the data having only flicker noise is obtained, and the first frame and the (C120 + 1) After the average of the samples of the frame is obtained, the data obtained by subtracting the value obtained from the sample of (C120 + 1) again to obtain data having only flicker noise (S606). The data in which only such flicker noise components exist is normalized (S608). In this case, a zero mean or unit variance method may be used.

Then, the normalized data is multiplied by the coefficients of 100 Hz and 120 Hz, respectively, real numbers are added to real numbers, imaginary numbers are added to each other, then the squares of the real part and the imaginary part are added and divided by the number of samples to obtain a 100 Hz flicker component and a 120 Hz flicker component (S610). If there is a case larger than the threshold value (S612), the frequency at which the flicker component appears at that time is determined as the flicker noise frequency, and the exposure time is changed to remove it (S614). The exposure time can be determined by the above equation (6).

If none of the flicker components is larger than the threshold (S612), the flicker noise may be regarded as no flicker noise and the current exposure time may be maintained as it is (S616).

The flicker noise canceling apparatus of the present invention may determine flicker noise frequency and control the exposure time by repeatedly performing steps S602 to S616.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes can be made in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

As described above, the present invention extracts data containing only the flicker noise component by using the image when the flicker noise of the opposite phase is generated in order to exclude the natural frequency component of the input image. Compared to this, the flicker noise frequency can be measured and eliminated.

In addition, the present invention, by setting a specific frame rate that requires a small number of frames in advance for each section even when the frame rate is not fixed to one, it is possible to detect flicker noise without change even in a variable frame rate There is.

Claims (10)

A flicker noise canceller, A preprocessor extracting the resampled luminance data from the input Bayer pattern image; A flicker noise detector which extracts data having flicker noise from the resampled luminance data using a frame in which flicker noise of an opposite phase appears; And And an exposure time control unit configured to determine a flicker noise frequency from data in which there is flicker noise obtained by the flicker noise detector, and to control the exposure time using the flicker noise detection device. The method of claim 1, The preprocessing unit, A luminance component extractor for extracting a luminance component from an input Bayer pattern image; An average value calculator for obtaining an average value in the horizontal direction of the luminance image obtained from the luminance component extractor; And And a resampling performer for acquiring samples at intervals of sampling intervals in the vertical direction from among the average values in the horizontal direction obtained from the average calculator. The method of claim 2, The resampling performer, And if the sampling interval is a real number, a flicker noise canceling device characterized in that interpolation is obtained through bilinear interpolation. The method of claim 1, The flicker noise detector, A frame rate adjuster for measuring a current frame rate and adjusting the frame rate to be close to the present value without being divided into 100 Hz and 120 Hz; A frame extractor for extracting frames C100 and C120, respectively, in which flicker noise of an opposite phase appears; Obtain the average of the first frame and (C100 + 1) frame sample and subtract it from the sample of the (C100 + 1) frame, and average the first frame and the (C120 + 1) frame sample to find the sample of the (C120 + 1) frame. A flicker noise detector for obtaining data in which only flicker noise is present; And And a normalizer for normalizing the data received from the flicker noise detector. The method of claim 4, wherein The frame extractor, In order to extract the frames C100 and C120 in which the flicker noise of the opposite phase appears, respectively, a positive integer whose product of '100 / frame rate' is close to a predetermined value is determined as C100, and is determined as '120 / frame rate'. A positive integer whose product is close to a predetermined value is determined as C120, When the predetermined value is a value including a decimal point, it is determined to be a positive integer that is close to the base value of the decimal point or less, and the positive integer determined by the C100 and the positive integer determined by the C120 is matched or inconsistent. Flicker noise canceller. The method of claim 1, The exposure time control unit, The 100 Hz flicker component and the 120 Hz flicker component are obtained by using the coefficients obtained from the following equations, and if a frequency having the flicker component larger than a predetermined threshold exists among them, the frequency is determined as the flicker noise frequency, and the exposure is used. Flicker noise canceller, characterized in that changing the time.

Where n is an integer from 0 to (sample-1) In the flicker noise cancellation method, Extracting luminance data resampled from the input Bayer pattern image (a); Measuring the current frame rate and adjusting the frame rate to be close to the present value without dividing into 100 Hz and 120 Hz; (C) extracting frames C100 and C120 in which flicker noise of opposite phases appears; Obtain the average of the first frame and (C100 + 1) frame sample and subtract it from the sample of the (C100 + 1) frame, and average the first frame and the (C120 + 1) frame sample to find the sample of the (C120 + 1) frame. Subtracting at and obtaining data each having only flicker noise; (E) normalizing data in which only flicker noise is present; (F) determining a 100 Hz flicker component and a 120 Hz flicker component, and if there is a frequency having a flicker component larger than a predetermined threshold, determining the frequency as the flicker noise frequency and using the same to change the exposure time; Flicker noise removal method comprising a. The method of claim 7, wherein And (g) repeatedly performing the steps (a) to (f). The method according to claim 7 or 8, Step (c) is, In order to extract the frames C100 and C120 in which the flicker noise of the opposite phase appears, respectively, a positive integer whose product of '100 / frame rate' is close to a predetermined value is determined as C100, and is determined as '120 / frame rate'. A positive integer whose product is close to a predetermined value is determined as C120, When the predetermined value is a value including a decimal point, it is determined to be a positive integer that is close to the base value of the decimal point or less, and the positive integer determined by the C100 and the positive integer determined by the C120 is matched or inconsistent. Characterized by flicker noise cancellation. The method according to claim 7 or 8, 100 Hz flicker component and 120 Hz flicker component of step (f), A flicker noise canceling method characterized by obtaining each of the coefficients extracted from the following equation.

Where n is an integer from 0 to (sample-1)

Images