JP2012235442A - Technique for reducing noise for machine vision system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for preferably reducing noise of an image captured by an imaging device.SOLUTION: A method for reducing noise of an image captured by an imaging device includes: a first step of acquiring an image from the imaging device; a second step of reducing offset components of fixed pattern noise from the image; a third step of reducing gain components of the fixed pattern noise from the image; and a fourth step of reducing residual components of the fixed pattern noise from the image using a filter which reduces impulsive noise included in the image after the second step and the third step.

Description

  The present invention relates to a technique for reducing noise in an image captured by a camera.

  A digital camera captures an image using a CCD sensor or a CMOS sensor. A CCD sensor or a CMOS sensor has a two-dimensional array of sensor elements. CCD or CMOS elements convert light into electrons (photoelectric conversion). The electrons are then converted into (digital) signals. Each such element corresponds to a pixel of an image or frame imaged by the camera. Here, the pixel data or image output from the sensor or camera always includes undesirable noise in addition to the target signal. There are various types of noise originating from sensors or cameras. For example, fixed pattern noise, readout noise, and shot noise. In order to accurately identify the true appearance of the image content, noise must be suppressed or eliminated.

  The dark current is a cause of fixed pattern noise (FPN). The dark current is mainly derived from a false charge accumulated in each element of the CCD sensor or the CMOS sensor even in the absence of light. Different elements accumulate different amounts of false charge. Therefore, fixed pattern noise results in a spatially nonuniform (unevenness) pattern in the captured image. However, the spatial distribution is substantially unchanged in time. Therefore, some of the dark current noise can be classified as fixed pattern noise. This type of fixed pattern noise can be referred to as dark signal non-uniformity (DSNU). In addition to such DSNU noise, there is also dark current shot noise. The dark current shot noise is a spatially and temporally random noise added to the dark current noise. The dark current noise is temperature dependent. This makes it somewhat difficult to predict what can happen when the temperature changes. In general, dark current noise increases with increasing temperature.

  Further, shot noise is basically related to a path through which photons reach the detection element. Light shot noise is a spatially and temporally random event. The dispersion of light shot noise generally depends on the signal intensity. Also, readout noise is generally independent of signal strength and can be considered to encompass other random noise sources similar to additional white Gaussian noise. Such random noise sources include amplifier noise, analog-to-digital conversion (ADC) noise, and some other types of noise.

JP 2006-020354 A (published January 19, 2006) JP2011-159178A (released on August 18, 2011)

  As described above, it is desirable to reduce various types of noise originating from sensors and cameras, such as fixed pattern noise, readout noise, and / or shot noise, in order to accurately identify the true appearance of the image. .

  The above objects, features and advantages of the present invention, as well as other objects, features and advantages of the present invention will be more readily understood with reference to the detailed description of the invention and the drawings.

  In order to solve the above problem, a method for reducing noise in an image captured by an imaging apparatus according to the present invention includes a first step of acquiring the image from the imaging apparatus, and an offset component of fixed pattern noise from the image. A second step of reducing the gain component of the fixed pattern noise from the image, and impulsive noise included in the image after the second step and the third step. And a fourth step of reducing a residual component of the fixed pattern noise from the image by using a filter.

  Even after the offset component and the gain component of the fixed pattern noise are removed from the captured image, an impulsive residual component may remain. According to the above configuration, since the remaining component can be removed, the fixed pattern noise of the captured image can be more preferably reduced.

  In the above method, it is preferable that the filter ranks each pixel in the window based on a pixel value in the window set in the image. The window is preferably a 3 × 3 pixel window. The ranking is preferably ranking of pixel values in the window. Further, it is preferable that the filter sets the pixel value of the center pixel of the window as a modification target. The condition for modifying the pixel value of the central pixel preferably includes that the pixel value of the central pixel is the minimum value or the maximum value of each pixel in the window. Further, the condition for modifying the pixel value of the central pixel preferably includes a condition based on a difference between the pixels in the window. In the above method, the condition for modifying the pixel value of the central pixel is that the difference between the pixel value of the central pixel and the pixel value of each pixel other than the central pixel in the window is specified in advance. It is preferable that it is more than the set threshold value. As a result, the pixel value of the central pixel can be modified when the pixel value of the central pixel differs greatly from the other pixels. The condition for modifying the pixel value of the central pixel preferably includes that the difference between the pixel values of the pixels other than the central pixel in the window is smaller than a predetermined threshold value. Thereby, the pixel value of the center pixel can be modified when the pixel values other than the center pixel are sufficiently close to each other. Moreover, it is preferable that the filter performs image processing based on the following formula.

_{(However, f 1 to 9} represents each pixel value corresponding to each position in the window _{(f 5} represents a _{central), r 1 to 9} were sorted on the basis of the pixel value _{(r 1} Indicates a minimum value, r ₉ indicates a maximum value), T _RD and T _RV indicate pre-specified threshold values, otherwise indicates a case where the above two expressions do not hold, and a left arrow Indicates assignment.)
According to said structure, the said filter can be implement | achieved suitably. In particular, the residual component of the fixed pattern noise can be suitably reduced without removing image features such as defective picture elements of an LCD panel.

  ADVANTAGE OF THE INVENTION According to this invention, the noise originating in a sensor and a camera can be reduced suitably.

It is a figure explaining the schematic structure of the system for analysis of noise characteristics and noise reduction. It is a figure explaining the sub stage of noise reduction. It is a figure explaining the measurement of the gain component of fixed pattern noise. It is a figure explaining the measurement of the offset component of fixed pattern noise. It is a figure explaining estimation of the parameter of random noise. It is a figure explaining the noise reduction in an "online" stage. It is a figure explaining the filter of the residual component of fixed pattern noise. It is a figure explaining the other filter of the residual component of fixed pattern noise. It is a figure explaining the filter of M-AND noise. It is a figure which shows an example of a computer system.

  Hereinafter, a case where the noise reduction technique of the present invention is applied to image processing of a captured image for inspection of defective picture elements of an LCD panel will be described. Since the present invention can distinguish a signal to be detected, such as a characteristic portion of an image caused by a defective picture element, from noise, it can be suitably applied to such a use. However, it goes without saying that the present invention can be applied to general image processing for removing noise from a captured image.

  FIG. 1 is a diagram illustrating a schematic configuration of a system for noise characteristic analysis and noise reduction (hereinafter also referred to as the present system) according to an embodiment of the present invention. The system includes an “offline” stage and an “online” stage. The “offline” stage consists of noise measurement and characterization using the same camera (or similar camera) used in the “online” (production) stage. The “offline” stage is typically performed irregularly (or only once) for each specific camera and / or specific camera placement and / or specific camera setting. As such, the “offline” stage need not be executed under real-time constraints. In the “offline” stage, the test object (test target) is placed at a location where it is imaged using a camera. A series of images are then captured, processed and analyzed to extract the relevant noise characteristics. The obtained noise characteristics are stored in an appropriate recording medium (for example, a hard disk of a computer). The recording medium may be connected to or part of a host computer system that is separate from the camera itself.

  The “online” stage refers to the actual use of cameras and image processing systems in a production environment. In the noise reduction stage, an image of the analysis target is processed, and noise in each image is reduced before the image is subjected to an actual analysis stage (for example, an inspection stage). In the “online” noise reduction stage, the noise characteristics stored in the recording medium are repeatedly used. The “online” stage is typically subject to real-time constraints on operation. Therefore, an image processing stage for noise reduction typically needs to be executed within a limited processing time. This stage can be realized by an appropriate combination of hardware and software. For example, it can be realized by a personal computer running appropriate image processing software for noise reduction.

  FIG. 2 is a diagram for explaining an outline of a substage for noise reduction. The substage includes a substage that reduces the offset component of fixed pattern noise (offset fixed-pattern noise), and the gain component of fixed pattern noise (gain fixed-pattern noise). A sub-stage is included to reduce. These two substages may be separate stages, or may be combined into a single substage. These substages can substantially reduce fixed pattern noise, but unfortunately some noise remains. For example, fixed pattern noise may remain after performing these initial stages. This remaining fixed pattern noise is generally referred to herein as a residual component of fixed pattern noise (remnant fixed-pattern noise). In some cases, the residual component of this fixed pattern noise may be due to dark current shot noise. This residual component of fixed pattern noise can also be attributed to differences in operating temperature when compared to measurements for fixed pattern noise calibration. In some implementations, it is difficult to adequately manage the operating temperature, and even measuring the operating temperature can be difficult. Therefore, the fixed pattern noise may be compensated excessively or insufficiently. It is therefore useful to provide additional processing to remove the residual component of fixed pattern noise. Additional substages can be used to remove or reduce residual components of fixed pattern noise. Any appropriate filter can be applied to the residual component of the fixed pattern noise. For example, an extended rank-conditioned median filter can be used. In the last substage, read noise and shot noise are suppressed. Any appropriate filter can be applied to read noise and shot noise. For example, an extended adaptive noise smoothing filter can be used.

  A mathematical model of noise is described below. In this model, each image can be considered as a two-dimensional (2D) array with a size of M × N pixels. The position of the pixel in the two-dimensional image is indicated by the notation (i, j). When the system handles a series of images, the pixel value at each position in the kth image can be indicated by the notation (i, j, k). The model for images and noise is as follows.

Where f (i, j, k) represents the “true” image (“desired signal”) that the system is to measure; g (i, j, k) is the “real” image captured by the camera. D _FPN (i, j) indicates the non-uniform offset / bias component of the fixed pattern noise due to DSNU (Dark Signal Non-Uniformity). c _FPN (i, j) indicates a non-uniform gain component of fixed pattern noise caused by PRNU (Pixel Responsivity Non-Uniformity); n _T (i, j, k) is , Random noise in time, including read noise (n _READ (i, j, k)) and shot noise (n _SHOT (i, j, k)).

  The readout noise mainly consists of a signal-independent noise source. The average value of the readout noise is approximately 0, and the characteristics are mainly indicated by the dispersion.

  The variance of the readout noise is assumed to be a constant. For example, it is the same in each pixel of all images.

  Shot noise is signal-dependent, and is characterized by dispersion depending on the signal value of each pixel.

  Here, α represents the total gain (digital number (DN) / number of electrons) in the pixel.

  The variances of the two noise sources are summed to give a total variance of temporal random noise.

  Note that the non-uniform offset component and gain component appear randomly in space, but are substantially fixed in time, and are therefore substantially the same in all images.

("Offline" stage)
FIG. 3 is a diagram for explaining the measurement of the gain component (C _FPN ) of fixed pattern noise. The characteristic of the non-uniform gain component of the fixed pattern noise can be indicated by a two-dimensional array (M × N pixels) c _FPN (i, j). For a camera with a perfectly uniform response, c _FPN (i, j) = 1 for all pixels. Deviation from value 1 represents non-uniformity in the gain component. This gain component can be measured. The system uses a spatially uniform light source, directs the light source at the camera, and can image multiple uniform fields (“flat fields”). Such a uniform light source can be supplied by, for example, an integrating sphere. Here, let K1 be the number of images captured by the camera. K1 should be a sufficiently large number (eg, K1> 9). For each image, the camera setting parameters and imaging conditions should be the same (or generally the same) as the parameters and conditions used in the “online” (actual operation) stage. At this time, since a flat field is presented to the camera, the actual signal should be a steady value between images.

  Also, the images should be the same except for temporal noise between K1 captured images (also referred to as “flat light” images). Here, if the system averages K1 captured images, random temporal noise will be suppressed. Therefore, in consideration of the equations (1) and (6), this averaged image (also referred to as “average flat light image”) is defined by the following equation.

  This averaged image includes a gain component of fixed pattern noise, and also includes an offset component. In order to remove and / or reduce the offset component of fixed pattern noise, another set of images (“flat” with the same conditions except that no light is supplied to the camera / sensor by the same camera. (Also referred to as “dark” images). The exposure time used to capture this series of “dark” images is the same (or substantially the same) as the exposure time used when imaging a flat field under light. Here, let K2 be the number of such “dark” images taken. K2 should be a sufficiently large number (eg, K2> 9). Since there is no light, f (i, j) = 0. Therefore, considering equation (1), when the system averages these K2 images, this averaged image (also referred to as “average flat dark image”) is defined by the following equation: .

  Note that random noise will be suppressed by averaging.

  The gain component of the fixed pattern noise can be calculated by subtracting the “average flat dark image” from the “average flat light image” for each pixel, as shown in Expression (9).

  The system can perform post-processing on the “fixed pattern noise gain component image (FPN gain image)” or “flat field image”. That is, the system can calculate its overall average and overall standard deviation. The system can clip the image so that the finally estimated flat field pixel value falls between the minimum value and the maximum value based on the steady parameter. For example, the minimum value may be an overall average value−3.0 × overall standard deviation, and the maximum value may be an overall average value + 3.0 × overall standard deviation. . The flat field image FF (i, j) is stored for correction of the gain component of the fixed pattern noise in the later “online” stage.

The non-uniform offset component of the fixed pattern noise can be indicated by a two-dimensional array (M × N pixels) d _FPN (i, j). A camera without an offset component of fixed pattern noise may have a value of d _FPN (i, j) that is equal to 0 or constant for all pixels. This offset component can be measured using any suitable technique. For example, the technique shown in FIG. 4 can be used.

FIG. 4 is a diagram for explaining the measurement of the offset component d _FPN of fixed pattern noise by this system. The system can capture a series of “dark frames”. That is, an image is taken without providing light to the camera or sensor (using a lens cap or closing the shutter). This should be done for several exposure times. Here, mark the specific exposure time t _m. For example, t _m = 100 ms. The offset component of the fixed pattern noise associated with the exposure time is denoted as d _{FPN, tm} (i, j) (in the left column, the subscript tm represents t _m ). Here, let K3 be the number of such “dark” images taken at the exposure time t _m . K3 should be as large as possible (for example, K3> 9). Since there is no light, f (i, j) = 0. Therefore, considering equation (1), if the system averages these K3 images, this averaged image is defined by the following equation:

Note that random noise will be suppressed by averaging. This image can be referred to as a “fixed pattern noise offset component image (FPN offset image)” or “average fark frame”. This average dark frame DF _tm (i, j) (in the left column, the subscript tm represents t _m ) should be obtained for each required exposure time. The average dark frame DF _tm (i, j) is stored for correction of the offset component of the fixed pattern noise in a later “online” stage. This completes the process used for fixed pattern noise characteristic analysis in the “offline” stage.

Also, other processes of the “offline” stage related to the characteristic analysis of the temporal random noise component n _T (i, j, k) can be performed. The variance of random noise is described in equation (5). The parameters indicating the dispersion characteristics of the readout noise and the shot noise are the readout noise variance σ ² _READ and the shot noise gain α. In order to estimate these parameters for the particular camera used, one should obtain data points of noise variance versus signal level. For example, a series of data frames can be acquired while gradually increasing the exposure time. That is, the present system can set the exposure time to a value that gradually increases (for example, 1, 2, 3,..., 100,..., 200,... Ms). The system can image two (or more) data frames for a uniform (blank) target at each exposure time. The system can also capture a set of dark frames at the exposure time. Then, one of the data frames is subtracted from the other. The system can set a plurality of measurement windows in an image obtained as a result of subtraction. Each measurement window may be a square area composed of a plurality of pixels. For example, it may be an area of 64 × 64 pixels, 128 × 128 pixels, or 256 × 256 pixels. In each window, the system calculates the variance and divides by 2.0 (correction for subtraction). The system can also calculate a “true” signal value for each measurement window. This can be done by subtracting the average value of the measurement window in the (averaged) dark frame from the average value of the measurement window in the corresponding data frame. From the above, the system can acquire a single <signal, variance> data point for the measurement window. By combining all data points for multiple measurement windows, the system can collect data points at various levels of signal (and corresponding variance values). These data points can be plotted on a graph. An example is shown in FIG. As shown in FIG. 5, the data points exhibit a linear behavior. This is as in the model predicted in Equation (5). The parameters of the model can be easily estimated from the collected data points by approximating them to a linear model (straight line). For example, a standard linear least square approximation method can be used. Thereby, the estimated values of σ ² _READ and α can be obtained for the camera.

("Online" stage)
FIG. 6 shows an outline of noise reduction processing in the “online” stage. In the noise reduction processing in the "online" stage, first, the first image is captured using the desired exposure time T _e. Subsequently, the gain component and the offset component of the fixed pattern noise in the captured image are removed by calibration. For the calibration, the above-described “fixed pattern noise gain component image” and “fixed pattern noise offset component image” for an appropriate exposure time are used. As shown in FIG. 2, the present system first removes the offset component and then removes the gain component. These two sub-steps may be executed simultaneously or in reverse order. Here, the captured _{image, g te (i, j,} k) ( in the left, the te subscript denote the _{t e)} and referred. The first estimated image of the “true” image f (i, j, k) can be calculated as follows.

  The average value of the flat field can be estimated by taking the spatial average of the flat field image.

  The fixed pattern noise calibration step can significantly reduce the fixed pattern noise in the image. However, as described above, the fixed pattern noise component cannot be predicted completely. Fixed pattern noise in the image can be undercompensated or overcompensated. Therefore, fixed pattern noise can clearly remain in the image at this stage. The remaining component of the fixed pattern noise mainly appears as spike noise or impulsive noise in the image. That is, it consists of a plurality of isolated spikes. Spikes can have significantly higher or lower values than surrounding pixels. The fixed pattern noise reduction step intends to remove or reduce the remaining spike-like noise (impulse noise) and to prevent any feature of the image (scenery) from being removed.

  One approach to removing impulsive noise is to use a median filter. This is performed using, for example, a 3 × 3 pixel window. However, the median filter does not distinguish impulsive noise from image characteristics. Therefore, there is a risk of removing small features of the image. The system can use a specific filter belonging to a so-called rank-conditioned rank-selection filter family. Specifically, a rank-conditioned median filter is used that has been expanded to more reliably remove small features of the image.

(Media rank filter with extended rank condition)
The extended rank conditional median filter can be defined as follows. The system may use a 3x3 window that slides across all pixels of the image. At any position, the system can acquire nine pixel values within a 3 × 3 window. This may be described as a vector [f ₁ , f ₂ , f ₃ , f ₄ , f ₅ , f ₆ , f ₇ , f ₈ , f ₉ ]. Note that f ₅ is ordered so as to be in the center of the 3 × 3 window. These pixel values can be ranked from a smaller value to a larger value. The resulting ranked pixel value vector is [r ₁ , r ₂ , r ₃ , r ₄ , r ₅ , r ₆ , r ₇ , r ₈ , r ₉ ] (where r ₁ ≦ r ₂ ≦ r ₃ ... ≦ r ₉ ). In other words, r ₁ is rank 1 in the window, that is, the smallest pixel value, r ₉ is rank 9 in the window, that is, the maximum pixel value, and r ₅ is rank 5 in the window, that is, the pixel value. Median value. In this system, the isolated spike-like noise first determines whether the pixel value at the center of the window is either rank 1 (minimum value in the window) or rank 9 (maximum value in the window). Is detected by If the central pixel has neither rank 1 nor rank 9, the central pixel is not spiked noise and no processing is performed. In this case, the central pixel maintains its original value without being replaced. If the central pixel has rank 1, the system calculates the difference between the pixel value (r ₁ ) of the central pixel and the next smaller pixel value (r ₂ ). Alternatively, if the central pixel has rank 9, the system calculates the difference between the pixel value (r ₉ ) of the central pixel and the next largest pixel value (r ₈ ). When the absolute value of the difference is larger than the threshold _TRCM specified in advance, the filter classifies the central pixel as spike-like noise caused by the residual component of the fixed pattern noise. In this case, the pixel value of the center pixel is replaced by the center value (r ₅ ). The extended rank conditional median filter can be specified as follows.

However, f _c (in the left column, it is assumed that a caret (^) is attached on f. Hereinafter, f _c (caret) is also indicated) indicates an estimated value of the center pixel. As an example of the threshold, T _RCM = 8. Note that the threshold may be fixed. Alternatively, the threshold value may be adaptively determined based on the estimation of the strength of the remaining component of the fixed pattern noise.

  The sub-step flow for calculating the output of the extended rank conditional median filter may be modified as desired. For example, different ranking (ordering) techniques may be used. As another example, in order to realize the above-described filter, the filter may be optimized from the viewpoint that complete ranking is not always necessary. As yet another example, the filter may be further optimized to consider and process multiple neighboring window pixel values simultaneously.

  FIG. 7 shows an example of the operation of the filter. In FIG. 7, a 3 × 3 window is shown superimposed on a portion of the image. Nine pixel values in the window are represented as vectors. The pixel value of the center pixel is 71, which can be regarded as a positive single pixel spike compared to all surrounding pixel values. The ranked pixel values are also expressed as vectors. The filter detects spike noise as follows. That is, in the filter, the pixel value of the center pixel has a rank 9 (maximum value), and the difference between the pixel value of the center pixel and the next largest pixel value (47 in this example) is the threshold value. By determining that it is larger than (for example, 8), it is detected that the central pixel is spike-like noise. In this case, the pixel value of the center pixel can be replaced with the center value (46 in this example). Here, if the pixel value of the center pixel is 48, the pixel value of the center pixel still has rank 9, but the difference from the pixel value of the rank 8 pixel is the threshold value. Can be smaller. In this case, there is no significant spike and the filter can leave the pixel value of the center pixel unchanged.

(Rank-based impulse rejection filter)
Here, in some scenes during the inspection of the liquid crystal panel, a small luminance peak that occupies a part of the image may represent a small defective picture element. Therefore, it is desirable that such a small defective pixel can be discriminated even if the remaining component of the fixed pattern noise is removed. Even if the residual components of fixed pattern noise are removed, a rank-based impulse removal (RIR) filter can be used as one technique for further distinguishing such small defective picture elements. is there. The RIR filter can be characterized as using a 3 × 3 window that slides across all pixels of the image. At any position, the system can acquire nine pixel values within a 3 × 3 window. This may be described as a vector [f ₁ , f ₂ , f ₃ , f ₄ , f ₅ , f ₆ , f ₇ , f ₈ , f ₉ ]. Note that f ₅ is ordered so as to be in the center of the 3 × 3 window. Other pixels are surrounding pixels. These pixel values can be ranked from a smaller value to a larger value. The resulting ranked pixel value vector is [r ₁ , r ₂ , r ₃ , r ₄ , r ₅ , r ₆ , r ₇ , r ₈ , r ₉ ] (where r ₁ ≦ r ₂ ≦ r ₃ ... ≦ r ₉ ). In other words, r ₁ is rank 1 in the window, that is, the smallest pixel value, r ₉ is rank 9 in the window, that is, the maximum pixel value, and r ₅ is rank 5 in the window, that is, the pixel value. Median value. In order to classify the central pixel (f5) as spike-like noise caused by the residual component of the fixed pattern noise, the following three conditions should be satisfied.

  First, to classify the center pixel as spike noise, the system determines whether the center pixel value is either rank 1 (minimum value in the window) or rank 9 (maximum value in the window). Determine whether or not. If the central pixel has neither rank 1 nor rank 9, the central pixel is not spiked noise and no processing is performed. In this case, the central pixel maintains its original value without being replaced.

Second, in order to classify the central pixel as spike noise, the system makes a determination on the difference in pixel values. In order for the central pixel to be classified as spiked noise, the central pixel should have a pixel value that is significantly different from the pixel values of the other pixels in the window. That is, when the central pixel has rank 1, the system calculates the difference between the pixel value (r ₁ ) of the central pixel and the next smaller pixel value (r ₂ ). Alternatively, if the central pixel has rank 9, the system calculates the difference between the pixel value (r ₉ ) of the central pixel and the next largest pixel value (r ₈ ). If the absolute value of the difference is not greater than the threshold _TRCM specified in advance, the central pixel is not classified as spike noise and no processing is performed.

Third, in order to classify the center pixel as spike noise, the system makes a further determination on the difference in pixel values. Pixel values other than the central pixel should be close enough to each other. In other words, when the central pixel has rank 1, the system uses the pixel value (r ₂ ) of the rank ₂ pixel, the pixel value (r ₈ ) of the rank ₈ pixel, and the pixel of the rank 9 pixel. The difference from the value (r ₉ ) is calculated. Alternatively, if the central pixel has a rank of 9, the system will calculate the pixel value of the rank 8 pixel (r ₈ ), the pixel value of the rank 1 pixel (r ₁ ), and the pixel of the rank 2 pixel. The difference from the value (r ₂ ) is calculated. The system then calculates the sum of the squares of the differences. When the sum of the squares of the differences is not smaller than the threshold value _TRV specified in advance, the central pixel is not classified as spike noise and no processing is performed.

  When all the above three conditions are satisfied, the center pixel can be classified as spike-like noise caused by the remaining component of the fixed pattern noise, and the pixel value of the center pixel is the pixel value of the rank 2 pixel or It can be replaced by the pixel value of a rank 8 pixel. That is, if the central pixel has a rank 1, the pixel value of the central pixel is replaced with the pixel value of the rank 2 pixel; if the central pixel has a rank 9, the central pixel Is replaced with the pixel value of the rank 8 pixel. The rank-based impulse rejection filter can be defined as follows.

However, f _c (caret) indicates an estimated value of the center pixel. As an example of the threshold, T _RCM = 8 and T _RV = 20. Note that the threshold may be fixed. Alternatively, the threshold value may be adaptively determined based on the estimation of the strength of the remaining component of the fixed pattern noise.

  The sub-step flow for calculating the output of the rank-based impulse rejection filter may be modified as desired. For example, different ranking (ordering) techniques may be used. As another example, in order to realize the above-described filter, the filter may be optimized from the viewpoint that complete ranking is not always necessary. As yet another example, the filter may be further optimized to consider and process multiple neighboring window pixel values simultaneously.

  FIG. 8 shows an example of the operation of the filter. In FIG. 7, a 3 × 3 window is shown superimposed on a portion of the image. Nine pixel values in the window are represented as vectors. The pixel value of the center pixel is 71, which can be regarded as a positive single pixel spike compared to all surrounding pixel values. The ranked pixel values are also expressed as vectors. The filter detects spike noise as follows.

(1) It is determined that the pixel value of the center pixel has rank 9 (condition 1)
(2) Determine that the difference between the pixel value of the center pixel and the next largest pixel value (47 in this example) is greater than the threshold (condition 2), and (3) Other than the center pixel It is determined that the pixel values of the pixels are close to each other (condition 3); this is to calculate the difference between r ₈ and r ₁ and the difference between r ₈ and r _2, and the sum of the squares of the differences is calculated from the threshold value Can also be determined by determining that they are smaller.

  In this case, the central pixel is replaced by the value 47.

(Extended adaptive noise smoothing filter)
The final step in the “online” noise reduction stage is to suppress temporal random noise after removal of fixed pattern noise. The temporal random noise includes read noise and shot noise. Residual random noise can be suppressed using a spatial neighborhood filter. For this purpose, the image data at this stage can be modeled as follows.

Where f (i, j, k) indicates a “true” image; g (i, j, k) indicates the given data; n _T (i, j) is a temporal random Indicates noise. “True” image features should not be suppressed at this stage, and processing time should be kept short. The system can use a so-called adaptive noise smoothing filter that has been extended to improve performance. The extended adaptive noise smoothing filter is based on local statistics and can be defined as follows. That is, the filter uses a sliding window (for example, a 5 × 5 square window) centered on the pixel to be processed. The size W of the filter window is a parameter. The sliding window is assumed to be centered on the pixel at position (i, j). Here, g (i, j) (in the left column, it is assumed that a bar (−) is attached on g. Hereinafter, g (bar) will be also described). Let i, j) be an estimate of the local average. The system can determine this value by calculating the average pixel value in a local window surrounding (i, j). The estimate can be verified by limiting the set of pixels used to calculate the local average within the window. That is, only the pixel value g (m, n) in which the absolute value of the difference from the pixel value of the center pixel is smaller than the threshold value T _ANS is included. That is, if | g (m, n) -g (i, j) | <T _ANS , g (m, n) serves to estimate the local average. An example of the threshold is T _ANS = 10. The local average theoretically matches the local average of an image without noise (a “true” image f (i, j)). This is because the average value of noise is zero. This local average g (bar) (i, j) can itself serve as an estimate of the noise-free image f (i, j). However, this adaptive noise smoothing filter improves this by adaptively weighting this local average using the input pixel value g (i, j). This weighting process works to prevent local image feature suppression. Suppression of local image features can occur because g (bar) (i, j) is obtained by spatial local averaging. In this weighting process, the adaptive noise smoothing filter considers local noise intensity and local signal variance. This is shown below.

Here, V _g (i, j) is an estimated value of local variance of g (i, j). The system can determine this value by calculating the local variance statistic of the pixel values in the local window surrounding (i, j). The system can improve this estimate by again limiting the set of pixels used to calculate this local variance within the window. That is, only the pixel value g (m, n) in which the absolute value of the difference from the pixel value of the center pixel is smaller than the threshold value T _ANS is included. That is, if | g (m, n) -g (i, j) | <T _ANS , g (m, n) serves to estimate the local variance.

Here, V _n (i, j) is an estimated value of temporal noise variance at (i, j). According to the noise model, the present system can grasp this variance as follows.

Here, the system uses the random noise parameters estimated in the “offline” stage. Thereby, this system can estimate the local dispersion _| variation _Vf (i, j) in an image without a noise as follows.

  In other words, the present system subtracts the estimated value of the variance of noise from the estimated value of variance of the given image, and performs clipping processing using a zero value as a threshold (to avoid negative values).

  The final estimate of the local pixel value of the image without noise is calculated as follows.

As shown in Expression (17), the adaptive noise smoothing filter calculates a balance between the local average g (bar) (i, j) and the input value g (i, j) including noise. To do. This weighting is determined locally based on the local noise intensity relative to the local signal variance. Near the image features, the local signal variance is large. Therefore, the input values are weighted more heavily than the local average and the features are maintained. In an image area (for example, a flat area) having no characteristic portion, a noise suppression effect is provided. Note that the threshold T _ANS can be fixed. Alternatively, the threshold value may be adaptively determined based on an estimation of noise intensity. For example, the threshold value may be set to the local standard deviation of multiple noises or the estimated value itself.

(Modified adaptive noise smoothing filter)
Another suitable filter is a modified adaptive noise smoothing filter (M-ANS). This filter suitably maintains a small luminance peak in the input image. A small luminance peak can represent a luminance peak in the image and can represent a feature portion of the image. The characteristic part of the image corresponds to, for example, a defective picture element in the LCD panel. The filter may be based on calculating a modified local variance. This modified local dispersion can be used to store small luminance peaks. In the luminance peak region, the modified local variance becomes large, so the modified adaptive noise smoothing filter hardly performs noise smoothing. In other regions, such as the background region, the modified adaptive noise smoothing filter provides a noise smoothing effect equivalent to an unmodified modified adaptive noise smoothing filter.

  The modified adaptive noise smoothing filter is based on local statistics and can be defined as follows. That is, the filter uses a sliding window (for example, a 7 × 7 square window) centered on the pixel to be processed. The size Q1 of the filter window is a parameter. The sliding window is assumed to be centered on the pixel at position (i, j). Within this first filter window is a second, smaller window. The size of the smaller filter window W2 is a parameter, for example, a 3 × 3 square window. These local windows are shown in FIG. The filter can use a central region (C) defined as a group of pixels within a small window. The filter can also use a surrounding area (S) defined as a group of pixels within a large window and not within a small window.

The filter can calculate the average _μs of the pixel values in the surrounding area as follows.

  Subsequently, the present system can calculate the modified local variance based on the difference between the pixel value in the central region C and the average value of the pixel values in the surrounding region as follows.

This modified local variance can be denoted as “center-ambient variance” CSV _g .

  Here, g (bar) (i, j) is an estimated value of the local average of the given image g (i, j). The system can determine this value by calculating an average pixel value in a local window (eg, a window as shown in FIG. 9) surrounding (i, j). The modified adaptive noise smoothing filter adaptively weights the local average g (bar) (i, j) using the input pixel value g (i, j), thereby obtaining a pixel value f having no noise. An estimated value of (i, j) can be calculated. This weighting process works to prevent local image feature suppression. Suppression of local image features can occur because g (bar) (i, j) is obtained by spatial local averaging. In this weighting process, the modified adaptive noise smoothing filter takes into account local noise intensity and local signal variance. This is shown below.

Here, V _n (i, j) is an estimated value of temporal noise variance at (i, j). According to the noise model, the present system can grasp this variance as follows.

Here, the system uses the random noise parameters estimated in the “offline” stage. Thereby, this system can estimate the local dispersion _| variation _Vf (i, j) in an image without a noise as follows.

  In other words, the present system subtracts the estimated value of the variance of noise from the estimated value of variance of the given image, and performs clipping processing using a zero value as a threshold (to avoid negative values).

  The final estimate of the local pixel value of the image without noise is calculated as follows.

As described above, the modified adaptive noise smoothing filter calculates the balance between the local average g (bar) (i, j) and the input value g (i, j) including noise. This weighting is determined locally based on the local noise intensity relative to the modified local signal variance. If the local window is centered on a luminance peak (corresponding to a defective pixel on the LCD panel) and the luminance peak is small, this peak is included in the central area (small window), and the surrounding area contains only background and noise. Including. Since μ _s is the average of the background (non-peak), the modified local variance (“center-ambient variance”) is large in the peak region. In this case, the modified adaptive noise smoothing filter will use the original input pixel value; the luminance peak is preserved. A basic adaptive noise smoothing filter is not efficient for detecting this luminance peak.

  If the local window is not centered on the luminance peak but centered on the background area, the average value of the surrounding area should be the same as the average value of the central area. Therefore, in the background region, the local variance is small, and the modified adaptive noise smoothing filter performs more noise smoothing. Thus, the modified adaptive noise smoothing filter provides the same noise smoothing effect as the unmodified adaptive noise smoothing filter; however, the modified adaptive noise smoothing filter is not modified adaptive noise. Stores luminance peaks better than smoothing filters. The size of the small window (center area) can be adapted to the size of the smallest luminance peak to be preserved. The size of the large window should be several pixels larger.

Near the image features, the local signal variance is large. Therefore, the input values are weighted more heavily than the local average and the features are maintained. In an image area (for example, a flat area) having no characteristic portion, a noise suppression effect is provided. Note that the threshold T _ANS can be fixed. Alternatively, the threshold value may be adaptively determined based on an estimation of noise intensity. For example, the threshold value may be set to the local standard deviation of multiple noises or the estimated value itself.

  The terms and expressions used in the above detailed description are used for explanation and not for limitation. In addition, use of such terms and expressions is not intended to exclude equivalents or portions of the features described and illustrated. It is recognized that the scope of the invention should be defined and limited in the claims.

(Computer system)
FIG. 10 is a diagram illustrating an example of the computer system described above. As shown in FIG. 10, the computer system 2 includes a main control unit 3 and a recording medium 4. The main control unit 3 includes an “offline” stage execution unit 5 and an “online” stage execution unit 6. The “online” stage execution unit 6 includes a fixed pattern noise offset component correction unit 7, a fixed pattern noise gain component correction unit 8, a fixed pattern noise residual component filter unit 9, and a shot noise and readout noise filter unit 10. A CMOS / CCD inspection camera 1 is connected to the computer system 2.

  The “offline” stage execution unit 5 performs noise measurement and characteristic analysis in the “offline” stage described above based on the image captured by the CMOS / CCD inspection camera 1. Specifically, as described above, a “fixed pattern noise gain component image” and a “fixed pattern noise offset component image” are calculated based on the captured images captured under different conditions, and recorded on the recording medium 4.

  As described above, the fixed pattern noise offset component correction unit 7 and the fixed pattern noise gain component correction unit 8 are the “fixed pattern noise gain component image” and “fixed” for the captured image captured by the CMOS / CCD inspection camera 1. Calibration is performed using the “pattern noise offset component image” to remove the offset component and the gain component of the fixed pattern noise.

  The fixed pattern noise residual component filter unit 9 removes the residual component of the fixed pattern noise. The fixed pattern noise residual component filter unit 9 uses, for example, an extended rank-conditional median filter or rank-based impulse removal filter as described above.

  The shot noise and readout noise filter unit 10 reduces shot noise and readout noise. The shot noise and readout noise filter unit 10 uses, for example, an extended adaptive noise smoothing filter or a modified adaptive noise smoothing filter as described above.

  As described above, according to the computer system 2, the above-described noise reduction technique can be suitably implemented.

(Program and recording medium)
Finally, each block of the main control unit 3 may be realized by hardware by a logic circuit formed on an integrated circuit (IC chip) or by software using a CPU (Central Processing Unit). May be.

  In the latter case, the main control unit 3 includes a CPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, a RAM (Random Access Memory) that expands the program, the program, A storage device (recording medium) such as a memory for storing various data is provided. An object of the present invention is a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the main control unit 3 which is software that realizes the above-described functions is recorded so as to be readable by a computer This can also be achieved by supplying the program to the main control unit 3 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. IC cards (including memory cards) / optical cards, semiconductor memories such as mask ROM / EPROM / EEPROM / flash ROM, PLD (Programmable logic device), FPGA (Field Programmable Gate Array), etc. Logic circuits can be used.

  The main control unit 3 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited as long as it can transmit the program code. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, and the like can be used. The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, even with wired lines such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA and remote control, Bluetooth (registered trademark), IEEE 802.11 wireless, HDR ( It can also be used by radio such as High Data Rate (NFC), Near Field Communication (NFC), Digital Living Network Alliance (DLNA), mobile phone network, satellite line, and digital terrestrial network.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  The present invention can be suitably used for an imaging technique or the like in a machine vision system.

DESCRIPTION OF SYMBOLS 1 CMOS / CCD inspection camera 2 Computer system 3 Main control part 4 Recording medium 5 "Offline" stage execution part 6 "Online" stage execution part 7 Fixed pattern noise offset component correction part 8 Fixed pattern noise gain component correction part 9 Fixed pattern noise Residual component filter unit 10 Shot noise and readout noise filter unit

Claims (10)

A method for reducing noise in an image captured by an imaging device,
A first step of acquiring the image from the imaging device;
A second step of reducing an offset component of fixed pattern noise from the image;
A third step of reducing the gain component of the fixed pattern noise from the image;
After the second step and the third step, a fourth step of reducing a residual component of the fixed pattern noise from the image using a filter that reduces impulsive noise included in the image is included. A method characterized by.   The method according to claim 1, wherein the filter ranks each pixel in the window based on a pixel value in a window set in the image.   The method of claim 2, wherein the window is a 3x3 pixel window.   The method of claim 2, wherein the ranking is a ranking of pixel values within the window.   The method according to claim 4, wherein the filter targets a pixel value of a central pixel of the window.   The condition for modifying the pixel value of the central pixel includes that the pixel value of the central pixel is a minimum value or a maximum value in each pixel in the window. Method.   The method according to claim 6, wherein the condition for modifying the pixel value of the central pixel includes a condition based on a difference between pixels in the window.   The condition for modifying the pixel value of the central pixel is that a difference between the pixel value of the central pixel and the pixel value of each pixel other than the central pixel in the window is equal to or greater than a predetermined threshold value. The method according to claim 7, further comprising:   The condition for modifying the pixel value of the central pixel includes that a difference between pixel values of pixels other than the central pixel in the window is smaller than a predetermined threshold value. Item 9. The method according to Item 8. The method according to claim 9, wherein the filter performs image processing based on the following formula.
_{(However, f 1 to 9} represents each pixel value corresponding to each position in the window _{(f 5} represents a _{central), r 1 to 9} were sorted on the basis of the pixel value _{(r 1} Indicates a minimum value, r ₉ indicates a maximum value), T _RD and T _RV indicate pre-specified threshold values, otherwise indicates a case where the above two expressions do not hold, and a left arrow Indicates assignment.)