KR20120115729A

KR20120115729A - A preprocessing to reduce influence of outliers for spectrum data

Info

Publication number: KR20120115729A
Application number: KR1020110033210A
Authority: KR
Inventors: 천선일; 양상훈; 박동선; 박정권
Original assignee: 전북대학교산학협력단; 주식회사 아이에스피
Priority date: 2011-04-11
Filing date: 2011-04-11
Publication date: 2012-10-19

Abstract

The present invention relates to a method for identifying a more clear pattern for values affected by an outlier when applying the proposed preprocessing method and further obtaining a value having a high signal-to-noise ratio (SNR).

Description

Preprocessing to reduce influence of outliers for spectrum data

The present invention adds the proposed work to the values affected by the outliers when applying the preprocessing algorithm to the observed signal to identify a clearer pattern and further to obtain a value with a high signal-to-noise ratio. It is about.

In statistical or image computation and signal processing, preprocessing is often done in an attempt to obtain and verify a specific pattern of noise-containing values.

Commonly used types of preprocessing tasks include a moving average algorithm that obtains the mean value of a specified range for a given signal, a Savitzky Golay Smooth algorithm that obtains a partial polynomial regression value for that signal, and an intermediate value for removing impulse noise in the observed signal. Algorithms, etc.

Since preprocessing algorithms are generally not linear, given values are not explicitly represented in functional terms, but rather as processed values themselves. In addition, the goal is to obtain an approximate value without noise, rather than inferring the value of the signal accurately through preprocessing.

In this respect, the resulting signal is finally expressed in some linear function form, which is different from Curve Fitting, which attempts to restore the original signal more accurately while reducing the error rate.

During the development of a compact X-ray fluorescence spectrometer, it was often found that unexpected anomalies occurred in the energy spectral value of the second X-ray actually obtained [Figure 1]. 2] If spectral analysis aimed at high recall rate does not reduce the influence of outliers, the result is distorted results.

Curve fitting and preprocessing algorithms introduced above can be used to reduce the effects of outliers. If you can overcome this, you will get a fairly accurate result after the signal processing.

Among the preprocessing algorithms, the moving average algorithm is a simple algorithm that determines the average value of the specified window at each position of the observed value as the smooth value of the position.

The Savitzky Golay Smooth filter (SG) determines the polynomial regression (partial polynomial regression) of the points and surrounding values to be smoothed as shown in Figure 3.

When the obtained value (y) is a form in which the noise (ε) is added to the coefficient (w) and the position (X) of the polynomial, the optimal coefficient that minimizes the error rate within the corresponding range is the pseudo inverse of X and the obtained value (y Can be found as the product of

SG filter has the advantage that it is strong against noise because it performs partial polynomial regression for each window and finds optimal smooth value by varying order of polynomial to apply and window size to apply partial polynomial regression. Filter.

The present invention is to provide a pretreatment method that can be processed more accurately when the value observed by a widely used pretreatment method contains an unexpected abnormal value.

The present invention provides the following means for solving the above problems.

The preprocessing method for minimizing the influence of the outliers of the spectral data according to the present invention,

Selecting a model of a signal to be processed to obtain a difference value between the signal to be processed and the model;

Applying a median filter to the difference to remove outliers and noise outside of a predetermined range;

Reconstructing the preprocessed signal by adding the signal and the model processed in the step;

And applying a smooth algorithm to the restored signal to perform a preprocessing result.

Through the present invention, the influence of the outliers found in the spectrum and other observed signals can be reduced, and preprocessing can be performed at a lower calculation cost to obtain a result closer to the original signal.

1 is a graph of signals obtained by experiments.
2 is a graph showing a difference value between a model and a signal for an experimental signal.
3 is a graph showing smoothing points and partial polynomial regression.
4 is a flow chart of a preprocessing method for minimizing the effect of outliers on spectral data in accordance with the present invention.
5 is a graph of values using median filters.
Figure 6 is a graph of the value applying the actual Smooth algorithm.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. Even if the terms are the same, it is to be noted that when the portions to be displayed differ, the reference signs do not coincide.

The terms to be described below are terms set in consideration of functions in the present invention, and may be changed according to a user's intention or custom such as an experimenter and a measurer, and the definitions should be made based on the contents throughout the present specification.

The preprocessing method for minimizing the influence of the outliers of the spectral data according to the present invention goes through the following steps.

1. Obtain a model of the signal obtained from the experiment and list the difference between the signal and the model.

2. The median filter is applied to the results obtained above to remove some of the outliers and noise.

3. Add the processed signal and the model to restore the preprocessed signal.

4. Finally, the Smooth algorithm is applied to the result obtained above to obtain the pretreatment result.

4, there is shown a preprocessing method for minimizing the effect of outliers on spectral data according to the present invention. This is done by obtaining a model of the observed signal, applying a median filter to the difference between the model and the observed signal, and then applying a smooth algorithm by adding the filter applied value and the model.

3 shows a diagram illustrating an example of smoothing points and partial polynomial regression.

The present invention uses a method to grasp the characteristics of the value to be applied to the preprocessing and to obtain a preprocessing result closer to the original signal by adding one method to the already proposed good smooth algorithm. The value to which the Smooth algorithm is applied is characterized by the addition of an outlier to a signal with Gaussian distribution plus white noise. The goal is to eliminate the effects of outliers and obtain preprocessing results as close as possible to the original signal.

Since it is found that the outlier part of the observed signal has a particularly large noise, a median filter that removes the noise having an impulse characteristic in a constant signal is applied to remove the influence of the outlier.

Referring to FIG. 5, since the median filter has a characteristic of obtaining the best result when the value to be applied has a constant value without a linear polynomial or an exponential shape, the sine, cosine, and second order actually obtained. To apply to a spectral signal having a polynomial or Gaussian shape, the curved portion has a limit of flattening.

To apply the median filter, we create a model that differs from curve fitting, which best represents the observed signal, and finds the difference between the signal and the model. As a result, the difference value was changed into a pattern to which a median filter can be applied.

Finally, adding the median filter and the model again adds a signal with subtle noise but little noise.

Applying the actual Smooth algorithm to the result preprocessed with the above process, the signal-to-noise ratio result can be definitely improved compared to applying the Smooth algorithm only to the signal with the outlier. These results are shown in detail in FIG.

The present invention is not limited to the scope of the embodiments by the above embodiments, all having the technical spirit of the present invention can be seen to fall within the scope of the present invention, the present invention is the scope of the claims by the claims Note that is determined.

1: graph

Claims

A preprocessing method for minimizing the influence of outliers on spectral data,
Selecting a model of a signal to be processed to obtain a difference value between the signal to be processed and the model;
Applying a median filter to the difference to remove outliers and noise outside of a predetermined range;
Reconstructing the preprocessed signal by adding the signal and the model processed in the step;
And applying a smooth algorithm to the restored signal to perform a preprocessing result.
Preprocessing method to minimize outlier effects of spectral data.