CN106874651A - Room air data preprocessing method based on local weighted recurrence - Google Patents

Abstract

The present invention relates to the air parameter data prediction for changing over time, it is that the data that room air is changed over time are pre-processed with simple but effective method, including filling up for data vacancy long in short-term, the removal of data exception hop value, at the same time ensure that the data related to people's Behavioral change significantly change and be not identified as exceptional value, finally carry out the correction of zero migration.The technical solution adopted by the present invention is, room air data preprocessing method based on local weighted recurrence, filling up for data vacancy long in short-term is carried out first, ensure 0 value of the whole data in the absence of vacancy, then the removal of data exception hop value is carried out again, when ensureing no longer to have the data jump point of needle-like, then the correction of zero migration is carried out, the data that will be handled well are updated in calibration curve.Present invention is mainly applied to the air parameter data prediction for changing over time.

Description

Room air data preprocessing method based on local weighted recurrence

Technical field

This algorithm can be to the air parameter (temperature, humidity, concentration of formaldehyde, the PM2.5 concentration, carbon dioxide that change over time Concentration etc.) in data vacancy filled up, it is possible to the abnormal hop value in data is removed, and can be to data Carry out the amendment of zero migration.Belong to the field of specific data prediction.Concretely relate to the interior based on local weighted recurrence Air data preprocess method.

Background technology

The technical method difficulty or ease for being pre-processed to data at present all have, but simple preprocess method has been difficult to Effect, and effective preprocess method is often more complicated [1].The data object of this technology pretreatment is room air data：The One, but this data to be had slowly changed with the time on the whole the characteristics of suffering from different degrees of random noise all the time (such as Fig. 1)；Second, because hardware system has data vacancy warning function, therefore may insure that the duration of data vacancy is very short；The Three, already have the calibration curve of amendment zero migration.So filled up and zero migration compared to data vacancy, this The core of technology is that the saltus step exceptional value in data is removed, and ensure that the data related to people's Behavioral change are big Amplitude variation is not identified as exceptional value and is removed.

In the method rejected to data outliers, most commonly the direct use C4.5 decision trees of data are carried out Classification judges [2], but the algorithm easily will be because of data caused by people's Behavioral change significantly change value and abnormal hop value one And it is classified into exceptional value；Secondly CD (Curve Description) method is also used for the classification [3] to exceptional value, the method Judged by threshold value of the variable quantity and rate of change of adjacent numerical value, but for the problem that this patent to be solved, it There is similar defect with decision tree method, and it is also more complicated than decision tree method in program realization；It is external also to use noise data Filtration method (Filters) identification and rejecting abnormalities value, compare typically Ensemble Filter (EF) [4] and Iterative-Partitioning Filter (IPF) [5], both approaches are all more famous, but all more complicated, obtain To its extra setting multiple parameters [1], this problem faced to this technology it is not necessary that.

The content of the invention

To overcome the deficiencies in the prior art, it is contemplated that being changed over time to room air with simple but effective method Data pre-processed, including data vacancy long is filled up in short-term, the removal of data exception hop value, at the same time ensure with The related data of people's Behavioral change significantly change and are not identified as exceptional value, finally carry out the correction of zero migration.The present invention The technical scheme of use is that the room air data preprocessing method based on local weighted recurrence carries out data long in short-term first Vacancy is filled up, it is to be ensured that then whole data carry out the removal of data exception hop value again in the absence of 0 value of vacancy, are protecting When no longer there is the data jump point of needle-like in card, then carry out the correction of zero migration, and the data that will be handled well are updated to demarcation In curve.

The removal for carrying out data exception hop value is comprised the concrete steps that, is fitted significant information using matched curve Come, while needle-like data jump and all of high-frequency noise are not fitted, specifically from local weighted recurrence (Local Weight Regression) fitting of useful information is carried out, then subtract matched curve with former data and curves and obtain noise curve, Solve the interference that useful information is removed to hop value.

Local weighted Regression comprises the concrete steps that, first with the reference point on the transverse axis of certain amount by whole data decile Come, and ask calculation line locality to return respectively centered on these points, when regression parameter is solved using least square method, in Flexible strategy shared by the more remote data point of heart point are smaller, finally obtain the recurrence numerical value of these points, then returned these with interpolation Numerical point is connected, used here as linear interpolation；

Further, to each training data point, will cause：

∑_iw⁽ⁱ⁾(y⁽ⁱ⁾-θ^Tx⁽ⁱ⁾)² (1)

It is minimum；

Wherein i is the number footmark of training data；X refers to the time value of time shaft；Y is desired value；θ is that regression equation is Number vector, using quadratic regression, therefore θ is a three-dimensional vector；W is Gauss flexible strategy, is expressed as：

The reference point on selected transverse axis is referred to without superscript x, τ is bandwidth (bandwidth), and τ is bigger, The intensity of local regression is bigger；

It is local weighted to return many Gauss power before each residuals squares, require secondary to each reference point Regression curve, and parameter of curve must be different, to any one reference point x, have：

θ=(X^TWX)^-1X^TWy (3)

Wherein, X is by 1, x⁽ⁱ⁾, (x⁽ⁱ⁾)²The m of composition ties up matrix, and referred to as design matrix (design matrix) m is Training data quantity, X writings：

W is m rank diagonal matrix, writing diag (w⁽¹⁾…w⁽ⁱ⁾…w⁽ⁿ⁾)；Y is the m rank column vectors that desired value is lined up, and is denoted as (y⁽¹⁾…y⁽ⁱ⁾…y⁽ⁿ⁾)^T；The θ for finally giving is the matrix of 3 × 3, takes three elements from top to bottom in first row in θ Respectively as coefficient before the constant term in quadratic regression curve, coefficient before coefficient and quadratic term before first order, for each ginseng Examination point xck^(j), generation returns regression curve its corresponding regressand value yck^(j), wherein j is the number footmark with reference to point data, this Sample just forms a regression point (xck^(j),yck^(j))；

Adjacent regression point is carried out into the regression curve that linear interpolation just obtains returning whole data and curves.

The flow of hop value is recognized and rejected using local weighted recurrence：

A. former data and curves are carried out into local weighted recurrence, generates matched curve；

B. former data are subtracted into matched curve and obtains residual error curve；

C. the average value and standard deviation of residual error curve are sought；

D. all of residual error data is traveled through, using Pauta criterion, all data beyond limitation is picked out：

E. the label of the data chosen in d is obtained, and the former data in correspondence label is substituted for saltus step data two ends Interpolation between normal data, reaches smooth purpose.

The features of the present invention and beneficial effect are：

The present invention has principle simple, the characteristics of calculate quick and effect is significant.The invention is to IAQ (IAQ) There is good beneficial effect with the data of time：The present invention effectively can isolate noise from former data；Can pass through Analyze noise feature is removed to the saltus step exceptional value in data, and ensure that the data related to people's Behavioral change Significantly change and be not identified as exceptional value and be removed.

Brief description of the drawings：

Fig. 1：Office uses the actual measurement room air data of sensor one, and transverse axis is the time, in units of 1 second； The longitudinal axis is corresponding numerical value intensity.1-2 and 1-3 have the data jump of needle-like in figure.

Fig. 2：Actual measurement formaldehyde with data vacancy and data jump changes with time data and curves.

Fig. 3 data smoothing flow charts.

Fig. 4：The actual measurement formaldehyde that data vacancy is interpolated changes with time data and curves.

Fig. 5：The actual measurement formaldehyde after local weighted recurrence is carried out to Fig. 1 curves to change with time data matched curve

Fig. 6：Formaldehyde is changed with time the residual error curve of data, it is seen that data jump wherein, and is not contained and any had Use information.

Fig. 7：The actual measurement formaldehyde that data jump is smoothed changes with time data and curves, it is seen that useful information is all retained Get off.

Specific embodiment

In order to prevent complexity and the amplification of error of algorithm, the Interior Space destiny to short time data vacancy may be carried During according to being pre-processed, filling up for data vacancy long in short-term is carried out first, it is to be ensured that whole data in the absence of vacancy 0 value, so Carry out the removal of data exception hop value again afterwards, when ensureing no longer to have the data jump point of needle-like, then carry out zero migration Correction, the data that will be handled well are updated in calibration curve.

1) in initial data, every vacancy value has all been replaced with 0.

Select the time series data of n numerical value composition first, n needs to be easy divided evenly number here, such as 2000, this is The removal of data jump value can effectively be carried out.Then initial data is carried out into filling up for vacancy value, and according in (three) The data character mentioned, herein from interpolation is carried out to data, result is replaced 0 value of correspondence position.

2) when ensuring there is no 0 value in whole data, start to be removed hop value.

It is noted here that removal will can not be significantly changed because of the data of people's Behavioral change, with data vacancy, number According to saltus step and because of the actual measurement formaldehyde data instance (such as Fig. 2) that the data of people's Behavioral change significantly change, data jump is found There is very big difference with the data movement because of people's Behavioral change：

Such as right side dashed curve part in Fig. 2, corresponding moment herein, office door is opened so that the experiment of opposite house Experiment formaldehyde gas part inside room pours in office, result in indoor formaldehyde concentration rising, and this belongs to typical people's behavior Change.The change of this concentration of formaldehyde can not be taken as outlier identification and reject.Secondly, the part of the red circle of solid line in left side is Shortage of data, is 0 value.

Herein, it is not suitable for that data are directly carried out the rejecting of exceptional value, one side data entirety tendency takes on slow Slow variation, the 0.047 of right side is changed to from the 0.07 of left side, and on the other hand the curve of the exactly right side red circle of dotted line is easy to be worked as Make exceptional value, these are all the significant information in data and curves.It is therefore desirable to by significant information in data and curves Retain, so suitable strategy is exactly to fit significant information to come using matched curve, while not being fitted needle-like Data jump and all of high-frequency noise, are had from local weighted recurrence (Local Weight Regression) here With the fitting of information, then subtract matched curve with former data and curves and obtain noise curve, just can be fully solved useful information pair The interference of hop value removal.

3) local weighted Regression

Local weighted recurrence is the modified version that general linear is returned, and can overcome the defect of the latter's poor fitting or over-fitting. Its step of is first to be separated whole data etc. with the reference point on the transverse axis of certain amount, and centered on these points respectively Ask calculation line locality to return, when regression parameter is solved using least square method, from central point more away from data point shared by power Number is smaller.The recurrence numerical value of these points can be finally obtained, these then are returned into numerical point with interpolation is connected, used here as line Property interpolation.

To each training data point, will cause：

∑_iw⁽ⁱ⁾(y⁽ⁱ⁾-θ^Tx⁽ⁱ⁾)² (1)

It is minimum.

Wherein i is the number footmark of training data；X is characteristic value, and the time value of time shaft is referred to herein；Y is mesh Scale value, is herein concentration of formaldehyde value；θ is the coefficient vector of regression equation, and this method uses quadratic regression, therefore θ is a three-dimensional Vector；W is Gauss flexible strategy, is expressed as：

The reference point on selected transverse axis is referred to without superscript x, τ is bandwidth (bandwidth), and τ is bigger, The intensity of local regression is bigger.

Local weighted to return many Gauss power before each residuals squares, this will greatly weaken from reference point ratio Influence of the data farther out to being fitted, and then reach the purpose of local regression.The recurrence for requiring secondary to each reference point Curve, and parameter of curve must be different, to any one reference point x, have：

θ=(X^TWX)^-1X^TWy (3)

Wherein, X is by 1, x⁽ⁱ⁾, (x⁽ⁱ⁾)²The m of composition ties up matrix, and referred to as design matrix (design matrix) m is Training data quantity, and X writings (now in order to seek quadratic regression curve, so matrix only has three row, if asking n regression curve, square Battle array just has n+1 to arrange)：

W is m rank diagonal matrix, writing diag (w⁽¹⁾…w⁽ⁱ⁾…w⁽ⁿ⁾)；Y is the m rank column vectors that desired value is lined up, and is denoted as (y⁽¹⁾…y⁽ⁱ⁾…y⁽ⁿ⁾)^T；The θ for finally giving is the matrix of 3 × 3, takes three elements from top to bottom in first row in θ Respectively as coefficient before the constant term in quadratic regression curve, coefficient before coefficient and quadratic term before first order.For each ginseng Examination point xck^(j), generation returns regression curve its corresponding regressand value yck^(j), wherein j is the number footmark with reference to point data, this Sample just forms a regression point (xck^(j),yck^(j))。

Adjacent regression point is carried out into linear interpolation just can more accurately obtain returning whole data and curves Regression curve, such curve can retain all of useful information.

4) the local weighted flow for returning and recognizing and reject hop value is used

A. former data and curves are carried out into local weighted recurrence, generates matched curve.

B. former data are subtracted into matched curve and obtains residual error curve.

C. the average value and standard deviation of residual error curve are sought.

D. all of residual error data is traveled through, using Pauta criterion, all data beyond limitation is picked out：

Pauta criterion specifies：Data within the scope of all standard deviations beyond three times are all removed.I.e.：Remove all of X, if x meets | x- μ |<3σ.Because the data volume of disposable treatment is larger, exceed well under the use data of Pauta criterion Limit：100 data.So taking relatively simple Pauta criterion here.

E. the label of the data chosen in d is obtained, and the former data (saltus step data) in correspondence label is substituted for saltus step Interpolation between the normal data at data two ends, reaches smooth purpose.

5) after being recognized with local weighted recurrence and rejecting hop value, data are updated to returning for demarcation after the treatment that will be obtained Return in equation, and then obtain complete pre-processed results data.

1) name of SQL, input variable and output variable.

Input variable has two：One is " initial data ", is imported from excel forms, is existed in the form of column vector； Another is " to the segmentation space-number of time dimension ".

Output variable has one：It is " by the result data after abnormal data smoothing algorithm ", and fills out back excel tables Lattice.

2) former data are carried out into the smooth flow of the room air data outliers based on local weighted recurrence (in former data Can not have

Vacancy value for 0), as shown in Figure 3.

Here the actual measurement formaldehyde after being padded changes with time data and curves as example (see Fig. 4)：Take containing 1500 original data vectors of data, " to the segmentation space-number of time dimension " takes 50；Fig. 5 gives the office to Fig. 4 curves The matched curve of portion's weighted regression, it can be seen that, all of useful information all has been retained, and it is bent that Fig. 6 gives Fig. 4 data Line subtracts the residual error curve that Fig. 5 data and curves are obtained, also referred to as noise curve, it can be seen that, the data jump of needle-like is all by from original Separated in data out, residual error curve has been judged according to Pauta criterion, removed and substituted with interpolation the jump for not meeting judgement Become data, then it is added with the matched curve of Fig. 5, obtain the data and curves (such as Fig. 7) after smooth by hop value.

Bibliography：

[1]Salvador García,Julian Luengo,Tutorial on practical tips of the most influential data preprocessing algorithms in data mining.Knowledge-Based Systems,2016；98:1-29..

[2]J.R.Quinlan,C4.5:Programs for Machine Learning,Morgan Kaufmann Pub-lishers Inc.,1993.[3]Hao Zhou；Lifeng Qiao,Ph.D.；Yi Jiang,Ph.D.；Hejiang Sun,Ph.D.；Qingyan Chen,Ph.D.Recognition of air-conditioner operation from indoor air temperature and relative humidity by a data mining approach.Energy and Buildings,2016；111:233-241.

[4]C.E.Brodley,M.A.Friedl,Identifying mislabeled training data, J.Artif.Intell.Res.1999；11:131–167.

[5]T.M.Khoshgoftaar,P.Rebours,Improving software quality prediction by noise filtering techniques,J.Comput.Sci.Technol.2007；22:387–396.

Claims (4)

1. a kind of room air data preprocessing method based on local weighted recurrence, it is characterized in that, long number in short-term is carried out first According to filling up for vacancy, it is to be ensured that then whole data carry out the removal of data exception hop value again in the absence of 0 value of vacancy, When no longer there is the data jump point of needle-like in guarantee, then carry out the correction of zero migration, and the data that will be handled well are updated to mark In determining curve.

2. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 1, it is characterized in that, carry out The removal of data exception hop value comprises the concrete steps that, fits significant information to come using matched curve, while not Fitting needle-like data jump and all of high-frequency noise, specifically from local weighted recurrence (Local Weight Regression the fitting of useful information) is carried out, then matched curve is subtracted with former data and curves to obtain noise curve, solve useful The interference that information is removed to hop value.

3. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 2, it is characterized in that, it is local Weighted regression principle comprises the concrete steps that, first separated whole data etc. with the reference point on the transverse axis of certain amount, and with this Ask calculation line locality to return centered on a little points respectively, when regression parameter is solved using least square method, from central point more away from Flexible strategy shared by data point are smaller, finally obtain the recurrence numerical value of these points, and these then are returned into numerical point with interpolation is connected, Used here as linear interpolation；

Further, to each training data point, will cause：

∑_iw⁽ⁱ⁾(y⁽ⁱ⁾-θ^Tx⁽ⁱ⁾)² (1)

It is minimum；

Wherein i is the number footmark of training data；X refers to the time value of time shaft；Y is desired value；θ be regression equation coefficient to Amount, using quadratic regression, therefore θ is a three-dimensional vector；W is Gauss flexible strategy, is expressed as：

$w^{\left(i\right)}=\exp (-\frac{{(x^{\left(i\right)}-x)}^2}{2{\mathrm{\&tau;}}^2})---\left(2\right)$

The reference point on selected transverse axis is referred to without superscript x, τ is bandwidth (bandwidth), and τ is bigger, it is local The intensity of recurrence is bigger；

Local weighted to return many Gauss power before each residuals squares, require secondary to each reference point returns Return curve, and parameter of curve must be different, to any one reference point x, have：

θ=(X^TWX)^-1X^TWy (3)

Wherein, X is by 1, x⁽ⁱ⁾, (x⁽ⁱ⁾)²The m dimension matrixes of composition, referred to as design matrix (design matrix) m is training number Data bulk, X writings：

$X=\left(\begin{array}{ccc}1& x^{\left(1\right)}& {\left(x^{\left(1\right)}\right)}^2\\ .& .& .\\ .& .& .\\ .& .& .\\ 1& x^{\left(i\right)}& {\left(x^{\left(i\right)}\right)}^2\\ .& .& .\\ .& .& .\\ .& .& .\\ 1& x^{\left(m\right)}& {\left(x^{\left(m\right)}\right)}^2\end{array}\right)---\left(4\right)$

W is m rank diagonal matrix, writing diag (w⁽¹⁾…w⁽ⁱ⁾…w⁽ⁿ⁾)；Y is the m rank column vectors that desired value is lined up, and is denoted as (y⁽¹⁾…y⁽ⁱ⁾…y⁽ⁿ⁾)^T；The θ for finally giving is the matrix of 3 × 3, takes three elements point from top to bottom in first row in θ Not as coefficient before the constant term in quadratic regression curve, coefficient before coefficient and quadratic term before first order, for each reference Point xck^(j), generation returns regression curve its corresponding regressand value yck^(j), wherein j is the number footmark with reference to point data, so Just a regression point (xck is formed^(j),yck^(j))；

By returning that adjacent regression point carries out that linear interpolation just can more accurately obtain returning whole data and curves Return curve.

4. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 2, it is characterized in that, use Local weighted recurrence is recognized and rejects the flow of hop value：

A. former data and curves are carried out into local weighted recurrence, generates matched curve；

B. former data are subtracted into matched curve and obtains residual error curve；

C. the average value and standard deviation of residual error curve are sought；

D. all of residual error data is traveled through, using Pauta criterion, all data beyond limitation is picked out：

E. the label of the data chosen in d is obtained, and the former data in correspondence label is substituted for the normal of saltus step data two ends Interpolation between data, reaches smooth purpose.