CN101634991A - Telemetering data cleaning method for data acquisition and monitoring control system - Google Patents

Abstract

The telemetering data cleaning method for a data acquisition and monitoring control system relates to the technical field of data processing, and solves the technical problem of data quality improvement. The method is characterized in that a pattern list is set up at first, then real-time data is cleaned according to the pattern list, wherein the establishment of the pattern list comprises the following steps: the historical data over one month is grouped according to acquisition time, the arithmetic mean value of each group is calculated, then the historical data is revised according to an admission relation, the arithmetic mean value of each group is calculated again, and finally, a pattern list is set up according to the evaluated arithmetic mean value of each group; the cleaning of real time data comprises the following steps: finding a pattern value matched with the real-time data in the pattern list according to acquisition time, and then carrying out error judgment on the real-time data through the pattern value and maximal permissible error value so as to directly remove polluted data and provide reference data. By using the telemetering data cleaning method of the invention, higher data quality can be guaranteed.

Description

The telemetry cleaning method of data acquisition and supervisor control

Technical field

The present invention relates to data processing technique, particularly relate to the technology of the telemetry cleaning method of a kind of data acquisition that is used for electric system and supervisor control (SCADA).

Background technology

In the data acquisition and supervisor control of electric system, all set up data warehouse according to the real-time telemetry data of being gathered, and with this data source as most of real-time analysis software in the system.Because the load data of electric system is the nonlinear function of time, be difficult to express with a general nonlinear function, simultaneously owing to reasons such as harvester, data channel, in gatherer process, may produce Acquisition Error, data are polluted, and the data of this pollution may impact real-time analysis software, cause inaccurate analysis result, therefore in setting up process data warehouse, effectively clear up with the maturity that guarantees data significant to data.

Present data acquisition and supervisor control are all cleaned telemetry by the quality sign indicating number of analyzing institute's image data.This cleaning method is not considered property time correlation of data, so its cleaning performance is very poor, and the quality of data is lower.

Summary of the invention

At the defective that exists in the above-mentioned prior art, it is good that technical matters to be solved by this invention provides a kind of cleaning performance, the telemetry cleaning method of data acquisition that the quality of data is high and supervisor control.

In order to solve the problems of the technologies described above, the telemetry cleaning method of a kind of data acquisition provided by the present invention and supervisor control is characterized in that, sets up pattern list earlier, cleans real time data according to pattern list again;

Wherein, it is as follows to set up the concrete steps of pattern list:

1) every certain interval of time data acquisition that takes remote measurement, and the continuous acquisition time more than one month, with the data of gathering at every turn and gather the date, acquisition time deposits tables of data DTable in;

2) be that a basic point is divided into 1440 time points with acquisition time with 1 minute, according to 1440 time points the data of DTable are divided into 1440 groupings, and to each data of DTable form unique coordinate (d, i), wherein d is the collection date of these data, and i is the grouping under these data;

3) data of each grouping are asked arithmetic mean, the arithmetic mean that i is organized is designated as Ei, wherein 1440＞=i＞0;

4) each data with DTable are indexed to its corresponding arithmetic mean automatically by its acquisition time;

5) set 3 times the relation of allowing, each data of DTable are revised, its modification rule is:

$\left\{\begin{array}{cc}\mathrm{DTable}(d,i)=\mathrm{Ei}& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)>3\&times;\mathrm{Ei})\\ \mathrm{DTable}(d,i)=\mathrm{DTable}(d,i)& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)<=3\&times;\mathrm{Ei})\end{array}\right.;$

6) according to correction result the data of each grouping are asked arithmetic mean once more, make the arithmetic mean of each grouping be updated to the new value of trying to achieve;

7) set up pattern list ModelTable, the arithmetic mean of 1440 groupings that step 6 is tried to achieve is as mode value, and deposits pattern list ModelTable in according to time sequencing;

Wherein, the concrete steps of cleaning real time data are as follows:

1), be that a basic point is set up index value for each real time data Real (i) with 1 minute, and, find the mode value ModelTable (i) that is complementary with it that error of calculation Eri is according to its index value indexing model table ModelTable according to acquisition time:

Eri＝(real(i)-ModelTable(i))/ModelTable(i)；

2) each real time data Real (i) is carried out error judgment, and clean, its cleaning rule is:

$\left\{\begin{array}{cc}\mathrm{real}\left(i\right)=\mathrm{Ei}& \mathrm{if}(\mathrm{Ei}>E\max )\\ \mathrm{real}\left(i\right)=\mathrm{real}\left(i\right)& \mathrm{if}(\mathrm{Ei}<=E\max )\end{array}\right.;$

Wherein, Emax is the maximum error permissible value;

3) data that will clean are saved to data warehouse.

The telemetry cleaning method of data acquisition provided by the invention and supervisor control, according to acquisition time historical data is divided into groups earlier, and set up the high pattern list of the quality of data according to the arithmetic mean of each grouping, utilize pattern list and maximum error permissible value that real time data is cleaned again, directly reject contaminated telemetry, and providing reference data, its cleaning performance is good, can guarantee the higher data quality.

Description of drawings

Fig. 1 is the process flow diagram that the method for the embodiment of the invention is set up pattern list;

Fig. 2 is that the method for the embodiment of the invention is cleaned the process flow diagram of real time data.

Embodiment

Below in conjunction with description of drawings embodiments of the invention are described in further detail, but present embodiment is not limited to the present invention, every employing analog structure of the present invention and similar variation thereof all should be listed protection scope of the present invention in.

A kind of data acquisition that the embodiment of the invention provided and the telemetry cleaning method of supervisor control is characterized in that, set up pattern list earlier, clean real time data according to pattern list again;

Wherein, set up the concrete steps following (referring to Fig. 1) of pattern list:

1) every certain interval of time data acquisition that takes remote measurement, and the continuous acquisition time more than one month, with the data of gathering at every turn and gather the date, acquisition time deposits tables of data DTable in;

2) be that a basic point is divided into 1440 time points with acquisition time with 1 minute, according to 1440 time points the data of DTable are divided into 1440 groupings, and to each data of DTable form unique coordinate (d, i), wherein d is the collection date of these data, and i is the grouping under these data;

3) data of each grouping are asked arithmetic mean, the arithmetic mean that i is organized is designated as Ei, wherein 1440＞=i＞0;

4) each data with DTable are indexed to its corresponding arithmetic mean automatically by its acquisition time;

5) set 3 times the relation of allowing, each data of DTable are revised, its modification rule is:

$\left\{\begin{array}{cc}\mathrm{DTable}(d,i)=\mathrm{Ei}& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)>3\&times;\mathrm{Ei})\\ \mathrm{DTable}(d,i)=\mathrm{DTable}(d,i)& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)<=3\&times;\mathrm{Ei})\end{array}\right.;$

6) according to correction result the data of each grouping are asked arithmetic mean once more, make the arithmetic mean of each grouping be updated to the new value of trying to achieve;

7) set up pattern list ModelTable, the arithmetic mean of 1440 groupings that step 6 is tried to achieve is as mode value, and deposits pattern list ModelTable in according to time sequencing;

Wherein, clean the concrete steps following (referring to Fig. 2) of real time data:

1), be that a basic point is set up index value for each real time data Real (i) with 1 minute, and, find the mode value ModelTable (i) that is complementary with it that error of calculation Eri is according to its index value indexing model table ModelTable according to acquisition time:

Eri＝(real(i)-ModelTable(i))/ModelTable(i)；

2) each real time data Real (i) is carried out error judgment, and clean, its cleaning rule is:

$\left\{\begin{array}{cc}\mathrm{real}\left(i\right)=\mathrm{Ei}& \mathrm{if}(\mathrm{Ei}>E\max )\\ \mathrm{real}\left(i\right)=\mathrm{real}\left(i\right)& \mathrm{if}(\mathrm{Ei}<=E\max )\end{array}\right.;$

Wherein, Emax is the maximum error permissible value;

3) data that will clean are saved to data warehouse.

Claims (1)

1, the telemetry cleaning method of a kind of data acquisition and supervisor control is characterized in that, sets up pattern list earlier, cleans real time data according to pattern list again;

Wherein, it is as follows to set up the concrete steps of pattern list:

1) every certain interval of time data acquisition that takes remote measurement, and the continuous acquisition time more than one month, with the data of gathering at every turn and gather the date, acquisition time deposits tables of data DTable in;

2) be that a basic point is divided into 1440 time points with acquisition time with 1 minute, according to 1440 time points the data of DTable are divided into 1440 groupings, and to each data of DTable form unique coordinate (d, i), wherein d is the collection date of these data, and i is the grouping under these data;

3) data of each grouping are asked arithmetic mean, the arithmetic mean that i is organized is designated as Ei, wherein 1440＞=i＞0;

4) each data with DTable are indexed to its corresponding arithmetic mean automatically by its acquisition time;

5) set 3 times the relation of allowing, each data of DTable are revised, its modification rule is:

$\left\{\begin{array}{cc}\mathrm{DTable}(d,i)=\mathrm{Ei}& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)>3\&times;\mathrm{Ei})\\ \mathrm{DTable}(d,i)=\mathrm{DTable}(d,i)& \mathrm{if}(\mathrm{fabs}\left(\mathrm{DTable}(d,i)\right)<=3\&times;\mathrm{Ei})\end{array}\right.;$

6) according to correction result the data of each grouping are asked arithmetic mean once more, make the arithmetic mean of each grouping be updated to the new value of trying to achieve;

7) set up pattern list ModelTable, the arithmetic mean of 1440 groupings that step 6 is tried to achieve is as mode value, and deposits pattern list ModelTable in according to time sequencing;

Wherein, the concrete steps of cleaning real time data are as follows:

1), be that a basic point is set up index value for each real time data Real (i) with 1 minute, and, find the mode value ModelTable (i) that is complementary with it that error of calculation Eri is according to its index value indexing model table ModelTable according to acquisition time:

Eri＝(real(i)-ModelTable(i))/ModelTable(i)；

2) each real time data Real (i) is carried out error judgment, and clean, its cleaning rule is:

$\left\{\begin{array}{cc}\mathrm{real}\left(i\right)=\mathrm{Ei}& \mathrm{if}(\mathrm{Ei}>E\max )\\ \mathrm{real}\left(i\right)=\mathrm{real}\left(i\right)& \mathrm{if}(\mathrm{Ei}<=E\max )\end{array}\right.;$

Wherein, Emax is the maximum error permissible value;

3) data that will clean are saved to data warehouse.

Images