CN115017147A - Method and device for processing dynamic measurement data - Google Patents
Method and device for processing dynamic measurement data Download PDFInfo
- Publication number
- CN115017147A CN115017147A CN202210752869.6A CN202210752869A CN115017147A CN 115017147 A CN115017147 A CN 115017147A CN 202210752869 A CN202210752869 A CN 202210752869A CN 115017147 A CN115017147 A CN 115017147A
- Authority
- CN
- China
- Prior art keywords
- data
- dynamic measurement
- measurement data
- value
- error
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005259 measurement Methods 0.000 title claims abstract description 225
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012545 processing Methods 0.000 title claims abstract description 46
- 230000009897 systematic effect Effects 0.000 claims abstract description 22
- 230000008569 process Effects 0.000 claims abstract description 17
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 238000004458 analytical method Methods 0.000 claims abstract description 8
- 230000000737 periodic effect Effects 0.000 claims description 33
- 238000010276 construction Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 4
- 238000004140 cleaning Methods 0.000 abstract description 6
- 238000000926 separation method Methods 0.000 abstract description 5
- 238000012937 correction Methods 0.000 abstract description 2
- 238000011160 research Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Water Supply & Treatment (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Public Health (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Complex Calculations (AREA)
Abstract
The application discloses a method and a device for processing dynamic measurement data, wherein the method comprises the following steps: acquiring original data from a new energy cloud database, wherein the original data are all data acquired in the dynamic measurement process; preprocessing the original data to obtain dynamic measurement data; establishing a combined model of the dynamic measurement data, wherein the combined model is used for carrying out characteristic analysis on the dynamic measurement data so as to obtain a system error and a random error of the dynamic measurement data; separating data containing the system errors in the dynamic measurement data according to the system errors; separating data containing random errors in the dynamic measurement data according to the random errors; and according to the separated data containing the system error and the random error, obtaining the true value of the dynamic measurement data from the dynamic measurement data. By using the scheme provided by the embodiment of the application, data cleaning can be realized by bad data positioning and error correction and systematic and random error separation, and high-quality dynamic measurement data can be obtained.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing dynamic measurement data.
Background
At present, the data in the new energy industry is huge in quantity, various in types and complex in composition, and more importantly, the data quality is good and uneven, and the phenomena of data loss, abnormality, confusion, repetition and the like are obvious, so that the problems that the data value density of national grid absorption clouds is low, the data quality is to be improved, the data island and the potential value of the data are not fully mined and the like are directly caused to be prominent.
However, the currently adopted "data cleaning" is to remove null data and data which does not meet the validity requirement before data introduction, still belongs to the category of data screening, is not data cleaning in the true sense, and cannot obtain high-quality dynamic measurement data which is enough for research.
Disclosure of Invention
Based on the above problems, the present application provides a method and an apparatus for processing dynamic measurement data, which implement data cleaning by bad data positioning and error correction, systematic and random error separation.
One aspect of the present application provides a method for processing dynamic measurement data, including:
acquiring original data from a new energy cloud database, wherein the original data are all data acquired in the dynamic measurement process;
preprocessing the raw data to obtain dynamic measurement data, wherein the preprocessing at least comprises one of the following steps: data truncation, discretization, initial identification statistical property and point picking processing;
establishing a combined model of the dynamic measurement data, wherein the combined model is used for carrying out characteristic analysis on the dynamic measurement data to obtain a system error and a random error of the dynamic measurement data;
separating data containing system errors in the dynamic measurement data according to the system errors output by the combined model;
separating data containing random errors in the dynamic measurement data according to the random errors output by the combined model;
and obtaining the real value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the system error and the separated data containing the random error.
Preferably, the data truncation is performed on the original data, and includes:
truncating the original data according to a preset truncation length to obtain the dynamic measurement data;
wherein, the interception length is the maximum data length required to be measured in one or more dynamic measurement processes.
Preferably, the performing a checkpoint rejecting process on the original data includes:
constructing a first median sequence according to the original data;
forming a second median sequence of three data adjacent to the first median sequence according to the first median sequence;
constructing a third sequence according to a second median sequence of the adjacent three data, wherein the construction method of the third sequence is shown by the following formula:
wherein, { x' i Is the third sequence, { x ″ ] i Is a second median sequence;
calculating an absolute value of a difference obtained by subtracting a first value in the third sequence from a first value in the first median sequence to obtain a difference value, and judging whether the difference value is greater than a preset threshold value;
if the difference value is larger than a preset threshold value, removing a first value in the first median sequence from the original data, calculating a smooth linear interpolation according to adjacent data of the first value in the first median sequence, and replacing the first value in the first median sequence with the linear interpolation.
Preferably, the combined model of the dynamic measurement data includes: deterministic function f (n), random function y (n);
performing feature extraction on the dynamic measurement data X (n) by the following formula;
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)；
the random function Y (n) is used for calculating errors of external factors on data measurement; the deterministic function f (n) is divided into a non-periodic function d (n) and a periodic function p (n), the periodic function p (n) is used for calculating regularly fluctuating data in the dynamic measurement data, and the non-periodic function d (n) is used for calculating irregularly fluctuating data in the dynamic measurement data;
the dynamic measurement data X (n) includes the true value X of the dynamic measurement data 0 (n) and the measurement error e (n), the true value X of the dynamic measurement data 0 (n) a corresponding true value f comprising said deterministic function f (n) 0 (n) and the true value Y of said randomness function 0 (n); the measurement error e (n) is represented by a system error e s (n) and random error e r (n) composition, represented by the formula:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；
wherein d is 0 (n) the true value of the non-periodic function; p is a radical of 0 (n) is the true value of the periodic function.
Preferably, the combined model of the dynamic measurement data is further used for calculating a system error, in generalThe method is realized by the following formula: e [ E (n)]＝e s (n) wherein E [ E (n)]Is the expected value of the measurement error e (n);
the systematic error is used to separate data containing the systematic error from the dynamic measurement data.
Preferably, the combined model of the dynamic measurement data is further used for calculating a random error, which is implemented by the following formula:
e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]；
wherein E [ x (n) ] is calculated by the following formula:
X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n)；
wherein, E [ X ] 0 (n)+e(n)]For the true value X of the dynamic measurement data 0 (n) a desired value of the sum of said measurement error e (n);
the random error is used for separating data containing the random error in the dynamic measurement data.
Preferably, the combined model of the dynamic measurement data is further used for calculating a variance value of the random error, and is implemented by the following formula:
One aspect of the present application provides a device for processing dynamic measurement data, the device including: the system comprises an acquisition module, a processing module and a model establishing module;
the acquisition module is used for acquiring original data from a new energy cloud database, wherein the original data are all data acquired by dynamic measurement;
the processing module is configured to perform preprocessing on the raw data to obtain dynamic measurement data, where the preprocessing at least includes one of: data truncation, discretization, initial identification statistical property and point picking processing;
the model establishing module is used for establishing a combined model of the dynamic measurement data, and the combined model is used for performing characteristic analysis on the dynamic measurement data to obtain a system error and a random error of the dynamic measurement data;
the processing module is further configured to separate data including a system error from the dynamic measurement data according to the system error output by the combined model;
the processing module is further configured to separate data including random errors from the dynamic measurement data according to the random errors output by the combination model;
and the processing module is further used for obtaining a true value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the system error and the separated data containing the random error.
Preferably, the processing module is further configured to:
truncating the original data according to a preset truncation length to obtain the dynamic measurement data;
wherein the truncation length is the maximum data length required to be measured in one or more dynamic measurement processes.
Preferably, the processing module is further configured to:
constructing a first median sequence according to the original data;
forming a second median sequence of three data adjacent to the first median sequence according to the first median sequence;
constructing a third sequence according to a second median sequence of the three adjacent data, wherein the construction method of the third sequence is shown by the following formula:
wherein, { x' i Is the third sequence, { x ″ ] i Is a second median sequence;
calculating an absolute value of a difference obtained by subtracting a first value in the third sequence from a first value in the first median sequence to obtain a difference value, and judging whether the difference value is greater than a preset threshold value;
if the difference value is larger than a preset threshold value, removing a first value in the first median sequence from the original data, calculating a smooth linear interpolation according to adjacent data of the first value in the first median sequence, and replacing the first value in the first median sequence with the linear interpolation.
Preferably, the model building module is specifically configured to build a combined model of the dynamic measurement data, and includes: deterministic function f (n), random function y (n);
performing feature extraction on the dynamic measurement data X (n) by the following formula;
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)；
the random function Y (n) is used for calculating errors of external factors on data measurement; the deterministic function f (n) is divided into a non-periodic function d (n) and a periodic function p (n), the periodic function p (n) is used for calculating regularly fluctuating data in the dynamic measurement data, and the non-periodic function d (n) is used for calculating irregularly fluctuating data in the dynamic measurement data;
the dynamic measurement data X (n) includes the true value X of the dynamic measurement data 0 (n) and the measurement error e (n), the true value X of the dynamic measurement data 0 (n) a corresponding true value f comprising said deterministic function f (n) 0 (n) the true value Y corresponding to said randomness function Y (n) 0 (n); the measurement error e (n) is represented by a system error e s (n) and random error e r (n) composition, represented by the formula:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；
wherein d is 0 (n) the true value of the non-periodic function; p is a radical of 0 (n) is the true value of the periodic function.
Preferably, the model building module further comprises a calculating module, which is used for calculating the system error, and is implemented by the following formula: e [ E (n)]＝e s (n) wherein E [ E (n)]Is the expected value of the measurement error e (n).
Preferably, the calculating module is further configured to calculate a random error, which is implemented by the following formula:
e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]；
wherein E [ x (n) ] is calculated by the following formula:
X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n)；
wherein, E [ X ] 0 (n)+e(n)]For the true value X of the dynamic measurement data 0 (n) and the sum of the measurement errors e (n).
Preferably, the calculating module is further configured to calculate a variance value of the random error, which is implemented by the following formula:
This application is through avoiding introducing bad data, rejecting the value that contains the data of gross error and calculation system error and random error, and the separation contains the data of system error and random error, realizes rinsing dynamic measurement data's data, can obtain high-quality dynamic measurement data.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of a method for processing dynamic measurement data according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of an exemplary scenario of a method for processing dynamic measurement data according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a device for processing dynamic measurement data according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present application will now be described with reference to the accompanying drawings, and it is to be understood that the described embodiments are merely illustrative of some, but not all, embodiments of the present application. As can be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.
The embodiment of the application provides a method for processing dynamic measurement data, which is used for realizing data cleaning of original dynamic measurement data. The following are detailed below.
Referring to fig. 1, the figure is a schematic flowchart of a processing method for dynamic measurement data according to an embodiment of the present application. The processing method of dynamic measurement data provided by the embodiment of the present application can be implemented, for example, by the following steps S101 to S106.
S101: and acquiring original data from the new energy cloud database.
The method comprises the steps that original data are obtained from a new energy cloud database, and the data are not processed, so that the problems of excessive original data, gross errors, system errors, random errors and the like exist.
Specifically, all data acquired in the dynamic measurement process of the original data have problems of excessive data and the like, and accurate data which can be used for scientific research can be acquired only by performing data cleaning.
S102: and preprocessing the original data to obtain dynamic measurement data.
In the original data, there are obvious problems of data redundancy, large error of part of data, etc., and the original data needs to be preprocessed, and the data with small error of part is reserved.
Specifically, the preprocessing includes data truncation, discretization, initial identification statistical characteristics, point picking processing and the like.
Specifically, the data truncation of the original data includes:
truncating the original data according to a preset truncation length to obtain dynamic measurement data;
the interception length is the maximum data length required to be measured in one or more dynamic measurement processes.
By adopting data truncation, the problems of excessive original data and introduction of bad data can be avoided, and the problem of original data redundancy is solved.
Specifically, the method for eliminating points of the original data comprises the following steps:
first from the original data { x i A first sequence of median { x' i }; take { x i The first 5 in the sequence x 1 、x 2 、x 3 、x 4 、x 5 Rearranging to x by magnitude (1) ≤x (2) ≤x (3) ≤x (4) ≤x (5) Taking the median digit x (3) Is recorded as x' 3 Then x is truncated 1 Adding x 6 Taking x 2 、x 3 、x 4 、x 5 、x 6 Of median x' 4 . And so on to obtain N-3 median, and finally forming a first median sequence { x 'of adjacent 5 original data' i }(i＝3，4，3，N-1)。
According to the above method, a first sequence of median { x' i Form a second median sequence { x ″ } i }(i＝4，5，…，N-2)。
Specifically, the second median sequence is formed by three adjacent data of the first median sequence.
According to a second sequence of median { x ″) i Construction of the third sequence { x' i { x' i The construction method of the third sequence is shown by the following formula:
the value Δ is preset as required H If | x i -x″′ i |＞Δ H If x is to be eliminated i And calculating a coupled smoothed linear interpolation from the neighboring data, replacing x with the linear interpolation i 。
And the effect of eliminating data containing large errors can be achieved by adopting point eliminating processing.
Specifically, the initial identification statistical characteristics can preliminarily identify the statistical characteristics (independence, stationarity, ergodicity and the like) of the random data and the change rule (linearity, periodicity and the like) of the deterministic component, and can establish a combined model representing data composition according to the statistical characteristics in the follow-up process.
S103: and establishing a combined model of the dynamic measurement data.
The dynamic measurement data x (n) is composed of a deterministic function f (n) and a stochastic function y (n). The deterministic function f (n) is further divided into a non-periodic function d (n) and a periodic function p (n), namely:
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)
specifically, the random function y (n) is used for calculating an error generated by an external factor on data measurement, the periodic function p (n) is used for calculating regularly fluctuating data in dynamic measurement data, and the aperiodic function d (n) is used for calculating irregularly fluctuating data in the dynamic measurement data;
in particular, the dynamic measurement data X (n) can be further divided into the true value X of the measured variable 0 (n) and the error e (n) (hereinafter, the actual value is represented by subscript 0), and the actual value X 0 (n) true values f corresponding to deterministic functions f (n) 0 (n) true value Y corresponding to randomness function Y (n) 0 (n) error e (n) is composed of systematic error e s (n) and random error e r (n) composition, i.e.:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；
wherein d is 0 (n) and p 0 (n) true values f which are respectively deterministic components 0 The non-periodic component and the periodic component of (n).
Specifically, the combined model of the dynamic measurement data is further used for calculating a system error e (n), and is implemented by the following formula: e [ E (n)]＝e s (n) wherein E [ E (n)]Is the expected value of the measurement error e (n).
The systematic error is used to separate data including the systematic error e (n) from the dynamic measurement data.
In particular, the combined model of the dynamic measurement data is also used for calculating the random error e r (n) by the following formula:
e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]；
wherein E [ x (n) ] is calculated by the formula:
X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n)；
wherein, E [ X ] 0 (n)+e(n)]For the actual value X of the dynamic measurement data 0 (n) a desired value of the sum of (n) and the measurement error e (n);
the random error is used for separating the dynamic measurement data and contains the random error e r (n) of (d).
In particular, the combined model of the dynamic measurement data is also used for calculating the random error e r The variance value of (n) is realized by the following formula:
The variance of the random error is an evaluation standard of the random error and is used for judging whether the obtained data is stable or not.
Establishing a combination model of the dynamic measurement data for performing characteristic analysis on the dynamic measurement data to obtain a system error e of the dynamic measurement data s (n) and random error e r (n)。
S104: the system error e output according to the combined model s (n) separating data containing systematic errors in the dynamic measurement data.
Based on the systematic error e obtained in step S103 s (n) subtracting the system error e from the dynamic measurement data X (n) s Obtaining the true value X of the dynamic measurement data after (n) 0 (n) and random error e r (n) to facilitate subsequent separation of data containing random errors.
S105: random error e output according to the combined model r (n) separating data containing random errors in the dynamic measurement data.
Based on the random error e found in step S103 r (n) obtaining the true value X of the dynamic measurement data from the step S104 0 (n) and random error e r (n) the sum minus the random error e r (n)。
The step can remove the dynamic measurement data influenced by external factors to obtain the true value of the dynamic measurement data, and the obtained data can be used for researching corresponding scientific problems.
S106: and obtaining the true value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the systematic error and the separated data containing the random error.
The data including the systematic error and the data including the random error separated in steps S105 and S106 are discarded to obtain the true value of the dynamic measurement data.
The real value of the dynamic measurement data is high-quality dynamic measurement data, and can be directly used for scientific research.
According to the embodiment of the application, the original data of the dynamic measurement data are preprocessed, the situation that excessive original data and bad data are introduced and data containing gross errors are eliminated is avoided, then the data containing the system errors and the random errors are separated from the preprocessed dynamic measurement data, the data of the dynamic measurement data are cleaned, high-quality dynamic measurement data are obtained, and the research on corresponding scientific problems is facilitated.
Fig. 2 is a flowchart illustrating an exemplary scenario of a method for processing dynamic measurement data according to an embodiment of the present application, and may be implemented through the following steps S201 to S206.
The dynamic measurement data is assumed to be the voltage values of the wind turbine, wherein the voltage values are tested every 5 seconds.
S201: and acquiring original data from the new energy cloud database.
Raw data of dynamic measurement data are introduced, assuming that dynamic measurement data of voltage values of wind turbines of one day are introduced.
S202: the raw material is pre-processed.
The preprocessing comprises data truncation, discretization, initial identification statistical property, point picking processing and the like.
Since measurement is performed every 5 seconds, including 17280 seconds in one day, introducing 17280 measurement data at a time is too complicated, and intercepting a part of the data by using a data truncation method, for example, intercepting 1 hour of the data, only 720 data are left. Too much original data can be avoided, and subsequent calculation is not facilitated.
Suppose that 720 data include 1.2,1.3, 1.4,10800,1.5, etc., and "10800" is too different from other data. In order to avoid introducing such data containing gross errors, a method of eliminating points is adopted to eliminate the data such as "10800".
S203: and establishing a combined model of the dynamic measurement data.
Dividing dynamic measurement data X (n) into a deterministic function f (n) and a stochastic function Y (n), and further dividing f (n) into a non-periodic function d (n) and a periodic function p (n) according to the following formula;
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)
wherein, the random function Y (n) refers to the influence of external factors on data measurement, such as the influence of wind power on data measurement, running animals and the like; p (n) refers to the part of data with regular fluctuation in data, such as the assumed voltage value is between 1.2 and 1.5; d (n) refers to the part of the data that does not fluctuate regularly.
In particular, the dynamic measurement data X (n) can be further divided into the true value X of the measured variable 0 (n) and the error e (n) (hereinafter, the actual value is represented by subscript 0), and the actual value X 0 (n) from the deterministic true value f 0 (n) and the true randomness value Y 0 (n) error e (n) is composed of systematic error e s (n) and random error e r (n) composition, i.e.:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；
wherein d is 0 (n) and p 0 (n) true values f which are respectively deterministic components 0 The non-periodic component and the periodic component of (n).
According to formula E [ E (n)]＝e s (n) calculating the systematic error e s (n) the system error of the obtained voltage is 0.9, for example.
According to formula X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n) calculating E [ X (n)]。
Then according to the formula e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]Calculating a random error e r (n) the random error of the obtained voltage is 0.1, for example.
According to the formula
Calculating a random error e r Variance value of (n), e.g.The variance value of the random error of the voltage was found to be 0.05, and it was found that the obtained data tended to be stable.
Establishing a combination model of the dynamic measurement data for performing characteristic analysis on the dynamic measurement data to obtain a system error e of the dynamic measurement data s (n) and random error e r (n)。
S204: a systematic error e output according to the combined model s (n) separating data containing systematic errors in the dynamic measurement data.
Based on the system error e obtained in step S203 s (n) is 0.9, and the dynamic measurement data X (n) is subtracted by the system error e s Obtaining the true value X of the dynamic measurement data after (n) 0 (n) and random error e r (n) to facilitate subsequent separation of data containing random errors.
S205: random error e output according to the combined model r (n) separating data containing random errors in the dynamic measurement data.
Based on the random error e found in step S103 r (n) is 0.1, and the true value X of the dynamic measured data is obtained in step S104 0 (n) and random error e r (n) the sum minus the random error e r (n)。
The step can remove the dynamic measurement data influenced by external factors to obtain the true value of the dynamic measurement data, and the obtained data can be used for researching corresponding scientific problems.
S206: and obtaining the true value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the systematic error and the separated data containing the random error.
The data including the systematic error and the data including the random error separated in steps S205 and S206 are truncated to obtain the true value of the dynamic measurement data.
The actual value of the dynamic measurement data is high-quality dynamic measurement data, and can be directly used for researching the point voltage change of the wind driven generator in one day.
The above describes a method for processing dynamic measurement data provided by an embodiment of the present application, and a processing apparatus for dynamic measurement data provided by an embodiment of the present application is described below with reference to the accompanying drawings.
As shown in fig. 3, an embodiment of the processing apparatus for dynamic measurement data provided in the embodiment of the present application includes:
the acquiring module 301 is configured to acquire original data from a new energy cloud database, where the original data are all data acquired through dynamic measurement;
a processing module 302, configured to perform preprocessing on the raw data to obtain dynamic measurement data, where the preprocessing includes at least one of: data truncation, discretization, initial identification statistical property and point picking processing;
a model establishing module 303, configured to establish a combined model of the dynamic measurement data, where the combined model is used to perform feature analysis on the dynamic measurement data to obtain a system error and a random error of the dynamic measurement data;
the processing module 302 is further configured to separate data including a systematic error from the dynamic measurement data according to the systematic error output by the combination model;
the processing module 302 is further configured to separate data including random errors from the dynamic measurement data according to the random errors output by the combination model;
the processing module 302 is further configured to obtain an actual value of the dynamic measurement data from the dynamic measurement data according to the separated data including the systematic error and the separated data including the random error.
Specifically, the processing module is further configured to:
constructing a first median sequence according to the original data;
forming a second median sequence of three data adjacent to the first median sequence according to the first median sequence;
and constructing a third sequence according to a second median sequence of three adjacent data, wherein the construction method of the third sequence is shown by the following formula:
wherein, { x' i Is the third sequence, { x ″ ] i Is a second median sequence;
calculating the absolute value of the difference between the first value in the first median sequence and the corresponding first value in the third sequence to obtain a difference value, and judging whether the difference value is greater than a preset threshold value;
if the difference is larger than a preset threshold value, a first value in the first median sequence is removed from the original data, a smooth linear interpolation is calculated according to adjacent data of the first value in the first median sequence, and the linear interpolation is used for replacing the first value in the first median sequence.
Specifically, the model building module 303 is specifically configured to build a combined model of dynamic measurement data, and includes: deterministic function f (n), random function y (n);
performing feature extraction on the dynamic measurement data X (n) by the following formula;
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)；
the random function Y (n) is used for calculating errors generated by external factors on data measurement; the deterministic function f (n) is divided into a non-periodic function d (n) and a periodic function p (n), the periodic function p (n) is used for calculating regularly fluctuating data in the dynamic measurement data, and the non-periodic function d (n) is used for calculating irregularly fluctuating data in the dynamic measurement data;
the dynamic measurement data X (n) includes the true value X of the dynamic measurement data 0 (n) and measurement errors e (n), the true value X of the dynamic measurement data 0 (n) true values f corresponding to functions f (n) comprising determinism 0 (n) true value Y corresponding to randomness function Y (n) 0 (n); measurement error e (n) is derived from system error e s (n) and random error e r (n) composition, represented by the formula:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；；
wherein d is 0 (n) the true value of the non-periodic function; p is a radical of formula 0 (n) is the true value of the periodic function.
Specifically, the model building module 303 further includes a calculating module 304, where the calculating module 304 is configured to calculate a system error, and is implemented by the following formula: e [ E (n)]＝e s (n) wherein E [ E (n)]Is the expected value of the measurement error e (n).
Specifically, the calculating module 304 is further configured to calculate a random error, which is implemented by the following formula:
e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]；
wherein E [ x (n) ] is calculated by the following formula:
X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n)；
wherein, E [ X ] 0 (n)+e(n)]For the actual value X of the dynamic measurement data 0 (n) and the sum of the measurement errors e (n).
Specifically, the calculating module 304 is further configured to calculate a variance value of the random error, which is implemented by the following formula:
It should be noted that, since the above-described processing apparatus for dynamic measurement data is based on the same concept as the method embodiment shown in fig. 1 of the present application, the technical effect thereof is the same as the method embodiment of the present application, and for specific contents, reference may be made to the description of the technical effect of the method embodiment shown in fig. 1 of the present application, and details are not repeated here.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The above embodiments are intended to explain the objects, aspects and advantages of the present invention in further detail, and it should be understood that the above embodiments are merely illustrative of the present invention.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A method for processing dynamic measurement data, the method comprising:
acquiring original data from a new energy cloud database, wherein the original data are all data acquired in the dynamic measurement process;
preprocessing the raw data to obtain dynamic measurement data, wherein the preprocessing at least comprises one of the following steps: data truncation, discretization, initial identification statistical property and point picking processing;
establishing a combined model of the dynamic measurement data, wherein the combined model is used for carrying out characteristic analysis on the dynamic measurement data to obtain a system error and a random error of the dynamic measurement data;
separating data containing system errors in the dynamic measurement data according to the system errors output by the combined model;
separating data containing random errors in the dynamic measurement data according to the random errors output by the combined model;
and obtaining the real value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the system error and the separated data containing the random error.
2. The method of claim 1, wherein performing data truncation on the raw data comprises:
truncating the original data according to a preset truncation length to obtain the dynamic measurement data;
wherein, the interception length is the maximum data length required to be measured in one or more dynamic measurement processes.
3. The method of claim 1, wherein performing a checkpointing process on the raw data comprises:
constructing a first median sequence according to the original data;
forming a second median sequence of three data adjacent to the first median sequence according to the first median sequence;
constructing a third sequence according to a second median sequence of the three adjacent data, wherein the construction method of the third sequence is shown by the following formula:
wherein, { x' i Is the third sequence, { x ″ ] i Is a second median sequence;
calculating an absolute value of a difference obtained by subtracting a first value in the third sequence from a first value in the first median sequence to obtain a difference value, and judging whether the difference value is greater than a preset threshold value;
if the difference value is larger than a preset threshold value, removing a first value in the first median sequence from the original data, calculating a smooth linear interpolation according to adjacent data of the first value in the first median sequence, and replacing the first value in the first median sequence with the linear interpolation.
4. The method of claim 1, wherein the combined model of dynamic measurement data comprises: deterministic function f (n), random function y (n);
performing feature extraction on the dynamic measurement data X (n) by the following formula;
X(n)＝f(n)+Y(n)＝d(n)+p(n)+Y(n)；
the random function Y (n) is used for calculating an error of an external factor on a dynamic measurement process; the deterministic function f (n) is divided into a non-periodic function d (n) and a periodic function p (n), the periodic function p (n) is used for calculating regularly fluctuating data in the dynamic measurement data, and the non-periodic function d (n) is used for calculating irregularly fluctuating data in the dynamic measurement data;
the dynamic measurement data X (n) includes the true value X of the dynamic measurement data 0 (n) and measurement errors e (n), the true value X of the dynamic measurement data 0 (n) a corresponding true value f comprising said deterministic function f (n) 0 (n) the true value Y corresponding to said randomness function Y (n) 0 (n); the measurement error e (n) is determined by the system error e s (n) and random error e r (n) composition, represented by the formula:
X(n)＝X 0 (n)+e(n)＝f 0 (n)+Y 0 (n)+e s (n)+e r (n)＝d 0 (n)+p 0 (n)+Y 0 (n)+e s (n)+e r (n)；
wherein d is 0 (n) the true value of the non-periodic function; p is a radical of 0 (n) is the true value of the periodic function.
5. The method of claim 4, wherein the combined model of dynamic measurement data is further used to calculate a system error by: e [ E (n)]＝e s (n) wherein E [ E (n)]Is the expected value of the measurement error e (n);
the systematic error is used to separate data containing the systematic error from the dynamic measurement data.
6. The method of claim 4, wherein the combined model of dynamic measurement data is further used to calculate a random error, which is implemented by the following formula:
e r (n)＝X(n)-X 0 (n)＝X(n)-E[X(n)]；
wherein E [ x (n) ] is calculated by the following formula:
X 0 (n)＝E[X(n)]＝E[X 0 (n)+e r (n)]＝d 0 (n)+p 0 (n)；
wherein, E [ X ] 0 (n)+e(n)]For the true value X of the dynamic measurement data 0 (n) a desired value of the sum of said measurement error e (n);
the random error is used for separating data containing the random error in the dynamic measurement data.
7. The method of claim 6, wherein the combined model of the dynamic measurement data is further used to calculate a variance value of the random error, which is implemented by the following formula:
8. An apparatus for processing dynamic measurement data, the apparatus comprising: the system comprises an acquisition module, a processing module and a model establishing module;
the acquisition module is used for acquiring original data from a new energy cloud database, wherein the original data are all data acquired by dynamic measurement;
the processing module is configured to perform preprocessing on the raw data to obtain dynamic measurement data, where the preprocessing at least includes one of: data truncation, discretization, initial identification statistical property and point picking processing;
the model establishing module is used for establishing a combined model of the dynamic measurement data, and the combined model is used for performing characteristic analysis on the dynamic measurement data to obtain a system error and a random error of the dynamic measurement data;
the processing module is further configured to separate data including a system error from the dynamic measurement data according to the system error output by the combined model;
the processing module is further configured to separate data including random errors from the dynamic measurement data according to the random errors output by the combination model;
and the processing module is further used for obtaining a true value of the dynamic measurement data from the dynamic measurement data according to the separated data containing the system error and the separated data containing the random error.
9. The apparatus of claim 8, wherein the processing module is further configured to:
truncating the original data according to a preset truncation length to obtain the dynamic measurement data;
wherein the truncation length is the maximum data length required to be measured in one or more dynamic measurement processes.
10. The apparatus of claim 8, wherein the processing module is further configured to:
constructing a first median sequence according to the original data;
forming a second median sequence of three data adjacent to the first median sequence according to the first median sequence;
constructing a third sequence according to a second median sequence of the three adjacent data, wherein the construction method of the third sequence is shown by the following formula:
wherein, { x' i Is the third sequence, { x i Is a second median sequence;
calculating an absolute value of a difference obtained by subtracting a first value in the third sequence from a first value in the first median sequence to obtain a difference value, and judging whether the difference value is greater than a preset threshold value;
if the difference value is larger than a preset threshold value, removing a first value in the first median sequence from the original data, calculating a smooth linear interpolation according to adjacent data of the first value in the first median sequence, and replacing the first value in the first median sequence with the linear interpolation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210752869.6A CN115017147A (en) | 2022-06-29 | 2022-06-29 | Method and device for processing dynamic measurement data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210752869.6A CN115017147A (en) | 2022-06-29 | 2022-06-29 | Method and device for processing dynamic measurement data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115017147A true CN115017147A (en) | 2022-09-06 |
Family
ID=83079196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210752869.6A Pending CN115017147A (en) | 2022-06-29 | 2022-06-29 | Method and device for processing dynamic measurement data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115017147A (en) |
-
2022
- 2022-06-29 CN CN202210752869.6A patent/CN115017147A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109726872B (en) | Energy consumption prediction method and device, storage medium and electronic equipment | |
| CN110969285B (en) | Prediction model training method, prediction device, prediction equipment and medium | |
| CN109636032B (en) | Precipitation forecast method, system, terminal and storage medium based on multi-mode integration | |
| CN111832222A (en) | Pollutant concentration prediction model training method, prediction method and device | |
| CN109687447B (en) | Electric power energy consumption prediction method and device | |
| CN112330050A (en) | Power system load prediction method considering multiple features based on double-layer XGboost | |
| CN114819374A (en) | Regional new energy ultra-short term power prediction method and system | |
| CN110991761B (en) | Heat supply load prediction method and device | |
| CN110110339A (en) | A kind of hydrologic forecast error calibration method and system a few days ago | |
| CN112700050B (en) | Method and system for predicting ultra-short-term 1 st point power of photovoltaic power station | |
| CN111435472A (en) | Method, device, equipment and storage medium for predicting quantity of components | |
| CN115017147A (en) | Method and device for processing dynamic measurement data | |
| CN111814345B (en) | Method for interpolating and observing missing wind speed data by utilizing various mesoscale wind speed data | |
| CN110781569B (en) | Abnormality detection method and system based on multi-resolution grid division | |
| CN113379279A (en) | Deep reinforcement learning short-term voltage stability evaluation method based on incomplete data | |
| CN111382891B (en) | Short-term load prediction method and short-term load prediction device | |
| CN111143776B (en) | Electric quantity load prediction method and device | |
| CN108363789B (en) | Short-term missing repairing method and device for electricity consumption meter data of industrial and commercial users | |
| CN114547829A (en) | Theoretical power calculation method and system for wind power plant | |
| CN111694820A (en) | Electric power abnormal data multiple filtering method and device, electronic equipment and storage medium | |
| JP2019208350A (en) | Diagnostic device for photovoltaic power generation facility | |
| CN112613674B (en) | Medium-and-long-term wind power generation capacity prediction method and device, electronic equipment and storage medium | |
| CN111339156B (en) | Method, apparatus and computer readable storage medium for long-term determination of business data | |
| CN115329148B (en) | Data screening and integrating method and system based on multiple big data processing | |
| CN117592651A (en) | Method, device, equipment and medium for calculating fault loss electric quantity of photovoltaic power station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |