CN107958089A - Build the method and apparatus of model and the detection method and device of abnormal data - Google Patents
Build the method and apparatus of model and the detection method and device of abnormal data Download PDFInfo
- Publication number
- CN107958089A CN107958089A CN201711375240.XA CN201711375240A CN107958089A CN 107958089 A CN107958089 A CN 107958089A CN 201711375240 A CN201711375240 A CN 201711375240A CN 107958089 A CN107958089 A CN 107958089A
- Authority
- CN
- China
- Prior art keywords
- state
- data
- dimensionality reduction
- vector
- historical data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides a kind of detection method and device of the method and apparatus and abnormal data for building model, builds state transition model with the relevance based on data and realizes the detection of abnormal data.The method of the structure state transition model includes：Historical data is obtained, the historical data includes the multi-group data between each other with relevance；Dimensionality reduction is carried out to the historical data；And the dimensionality reduction result structure state transition model using the historical data.
Description
Technical field
The present invention relates to technical field of data processing, more particularly, to the method and apparatus of structure state transition model
And the detection method and device of abnormal data.
Background technology
Anomaly data detection is all important problem in all industrial circles including wind-powered electricity generation industry.
Existing anomaly data detection technology has very much, be typically based on data characteristics to normal data and abnormal data into
Row is distinguished, and is realized using the method for unsupervised learning.
But in the industrial circles such as wind-powered electricity generation industry, data often do not isolate, but can in time or spatially deposit
In variation tendency, mutually there is relevance, and existing abnormal deviation data examination method is due to the characteristics of only relying upon current data
Carry out the judgement of abnormal data and do not consider the data correlation on the either time still spatially, so frequently can lead to
Erroneous judgement.
For example, as can be seen that the point in three circles does not meet sine from the sensing data curve shown in Fig. 1
Characteristic distributions, should be regarded as abnormal point.But if without considering the overall distribution of data as existing abnormal deviation data examination method,
And data exception is individually judged according to amplitude, then because the point in the first two circle is within normal amplitude range,
It is normal point that can be mistakenly considered the two points, so as to draw the wrong conclusion.
The content of the invention
The present invention be in view of problem above and propose, its purpose is to provide a kind of structure of the relevance based on data
The method and apparatus of state transition model and the detection method of abnormal data and device.
According to an aspect of the present invention, there is provided a kind of method for building state transition model, including：Obtain historical data,
The historical data includes the multi-group data between each other with relevance；Dimensionality reduction is carried out to the historical data；And utilize
The dimensionality reduction result structure state transition model of the historical data.
According to another aspect of the present invention, there is provided a kind of detection method of abnormal data, including：Use above-mentioned structure shape
The state transition model that the method for state metastasis model is built according to historical data, predicts current data, obtains predicted value；Obtain
Treat the current data of abnormality detection, the current data for treating abnormality detection is have relevance with the historical data one group
Data；Dimensionality reduction is carried out to the current data for treating abnormality detection；And the predicted value is treated into working as abnormality detection with described
The dimensionality reduction result of preceding data is compared, to judge whether the current data for treating abnormality detection is abnormal.
According to another aspect of the present invention, there is provided a kind of device for building state transition model, including：Data acquisition list
Member, it obtains historical data, and the historical data includes the multi-group data between each other with relevance；Dimensionality reduction unit, its is right
The historical data carries out dimensionality reduction；State transition model construction unit, it builds shape using the dimensionality reduction result of the historical data
State metastasis model.
According to another aspect of the present invention, there is provided a kind of detection device of abnormal data, including：Predicting unit, it is used
The state transition model that the device of above-mentioned structure state transition model is built according to historical data, predicts current data, obtains
To predicted value；Data capture unit, it, which is obtained, treats the current data of abnormality detection, it is described treat abnormality detection current data be with
The historical data has one group of data of relevance；Dimensionality reduction unit, it drops the current data for treating abnormality detection
Dimension；And identifying unit, it compared with the dimensionality reduction result of the current data for treating abnormality detection, comes the predicted value
Treat whether the current data of abnormality detection is abnormal described in judgement.
According to the present invention, historical data is being carried out to build state transition model on the basis of dimensionality reduction, drop can be being passed through
On the basis of dimension removes the correlation between historical data, structure more accurately expresses the transfer from historic state to succeeding state
Model.And then the comparison of the model prediction result and real data for passing through constructed state transition model carries out abnormal number
According to detection, can be based on the relevance of data, the characteristics of making full use of the variation tendency of data, judges the exception of data so that
It can more objective, more accurately detect the generation of data exception.
Brief description of the drawings
Fig. 1 is the figure for showing sensor abnormality data instance；
Fig. 2 is the flow chart for the method for showing structure state transition model according to the embodiment of the present invention；
Fig. 3 is the flow chart of detailed process the step of state transition model is built in the method for show Fig. 2；
Fig. 4 is the flow chart for the detection method for showing abnormal data according to the embodiment of the present invention；
Fig. 5 is the schematic diagram for illustrating to replicate neural net method；
Fig. 6 is the block diagram for the device for showing structure state transition model according to the embodiment of the present invention；
Fig. 7 is the block diagram formed in detail of state transition model construction unit in the device for show Fig. 6；
Fig. 8 is the block diagram for the detection device for showing abnormal data according to the embodiment of the present invention.
Embodiment
Hereinafter, with reference to the accompanying drawings of embodiments of the present invention.
The present invention is directed to the data of relevant property, makes full use of the relevances of data to realize the structure of state transition model
With the detection of abnormal data.
In an embodiment of the invention, there is provided a kind of method for building state transition model.It is carried out below
Describe in detail.
Fig. 2 is the flow chart for the method for showing structure state transition model according to the embodiment of the present invention.
With reference to Fig. 2, first in step S210, historical data is obtained.Here, historical data includes has association between each other
The multi-group data of property.So-called relevance, can be that temporal relevance can also be relevance spatially, and then can also
It is other kinds of relation.So-called temporal relevance, can be the temporal precedence relationship of acquisition of data, so-called space
On relevance, can be data obtain position on proximity relation.In addition, so-called multi-group data, is more than 2 groups and each group
The data of multiple data are included respectively, and then the data amount check of every group of data is preferably identical.
In one embodiment, in this step, multi-group data is obtained in chronological order from data source, formation has the time
On relevance historical data.
In another embodiment, in this step, obtained from multiple data source opsition dependents order of diverse location multigroup
Data, form the historical data with relevance spatially.
Data source in said one embodiment and another embodiment can be one or more monitoring devices, for example pass
Sensor.
Then, in step S220, dimensionality reduction is carried out to above-mentioned historical data.
In one embodiment, by carrying out principal component analysis (Principle Components to historical data
Analysis (PCA)), to carry out the dimensionality reduction of historical data.
Specifically, above-mentioned history data set is become to the matrix A of corresponding m rows n row first, then (1) is right according to the following formula
Matrix A carries out singular value decomposition.
A=USV*≈UkSkVk * (1)
Wherein, U, V are the unitary matrice after the singular value decomposition of matrix A；S is diagonal to be made of the singular value of matrix A
Matrix, its diagonal element arrange from big to small；V* represents the associate matrix of V, UkAnd VkThe preceding k row compositions of respectively U and V
Matrix, SkFor the diagonal matrix being made of the preceding k singular value of matrix A.
Then, to the singular value decomposition of above-mentioned matrix A, according to the following formula (2), the dimensionality reduction of data is carried out.
Thus, by above formula (2), matrix A is mapped to k dimension spaces (k<<N), the matrix B after being mapped, i.e. dimensionality reduction
As a result.
It should be noted that in the present invention, the purpose for carrying out dimensionality reduction to historical data will be removed in historical data
Correlation between data.Because if not dimensionality reduction and directly using historical data structure state transition model, to a certain state
Influence factor may accumulate NextState, and then build up down, final accumulation can cause institute's structure to a certain extent
The output result error of established model is excessive, or result mistake.
Then, in step S230, state transition model is built using the dimensionality reduction result of above-mentioned historical data.
Fig. 3 is the flow chart of detailed process the step of showing structure state transition model.
First in step S310, ask for corresponding with the multi-group data of the historical data after the dimensionality reduction in step S220 multiple
State vector.Specifically, in this step, the dimensionality reduction result of each group of data in the multi-group data of above-mentioned historical data is made
For a state vector, so as to obtain multiple state vectors corresponding with the multi-group data of historical data.
In the case of the principal component analysis of above-described embodiment, the transposed matrix B of the i.e. matrix B of dimensionality reduction result is obtained first*,
Then by transposed matrix B*In each row be defined as state vector xi, i=1,2 ..., m, so as to obtain corresponding multiple states
Vector.
Then, in step S320, state is obtained based on each state vector in multiple state vectors and previous state vector and is turned
Move matrix.
Specifically, by minimizing each state vector X in multiple state vectors2,mWith previous state vector X1,m-1With state
The product TX of transfer matrix T1,m-1Between difference object function, that is, following formula (3) shown in object function, can be calculated
State-transition matrix T.
Wherein, X2,m=[x2,x3,…,xm], X1,m-1=[x1,x2,…,xm-1]。
Then, in step S330, the shape of state-transition matrix structure vector from historic state vector to current state is utilized
State metastasis model.
Specifically, using above-mentioned state-transition matrix T, the Markov state transfer mould represented by structure following formula (4)
Type, the state transition model is by by state-transition matrix T and historic state vector xiIt is multiplied and then is obtained plus modeling error
To NextState vector xi+1。
xi+1=Txi+wi (4)
Wherein, T is state-transition matrix, wiFor modeling error.Thus, constructed state transition model can express from
Historic state and then according to constructed state transition model, can be predicted not to the transfer of current state according to historic state
Carry out state.
It should be noted that the follow-up data of the state transition model prediction history data of structure is stated in actual use
When, by such as following formula (5) Suo Shi by from state-transition matrix that historical data obtains and last in multiple state vectors
State vector is multiplied, and can obtain the predicted value of next data of historical data.
Wherein,For the pre- of the i.e. current data of the next data to historical data represented in the form of state vector
Measured value；T represents state-transition matrix；xiCorresponding to current state vector preceding state vector, i.e., above-mentioned multiple states to
Last state vector in amount.
But not follow historical data closely but also there are it between historical data and current data in current data
In the case of his data, historic state x can be based oniThe prediction of Future Data as (6) progress according to the following formula.
Wherein,For the predicted value to the i-th+j states, TjRepresent j power of state-transition matrix.Thus, pass through
State xiWith the j power T of state-transition matrix TjProduct, can predict the values of the i-th+j states in future.
Thus, using constructed state transition model, arbitrary Future Data can be predicted.
The method of structure state transition model according to the present embodiment, it is first that historical data dimensionality reduction is empty to low-dimensional state
Between, and then build low-dimensional state transition model.Thus, due to eliminating the data in historical data by the dimensionality reduction of historical data
Between correlation, the possibility to next state cumulative to the factor of a certain state is avoided, so constructed state transfer
Model can express the transfer from historic state to succeeding state exactly.And then the state built according to historical data shifts
Model can provide support for prolonged fault pre-alarming.
In an embodiment of the invention, there is provided a kind of detection method of abnormal data.It is carried out below in detail
Explanation.
Fig. 4 is the flow chart for the detection method for showing abnormal data according to the embodiment of the present invention.
As shown in figure 4, in step S410, using the method for the structure state transition model shown in above-mentioned Fig. 2 according to history
The state transition model of data structure, predicts current data, obtains predicted value.
Specifically, in this step, such as with shown in the following formula (7) of above formula (5) equivalence, by will be obtained from historical data
The preceding state multiplication of vectors of state-transition matrix and current state vector, to obtain the predicted value of current data.Herein,
The preceding state vector of so-called current state vector, is from last shape in multiple state vectors that historical data obtains
State vector.
Wherein,For the predicted value to current data represented in the form of state vector；T represents state-transition matrix；
xt-1It is vectorial corresponding to preceding state, i.e., last state vector in above-mentioned multiple state vectors.
In step S420, the current data for treating abnormality detection is obtained.Here, it is described treat abnormality detection current data be with
Above-mentioned historical data has one group of data of relevance, and the relevance which is with the historical data is mutual is identical
The relevance of type.In the present embodiment, to treat that the current data of abnormality detection is to follow the data, i.e. of above-mentioned historical data closely
The current data for treating abnormality detection is illustrated in case of next data of historical data, herein so-called next data,
It is the sky of the temporal next data or historical data of historical data according to historical data and the type of the relevance of current data
Between on next data.
In one embodiment, in the case of relevance in time, in this step, from above-mentioned steps S210 phases
Same data source obtains the above-mentioned current data for treating abnormality detection after above-mentioned historical data.
In another embodiment, in the case of relevance spatially, in this step, from the number with historical data
The above-mentioned current data for treating abnormality detection is obtained according to other close data sources of source position.
In step S430, the dimensionality reduction identical with historical data is carried out to the above-mentioned current data for treating abnormality detection.I.e. at this
In step, this is treated that the current data of abnormality detection projects to the lower dimensional space identical with the dimensionality reduction step S220 in Fig. 2.
In one embodiment, in this step, the step with above-mentioned Fig. 2 is carried out to the above-mentioned current data for treating abnormality detection
Rapid principal component analysis identical S220, is projected into identical k dimension spaces.On the dimensionality reduction, due to in step S220
Reduction process is identical, so in this description will be omitted.
And then due to treating that the current data of abnormality detection can be expressed as the matrix of a line n row, so according to principal component point
The dimensionality reduction of analysis is as a result, the corresponding state vector of the current data with treating abnormality detection shown in following formula (8) can be obtained：
Wherein, atTo treat the column vector of the current data of abnormality detection composition, xtException is treated for what is obtained based on dimensionality reduction result
The corresponding states vector of the current data of detection.
In step S440, by the predicted value compared with the dimensionality reduction result of the current data for treating abnormality detection,
To judge that this treats whether the current data of abnormality detection is abnormal.Specifically, when predicted value and the current data for treating abnormality detection
When difference between dimensionality reduction result exceedes predetermined threshold value, it is determined as that this treats that the current data of abnormality detection is abnormal data.
In one embodiment, in this step, as shown in following formula (9), take the predicted value that obtains in step S410 with
The norm of error vector between the dimensionality reduction result for the current data of abnormality detection that what step S430 was obtained treat as predicted value with
The difference between the dimensionality reduction result of the current data of abnormality detection is treated to be judged.
When the norm is more than predetermined threshold value, the current data for being judged to treating abnormality detection is abnormal data, otherwise
It is normal data, as shown in following formula (10).
Wherein, τ is predetermined threshold value.
In addition, except norm, can also be using the function of distance between other two state vectors of measurement as predicted value
With treat abnormality detection current data dimensionality reduction result between difference carry out the judgement of data exception, such as vector angle,
Shape similarity etc..
It should be noted that, although above with treat the current data of abnormality detection be follow closely above-mentioned historical data data,
It is illustrated in case of the next data for being historical data, but is not followed closely in the current data for treating abnormality detection
Above-mentioned historical data but between historical data and the current data for treating abnormality detection also there are other data in the case of,
It can be based on historic state xiAccording to the prediction of Future Data as above formula (6) progress.
So as to using state transition model of the method by above-mentioned Fig. 2 according to constructed by historical data, predict and appoint
The Future Data of meaning and then the detection for realizing abnormal data.
The detection method of abnormal data according to the present embodiment, utilizes the method root of the structure state transition model of Fig. 2
The low-dimensional state transition model built according to historical data is predicted Future Data, when the Future Data of actual measurement is in low-dimensional state
There are during larger difference, judge the Future Data of the actual measurement for abnormal data with prediction result for the projection in space.Thus, base is passed through
Abnormality detection is carried out to current data in the variation tendency of historical data, can it is more objective, more accurately judge data exception
Occur.
In addition, although the dimensionality reduction for the historical data being described above in the method for Fig. 2 and treating in the method for Fig. 4 are different
The situation of dimensionality reduction is carried out using the method for principal component analysis in the dimensionality reduction of the current data often detected, but can also be used artificial
The dimensionality reduction that neural net method carries out, the artificial neural network of this restructural initial data are known as replicating neutral net
(Replicator Neural Networks).Fig. 5 is the schematic diagram for illustrating the method for replicating neutral net, wherein most left
Side is initial data input, the rightmost side for reconstruct initial data, this two groups of data should correspondent equal it is (actual often to have one
Fixed error), the output of middle hidden node is the state value after dimensionality reduction.The example provided from Fig. 5 can be seen that original
Beginning data are 6 dimensional vectors, become 2 dimensional vectors after neutral net is compressed, then by reconstruct, revert to 6 new dimensional vectors, god
Through network by training so that 6 dimensional vectors that it is reconstructed are equal or close as best one can with 6 original dimensional vectors.
And then as dimension reduction method, in addition to the method for principal component analysis described above and duplication neutral net, go back
It can use the methods of independent component analysis, LLE (Locally linear embedding, be locally linear embedding into) to be gone through
History data and treat abnormality detection current data dimensionality reduction.
Under same inventive concept, the present invention provides the corresponding dress of method with above-mentioned structure state transition model
Put, be described in greater detail below.
Fig. 6 is the block diagram for the device for showing structure state transition model according to the embodiment of the present invention.
As shown in fig. 6, the device 600 of the structure state transition model of present embodiment includes：Data capture unit 610,
Dimensionality reduction unit 620 and state transition model construction unit 630.
Data capture unit 610 obtains historical data, and the historical data includes multigroup with relevance between each other
Data.The relevance is temporal relevance or relevance spatially.
Dimensionality reduction unit 620 carries out dimensionality reduction to the historical data.Specifically, dimensionality reduction unit 620 uses principal component analysis, answers
The method of neutral net processed, independent component analysis and any one method for being locally linear embedding into method are to the historical data
Carry out dimensionality reduction.
State transition model construction unit 630 builds state transition model using the dimensionality reduction result of the historical data.
Fig. 7 is the block diagram formed in detail for showing the state transition model construction unit 630 in Fig. 6.As shown in fig. 7, shape
State metastasis model construction unit 630 includes：State vector asks for unit 6301, it is by the multi-group data of the historical data
The dimensionality reduction result of each group of data obtains multiple shapes corresponding with the multi-group data of the historical data as a state vector
State vector；State-transition matrix asks for unit 6302, it is based on each state vector and previous state in the multiple state vector
Vector obtains state-transition matrix；Model construction unit 6303, its using the state-transition matrix build from historic state to
Measure the state transition model of current state vector.
And then state-transition matrix ask for unit 6302 by minimize in the multiple state vector each state vector and
Object function (the target i.e. shown in above formula (3) of the vectorial difference between the product of the state-transition matrix of previous state
Function), obtain the state-transition matrix.
It is by by the state-transition matrix and historic state vector phase shown in the state transition model such as above formula (4)
Multiply and then add modeling error and obtain the state transition model of NextState vector.
And then when predicting the follow-up data of the historical data using the state transition model, by by the shape
State transfer matrix is multiplied with last state vector in the multiple state vector, to obtain the pre- of the follow-up data
Measured value.
The device of the structure metastasis model of present embodiment can functionally realize above-mentioned structure state transition model
Method.
Fig. 8 is the block diagram for the detection device for showing abnormal data according to the embodiment of the present invention.
As shown in figure 8, the detection device 800 of the abnormal data of present embodiment includes：Predicting unit 810, data acquisition
Unit 820, dimensionality reduction unit 830 and identifying unit 840.
Predicting unit 810 is turned using the state that the device 800 of the structure state transition model of Fig. 6 is built according to historical data
Shifting formwork type, predicts current data, obtains predicted value.Specifically, predicting unit 810 such as shown in above formula (7) by will be from history
The state-transition matrix that data obtain is multiplied with last state vector in multiple state vectors, to obtain current data
Predicted value.
Data capture unit 820, which obtains, treats the current data of abnormality detection, it is described treat abnormality detection current data be with
The historical data has one group of data of relevance.The relevance is temporal relevance or relevance spatially,
And the relevance is the relevance of the relevance same type mutual with the historical data.
Dimensionality reduction unit 830 carries out dimensionality reduction to the current data for treating abnormality detection.Specifically, dimensionality reduction unit 830 uses
Principal component analysis, method, independent component analysis and any one method being locally linear embedding into method for replicating neutral net
Dimensionality reduction is carried out to the current data for treating abnormality detection.And then dimensionality reduction unit 830 is to the current data for treating abnormality detection
It is identical with the method used in the dimensionality reduction of the historical data to carry out method used in dimensionality reduction.
Identifying unit 840 by the predicted value compared with the dimensionality reduction result of the current data for treating abnormality detection,
To judge whether the current data for treating abnormality detection is abnormal.Specifically, when the predicted value and the abnormality detection for the treatment of
When difference between the dimensionality reduction result of current data exceedes predetermined threshold value, identifying unit 840 is judged to described treating abnormality detection
Current data be abnormal data.
And then the difference between the dimensionality reduction result of the predicted value and the current data for treating abnormality detection is described pre-
Norm, vector angle, the shape phase of error vector between measured value and the dimensionality reduction result of the current data for treating abnormality detection
Like any one in degree.
The detection device of the abnormal data of present embodiment can functionally realize the detection side of above-mentioned abnormal data
Method.
According to embodiment of the present invention, a kind of computer equipment is also provided.The computer equipment includes processing
Device and memory, memory storage have the computer program that can be performed on a processor, when the computer program is processed
When device performs, the step of realizing the detection method of abnormal data according to the embodiment of the present invention.
Moreover, it should be understood that the unit in the device of illustrative embodiments can be implemented hardware according to the present invention
Component and/or component software.Processing of the those skilled in the art according to performed by the unit of restriction, can be for example using existing
Field programmable gate array (FPGA) or application-specific integrated circuit (ASIC) realize unit.
In addition, the method for illustrative embodiments may be implemented as in computer readable recording medium storing program for performing according to the present invention
Computer program.Those skilled in the art can realize the computer program according to the description to the above method.When described
Computer program is performed the above method for realizing the present invention in a computer.
Although being particularly shown with reference to its illustrative embodiments and the invention has been described, those skilled in the art
Member can carry out shape it should be understood that in the case where not departing from the spirit and scope of the present invention that claim is limited to it
Various changes in formula and details.
Claims (20)
- A kind of 1. method for building state transition model, it is characterised in that including：Historical data is obtained, the historical data includes the multi-group data between each other with relevance；Dimensionality reduction is carried out to the historical data；AndState transition model is built using the dimensionality reduction result of the historical data.
- 2. according to the method described in claim 1, it is characterized in that, it is described structure state transition model the step of include：Using the dimensionality reduction result of each group of data in the multi-group data of the historical data as a state vector, acquisition and institute State the corresponding multiple state vectors of multi-group data of historical data；State-transition matrix is obtained based on each state vector in the multiple state vector and previous state vector；AndUtilize the state transition model of state-transition matrix structure vector from historic state vector to current state.
- 3. the method for structure state transition model according to claim 2, it is characterised in that obtaining state-transition matrix The step of in, the object function by minimizing difference obtains the state-transition matrix；Wherein, by the multiple state vector In each state vector subtract that previous state is vectorial to obtain the difference with the product of the state-transition matrix.
- 4. the method for the structure state transition model according to Claims 2 or 3, it is characterised in that the state shifts mould Type is by obtaining next shape by the state-transition matrix and the historic state multiplication of vectors, adding modeling error The state transition model of state vector.
- 5. the method for the structure state transition model according to Claims 2 or 3, it is characterised in that using the state When metastasis model predicts the historical data follow-up data, by by the state-transition matrix and the multiple state vector In last state vector be multiplied, obtain the predicted value of the follow-up data of the historical data.
- A kind of 6. detection method of abnormal data, it is characterised in that including：Usage right requires the shape that the method for the structure state transition model described in any one of 1-5 is built according to historical data State metastasis model, predicts current data, obtains predicted value；The current data for treating abnormality detection is obtained, the current data for treating abnormality detection is that have to associate with the historical data One group of data of property；Dimensionality reduction is carried out to the current data for treating abnormality detection；AndBy the predicted value compared with the dimensionality reduction result of the current data for treating abnormality detection, to judge described to treat exception Whether the current data of detection is abnormal.
- 7. the detection method of abnormal data according to claim 6, it is characterised in that in the determination step, work as institute When stating the difference between predicted value and the dimensionality reduction result of the current data when abnormality detection and exceeding predetermined threshold value, institute is judged State and treat that the current data of abnormality detection is abnormal data.
- 8. the detection method of abnormal data according to claim 7, it is characterised in that the predicted value treats exception with described Difference between the dimensionality reduction result of the current data of detection is the drop of the predicted value and the current data for treating abnormality detection Tie up any one in the norm of error vector between result, vector angle, shape similarity.
- 9. the method according to any one of claim 1-3,6-8, it is characterised in thatIn the step of dimensionality reduction, principal component analysis, method, independent component analysis and the local line of duplication neutral net are used Any one method in property embedding grammar carries out the dimensionality reduction of data.
- 10. the method according to any one of claim 1-3,6-8, it is characterised in that the relevance is on the time Relevance or relevance spatially.
- A kind of 11. device for building state transition model, it is characterised in that including：Data capture unit, it obtains historical data, and the historical data includes the multi-group data between each other with relevance；Dimensionality reduction unit, it carries out dimensionality reduction to the historical data；State transition model construction unit, it builds state transition model using the dimensionality reduction result of the historical data.
- 12. the device of structure state transition model according to claim 11, it is characterised in that the state transition model Construction unit includes：State vector asks for unit, it is using the dimensionality reduction result of each group of data in the multi-group data of the historical data as one A state vector, obtains multiple state vectors corresponding with the multi-group data of the historical data；State-transition matrix asks for unit, it is based on each state vector in the multiple state vector and is obtained with previous state vector State-transition matrix；AndModel construction unit, it builds the state of the vector from historic state vector to current state using the state-transition matrix Metastasis model.
- 13. the device of structure state transition model according to claim 12, it is characterised in that the state-transition matrix Asking for unit, each state vector and previous state in the multiple state vector are vectorial to shift square with the state by minimizing The object function of difference between the product of battle array, obtains the state-transition matrix.
- 14. the device of the structure state transition model according to claim 12 or 13, it is characterised in that the state transfer Model is by the way that the state-transition matrix is obtained next with the historic state multiplication of vectors, to add modeling error The state transition model of state vector.
- 15. the device of the structure state transition model according to claim 12 or 13, it is characterised in that using the shape When state metastasis model predicts the historical data follow-up data, by by the state-transition matrix and the multiple state to Last state vector in amount is multiplied, to obtain the predicted value of the follow-up data of the historical data.
- A kind of 16. detection device of abnormal data, it is characterised in that including：Predicting unit, its usage right require the device of the structure state transition model described in any one of 11-15 according to going through The state transition model of history data structure, predicts current data, obtains predicted value；Data capture unit, it, which is obtained, treats the current data of abnormality detection, it is described treat abnormality detection current data be with it is described Historical data has one group of data of relevance；Dimensionality reduction unit, it carries out dimensionality reduction to the current data for treating abnormality detection；AndIdentifying unit, its by the predicted value compared with the dimensionality reduction result of the current data for treating abnormality detection, to sentence Whether the fixed current data for treating abnormality detection is abnormal.
- 17. the detection device of abnormal data according to claim 16, it is characterised in that when the predicted value with it is described different When difference between the dimensionality reduction result of the current data often detected exceedes predetermined threshold value, the identifying unit is determined as described treat The current data of abnormality detection is abnormal data.
- 18. the detection device of abnormal data according to claim 17, it is characterised in that the predicted value with it is described treat it is different Difference between the dimensionality reduction result of the current data often detected is the predicted value and the current data for treating abnormality detection Any one in the norm of error vector between dimensionality reduction result, vector angle, shape similarity.
- 19. the device according to any one of claim 11-13,16-18, it is characterised in thatThe dimensionality reduction unit is using principal component analysis, the method for replicating neutral net, independent component analysis and is locally linear embedding into Any one method in method carries out the dimensionality reduction of data.
- 20. the device according to any one of claim 11-13,16-18, it is characterised in that when the relevance is Between on relevance or relevance spatially.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711375240.XA CN107958089A (en) | 2017-12-19 | 2017-12-19 | Build the method and apparatus of model and the detection method and device of abnormal data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711375240.XA CN107958089A (en) | 2017-12-19 | 2017-12-19 | Build the method and apparatus of model and the detection method and device of abnormal data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107958089A true CN107958089A (en) | 2018-04-24 |
Family
ID=61959375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711375240.XA Withdrawn CN107958089A (en) | 2017-12-19 | 2017-12-19 | Build the method and apparatus of model and the detection method and device of abnormal data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107958089A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327330A (en) * | 2018-09-11 | 2019-02-12 | 南京科思倍信息科技有限公司 | Chemical Manufacture Exceptional Slices management method based on data-driven |
| CN112241599A (en) * | 2020-11-18 | 2021-01-19 | 中国联合网络通信集团有限公司 | Method and device for establishing fault analysis model |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050193281A1 (en) * | 2004-01-30 | 2005-09-01 | International Business Machines Corporation | Anomaly detection |
| CN106873571A (en) * | 2017-02-10 | 2017-06-20 | 泉州装备制造研究所 | A kind of method for early warning based on data and Model Fusion |
-
2017
- 2017-12-19 CN CN201711375240.XA patent/CN107958089A/en not_active Withdrawn
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050193281A1 (en) * | 2004-01-30 | 2005-09-01 | International Business Machines Corporation | Anomaly detection |
| CN106873571A (en) * | 2017-02-10 | 2017-06-20 | 泉州装备制造研究所 | A kind of method for early warning based on data and Model Fusion |
Non-Patent Citations (2)
| Title |
|---|
| 许智慧: "马尔可夫状态转移概率矩阵的求解方法研究", 《中国优秀硕士学位论文全文数据库》 * |
| 钱腾: "基于网络活动的时间序列异常检测", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327330A (en) * | 2018-09-11 | 2019-02-12 | 南京科思倍信息科技有限公司 | Chemical Manufacture Exceptional Slices management method based on data-driven |
| CN109327330B (en) * | 2018-09-11 | 2022-02-22 | 南京科思倍信息科技有限公司 | Chemical production abnormal slice management method based on data driving |
| CN112241599A (en) * | 2020-11-18 | 2021-01-19 | 中国联合网络通信集团有限公司 | Method and device for establishing fault analysis model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zio | A study of the bootstrap method for estimating the accuracy of artificial neural networks in predicting nuclear transient processes | |
| Wang et al. | A grey prediction-based evolutionary algorithm for dynamic multiobjective optimization | |
| Giannakoglou et al. | Aerodynamic shape design using evolutionary algorithms and new gradient-assisted metamodels | |
| CN110686633B (en) | Landslide displacement prediction method and device and electronic equipment | |
| Ayodeji et al. | Causal augmented ConvNet: A temporal memory dilated convolution model for long-sequence time series prediction | |
| Devaraj et al. | Artificial neural network model for voltage security based contingency ranking | |
| JP6711323B2 (en) | Abnormal state diagnosis method and abnormal state diagnosis device | |
| Yeganeh et al. | Monitoring linear profiles using Artificial Neural Networks with run rules | |
| CN109143094A (en) | A kind of abnormal deviation data examination method and device of power battery | |
| KR20170127430A (en) | Method and system for detecting, classifying and / or mitigating sensor error | |
| JP5048748B2 (en) | Test table generation apparatus and test table generation method | |
| Lim et al. | Smart soft-sensing for the feedwater flowrate at PWRs using a GMDH algorithm | |
| CN107958089A (en) | Build the method and apparatus of model and the detection method and device of abnormal data | |
| Paikari et al. | Defect prediction using case-based reasoning: An attribute weighting technique based upon sensitivity analysis in neural networks | |
| Xiahou et al. | Measuring conflicts of multisource imprecise information in multistate system reliability assessment | |
| CN110096730B (en) | Method and system for rapidly evaluating voltage of power grid | |
| CN112767190B (en) | Method and device for identifying phase sequence of transformer area based on multilayer stacked neural network | |
| Kang et al. | CPS-based fault-tolerance method for smart factories: Simulation-based framework and fault management application | |
| CN108932197A (en) | Software failure time forecasting methods based on parameter Bootstrap double sampling | |
| Salvi et al. | Long short-term model for Brent oil price forecasting | |
| Alvarez et al. | Batch process monitoring in the original measurement's space | |
| Al-Dahidi et al. | Quantification of signal reconstruction uncertainty in fault detection systems | |
| CN110083849A (en) | Flexible measurement method, computer and the computer-readable medium of semiconductor manufacturing facility | |
| Chatterjee et al. | Transfer function modelling in software reliability | |
| CN112488282A (en) | Method, system, equipment and storage medium for predicting gas concentration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20180424 |
|
| WW01 | Invention patent application withdrawn after publication |