CN104869105A - Abnormal state online identification method - Google Patents

Abstract

The invention provides an abnormal state online identification method for detecting a potential abnormal point in a high dimensional data stream in a real-time and online way. Through analyzing the data characteristic of the data stream, a method based on angular distribution is disclosed to obtain the abnormal factor value corresponding to each data in the data stream. Combined with the need of a real-time monitoring data stream, the establishment of a small scale data stream type calculation set based on a normal set and a boundary set is provided so as to accelerate the computation speed of the abnormal state online identification method. For the concept transfer problem of a large data stream, the establishment of the real-time updating mechanism of the normal set and the boundary set is provided so as to ensure the detection accuracy of the abnormal state online identification method in a high dimensional space. By using the method, the consumption of time and physical memory can be greatly reduced, the potential abnormal point in the high dimensional data stream can be correctly detected in a real-time and online way, a condition is created for the realization of the real-time online assessment of the data stream, and the stability of a large data application system is enhanced.

Description

A kind of abnormality ONLINE RECOGNITION method

Technical field

The present invention relates to data mining, the technology such as outlier detection, particularly relate to a kind of abnormality ONLINE RECOGNITION method.

Background technology

Outlier detection is one of most important technical method in Data Mining.Along with the development of science and technology, as ecommerce, network flow monitoring, radio communication, many practical applications such as logistics transportation all can produce sequential, magnanimity, and vertiginous infinite data flow.Generally, mass data flow has the feature such as higher-dimension and concept drift.Usually, these features greatly hinder the abnormality detection in data flow.Therefore, how realizing effective excavation of unsafe factor from mass data is a very important problem.

Since the research of outlier detection is risen, famous research institutions more both domestic and external and academic unit have all carried out a large amount of research work in this field, and acquire a great achievement.Being summed up primarily of three kinds of method for detecting abnormality, is Corpus--based Method respectively, based on distance, and the method for detecting abnormality of density based.The method for detecting abnormality of Corpus--based Method generally needs the model of given data collection, distributed constant, and the abnormity point number of expection.But these parameters are often all not easy to be acquired.The data set higher to dimension based on the Outlier Detection Algorithm of distance has good effect, but often needs to set relevant parameter in advance, and needs to scan whole data set frequently, therefore can not meet the quick excavation requirement of data flow.Name-based Routing great majority have dependence to arest neighbors method, and most typical is used the thought of index data structure to improve algorithm performance, but computation complexity is still higher.In addition, along with the increase of dimension, in higher dimensional space, data will become more and more sparse.In this case, nearly all data are all abnormity point.Therefore, the method for density based is not suitable for the outlier detection in data flow yet.

According to above description, traditional abnormal point detecting method all cannot be adapted to the abnormality ONLINE RECOGNITION in data flow.Therefore in the urgent need to providing a kind of abnormal state detection method can supporting to dynamically update, the consumption to time and physical store can be reduced under the prerequisite ensureing accuracy of detection, thus the speed process realized High Dimensional Data Streams and detection in real time.

Summary of the invention

For Problems existing in above-mentioned background, the invention provides a kind of abnormality ONLINE RECOGNITION method, to solve the problem that traditional abnormal point detecting method is not suitable for the abnormality in ONLINE RECOGNITION data flow.

The step of the technical solution used in the present invention is as follows:

A kind of abnormality ONLINE RECOGNITION method, detects the potential abnormity point in High Dimensional Data Streams for real-time online, comprises step:

A. the data element in real-time data collection stream, obtains the high dimensional data sample set X containing certain data element, and carries out preliminary treatment to high dimensional data sample set X;

B. use the Outlier factor formula based on angular distribution to analyze each data element in set X, obtain the Outlier factor value of each data element in set X with this;

C. according to the Outlier factor value of each data element, and the proper set of setting, boundary set threshold value divides all data elements in set X.Namely be include data element in proper set, boundary set, the one of anomalous concentration.Thus construct initial proper set, boundary set;

D. latest data element X (i) in image data stream, with proper set, boundary set is set up dataflow-style on a small scale and is calculated collection;

E. use the Outlier factor formula based on angular distribution to analyze latest data element X (i), obtain the Outlier factor value of this data element with this;

F. according to the Outlier factor value of latest data element, and the proper set of setting, boundary set threshold value includes latest data element X (i) in proper set, the one in boundary set.If this data element is abnormity point, is then included in and extremely collected O, and it can be used as abnormity point to export;

G. detect proper set in real time, whether boundary set overflows.If overflow, then by proper set, boundary set upgrades by first in first out (FIFO) mode;

H. step D is jumped to, until detected all data elements.

Data element in described steps A real-time data collection stream, and the data element collected is stored into data set X successively.After the floor data in data set X reaches the upper limit, preliminary treatment is carried out to the data element in data set X.Preliminary treatment comprises and arranging in order the physics of each data element or mathematical feature, professional etiquette of going forward side by side model, simplify processes.

Described step B uses the Outlier factor formula based on angular distribution to carry out operational analysis to each data element in data set X, obtains Outlier factor value corresponding to each data element in data set X with this.Outlier factor formula based on angular distribution is as follows:

$\mathrm{VOA}\left(p\right)=\mathrm{Var}\left[{\mathrm{\Θ}}_{\mathrm{apb}}\right]={\mathrm{MOA}}_2\left(p\right)-{\left({\mathrm{MOA}}_1\left(p\right)\right)}^2;---\left(1\right)$

${\mathrm{MOA}}_1\left(p\right)=\frac{{\mathrm{\Σ}}_{\underset{a\≠b}{a,b\∈S\backslash \left\{p\right\}}}{\mathrm{\Θ}}_{\mathrm{apb}}}{\frac{1}{2}(n-1)(n-2)};---\left(2\right)$

${\mathrm{MOA}}_2\left(p\right)=\frac{{\mathrm{\Σ}}_{\underset{a\≠b}{a,b\∈S\backslash \left\{p\right\}}}{{\mathrm{\Θ}}^2}_{\mathrm{apb}}}{\frac{1}{2}(n-1)(n-2)};---\left(3\right)$

Wherein, the VOA (p) in formula (1) represents the Outlier factor value of some P.In formula (2) represent vector with vector the angle formed, some p and some a, some b inequality, n represents the number of data element in data set X.In formula (3) represent vector with vector between angled square of institute's structure, n represents the number of data element in data set X.

Described step C according to proper set, boundary set threshold value, and each data element that the Outlier factor value that obtains of step B is come in dividing data collection X.By the Outlier factor value of each data element respectively with proper set threshold value, boundary set threshold value compares.If this Outlier factor value is more than or equal to proper set threshold value, then this data element is included in proper set; If this Outlier factor value is less than proper set threshold value and is more than or equal to boundary set threshold value, then this data element is included in boundary set; If this Outlier factor value meet be less than proper set threshold value, and also meet be less than boundary set threshold value, then this element is included in abnormal collection.Initial normal sample set is obtained, boundary sample collection with this.

Described step D is that the Outlier factor value obtaining latest data element X (i) is prepared.Latest data element X (i) in image data stream, and by the proper set that this data element and step C obtain, boundary set composition on a small scale dataflow-style calculates collection.

Described step e is used and is analyzed latest data element X (i) that step D collects based on the Outlier factor formula of angular distribution, obtains the Outlier factor value of element X (i) with this.Analyzing this data element X (i) needs reference data flow pattern to calculate concentrated normal point and boundary point.Outlier factor formula based on angular distribution is consistent with the angular distribution Outlier factor formula in step B.

Described step F is followed according to proper set, boundary set threshold value, and the Outlier factor value of data element X (i) of step e acquisition decides the ownership of data element X (i).By the Outlier factor value of data element X (i) respectively with proper set threshold value, boundary set threshold value compares.If this Outlier factor value is more than or equal to proper set threshold value, then this data element is included in proper set; If this Outlier factor value between proper set threshold value and boundary set threshold value, then brings this data element into boundary set in; If this Outlier factor value not only meet be less than proper set threshold value, and also meet be less than boundary set threshold value, then this element is included in abnormal collection, and element X (i) is exported as abnormity point.Realize the real-time detection to data flow by the way.

Described step G detects the proper set set up in real time, and whether boundary set overflows.As long as one of them in proper set or boundary set is overflowed, then upgrade the set having occurred to overflow by first-in first-out (FIFO).Realize proper set with this, the real-time update of boundary set, solve the concept drift problem of high amount of traffic, thus ensure the detection accuracy of abnormality ONLINE RECOGNITION method at higher dimensional space.

Described step H realizes the real-time detection of abnormality in data flow to create necessary condition.Turn back to step D by step H, circulation step D is to step H, and what realize data flow is continuous, detects in real time.

The present invention proposes a kind of brand-new abnormality ONLINE RECOGNITION method, have the following advantages:

1., by using the Outlier factor formula based on angular distribution, each the high dimensional data element in data flow can be analyzed, and the Outlier factor value of each high dimensional data element can be obtained exactly.

2. by setting up based on proper set, the small-scale dataflow-style of boundary set calculates collection, significantly reduces the consumption of abnormality recognition methods to time and physical store, effectively improves abnormality and know the processing speed of another method to data flow data.For the real-time online assessment realizing data flow creates condition.

3. the proper set by setting up, boundary set real-time update mechanism, efficiently solves the concept drift problem of high amount of traffic, ensure that the recognition capability of abnormality recognition methods to abnormality potential in high amount of traffic.

Accompanying drawing explanation

Fig. 1 is the flow chart of the invention process steps A to step C.

Fig. 2 is the flow chart of the invention process step D to step H.

Embodiment

Below, by reference to the accompanying drawings the specific embodiment of the present invention is described further.

As shown in Figures 1 and 2, specific embodiment of the invention process and operation principle as follows:

A. the data element in real-time data collection stream, obtains the high dimensional data sample set X containing certain data element, and carries out preliminary treatment to high dimensional data sample set X;

B. use the Outlier factor formula based on angular distribution to analyze each data element in set X, obtain the Outlier factor value of each data element in set X with this;

C. according to the Outlier factor value of each data element, and the proper set of setting, boundary set threshold value divides all data elements in set X.Proper set is included in, boundary set, the one of anomalous concentration by data element.Thus construct initial proper set, boundary set;

D. latest data element X (i) in image data stream, with proper set, boundary set is set up dataflow-style on a small scale and is calculated collection;

E. use the Outlier factor formula based on angular distribution to analyze latest data element X (i), obtain the Outlier factor value of this data element with this;

F. according to the Outlier factor value of latest data element, and the proper set of setting, boundary set threshold value includes latest data element X (i) in proper set, the one in boundary set.If this data element is abnormity point, is then included in and extremely collected O, and it can be used as abnormity point to export;

G. detect proper set in real time, whether boundary set overflows.If overflow, then by proper set, boundary set upgrades by first in first out (FIFO) mode;

H. step D is jumped to, until detected all data elements.

Steps A pays no attention to each data on disconnected ground image data stream by certain sampling period, obtains the High Dimensional Data Set X containing certain data element.Each data element of data centralization has higher-dimension characteristic, includes physics or the mathematical feature of data element.By the continuous acquisition of a period of time, and after reaching the upper limit of set X, preliminary treatment is carried out to data set X.Preliminary treatment comprises and arranging in order the physics of each data element or mathematical feature, professional etiquette of going forward side by side model, simplify processes.

Step B uses and analyzes each data element in data acquisition system X based on the Outlier factor formula of angular distribution, and each data element will obtain the Outlier factor value corresponding with self.Need when analyzing some elements X (i) in data acquisition system X a, b of any inequality in reference set X and data element X (i) to form a vectorial angle at 2, and find out all this possibilities.Outlier factor formula based on angular distribution is as follows:

$\mathrm{VOA}\left(p\right)=\mathrm{Var}\left[{\mathrm{\Θ}}_{\mathrm{apb}}\right]={\mathrm{MOA}}_2\left(p\right)-{\left({\mathrm{MOA}}_1\left(p\right)\right)}^2;---\left(1\right)$

${\mathrm{MOA}}_1\left(p\right)=\frac{{\mathrm{\Σ}}_{\underset{a\≠b}{a,b\∈S\backslash \left\{p\right\}}}{\mathrm{\Θ}}_{\mathrm{apb}}}{\frac{1}{2}(n-1)(n-2)};---\left(2\right)$

${\mathrm{MOA}}_2\left(p\right)=\frac{{\mathrm{\Σ}}_{\underset{a\≠b}{a,b\∈S\backslash \left\{p\right\}}}{{\mathrm{\Θ}}^2}_{\mathrm{apb}}}{\frac{1}{2}(n-1)(n-2)};---\left(3\right)$

Wherein, the VOA (p) in formula (1) represents the Outlier factor value of some P.In formula (2) represent vector with vector the angle formed, some p and some a, some b inequality, n represents the number of data element in data set X.In formula (3) represent vector with vector between angled square of institute's structure, n represents the number of data element in data set X.

Step C according to proper set, boundary set threshold value, and the Outlier factor value of data element self divides all data elements in data set X.By step B, each data element can obtain the Outlier factor value VOA corresponding with self.By the Outlier factor value VOA of each data element respectively with proper set threshold value VOA1, boundary set threshold value VOA2 compares.If this Outlier factor value VOA is more than or equal to proper set threshold value VOA1, then this data element is included in proper set; If this Outlier factor value VOA is less than proper set threshold value VOA1 and is more than or equal to boundary set threshold value VOA2, then this data element is included in boundary set; If this Outlier factor value VOA meet be less than proper set threshold value VOA1, and also meet be less than boundary set threshold value VOA2, then this element is included in abnormal collection, and it can be used as abnormity point to export.By the way, the most initial proper set is obtained, boundary set.For inferring data element X (i) and the proper set of subsequent time, boundary set creates condition at the relative position of higher dimensional space.

Step D continues by certain sampling period data element X (i) gathering subsequent time, and by data element X (i) and proper set, boundary set is configured to dataflow-style on a small scale and calculates collection.Condition is created for analyzing data flow data continuously.

Step e is used and is analyzed latest data element X (i) collected based on the Outlier factor formula of angular distribution, obtains the Outlier factor value of element X (i) self with this.Analyze this data element X (i) to need with reference to the concentrated normal point of data stream type calculating on a small scale and boundary point.Outlier factor formula based on angular distribution is consistent with the angular distribution Outlier factor formula in step B.The element number concentrated due to the calculating of small-scale dataflow-style is limited, and only up-to-date element X (i) collected is analyzed, therefore ONLINE RECOGNITION method of the present invention has arithmetic speed as quick as thought, ensure that and carries out in real time data flow data, assess continuously.

Step F is followed according to proper set threshold value, boundary set threshold value, and the Outlier factor value of data element X (i) self decides the ownership of element X (i).Here, by the Outlier factor value VOA of data element X (i) respectively with proper set threshold value VOA1, boundary set threshold value VOA2 compares.If this Outlier factor value VOA is more than or equal to proper set threshold value VOA1, then this data element is included in proper set; If this Outlier factor value VOA is less than proper set threshold value VOA1 and is more than or equal to boundary set threshold value VOA2, then this data element is included in boundary set; If this Outlier factor value VOA meet be less than proper set threshold value VOA1, and also meet be less than boundary set threshold value VOA2, then this element is included in abnormal collection, and element X (i) is exported as abnormity point.By the process of step F, abnormality ONLINE RECOGNITION method achieves the real-time detection to data flow data.

Step G detects proper set in real time, and whether boundary set exists spillover.If proper set, some in boundary set at a time overflows, then by update mechanism, upgrade by first-in first-out (FIFO).To proper set, the real-time update of boundary set ensures that abnormality ONLINE RECOGNITION method detects the important method of accuracy.By to proper set, boundary set carries out the concept drift problem that real-time update can solve high amount of traffic.In addition, only have by first-in first-out upgrade data could be allowed as much as possible on time and amplitude to keep continuity.

Step H makes flow process jump to step D, achieves the continuous acquisition to data flow data by the circulation of step D to step H; Achieve dataflow-style on a small scale and calculate the foundation of collection; Achieve the uninterrupted analysis to data flow data; Achieve proper set, the real-time update of boundary set.

Claims (9)

1. an abnormality ONLINE RECOGNITION method, detects the potential abnormity point in High Dimensional Data Streams for real-time online, comprises step:

A. the data element in real-time data collection stream, obtains the high dimensional data sample set X containing certain data element, and carries out preliminary treatment to high dimensional data sample set X;

B. use the Outlier factor formula based on angular distribution to analyze each data element in set X, obtain the Outlier factor value of each data element in set X with this;

C. according to the Outlier factor value of each data element, and the proper set of setting, boundary set threshold value divides all data elements in set X.Namely be include data element in proper set, boundary set, the one of anomalous concentration.Thus construct initial proper set, boundary set;

D. latest data element X (i) in image data stream, with proper set, boundary set is set up dataflow-style on a small scale and is calculated collection;

E. use the Outlier factor formula based on angular distribution to analyze latest data element X (i), obtain the Outlier factor value of this data element with this;

F. according to the Outlier factor value of latest data element, and the proper set of setting, boundary set threshold value includes latest data element X (i) in proper set, the one in boundary set.If this data element is abnormity point, is then included in and extremely collected O, and it can be used as abnormity point to export;

G. detect proper set in real time, whether boundary set overflows.If overflow, then by proper set, boundary set upgrades by first in first out (FIFO) mode;

H. step D is jumped to, until detected all data elements.

2. a kind of abnormality ONLINE RECOGNITION method according to claim 1, is characterized in that: the data element in described steps A real-time data collection stream, and the data element collected is stored into data set X successively.After the floor data in data set X reaches the upper limit, preliminary treatment is carried out to the data element in data set X.Preliminary treatment comprises and arranging in order the physics of each data element or mathematical feature, professional etiquette of going forward side by side model, simplify processes.

3. a kind of abnormality ONLINE RECOGNITION method according to claim 1, it is characterized in that: described step B uses the Outlier factor formula based on angular distribution to carry out operational analysis to each data element in data set X, obtains Outlier factor value corresponding to each data element in data set X with this.

4. a kind of abnormality ONLINE RECOGNITION method according to claim 1, is characterized in that: described step C according to proper set, boundary set threshold value, and the Outlier factor value that obtains of step B carrys out each data element in dividing data collection X.By the Outlier factor value of each data element respectively with proper set threshold value, boundary set threshold value compares.If this Outlier factor value is more than or equal to proper set threshold value, then this data element is included in proper set; If this Outlier factor value is less than proper set threshold value and is more than or equal to boundary set threshold value, then this data element is included in boundary set; If this Outlier factor value meet be less than proper set threshold value, and also meet be less than boundary set threshold value, then this element is included in abnormal collection.Initial normal sample set is obtained, boundary sample collection with this.

5. a kind of abnormality ONLINE RECOGNITION method according to claim 1, is characterized in that: described step D is that the Outlier factor value obtaining latest data element X (i) is prepared.Latest data element X (i) in image data stream, and by the proper set that this data element and step C obtain, boundary set composition on a small scale dataflow-style calculates collection.

6. a kind of abnormality ONLINE RECOGNITION method according to claim 1, it is characterized in that: described step e is used and analyzed latest data element X (i) that step D collects based on the Outlier factor formula of angular distribution, obtains the Outlier factor value of element X (i) with this.Analyzing this data element X (i) needs reference data flow pattern to calculate concentrated normal point and boundary point.Outlier factor formula based on angular distribution is consistent with the angular distribution Outlier factor formula in step B.

7. a kind of abnormality ONLINE RECOGNITION method according to claim 1, it is characterized in that: described step F is followed according to proper set, boundary set threshold value, and the Outlier factor value of data element X (i) of step e acquisition decides the ownership of data element X (i).By the Outlier factor value of data element X (i) respectively with proper set threshold value, boundary set threshold value compares.If this Outlier factor value is more than or equal to proper set threshold value, then this data element is included in proper set; If this Outlier factor value between proper set threshold value and boundary set threshold value, then brings this data element into boundary set in; If this Outlier factor value not only meet be less than proper set threshold value, and also meet be less than boundary set threshold value, then this element is included in abnormal collection, and element X (i) is exported as abnormity point.Realize the real-time detection to data flow by the way.

8. a kind of abnormality ONLINE RECOGNITION method according to claim 1, it is characterized in that: described step G detects the proper set set up in real time, whether boundary set overflows.As long as one of them in proper set or boundary set is overflowed, then upgrade the set having occurred to overflow by first-in first-out (FIFO).Realize proper set with this, the real-time update of boundary set, solve the concept drift problem of high amount of traffic, thus ensure the detection accuracy of abnormality ONLINE RECOGNITION method at higher dimensional space.

9. a kind of abnormality ONLINE RECOGNITION method according to claim 1, is characterized in that: described step H realizes the real-time detection of abnormality in data flow to create necessary condition.Turn back to step D by step H, circulation step D is to step H, and what realize data flow is continuous, detects in real time.