CN108319699A - A real-time clustering method for evolutionary data flow - Google Patents

Abstract

The invention provides an online clustering method facing to an evolutionary data stream, which adopts the technical scheme that the method comprises the steps of ① establishing an effective class set, a vanishing class set and an outlier set, ② classifying points to be processed obtained at the current moment into a certain set, and ③ updating the outlier set, the effective class set and the vanishing class set.

Description

A real-time clustering method for evolutionary data flow

technical field

The invention belongs to the technical field of data stream clustering, and in particular relates to a dynamic clustering method for evolutionary data streams.

Background technique

Data flow refers to real-time incoming data, which is different from traditional batch-acquired data. It is usually divided into static data flow (data distribution does not change) and evolutionary data flow (data distribution changes) according to whether the data distribution changes. Evolutionary data flow Streams are also known as dynamic data streams. At present, data flow has become one of the main data forms in the information society, such as financial transaction data, communication record data, sensor observation data, etc. Data stream clustering technology refers to the analysis of data streams through some clustering means. It has become one of the main means of data stream mining due to its strong advantage of not relying on prior information.

Currently, data stream clustering methods are mainly aimed at static data streams. In fact, data streams in reality generally have evolutionary characteristics (or dynamic characteristics), that is, during the dynamic inflow of data stream data, such as the emergence of new types, the disappearance of old types, and the reappearance of old types of disappearance (hereinafter referred to as emergence, Disappearance, reappearance) and other evolutionary forms. In practical applications, detection of these ubiquitous evolutionary forms is usually of more significance to users, such as monitoring and observation purposes in astronomy, medicine, finance, networking, etc. Therefore, there is an urgent need to develop data stream clustering techniques for evolutionary data. The significance is that, on the one hand, this will improve users' comprehensive understanding of the current clustering patterns of data streams and the evolution of various types; on the other hand, it will also help users make accurate judgments before all data arrives, such as looking for network anomalies Invasion time, estimating the number of classes for a given time period, and finding the optimal adjustment time. Although scholars at home and abroad have made many attempts on data stream clustering technology for evolutionary data streams, they mainly focus on the evolution of new classes, which severely limits the application range of data stream clustering algorithms. Therefore, there are There is a need to extend the ability of data stream clustering techniques to handle multiple evolving forms of data.

Contents of the invention

The present invention provides an online clustering method oriented to evolutionary data streams. Aiming at three typical evolutionary forms of evolutionary data streams, that is, appearance, disappearance, and reappearance, detection strategies are designed respectively, and a processing framework is designed. These detection strategies are integrated, so as to realize the real-time clustering of evolutionary data streams in the present invention, which can make new classes in the data streams be added in time, disappearing classes be removed in time, and recurring classes be restored in time without the need for re-formation.

The technical scheme of the present invention is: a kind of real-time clustering method for evolution data flow, it is characterized in that, comprises the following steps:

① Steps to establish effective class sets, vanishing class sets, and outlier point sets;

② The step of classifying the points to be processed obtained at the current moment into a certain set;

③ The step of updating the outlier set, effective class set and disappearance class set.

in:

① Steps to establish effective class sets, vanishing class sets, and outlier point sets;

The initial value of the effective class set is the set of results obtained by collecting a certain amount of data and clustering the initial set by using the static clustering method; the initial value of the vanishing class set is an empty set; the initial value of the outlier set is empty set.

② The step of classifying the points to be processed obtained at the current moment into a certain set, including:

First calculate the Euclidean distance between the point to be processed and the effective class set and the class in the disappearing class set, and find the minimum value;

Then compare the above minimum value with the preset outlier threshold: if the minimum value is greater than the above threshold, the points to be processed will be divided into the outlier set; if the minimum value is less than or equal to the above threshold, the points to be processed will be divided into into the class corresponding to the minimum value.

③ Steps to update the outlier set, effective class set and disappearance class set:

(a) Steps to update the outlier set:

First calculate the number of elements in the outlier set; then compare the number with the predetermined threshold: if the number is greater than or equal to the threshold, use the static clustering method to classify all the elements in the outlier set , and add the classification result to the valid class set, and clear the outlier set; if the number is less than the threshold, no update is made to the outlier set.

(b) Steps to update the set of valid classes:

For each class in the valid class collection, do the following:

First, calculate the time interval between the current moment and the latest update of the class, if the time interval is greater than or equal to the predetermined disappearance threshold, delete the class from the effective class set and add it to the disappearing class set.

(c) Steps to update the collection of disappearing classes:

For each class in the disappearance class set, perform the following operations:

First calculate the number of processing points that have been divided into processing points since the class was added to the disappearing class set. If the number is greater than or equal to the predetermined recurrence threshold, delete the class from the disappearing class set and add it to the effective class set. .

The beneficial effects of the present invention are:

(1) The present invention respectively designs detection functions for three typical evolutionary forms (i.e. class appearance, disappearance and reappearance) in the evolutionary data stream (i.e. using the method of updating the outlier set, effective class set and disappearance class set). step), improving the processing capability of the data stream clustering method for evolutionary data streams;

(2) The present invention integrates and unifies three evolutionary forms of data streams by proposing an evolutionary data stream dynamic clustering processing algorithm, improves the stability of the data stream clustering method, and expands the application range of the data stream clustering method.

Description of drawings

Fig. 1 is a schematic flow chart of the principle of the present invention;

The flowchart of the concrete realization of Fig. 2 the present invention;

Figure 3 Data set information of simulation experiment 1;

The experimental result of Fig. 4 simulation experiment one;

Figure 5 Data set information of simulation experiment 2;

Fig. 6 Experimental results of the second simulation experiment.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Fig. 1 is a schematic flow chart of the principle of the present invention, at first carry out the step of setting up three types of sets (i.e. effective set, disappearance set and outlier point set), then carry out the step of dividing the points to be processed into sets, and finally carry out three types of sets Collection updates. Among them, the basic elements in the effective class set and the vanishing class set are categories; the basic elements in the outlier set are processing points, that is, the basic units that form data streams, also called data points. The effective class set stores the categories that still have clustering significance for the data flow at the current moment, and its initial value is the category obtained by using the static clustering method after collecting a certain amount of data to be processed; It is the category that loses the clustering meaning to the data flow and is deleted from the effective category set before the current moment, and its initial value is an empty set; the outlier point set stores outlier points, and the outlier point refers to the category with the following characteristics Processing point: Compare the minimum value of the distance between the processing point and the category in the effective class set and the disappearing class set with the predetermined outlier threshold. in the outlier set. The update of the three types of sets is carried out for three types of typical evolution forms. Specifically, the update of the outlier set is the evolution form corresponding to the emergence of new classes in the data stream, because the appearance of new classes will lead to a certain amount of new classes The points of the outliers are divided into the outlier set because no suitable class can be found. Therefore, when the number of points in the outlier set exceeds the predetermined threshold, it means that a new class has appeared, so it is necessary to classify the points in the outlier set Clustering; the update of the effective class set is the evolutionary form of the disappearance of the old class in the corresponding data stream. If a certain class in the effective class set has not been classified for a long time, after the time interval is greater than the predetermined disappearance threshold, it indicates that these classes has been "invalid", the class will be deleted from the effective class set and added to the disappearing class set; and the update of the disappearing class set is an evolutionary form of the reappearance of the old class in the corresponding data stream. If a certain class is still classified, and the number exceeds the predetermined recurrence threshold, it means that the class has changed from the "failed" state to the "valid" state, then this class will be deleted from the disappearing class collection and added to in the valid class collection. In the real-time processing stage of the data stream, the steps of dividing the points to be processed into collections and updating the three types of collections are repeated.

Fig. 2 is a flow chart of the specific implementation of the present invention. as shown in picture 2;

Firstly, three sets are established, Θ, O and Δ correspond to the valid set, the outlier set and the vanishing set respectively.

Among them, the initial values of the outlier set O and the vanishing class set Δ are empty sets respectively. When initializing the effective class set Θ, use the obtained data stream to obtain the initial clustering set Θ={c _j } through the existing clustering method, assuming that a total of J classes are obtained, then j=1,2,... .,J. The data volume of the obtained data stream is determined according to specific conditions.

Then, enter the real-time clustering processing stage:

Assuming that the multi-dimensional data point x _n arrives at any time t _n , the following real-time clustering processing steps are performed respectively:

The first step is to perform outlier detection;

①: If the effective class set Θ is a non-empty set, then calculate the Euclidean distance between each class c _j and x _n in the set, here it is recommended to obtain the Euclidean distance between the point x _n and the class center of the class c _j ; assuming that In the set, the class c _m is the closest to x _n , that is, the Euclidean distance between the two is the smallest, and the minimum distance is defined as d ₁ ;

If the effective class set Θ is an empty set, then d ₁ = ∞;

②: If the disappearance class set Δ is a non-empty set, assuming Δ={b _l }, a total of L elements, then l=1,2,...,L, calculate the Euclidean formula of each class b _l and x _n Distance; assuming that the class b _r in the disappearance class set Δ is the closest to x _n , that is, the Euclidean distance between the two is the smallest, then define the minimum distance as d ₂ ;

If Δ is an empty set, d ₂ =∞;

③: let d ₃ =min(d ₁ ,d ₂ ), which is the minimum value of both d ₁ and d ₂ . Compare d ₃ with the predetermined occurrence threshold d ₀ : when d ₃ > d ₀ , divide x _n into the outlier set O; if d ₃ ≤ d ₀ , divide x _n into the class of the corresponding set , where min is the minimum value function.

In the third step, assuming that the number of elements in the outlier set O is N, compare N with the predetermined emergence threshold θ _emerge , if N≥θ _emerge , execute the fourth step, otherwise execute the fifth step.

The fourth step is to use the existing static clustering method to cluster all the elements of the outlier set O, and add the result to the effective class set Θ, and clear the outlier set O.

The fifth step, for each class c _j in the effective class set Θ, perform the following operations:

First, calculate the time interval between the current moment and the latest update of this class, that is, use the following formula to obtain Δ _nj =t _n -t _j , where t _j is the last time when class c _j is classified into a processing point. Compare Δ _nj with the predetermined disappearance threshold θ _disp , if Δ _nj ≥ θ _disp , then delete the element c _j from the effective class set Θ, and add c _j as a class to the disappearing class set Δ, otherwise, execute the first six steps.

Step 6: For each class b _l in the disappearance class set Δ, detect the number f _l of its processing points since it was added to the set, and compare it with the predetermined recurrence threshold θ _recur , if f _l ≥ θ _recur , delete class b _l from the vanishing class set Δ, and add it to the valid class set Θ, otherwise, go back to the first step and wait for the next data point to be processed.

In order to illustrate that the present invention can dynamically cluster and process data streams with evolutionary characteristics, the following MATLAB simulation experiments have been carried out: experiment one is to use the present invention to process simulation data sets one, and the purpose is to demonstrate that the present invention has emergence and disappearance evolutionary characteristics The processing capability of the data flow; the second experiment is to use the present invention to process the simulation data set 2, and the purpose is to demonstrate the processing capability of the present invention to the data flow with the characteristics of emergence, disappearance and reappearance evolution.

Figure 3 shows the class number information of the simulation data set 1 used in Experiment 1. As shown in the figure, the horizontal axis represents the data points of the data stream, a total of 17,200 data points, sorted in order of inflow; the vertical axis Represents the number of the class corresponding to these data points, a total of 12, numbered 1 to 12. At the beginning of the data inflow, from the 1st data point to the 1600th data point, a total of 8 classes appeared, and then the 9th, 10th, and 11th classes began to appear; from the 5800th data point to the 8200th data point, All 12 classes appeared, and then, 12, 11, 10, 9, 8, 7, and 6 disappeared one after another. From the 16200th data point to the 17200th data point, only 5 classes existed.

Figure 4 shows the experimental results of the first simulation experiment. As shown in Figure 4, the horizontal axis represents data points, totally 17200, and the vertical axis represents the class number corresponding to each data point, and the class number of the simulation data set 1 is represented with a dotted line in the figure along with the real change of data points, the present invention The recognition result is represented by a solid line. The * mark is the time when the old class disappears, the △ mark is the time when the current model is rebuilt, and the × mark is the start time of stream processing. It can be seen from FIG. 4 that the present invention uses the first data point to the 1600th data point to complete the initialization at time 1600, and starts to process the data stream in real time from the 1601st data point. Comparing the solid line and dotted line in the figure, it can be found that the change trend of the two is basically the same, which shows that the present invention can effectively detect the emergence of new classes and the evolutionary form of old classes disappearing in the data stream, and this test has 17200 data points. The classification accuracy reaches 99.99%.

Figure 5 shows the number of categories of the simulation data set 2 used in Experiment 2. As shown in Figure 5, the horizontal axis represents the data points of the data stream, a total of 22,000 data points, sorted in order of inflow; The axes represent the numbers of the classes corresponding to these data points, there are 12 in total, and the numbers are 1~12 respectively. From the first data point to the 2400th data point stage, a total of 12 classes appeared, and then the 12th class began to disappear at the 2400th time, and began to reappear at the 3600th data point; Experienced the evolution of disappearing and reappearing.

Figure 6 shows the experimental results of the second simulation experiment. As shown in Figure 6, the horizontal axis represents data points, 22000 in total, and the vertical axis represents the class number corresponding to each data point, and the class number of the simulation data set two is represented with a dotted line in the figure along with the real change of data points, the present invention The recognition result is represented by a solid line. The * mark is the time when the old class disappears, the o mark is the reappearance time, and the × mark is the stream processing start time. It can be known from FIG. 6 that the present invention completes the initialization at time 2400 and starts to process the data stream in real time. Comparing the solid line and the dotted line in the figure, it can be found that the change trend of the two is basically the same, which shows that the present invention can effectively detect the evolution form of the disappearance of the old class and the reappearance of the old class in the data stream, and this test is performed on 22000 data The classification accuracy of points reaches 99.99%.

Claims (4)

1. a kind of real-time clustering method towards evolving data stream, which is characterized in that include the following steps：

1. the step of establishing effective class set, disappearance class set, the point set that peels off；

2. the step of pending point obtained to current time is included into some set；

3. update peels off the step of point set, effective class set and disappearance class set.

2. the real-time clustering method according to claim 1 towards evolving data stream, which is characterized in that update the point set that peels off The step of conjunction is：

The number of element in the point set that peels off is calculated first；Then is there is thresholding and be compared with predetermined in the number：If this Number is more than or equal to the thresholding, then is classified to all elements in the point set that peels off using static clustering processing method, and will divide Class result is added in effective class set, while emptying the point set that peels off；If the number is less than the thresholding, not to the point set that peels off It closes and carries out any update.

3. the real-time clustering method according to claim 2 towards evolving data stream, which is characterized in that update effective class set The step of conjunction is：

Every one kind in effective class set is proceeded as follows：

The time interval that calculating current time is updated away from such the last time first, if the time interval is more than or equal to predetermined disappear Thresholding is lost, then is deleted such from effective class set, and be added in disappearance class set.

4. the real-time clustering method according to claim 3 towards evolving data stream, which is characterized in that update disappearance class set The step of conjunction is：

Every one kind in being closed to disappearance class set proceeds as follows：

Such is calculated first from after being added in disappearance class set, is divided into the number of process points, is made a reservation for if the number is more than or equal to Reappear thresholding, then deletes such from disappearance class set, and be added in effective class set.