New! View global litigation for patent families

US20050210027A1 - Methods and apparatus for data stream clustering for abnormality monitoring - Google Patents

Methods and apparatus for data stream clustering for abnormality monitoring Download PDF

Info

Publication number
US20050210027A1
US20050210027A1 US10801420 US80142004A US2005210027A1 US 20050210027 A1 US20050210027 A1 US 20050210027A1 US 10801420 US10801420 US 10801420 US 80142004 A US80142004 A US 80142004A US 2005210027 A1 US2005210027 A1 US 2005210027A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
data
cluster
stream
statistical
clusters
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10801420
Inventor
Charu Aggarwal
Philip Yu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/62Methods or arrangements for recognition using electronic means
    • G06K9/6267Classification techniques
    • G06K9/6279Classification techniques relating to the number of classes
    • G06K9/6284Single class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/912Applications of a database
    • Y10S707/952Malicious software

Abstract

Techniques for monitoring abnormalities in a data stream are provided. A plurality of objects are received from the data stream and one or more clusters are created from these objects. At least a portion of the one or more clusters have statistical data of the respective cluster. It is determined from the statistical data whether one or more abnormalities exist in the data stream.

Description

    FIELD OF THE INVENTION
  • [0001]
    The present invention is related to techniques for clustering a data stream and, more particularly, techniques for monitoring data abnormalities in the stream through the clustering of the data stream.
  • BACKGROUND OF THE INVENTION
  • [0002]
    In general, large volumes of continuously evolving data, which may be stored, is referred to as a data stream. Data streams have received increased attention in recent years due to technological innovations, which have facilitated the creation, maintenance and storage of such data. A number of data mining studies have been conducted in the data stream context in recent years, see, e.g., C. C. Aggarwal, “A Framework for Diagnosing Changes in Evolving Data Streams,” ACM SIGMOD Conference, 2003; B. Babcock et al., “Models and Issues in Data Stream Systems,” ACM PODS Conference, 2002; P. Domingos et al., “Mining High-Speed Data Streams,” ACM SIGKDD Conference, 1998; S. Guha et al., “ROCK: A Robust Clustering Algorithm for Categorical Attributes,” Proceedings of the International Conference on Data Engineering, 1999; and L. O'Callaghan et al., “Streaming-Data Algorithms for High-Quality Clustering,” ICDE Conference, 2002.
  • [0003]
    Clustering is the partitioning of a given set of objects, such as data points, into one or more groups (clusters) of similar objects. The similarity of a data point with another data point is typically defined by a distance measure or objective function. In addition, data points that do not naturally fit into any particular cluster are referred to as outliers. Clustering has been widely studied by those in the database and data mining communities because of its applicability to a wide range of problems, see, e.g., P. Bradley et al., “Scaling Clustering Algorithms to Large Databases,” SIGKDD Conference, 1998; S. Guha et al., “CURE: An Efficient Clustering Algorithm for Large Databases,” ACM SIGMOD Conference, 1998; R. Ng et al., “Efficient and Effective Clustering Methods for Spatial Data Mining,” Very Large Data Bases Conference, 1994; A. Jain et al., “Algorithms for Clustering Data,” Prentice Hall, N.J., 1998; L. Kaufman et al., “Finding Groups in Data—An Introduction to Cluster Analysis,” Wiley Series in Probability and Math Sciences, 1990; E. Knorr et al., “Algorithms for Mining Distance-Based Outliers in Large Data Sets,” Proceedings of the VLDB Conference, September, 1998; E. Knorr et al., “Finding Intensional Knowledge of Distance-Based Outliers,” Proceedings of the VLDB Conference, September, 1999; S. Ramaswamy et al., “Efficient Algorithms for Mining Outliers from Large Data Sets,” Proceedings of the ACM SIGMOD Conference, 2000; and T. Zhang et al., “BIRCH: An Efficient Data Clustering Method for Very Large Databases,” ACM SIGMOD Conference, 1996.
  • [0004]
    The problem of categorical data clustering has also been recently studied, see, e.g., V. Ganti et al., “CACTUS-Clustering Categorical Data Using Summaries,” Proceedings of the ACM SIGKDD Conference, 1999; D. Gibson et al., “Clustering Categorical Data: An Approach Based on Dynamical Systems,” Proceedings of the VLDB Conference, 1998; and S. Guha et al., “ROCK: A Robust Clustering Algorithm for Categorical Attributes,” Proceedings of the International Conference on Data Engineering, 1999. However, these techniques cannot be utilized for clustering data streams, since they do not naturally scale well with increasing data size. Furthermore, a data stream clustering technique requires the appropriate mechanisms to deal with the temporal issues created by the evolution of the data stream.
  • [0005]
    Clustering and outlier monitoring present a number of unique challenges in an evolving data stream environment. For example, the continuous evolution of clusters makes it essential to quickly identify new patterns in the data. In addition, it is also important to provide end users with the ability to analyze the clusters in an offline fashion.
  • [0006]
    In the data stream environment, outlier and abnormality monitoring is especially problematic, since the temporal component of the data stream influences whether an outlier is defined as an abnormality. For example, the first arriving data point of a cluster may be considered an outlier at the moment of its arrival. However, as time passes, data points may join the newly created cluster, thereby initiating a new pattern of activity resulting from the evolution of the data stream. On the other hand, in many other cases, data points may not join the outlier or newly created cluster over time, thereby defining an abnormality. An important aspect of the data stream clustering process is the ability to identify and label such events effectively.
  • SUMMARY OF THE INVENTION
  • [0007]
    The present invention provides techniques for clustering a data stream and, more particularly, techniques for monitoring data abnormalities in the stream through the clustering of the data stream.
  • [0008]
    For example, in one aspect of the invention, a technique for monitoring abnormalities in a data stream comprises the following steps. A plurality of objects are received from the data stream, and one or more clusters are created from the plurality of objects. At least a portion of the one or more clusters have statistical data of the respective cluster. It is determined from the statistical data whether one or more abnormalities exist in the data stream.
  • [0009]
    Thus, a framework may be provided in which select statistical data may be stored at regular intervals. This results in a technique which is able to analyze different characteristics of the clusters in an effective manner. Advantageously, the inventive techniques may be useful for clustering different kinds of categorical data sets, and adapting to the rapidly evolving nature of a data stream.
  • [0010]
    Additional advantages of the inventive techniques of the present invention include the ability to explore the clusters in an online fashion, and store statistical data which may be utilized for a better understanding and analysis of the data stream. In applications in which the data stream evolves considerably, different kinds of clusters may assist in understanding the behavior of the data stream over different periods in time. This is advantageous since a fast data stream cannot be repeatedly processed in order to resolve different kinds of queries.
  • [0011]
    These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0012]
    FIG. 1 is a block diagram illustrating a hardware implementation suitable for employing methodologies, according to an embodiment of the present invention;
  • [0013]
    FIG. 2 is a flow diagram illustrating an abnormality monitoring methodology, according to an embodiment of the present invention;
  • [0014]
    FIG. 3 is a flow diagram illustrating a data stream and cluster maintenance methodology, according to an embodiment of the present invention;
  • [0015]
    FIG. 4 is a flow diagram illustrating a data point addition methodology, according to an embodiment of the present invention;
  • [0016]
    FIG. 5 is a flow diagram illustrating a statistical data update methodology, according to an embodiment of the present invention;
  • [0017]
    FIG. 6 is a flow diagram illustrating an abnormality discovery methodology, according to an embodiment of the present invention; and
  • [0018]
    FIG. 7 is a flow diagram illustrating a network intrusion detection methodology, according to an embodiment of the present invention.
  • DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
  • [0019]
    The following description will illustrate the invention using an exemplary data processing system architecture. It should be understood, however, that the invention is not limited to use with any particular system architecture. The invention is instead more generally applicable to any data processing system in which it is desirable to perform efficient and effective data stream clustering. It is to be understood that the phrase “data point,” illustratively used herein, is one example of a data “object.”
  • [0020]
    As will be illustrated in detail below, the present invention introduces techniques for clustering a data stream and, more particularly, techniques for monitoring data abnormalities in the stream through the clustering of the data stream. An abnormality, as referred to herein, is defined as an outlier cluster or outlier data point of the data stream having specifically defined values in the stored statistical data of the data point or cluster. The stored statistical data may include, for example, the number of pairwise attribute values, the number of categorical attribute values, the number of data points, the sum of the weights of the data points, and the time at which the last data point was added to the outlier. A more detailed description of the values of the statistical data required for abnormality determination are provided herein.
  • [0021]
    Referring initially to FIG. 1, a block diagram illustrates a hardware implementation suitable for employing methodologies, according to an embodiment of the present invention. As illustrated, an exemplary system comprises multiple client devices 10 coupled via a large network 20 to a server 30. Server 30 may comprise a central processing unit (CPU) 40 coupled to a main memory 50 and a disk 60. Server 30 may also comprise a cache 70 in order to speed up calculations. Multiple clients 10 can interact with server 30 over large network 20. It is to be appreciated that network 20 may be a public information network such as, for example, the Internet or World Wide Web, however, clients 10 and server 20 may alternatively be connected via a private network, a local area network, or some other suitable network.
  • [0022]
    Data points from a data stream are received at server 30 from an individual client 10 and stored on disk 60. All computations on the data stream are performed by CPU 40. The clustered data points and their corresponding statistical data are stored on disk 60, and are utilized for the purpose of answering a variety of user queries. For example, a data stream may relate to records of a credit card company corresponding to the transactions of their customers. Attributes of these records may include the age and sex of the customer.
  • [0023]
    In another example, the data points of the data stream may relate to records corresponding to user accesses, or customer connections, on a network. The queries for abnormalities in the data stream are searches for intrusions, or hacker actions. For example, a customer may attempt to bring down a web server by making millions of web accesses on the server using an automated machine, such as a crawler. The queries or searches for abnormalities may be initiated by a system administrator.
  • [0024]
    Referring now to FIG. 2, a flow diagram illustrates an abnormality monitoring methodology, according to an embodiment of the present invention. The inventive technique may be divided into two main steps:
      • (1) storage and maintenance of statistical data from the data stream (blocks 202 and 204); and
      • (2) use of statistical data for online abnormality querying (blocks 206-210).
  • [0027]
    The methodology begins at block 202, where data stream maintenance is performed. This maintenance involves receiving data points from the data stream and creating clusters, having associated statistical information. A more detailed description of cluster and data stream maintenance is provided in FIG. 3. In block 204, the statistical data of each cluster is stored by server 30. As described above, the stored statistical data may include the number of pairwise attribute values, the number of categorical attribute values, the number of data points, the sum of the weights of the data points, and the time at which the last data point was added. In accordance with this embodiment of the present invention, categorical data streams are pre-processed in such a way that the statistical information about each cluster is pre-stored at regular intervals. These intervals may be chosen based on a pyramidal distribution, as described in, for example, C. C. Aggarwal et al., “A Framework for Clustering Evolving Data Streams,” VLDB Conference, 2003, the disclosure of which is incorporated by reference herein. This condensed statistical data should satisfy two requirements:
      • (1) The statistical data may be easily updated for a fast data stream. The nature of the statistical information is chosen in such a way that it is possible to perform linear updates; and
      • (2) The statistical data allows for the computation of various analytical measures required by the user. Such measures may include clusters or outliers over a specific time horizon. It is also often desirable to determine the nature of a data stream evolution over a given time horizon.
  • [0030]
    In block 206, a user queries for abnormalities within a specified time horizon (t1, t2). Block 208 receives the query and resolves the query by retrieving stored statistical data of the clusters from block 204. The statistical data is used in order to respond to user queries for abnormalities in block 210, terminating the methodology. A more detailed description of block 210 is provided in FIG. 6.
  • [0031]
    Referring now to FIG. 3, a flow diagram illustrates a data stream and cluster maintenance methodology, according to an embodiment of the present invention. FIG. 3 may be considered a detailed description of block 202 in FIG. 2. The methodology begins at block 302, where a data point is received from the data stream. Similarity values for the data point are then computed, which relate to each existing cluster, in block 304. For example, when a new data point X arrives, its distance to each cluster is computed using a variety of known methods, such as the cosine distance. In block 306, the closest cluster is computed based on a comparison of the computed similarity values. The cluster with the maximum similarity value is chosen as the closest cluster.
  • [0032]
    In block 308, it is determined whether the data point should be added to the closest cluster. A more detailed description of block 308 is provided in FIG. 4. If it is determined that the data point should be added to the closest cluster, the addition is performed in block 310 and the statistical data of the cluster is updated in block 314, terminating the methodology. A more detailed description of block 314 is provided in FIG. 5. However, if it is determined that the data point should not be added to the cluster, a cluster is created containing the single data point in block 312. The statistical data of this cluster is generated using only this single data point in block 314, terminating the methodology.
  • [0033]
    A newly created cluster containing only a single data point may be referred to as a “trend-setter.” From the point of view of a user, a trend-setter is an outlier, until the arrival of other data points certify the fact that it is actually a cluster. If and when a sufficient number of new data points are added to the cluster, it is referred to as a mature cluster. The specific number of data points needed in order to make a mature cluster is application dependent, however, in the intrusion detection application described above, a mature cluster may contain 20-50 data points.
  • [0034]
    At a given moment in time, a mature cluster can either be “active” or “inactive.” A mature cluster is said to be active when it has received data points in the recent past. When a mature cluster has not received data points in the recent past, it is said to be inactive. Again, the specific amount of time that must pass in order for a mature cluster to become inactive is application dependent. However, in the intrusion detection application, an active mature cluster may be a mature cluster that has received data points in the last ten days. In some cases, a trend-setter cluster becomes inactive before it has a chance to mature. Such a cluster typically contains a small number of transient data points, which may typically be the result of an underlying abnormality that is short-term in nature.
  • [0035]
    A set of clusters may be dynamically maintained by effectively scaling with data size. In order to achieve better scalability during data stream maintenance, data structures may be constructed that allow for additive operations on the data points.
  • [0036]
    In order to achieve greater accuracy in the clustering technique, a high level of granularity is maintained in the maintenance of the underlying data structures. This may be achieved through a condensation technique in which groups of data clusters are condensed. These groups of clusters are referred to as cluster droplets.
  • [0037]
    A cluster droplet D(t, C) at time t, and a set of categorical data points C is referred to as a tuple (DF2, DF1, n, w(t), l), in which each statistical component is defined as follows:
      • vector DF2 contains the number of the pairwise attribute values;
      • vector DF1 contains the number of the categorical attribute values;
      • entry n contains the number of data points in the cluster;
      • entry w(t) contains the sum of the weights of the data points at time t (the value w(t) is a function of the time t and decays with time unless new data points are added to the droplet D(t)); and
      • entry l contains the time stamp of the last time that a data point was added to the cluster.
  • [0043]
    Cluster droplet maintenance involves storing the data at a high level of granularity so as to lose the least amount of information. The droplet update technique continuously maintains a set of cluster droplets C1 . . . Ck, which it updates as new data points arrive. For each cluster, the entire set of statistical data is maintained in the droplet. The maximum number of droplets k which are maintained is dependent upon the amount of available main memory 50. In receiving data points, it is first assumed that no clusters exist. As new data points arrive, unit clusters containing individual data points are created. Once a maximum number k of such clusters have been created, the online maintenance of the clusters may begin starting with a trivial set of k clusters which are updated over time with the arrival of new data points.
  • [0044]
    Referring now to FIG. 4, a flow diagram illustrates a data point addition methodology, according to an embodiment of the present invention. This may be considered a detailed description of block 308 of FIG. 3. The methodology begins at block 402, where similarity values of a given data point are computed relative to each cluster centroid. In block 404, it is determined whether the similarity value relating to the closest cluster is larger than a user-defined threshold. The user-defined threshold is chosen based on application dependent considerations regarding level of similarity desired in order for a data point to be considered a natural part of a given cluster. If the similarity value is greater than the user-defined threshold, the data point is reported as a non-outlier in block 408, and may added to the closest cluster, terminating the methodology. If the similarity value is less than or equal to the user-defined threshold, the data point is reported as an outlier in block 406, terminating the methodology.
  • [0045]
    In the case of cluster droplets described above, which maintain a maximum number of droplets k, the cluster with the maximum similarity value is defined as Cmindex. If a similarity value of S(X, Cmindex) is greater than the user-defined threshold, the point X is assigned to the cluster Cmindex. It is also determined whether an inactive cluster exists in the existing set of cluster droplets. If no such inactive cluster exists, then the data point X is added to Cmindex. In the even that the data point X is assigned to the cluster Cmindex, two steps are performed:
      • the statistics are updated to reflect the decay of the data points at the current moment in time; and
      • the statistics for each newly arriving data point are added to the statistics of Cmindex.
  • [0048]
    In the event that the newly arriving data point does not naturally fit in any of the cluster droplets and an inactive cluster does exist, then the most inactive cluster is replaced by a new cluster containing the solitary data point X. The most inactive cluster may be defined as the least recently updated cluster droplet. This new cluster is a potential outlier, or the beginning of a new trend. Further understanding of this new cluster droplet may only be obtained with the progress of the data stream.
  • [0049]
    Referring now to FIG. 5, a flow diagram illustrates a cluster statistical data update methodology, according to an embodiment of the present invention. This may be considered a detailed description of block 314 in FIG. 3. The methodology begins at block 502, where the number of attributes corresponding to pairwise values DF2, or second order statistics, are updated. In block 504, the number of attributes corresponding to individual categories DF1, or first order statistics, are updated. In block 506, the number of data points are updated, and in block 508 the decay statistics are updated, terminating the methodology. Decay statistics relate to weights of the data points and the time at which the last data point was added to the cluster.
  • [0050]
    In order to more fully describe decay statistics, a further description of the data stream is first required. The data stream comprises a set of multi-dimensional records Xl . . . Xk . . . arriving at time stamps Tl . . . Tk . . . . Each Xi is a multi-dimensional categorical record containing d dimensions which are denoted by Xi=(xl i . . . xd i). It is assumed that the ith categorical dimension contains vi possible values. Since the stream clustering technique should attribute greater importance to recent clusters, a time-sensitive weight is provided for each data point. It is assumed that each data point has a weight defined by f(t), which is also referred to as the fading function. The fading function f(t) is a non-monotonic decreasing function which decays uniformly with time t. In order to formalize this concept, the half-life of a point in the data stream is defined as the time at which f(t0)=(1/2) f(0).
  • [0051]
    Conceptually, the aim of defining a half life is to define the rate of decay of the weight assigned to each data point in the stream. Correspondingly, the decay-rate is defined as the inverse of the half life of the data stream. The decay-rate is denoted by λ=1/t0. In order for the half-life property to hold, the weight of each point in the data stream is defined by f(t)=2−λt, creating a half life of 1/λ. In the intrusion detection application described above, a decay rate may be 0.5 per day, thus, having a half-life of two days. However, the decay rate and half-life are application dependent, and therefore may differ from these examples.
  • [0052]
    By changing the value of λ, it is possible to change the rate at which the importance of the historical information in the data stream decays. The higher the value of λ, the lower the importance of the historical information compared to more recent data. By changing the value of this parameter, it is possible to obtain considerable control on the rate at which the historical statistics are allowed to decay. For more stable data streams, it is desirable to pick a smaller value of λ, whereas for rapidly evolving data streams, it is desirable to pick a larger value of λ.
  • [0053]
    Referring now to FIG. 6, a flow diagram illustrates an abnormality discovery methodology, according to an embodiment of the present invention. This may be considered a detailed description of block 210 of FIG. 2. These abnormalities are discovered in the time horizon (t1, t2). The methodology begins at block 602, where it is determined which clusters present at time t(2), were not present at time t(1). In block 604, it is determined which of the clusters satisfying the previous requirement, contain fewer than a defined number of points at time t(2). The clusters having fewer than the defined number of points are reported as the abnormal outliers in block 606, and the methodology terminates.
  • [0054]
    For example, when a new cluster is created during the streaming technique by a newly arriving data point, it is allowed to remain as a trend-setting outlier for at least one half-life. During that period, if at least one more data point is added to the newly formed cluster, it becomes an active and mature cluster. If no new points arrive during a half-life, then the trend-setting outlier is recognized as a true abnormality in the data stream, and the single point cluster is removed from the current set of clusters. Thus, a new cluster containing one data point is removed when the (weighted) number of points in the cluster is 0.5.
  • [0055]
    This criterion is also used for the removal of mature clusters. In other words, a mature cluster is removed when the weighted number of points in that cluster is larger than 0.5. This will happen only when the inactivity period in the cluster has exceeded the half life 1/λ. The greater the number of points in the cluster, the greater the level by which the inactivity period would need to exceed its half life in order to be considered an inactive cluster. This is a natural solution, since it is intuitively desirable to have stronger requirements (a longer inactivity period) for the elimination of a cluster containing a larger number of points.
  • [0056]
    The inventive techniques are applicable to a large number of applications such as systems diagnosis. For example, as described above, the techniques of the present invention may be utilized for online monitoring of network intrusions. Referring now to FIG. 7, a flow diagram illustrates a network intrusion detection methodology, according to an embodiment of the present invention. The goal of the methodology is to find intrusion attacks. The methodology begins by storing a stream of source IP (Internet Protocol) address data, destination IP address data, and signature data at the server, in block 702. Signature data refers to a field in the network logs defining the type of network access. The clustering technique of the invention is utilized in order to create summary or statistical data in block 704. Different weights can be utilized to make the clustering technique more effective. In block 706, this statistical data is used to discover abnormalities or customer actions considered intrusions, terminating the methodology. If desired, an online interface can also be utilized in order to diagnose the abnormalities.
  • [0057]
    Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made by one skilled in the art without departing from the scope or spirit of the invention.

Claims (31)

  1. 1. A method for monitoring abnormalities in a data stream, comprising the steps of:
    receiving a plurality of objects in the data stream;
    creating one or more clusters from the plurality of objects, wherein at least a portion of the one or more clusters comprise statistical data of the respective cluster; and
    determining from the statistical data whether one or more abnormalities exist in the data stream.
  2. 2. The method of claim 1, wherein the step of creating one or more clusters further comprises:
    computing one or more similarity values for a given object relating to one or more existing clusters; and
    determining a closest cluster for the object based on the one or more similarity values.
  3. 3. The method of claim 2, further comprising the steps of:
    determining whether to add the object to the closest cluster;
    adding the object to the closest cluster when determined and updating the statistical data of the closest cluster; and
    creating a new cluster comprising the object when the object is not added to the closest cluster, and generating statistical data of the new cluster.
  4. 4. The method of claim 3, wherein the step of determining whether to add the object to the closest cluster further comprises the step of determining if the similarity value is greater than a user-defined threshold.
  5. 5. The method of claim 1, wherein the step of determining from the statistical data whether one or more abnormalities exist further comprises the steps of:
    determining which clusters present at a first time were not present at a second time, wherein the second time is before the first time;
    determining which of the clusters, present at the first time and not present at the second time, contain fewer than a user-defined number of objects; and
    reporting clusters with fewer than the user-defined number of objects as abnormalities.
  6. 6. The method of claim 1, wherein the statistical data of each cluster is stored using an incremental updating process.
  7. 7. The method of claim 1, wherein the statistical data of each cluster comprises one or more statistical counts of each pairwise attribute.
  8. 8. The method of claim 1, wherein the statistical data of each cluster comprises one or more statistical counts of each categorical attribute.
  9. 9. The method of claim 1, wherein the statistical data of each cluster comprises a number of objects in each cluster.
  10. 10. The method of claim 1, wherein the statistical data is stored periodically at intervals chosen based on a pyramidal distribution.
  11. 11. The method of claim 1, wherein the step of creating one or more clusters further comprises the step of applying one or more weights to one or more attributes.
  12. 12. The method of claim 1, wherein abnormalities comprise intrusions in a network.
  13. 13. The method of claim 12, wherein the step of receiving a plurality of objects further comprises the step of collecting source IP (Internet Protocol) address data, destination IP address data and signature data.
  14. 14. The method of claim 12, wherein the step of creating one or more clusters further comprises the step of clustering source IP address data, destination IP address data and signature data.
  15. 15. The method of claim 12, wherein the step of determining from the statistical data whether one or more abnormalities exist further comprises the step of detecting one or more intrusions from statistical data of source IP address data, destination IP address data and signature data.
  16. 16. Apparatus for monitoring abnormalities in a data stream, comprising:
    a memory; and
    at least one processor coupled to the memory and operative to: (i) receive a plurality of objects in the data stream; (ii) create one or more clusters from the plurality of objects, wherein at least a portion of the one or more clusters comprise statistical data of the respective cluster; and (iii) determine from the statistical data whether one or more abnormalities exist in the data stream.
  17. 17. The apparatus of claim 16, wherein the operation of creating one or more clusters further comprises:
    computing one or more similarity values for a given object relating to one or more existing clusters; and
    determining a closest cluster for the object based on the one or more similarity values.
  18. 18. The apparatus of claim 17, further comprising:
    determining whether to add the object to the closest cluster;
    adding the object to the closest cluster when determined and updating the statistical data of the closest cluster; and
    creating a new cluster comprising the object when the object is not added to the closest cluster, and generating statistical data of the new cluster.
  19. 19. The apparatus of claim 18, wherein determining whether to add the object to the closest cluster further comprises determining if the similarity value is greater than a user defined threshold.
  20. 20. The apparatus of claim 17, wherein the operation of determining from the statistical data whether one or more abnormalities exist further comprises:
    determining which clusters present at a first time were not present at a second time, wherein the second time is before the first time;
    determining which of the clusters, present at the first time and not present at the second time, contain fewer than a user defined number of objects; and
    reporting clusters with fewer than a defined number of objects as abnormalities.
  21. 21. The apparatus of claim 16, wherein the statistical data of each cluster is stored using an incremental updating process.
  22. 22. The apparatus of claim 16, wherein the statistical data of each cluster comprises one or more statistical counts of each pairwise attribute.
  23. 23. The apparatus of claim 16, wherein the statistical data of each cluster comprises one or more statistical counts of each categorical attribute.
  24. 24. The apparatus of claim 16, wherein the statistical data of each cluster comprises a number of objects in each cluster.
  25. 25. The apparatus of claim 16, wherein the statistical data is stored periodically at intervals chosen based on a pyramidal distribution.
  26. 26. The apparatus of claim 16, wherein the operation of creating one or more clusters further comprises applying one or more weights to one or more attributes.
  27. 27. The apparatus of claim 16, wherein abnormalities comprise intrusions in a network.
  28. 28. The apparatus of claim 27, wherein the operation of receiving a plurality of objects further comprises collecting source IP address data, destination IP address data and signature data.
  29. 29. The apparatus of claim 27, wherein the operation of creating one or more clusters further comprises clustering source IP address data, destination IP address data and signature data.
  30. 30. The apparatus of claim 27, wherein the operation of determining from the statistical data whether one or more abnormalities exist further comprises detecting one or more intrusions from statistical data of source IP address data, destination IP address data, and signature data.
  31. 31. An article of manufacture for monitoring abnormalities in a data stream, comprising a machine readable medium containing one or more programs which when executed implement the steps of:
    receiving a plurality of objects in the data stream;
    creating one or more clusters from the plurality of objects, wherein at least a portion of the one or more clusters comprise statistical data of the respective cluster; and
    determining from the statistical data whether one or more abnormalities exist in the data stream.
US10801420 2004-03-16 2004-03-16 Methods and apparatus for data stream clustering for abnormality monitoring Abandoned US20050210027A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10801420 US20050210027A1 (en) 2004-03-16 2004-03-16 Methods and apparatus for data stream clustering for abnormality monitoring

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10801420 US20050210027A1 (en) 2004-03-16 2004-03-16 Methods and apparatus for data stream clustering for abnormality monitoring
US11753232 US7970772B2 (en) 2004-03-16 2007-05-24 Methods and apparatus for data stream clustering for abnormality monitoring

Publications (1)

Publication Number Publication Date
US20050210027A1 true true US20050210027A1 (en) 2005-09-22

Family

ID=34987581

Family Applications (2)

Application Number Title Priority Date Filing Date
US10801420 Abandoned US20050210027A1 (en) 2004-03-16 2004-03-16 Methods and apparatus for data stream clustering for abnormality monitoring
US11753232 Active 2024-12-30 US7970772B2 (en) 2004-03-16 2007-05-24 Methods and apparatus for data stream clustering for abnormality monitoring

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11753232 Active 2024-12-30 US7970772B2 (en) 2004-03-16 2007-05-24 Methods and apparatus for data stream clustering for abnormality monitoring

Country Status (1)

Country Link
US (2) US20050210027A1 (en)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060218144A1 (en) * 2005-03-28 2006-09-28 Microsoft Corporation Systems and methods for performing streaming checks on data format for UDTs
US20060227799A1 (en) * 2005-04-08 2006-10-12 Lee Man-Ho L Systems and methods for dynamically allocating memory for RDMA data transfers
US20070124801A1 (en) * 2005-11-28 2007-05-31 Threatmetrix Pty Ltd Method and System for Tracking Machines on a Network Using Fuzzy Guid Technology
US20070203903A1 (en) * 2006-02-28 2007-08-30 Ilial, Inc. Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface
US20070239636A1 (en) * 2006-03-15 2007-10-11 Microsoft Corporation Transform for outlier detection in extract, transfer, load environment
US20070245035A1 (en) * 2006-01-19 2007-10-18 Ilial, Inc. Systems and methods for creating, navigating, and searching informational web neighborhoods
EP1916828A1 (en) * 2006-10-27 2008-04-30 Sony France S.A. Event-detection in multi-channel sensor-signal streams
US20090281900A1 (en) * 2008-05-06 2009-11-12 Netseer, Inc. Discovering Relevant Concept And Context For Content Node
US7690037B1 (en) * 2005-07-13 2010-03-30 Symantec Corporation Filtering training data for machine learning
US20100114879A1 (en) * 2008-10-30 2010-05-06 Netseer, Inc. Identifying related concepts of urls and domain names
US7881255B1 (en) * 2004-09-29 2011-02-01 Google Inc. Systems and methods for relating network traffic using traffic-based signatures
GB2472289A (en) * 2009-07-27 2011-02-02 Ericsson Telefon Ab L M Outlier detection in streaming data
US8176178B2 (en) 2007-01-29 2012-05-08 Threatmetrix Pty Ltd Method for tracking machines on a network using multivariable fingerprinting of passively available information
CN102597992A (en) * 2009-07-22 2012-07-18 卡罗林·帕特里希亚·麦格瑞戈 System, method and computer program for multi-dimensional temporal data mining
US20120317117A1 (en) * 2011-06-08 2012-12-13 Hitachi Solutions, Ltd. Information Visualization System
US8380721B2 (en) 2006-01-18 2013-02-19 Netseer, Inc. System and method for context-based knowledge search, tagging, collaboration, management, and advertisement
US8763113B2 (en) 2005-11-28 2014-06-24 Threatmetrix Pty Ltd Method and system for processing a stream of information from a computer network using node based reputation characteristics
US8825654B2 (en) 2005-05-10 2014-09-02 Netseer, Inc. Methods and apparatus for distributed community finding
US9100344B2 (en) 2006-12-26 2015-08-04 Wipro Limited Label-based partitioning for network subscribers
US9110985B2 (en) 2005-05-10 2015-08-18 Neetseer, Inc. Generating a conceptual association graph from large-scale loosely-grouped content
US9444839B1 (en) 2006-10-17 2016-09-13 Threatmetrix Pty Ltd Method and system for uniquely identifying a user computer in real time for security violations using a plurality of processing parameters and servers
US9817902B2 (en) 2006-10-27 2017-11-14 Netseer Acquisition, Inc. Methods and apparatus for matching relevant content to user intention

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8484269B2 (en) 2008-01-02 2013-07-09 At&T Intellectual Property I, L.P. Computing time-decayed aggregates under smooth decay functions
US8391164B2 (en) * 2008-01-02 2013-03-05 At&T Intellectual Property I, L.P. Computing time-decayed aggregates in data streams
US20100306027A1 (en) * 2009-06-02 2010-12-02 International Business Machines Corporation Net-Metering In A Power Distribution System
US8135499B2 (en) 2009-07-07 2012-03-13 International Business Machines Corporation Load shedding of a selected substation by an electric utility
US9195713B2 (en) * 2009-11-08 2015-11-24 Hewlett-Packard Development Company, L.P. Outlier data point detection
US8892490B2 (en) * 2010-09-22 2014-11-18 Hewlett-Packard Development Company, L.P. Determining whether a point in a data stream is an outlier using hierarchical trees
US9342802B2 (en) * 2011-08-05 2016-05-17 Newswhip Media Limited System and method of tracking rate of change of social network activity associated with a digital object
CN102271091B (en) * 2011-09-06 2013-09-25 电子科技大学 Method for classifying network abnormal events
US9898513B2 (en) 2011-12-12 2018-02-20 University Of Ontario Institute Of Technology System, method and computer program for multi-dimensional temporal and relative data mining framework, analysis and sub-grouping
US8914371B2 (en) * 2011-12-13 2014-12-16 International Business Machines Corporation Event mining in social networks
US8909605B1 (en) * 2013-02-28 2014-12-09 Emc Corporation Method and system for accelerating data movement using change information concerning difference between current and previous data movements
CN104123448B (en) * 2014-07-14 2017-05-17 南京理工大学 Abnormality detection method based on multi-stream data context
US20170324634A1 (en) * 2016-05-09 2017-11-09 Level 3 Communications, Llc Monitoring network traffic to determine similar content

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020107858A1 (en) * 2000-07-05 2002-08-08 Lundahl David S. Method and system for the dynamic analysis of data
US20020161763A1 (en) * 2000-10-27 2002-10-31 Nong Ye Method for classifying data using clustering and classification algorithm supervised
US20030158855A1 (en) * 2002-02-20 2003-08-21 Farnham Shelly D. Computer system architecture for automatic context associations
US6625585B1 (en) * 2000-02-18 2003-09-23 Bioreason, Inc. Method and system for artificial intelligence directed lead discovery though multi-domain agglomerative clustering
US20040098617A1 (en) * 2002-11-18 2004-05-20 Research Foundation Of The State University Of New York Specification-based anomaly detection
US6947933B2 (en) * 2003-01-23 2005-09-20 Verdasys, Inc. Identifying similarities within large collections of unstructured data
US7072891B2 (en) * 2001-10-17 2006-07-04 Korea Advanced Institute Of Science & Technology Apparatus and method for hyper-rectangle based multidimensional data segmentation and clustering
US7227985B2 (en) * 2002-03-05 2007-06-05 Fuji Xerox Co., Ltd. Data classifier for classifying pattern data into clusters

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6460036B1 (en) * 1994-11-29 2002-10-01 Pinpoint Incorporated System and method for providing customized electronic newspapers and target advertisements

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6625585B1 (en) * 2000-02-18 2003-09-23 Bioreason, Inc. Method and system for artificial intelligence directed lead discovery though multi-domain agglomerative clustering
US20020107858A1 (en) * 2000-07-05 2002-08-08 Lundahl David S. Method and system for the dynamic analysis of data
US20020161763A1 (en) * 2000-10-27 2002-10-31 Nong Ye Method for classifying data using clustering and classification algorithm supervised
US7072891B2 (en) * 2001-10-17 2006-07-04 Korea Advanced Institute Of Science & Technology Apparatus and method for hyper-rectangle based multidimensional data segmentation and clustering
US20030158855A1 (en) * 2002-02-20 2003-08-21 Farnham Shelly D. Computer system architecture for automatic context associations
US7227985B2 (en) * 2002-03-05 2007-06-05 Fuji Xerox Co., Ltd. Data classifier for classifying pattern data into clusters
US20040098617A1 (en) * 2002-11-18 2004-05-20 Research Foundation Of The State University Of New York Specification-based anomaly detection
US6947933B2 (en) * 2003-01-23 2005-09-20 Verdasys, Inc. Identifying similarities within large collections of unstructured data

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7881255B1 (en) * 2004-09-29 2011-02-01 Google Inc. Systems and methods for relating network traffic using traffic-based signatures
US7571153B2 (en) * 2005-03-28 2009-08-04 Microsoft Corporation Systems and methods for performing streaming checks on data format for UDTs
US20060218144A1 (en) * 2005-03-28 2006-09-28 Microsoft Corporation Systems and methods for performing streaming checks on data format for UDTs
US20060227799A1 (en) * 2005-04-08 2006-10-12 Lee Man-Ho L Systems and methods for dynamically allocating memory for RDMA data transfers
US8838605B2 (en) 2005-05-10 2014-09-16 Netseer, Inc. Methods and apparatus for distributed community finding
US8825654B2 (en) 2005-05-10 2014-09-02 Netseer, Inc. Methods and apparatus for distributed community finding
US9110985B2 (en) 2005-05-10 2015-08-18 Neetseer, Inc. Generating a conceptual association graph from large-scale loosely-grouped content
US7690037B1 (en) * 2005-07-13 2010-03-30 Symantec Corporation Filtering training data for machine learning
US8141148B2 (en) * 2005-11-28 2012-03-20 Threatmetrix Pty Ltd Method and system for tracking machines on a network using fuzzy GUID technology
US8763113B2 (en) 2005-11-28 2014-06-24 Threatmetrix Pty Ltd Method and system for processing a stream of information from a computer network using node based reputation characteristics
US9449168B2 (en) 2005-11-28 2016-09-20 Threatmetrix Pty Ltd Method and system for tracking machines on a network using fuzzy guid technology
US8782783B2 (en) 2005-11-28 2014-07-15 Threatmetrix Pty Ltd Method and system for tracking machines on a network using fuzzy guid technology
US20070124801A1 (en) * 2005-11-28 2007-05-31 Threatmetrix Pty Ltd Method and System for Tracking Machines on a Network Using Fuzzy Guid Technology
US8380721B2 (en) 2006-01-18 2013-02-19 Netseer, Inc. System and method for context-based knowledge search, tagging, collaboration, management, and advertisement
US20070245035A1 (en) * 2006-01-19 2007-10-18 Ilial, Inc. Systems and methods for creating, navigating, and searching informational web neighborhoods
US8825657B2 (en) * 2006-01-19 2014-09-02 Netseer, Inc. Systems and methods for creating, navigating, and searching informational web neighborhoods
US9443018B2 (en) * 2006-01-19 2016-09-13 Netseer, Inc. Systems and methods for creating, navigating, and searching informational web neighborhoods
US20140351237A1 (en) * 2006-01-19 2014-11-27 Netseer, Inc. Systems and methods for creating, navigating, and searching informational web neighborhoods
US20070203903A1 (en) * 2006-02-28 2007-08-30 Ilial, Inc. Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface
US8843434B2 (en) 2006-02-28 2014-09-23 Netseer, Inc. Methods and apparatus for visualizing, managing, monetizing, and personalizing knowledge search results on a user interface
US20070239636A1 (en) * 2006-03-15 2007-10-11 Microsoft Corporation Transform for outlier detection in extract, transfer, load environment
US7565335B2 (en) * 2006-03-15 2009-07-21 Microsoft Corporation Transform for outlier detection in extract, transfer, load environment
US9444839B1 (en) 2006-10-17 2016-09-13 Threatmetrix Pty Ltd Method and system for uniquely identifying a user computer in real time for security violations using a plurality of processing parameters and servers
US9332020B2 (en) 2006-10-17 2016-05-03 Threatmetrix Pty Ltd Method for tracking machines on a network using multivariable fingerprinting of passively available information
US9444835B2 (en) 2006-10-17 2016-09-13 Threatmetrix Pty Ltd Method for tracking machines on a network using multivariable fingerprinting of passively available information
US7996189B2 (en) 2006-10-27 2011-08-09 Sony France S.A. Event-detection in multi-channel sensor-signal streams
US20090099820A1 (en) * 2006-10-27 2009-04-16 Sony France S.A. Event-detection in multi-channel sensor-signal streams
EP1916828A1 (en) * 2006-10-27 2008-04-30 Sony France S.A. Event-detection in multi-channel sensor-signal streams
US20100042371A2 (en) * 2006-10-27 2010-02-18 Sony France S.A. Event-detection in multi-channel sensor-signal streams
US9817902B2 (en) 2006-10-27 2017-11-14 Netseer Acquisition, Inc. Methods and apparatus for matching relevant content to user intention
US9100344B2 (en) 2006-12-26 2015-08-04 Wipro Limited Label-based partitioning for network subscribers
US8176178B2 (en) 2007-01-29 2012-05-08 Threatmetrix Pty Ltd Method for tracking machines on a network using multivariable fingerprinting of passively available information
US20090281900A1 (en) * 2008-05-06 2009-11-12 Netseer, Inc. Discovering Relevant Concept And Context For Content Node
US8417695B2 (en) 2008-10-30 2013-04-09 Netseer, Inc. Identifying related concepts of URLs and domain names
US20100114879A1 (en) * 2008-10-30 2010-05-06 Netseer, Inc. Identifying related concepts of urls and domain names
CN102597992A (en) * 2009-07-22 2012-07-18 卡罗林·帕特里希亚·麦格瑞戈 System, method and computer program for multi-dimensional temporal data mining
GB2472289A (en) * 2009-07-27 2011-02-02 Ericsson Telefon Ab L M Outlier detection in streaming data
US20120317117A1 (en) * 2011-06-08 2012-12-13 Hitachi Solutions, Ltd. Information Visualization System

Also Published As

Publication number Publication date Type
US20070226212A1 (en) 2007-09-27 application
US7970772B2 (en) 2011-06-28 grant

Similar Documents

Publication Publication Date Title
Cormode et al. Finding frequent items in data streams
Tao et al. Spatio-temporal aggregation using sketches
Levandoski et al. Lars: A location-aware recommender system
Das et al. Detecting anomalous records in categorical datasets
Liu et al. Partially supervised classification of text documents
Huang et al. Continuous skyline queries for moving objects
Cormode et al. An improved data stream summary: The count-min sketch and its applications
US7092941B1 (en) Clustering module for data mining
US20100280857A1 (en) Modeling multi-dimensional sequence data over streams
US6567936B1 (en) Data clustering using error-tolerant frequent item sets
Wang et al. Mining concept-drifting data streams using ensemble classifiers
US7523130B1 (en) Storing and retrieving objects on a computer network in a distributed database
Yen et al. An efficient approach to discovering knowledge from large databases
US7061874B2 (en) Method, system and computer program product for classifying packet flows with a bit mask
Chen et al. Density-based clustering for real-time stream data
Datar et al. Estimating rarity and similarity over data stream windows
US20060085592A1 (en) Method for distinct count estimation over joins of continuous update stream
Aggarwal et al. A framework for clustering uncertain data streams
Sun et al. Beyond streams and graphs: dynamic tensor analysis
Ordonez et al. FREM: fast and robust EM clustering for large data sets
Beyer et al. On synopses for distinct-value estimation under multiset operations
US5542089A (en) Method and apparatus for estimating the number of occurrences of frequent values in a data set
US7278156B2 (en) System and method for enforcing security service level agreements
Cormode et al. What's new: Finding significant differences in network data streams
Bulut et al. A unified framework for monitoring data streams in real time

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:AGGARWAL, CHARU C.;YU, PHILIP SHI-LUNG;REEL/FRAME:014696/0502

Effective date: 20040329