KR102136222B1 - System and method for clustering graph data and computer program for the same - Google Patents

Abstract

A system for clustering graph data includes: a graph database storage unit configured to store graph data including one or more nodes and one or more edges between the nodes; an event processing unit configured to receive a user request, an event occurrence, or a clustering request according to a preset calling period, and to provide a clustering result for the graph data; and a clustering processing unit configured to generate the clustering result from the graph data by using one selected from a plurality of preset community detection algorithms when called by the event processing unit. The clustering processing unit may determine one community detection algorithm by comparing the clustering result of each of the plurality of community detection algorithms with a preset correct answer. Accordingly, by automating the selection of the community detection algorithm, it is possible to overcome limitations on input and time that can be input, and continuously search for graph data without the intervention of an analyst.

Description

Graph data clustering system and method and computer program therefor {SYSTEM AND METHOD FOR CLUSTERING GRAPH DATA AND COMPUTER PROGRAM FOR THE SAME}

Embodiments relate to a graph data clustering system and method and a computer program for the same, and more specifically, to find a basic answer sheet through a continuous association of graph data that is stored and managed, and to detect various communities (community) detection) It is about the technique of comparing the cluster result of the algorithm with the correct answer sheet and automatically selecting the community detection algorithm through this.

With the recent development of measurement technology and the development of storage technology, various types of data that were not previously interested can be stored in high resolution, and an analysis technology that obtains new insights from these additional data has emerged as an important technology element.

This paradigm shift in information and communication technology has led to the development of various data models and data storage management technologies to represent various forms of the real world, and technologies to visualize them are also being advanced. Typical examples are graph data types and graph database technologies for dealing with complex data objects.

Compared to a relational database that stores each object according to a schema structure type that defines a table in advance, the graph database stores each data as one object and continuously maintains the relationship between objects without compromising the characteristics of each object. It is advantageous for the integrated storage management of structured and unstructured data.

For example, FIG. 1 shows an exemplary form of graph data stored in a graph database, and shows the relationship between [R1: A → B], [R2: A+B → C+D], and [R3: D → E]. Eggplant is a graph showing the chemical reaction. A to E represent reaction substances participating in the chemical reaction, which correspond to each node of the graph data. R1 to R3 represent chemical reactions, which correspond to each edge of the graph data.

The storage structure characteristics of graph data provide an environment suitable for storing nodes (or vertices) having various forms and edge data objects and managing relationships between different data. Such a large number of nodes can form a community, which is a cluster of data objects that have various interrelated or related characteristics according to the association, and through this, the nodes are classified into the same community even if they are not directly related. If you can, you can indirectly find a connection.

For example, when cyber security and threat information data is expressed as graph data, there are files, distribution paths, and attack IPs used by a specific hacker group A while invading, and files and distribution paths used by a specific hacker group B in the intrusion activity. It is assumed that there is an attack IP. In this case, if the hacker groups A and B share a specific resource or use the same diffusion path, it may be found through graph pattern search, but even if not, the hacker group among the numerous communities classified by applying the community detection algorithm If A and B belong to the same community, it is possible to infer the possibility that both groups are the same group through the results of clustering even though these two groups are actually known as different groups.

There are a number of such community detection algorithms, and the representative algorithms include the Louvain method, the Givan-Newman method, the Infomap algorithm, and the Newman-spectral algorithm. Each algorithm has a different method of utilizing each other, and the results derived according to the shape of the graph data are different. Therefore, it is not easy to select and apply a suitable community detection algorithm without the help of an expert in using it in a repository having different graph models, and it is not easy to verify its suitability even when applied.

Registered Patent Publication No. 10-1697875

According to an aspect of the present invention, a basic correct answer sheet is continuously found through a correlation with respect to graph data that is stored and managed, and a cluster result of various community detection algorithms is compared with the correct answer sheet and verified automatically. By selecting a community detection algorithm, it is possible to provide a graph data clustering system and method capable of optimized clustering, and a computer program therefor.

A graph data clustering system according to an embodiment includes: a graph database storage unit configured to store graph data including one or more nodes and one or more edges between the nodes; An event processing unit configured to receive a user request, an event occurrence, or a clustering request according to a preset call cycle, and provide clustering results for the graph data; And a clustering processing unit that is called by the event processing unit and configured to generate the clustering result from the graph data by using any one community detection algorithm selected from a plurality of preset community detection algorithms.

The clustering processing unit is further configured to determine a community detection algorithm by comparing a cluster result by each of the plurality of community detection algorithms with a preset correct answer sheet.

In one embodiment, the clustering processing unit is further configured to determine any one community detection algorithm based on the clustering result of each of the plurality of community detection algorithms and the normalized mutual information value of the correct answer sheet.

In one embodiment, the clustering processing unit, a clustering correct answer processor configured to receive and store the correct answer sheet; A community detection algorithm performer configured to generate a cluster result by each of the plurality of community detection algorithms; A verification calculator configured to generate the normalized cross information value; And an algorithm selector configured to determine a cluster result using the community detection algorithm having the largest normalized mutual information value among the plurality of community detection algorithms as the clustering result.

의 수학식에 의하여 I(A, B)를 상기 정규화된 상호 정보 값으로 결정하도록 더 구성된다.In one embodiment, the verification calculator, the first cluster result corresponding to the correct answer sheet A, the second cluster result by the community detection algorithm B, the number of clusters of the first cluster result c _A , the second The number of clusters in the cluster result is C _B , the number of nodes in the graph data is N, and the number of nodes belonging to the i th cluster, which is an arbitrary natural number in the first cluster result, is N _i , and the second cluster result is random. If the number of nodes belonging to the j-th cluster, which is a natural number, is N _j , and the number of nodes belonging to the j-th cluster of the second cluster result among the nodes belonging to the i-th cluster of the first cluster result is N _ij ,

It is further configured to determine I (A, B) as the normalized mutual information value by the equation of.

A graph data clustering method according to an embodiment may include storing graph data including one or more nodes and one or more edges between the nodes in a graph data clustering system; Receiving, by the graph data clustering system, a user's request, an event occurrence, or a clustering request by a preset call cycle; Determining, by the graph data clustering system, a community detection algorithm of one of the plurality of community detection algorithms by comparing a cluster result of each of the plurality of community detection algorithms with a preset correct answer sheet; Generating, by the graph data clustering system, the clustering result from the graph data using the determined community detection algorithm; And the graph data clustering system providing the clustering result.

In one embodiment, the step of determining any one of the community detection algorithms includes: calculating, by the graph data clustering system, a cluster result by each of the plurality of community detection algorithms and a normalized mutual information value of the correct answer sheet. Includes.

In one embodiment, the determining of any one of the community detection algorithms may include: the graph data clustering system, the community detection algorithm having the largest normalized mutual information value among the plurality of community detection algorithms, And determining with a detection algorithm.

의 수학식에 의하여 I(A, B)를 상기 정규화된 상호 정보 값으로 결정하는 단계를 포함한다.In one embodiment, the step of calculating the normalized cross-information value includes: the graph data clustering system A, a first cluster result corresponding to the correct answer sheet A, and a second cluster result by the community detection algorithm B, and The number of clusters in the first cluster result c _A , the number of clusters in the second cluster result c _B , the number of nodes in the graph data N, and the node belonging to the i th cluster, which is an arbitrary natural number in the first cluster result The number of N _i , the number of nodes belonging to the j th cluster, which is an arbitrary natural number in the second cluster result, is N _j , and the j th of the second cluster result among nodes belonging to the i th cluster of the first cluster result. If the number of nodes in the cluster is N _ij ,

And determining I(A, B) as the normalized mutual information value according to Equation of

The computer program according to an embodiment may be stored in a computer-readable recording medium as a combination of hardware and for executing a graph data clustering method according to the above-described embodiments.

Using the graph data clustering system and method according to an aspect of the present invention, an optimal community detection algorithm through an automated community detection algorithm selection process for a classification task of continuously changing graph data Can be applied, and as a result, it is possible to manage clusters in continuous and persistent graph data.

For example, according to the graph data clustering system and method according to an aspect of the present invention, whenever a new information is stored in a user's graph data, a user selects a suitable community detection algorithm automatically instead of manually classifying it and bases it By performing the clustering, it overcomes the limitations on manpower and time that can be put into analysis and has the advantage of efficient use of manpower and time.

The graph data clustering system and method according to an aspect of the present invention is a technology based on artificial intelligence (AI) technology that continuously searches for common information without an analyst's intervention through automated optimal algorithm search. Can be utilized.

1 is a conceptual diagram showing an exemplary form of graph data stored in a graph database.
2 is a schematic block diagram of a graph data clustering system according to an embodiment.
3 is a flowchart of a graph data clustering method according to an embodiment.
4 is exemplary original graph data processed by a graph data clustering method according to an embodiment.
FIG. 5 is an image showing a correct answer sheet for clustering on the original graph data of FIG. 4.
6A to 6C are images showing clustering results by various community detection algorithms for the original graph data of FIG. 4.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

2 is a schematic block diagram of a graph data clustering system according to an embodiment.

In one embodiment, the graph data clustering system includes an event processing unit 10, a clustering processing unit 20, and a graph database storage unit 30. In one embodiment, the clustering processing unit 20 includes a correct answer sheet processor 21, a community detection algorithm performer 22, a verification calculator 23, and an algorithm selector 24.

The graph data clustering system according to the embodiments may have aspects that are entirely hardware, or partially hardware and partially software. For example, each unit (10, 20, 30) included in the graph data clustering system may collectively refer to hardware and software related to processing data of a specific format and content and/or electronic communication. have. The terms "part", "module", "device", "terminal", "server" or "system" in this specification are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware can be a data processing device that includes a CPU or other processor. Also, software driven by hardware may refer to a running process, an object, an executable, a thread of execution, a program, or the like.

In addition, each element constituting the graph data clustering system according to the present embodiment is not intended to refer to separate devices that are physically separated from each other. That is, the event processing unit 10, the clustering processing unit 20 and the graph database storage unit 30 of FIG. 2 are only functionally classifying hardware constituting the graph data clustering system according to operations performed by the corresponding hardware. Each part does not necessarily have to be provided independently of each other. Of course, depending on the embodiment, one or more of the event processing unit 10, the clustering processing unit 20, and the graph database storage unit 30 may be implemented as separate devices that are physically separated from each other.

The graph database storage unit 30 stores a graph database including one or more graph data. The graph data of the graph database stored in the graph database storage unit 30 includes a plurality of nodes corresponding to each data object, and edges connecting each node and indicating an association between the data objects. . At this time, a plurality of nodes may be simultaneously connected to one edge. Also, each edge may be a directional vector. Further, in embodiments, the graph data may be a property graph in which multiple relationships and weights may exist at edges between objects, but is not limited thereto.

In the graph data clustering system according to the present embodiment, when clustering of graph data of the graph database as described above is to be performed, a community detection algorithm to be applied by comparing a cluster result by various community detection algorithms with a preset correct answer sheet Decide.

For this purpose, the event processing unit 10 may call the clustering processing unit 20 periodically or periodically according to a clustering request from a user occurring according to a specific time or specific event. In this case, the clustering request means a request for classifying data objects into one or a plurality of clusters (or clusters) based on an association between data objects in the graph database.

The clustering processing unit 20 is configured to perform various community detection algorithms, such as the Louvain method, the Givan-Newman method, the Infomap algorithm, and the Newman-spectral algorithm. When called by the event processing unit 10, the clustering processing unit 20 automatically compares a cluster result by each of these algorithms with a preset correct answer sheet, and automatically determines a community detection algorithm for applying to the clustering result to be provided to the user. It is configured to.

In this specification, the correct answer sheet refers to a result of clustering graph data by a classification method in which a certain level of effectiveness is verified for a plurality of information in the technical field to which the present invention belongs, and the correct answer sheet is a graph according to an analyst or embodiments It can be set in advance by a user of data clustering.

The event processing unit 10 may provide a user with a clustering result of graph data generated by the above process. For example, the clustering result may mean a result in which nodes of graph data are clustered into one or more groups. The event processing unit 10 may display the clustering result through a display unit (not shown) or provide the user with a data packet that can be displayed on a user device (not shown).

3 is a flowchart of a graph data clustering method according to an embodiment. For convenience of description, a graph data clustering method according to this embodiment will be described below with reference to FIGS. 2 and 3.

First, the event processing unit 10 may check whether an event processing cycle is needed to determine whether clustering of graph data stored in the graph database storage unit 30 is necessary (S1). At this time, the processing cycle does not necessarily mean that a certain time has arrived, and may be, for example, a user's request or a specific event such as modification of graph data. In addition, a user request for graph data clustering may be input from the user to the event processing unit 10 or may be received from the user device (not shown) to the event processing unit 10 in a communication method through a wired and/or wireless network. .

If the check result corresponds to the call cycle (when a user requests, when an event occurs, or when a predetermined time arrives, etc.) (S2), the event processing unit 10 may call the correct answer sheet processor 21 of the clustering processing unit 20 Yes (S3). The correct answer sheet processor 21 is a part in which the correct answer sheet is set in advance. If a correct answer sheet is input to the correct answer sheet processor 21 (S4), the community detection algorithm performer 22 performs a Louvain method, a Givan-Newman method, and an Infomap for comparison with the correct answer sheet. Clustering of data objects by various community detection algorithms such as an algorithm and a Newman-spectral algorithm is performed (S6).

On the other hand, if no correct answer sheet is input to the correct answer sheet processor 21, the correct answer sheet processor 21 may calculate a clustering result corresponding to the correct answer sheet based on the graph data stored in the graph database storage unit 30 (S5). . This may mean obtaining a cluster result by applying a predetermined criterion known to be useful in the field to the graph data stored in the graph database storage unit 30. For example, if there are different email information associated with a specific IP address, the email itself is different from each other, but can be identified as the same attacker through the use of the same IP address. Therefore, a correct answer sheet can be generated by making e-mail addresses having the same IP address as one cluster. However, this is an example, and the answer sheet may be generated using any other criteria known to be useful for the target graph data.

By the community detection algorithm performer 22, all stored in the community detection algorithm performer 22, such as the Louvain method, the Givan-Newman method, the Infomap algorithm, and the Newman-spectral algorithm, etc. If the algorithm is performed (S7), the execution result for each algorithm may be stored in the community detection algorithm performer 22 (S8).

Next, the verification calculator 23 and the algorithm selector 24 may determine the community detection algorithm to be applied by comparing the cluster result by each of the plurality of community detection algorithms described above with a preset correct answer sheet. In one embodiment, the verification calculator 23 may determine a community detection algorithm to be applied based on the normalized mutual information (NMI) value between the cluster result and the correct answer by each of the plurality of community detection algorithms.

Specifically, the verification calculator 23 may generate a two-dimensional list N _ij by configuring the number of clusters corresponding to the correct answer sheet as a row, and configuring the number of clusters by the community detection algorithm to be verified as a column. For example, if the number of clusters in the correct answer sheet is i and the number of clusters in the community detection algorithm to be verified is 9, N _ij is [[0, 16, 0, 0, 0, 0, 0, 0, 0], [ 0, 2, 7, 0, 0, 0, 0, 0, 0], ...] is a two-dimensional array having 9 columns and i rows. Next, the verification calculator 23 is to be verified The cluster result by the community detection algorithm can be compared with the correct answer sheet to calculate the NMI between the two in the range of 0 to 1 (S9). Specifically, the value of each term of the above-described two-dimensional array indicates how many clusters of the community detection algorithm are included in the i-th cluster of the correct answer sheet, and the same data exists in both clusters, Value increases. For example, the first term [0, 16, 0, 0, 0, 0, 0, 0, 0] of the 2D array is in the second cluster by the community detection algorithm that the data included in cluster 1 of the correct answer sheet will verify. This means that 16 have been found.

If the above process is automated and iteratively performed, it is possible to calculate the cluster result by the community detection algorithm to be verified and the NMI of the correct answer sheet. Specifically, the first cluster result corresponding to the correct answer sheet A, the second cluster result by the community detection algorithm B, the number of clusters of the first cluster result c _A , the number of clusters of the second cluster result c _B , the target graph N the number of nodes in the data, N the number of nodes belonging to the i th cluster, which is an arbitrary natural number in the first cluster result, N _i , N the number of nodes belonging to the j th cluster, which is an arbitrary natural number in the second cluster result. _j , If the number of nodes belonging to the j-th cluster of the second cluster result among the nodes belonging to the i-th cluster of the first cluster result is N _ij , the verification calculator calculates between the first and second cluster results by the following equation. The NMI of I (A, B) can be determined.

If the calculation of the NMI is performed by the verification calculator 23 for all the algorithms stored in the community detection algorithm performer 22 (S10), the algorithm selector 24 simulates based on the value returned from the verification calculator 23. A community detection algorithm having a high NMI value may be determined and the result may be a clustering result (S11).

The present inventors calculated the NMI of the correct answer sheet and each of the Leuven method, the Gervan-Newman algorithm, the Infomap algorithm, and the Newman spectrum algorithm on graph data having 18,071 nodes and 16,593 edges. As shown in the detection results shown in Table 1 below, among the verified algorithms, the algorithm by the Leuven method can be applied as an algorithm for obtaining the clustering result of the system by indicating the highest I(A, B) values.

Algorithm type Leuven Method Gaban-Newman
algorithm Info Map
algorithm Newman spectrum
algorithm Community detection result 571 cases 569 cases 599 cases 557 cases

-116220.2 -116220.2 -113868.61 -113748.61

-110008.07 -109692.95 -110191.03 -110092.05

-219920.46 -219339.23 -216567.33 -214256.27 I(A,B) 0.9721174 0.9709006 0.966561 0.957182

4 is exemplary original graph data processed by a graph data clustering method according to an embodiment. In addition, FIG. 5 is an image showing a correct answer sheet for clustering with respect to the original graph data of FIG. 4, and is an image showing the classification of clusters through color for convenience.

Referring to FIG. 4, the original graph data may include one or more nodes 400 and edges connecting each node 400. In the example of this figure, each node 400 of the graph data is classified into the first node 501 and the second node 502 belonging to different clusters as shown in FIG. 5, so that the graph data is divided into two groups. Can be clustered.

Meanwhile, FIGS. 6A to 6C are images showing clustering results by various community detection algorithms for the original graph data of FIG. 4.

First, nodes of the graph data are classified into a third node 601 and a fourth node 602 in the clustering result by any one algorithm shown in FIG. 6A. Since this is the same as the correct answer sheet in FIG. 5, only the cluster identification is different, so the clustering result shown in FIG. 6A and the NMI value of the correct answer sheet are 1, and it is shown that there is no difference in the clustering result of the correct answer sheet and the target algorithm as an output value.

Meanwhile, FIG. 6B shows the clustering result by another algorithm different from FIG. 6A, and in the clustering result, the algorithm grasps all the nodes 610 of the graph data as one cluster. When the clustering result shown in FIG. 6B is used as the input value of the verification calculator 23 (FIG. 2), the clustering result by the algorithm and the NMI of the correct answer sheet in FIG. 5 are 0, and information indicating that there is no correlation between the two is system Can be output from

Furthermore, FIG. 6C shows clustering results by another algorithm different from FIGS. 6A and 6B. In the clustering result shown in FIG. 6C, nodes of the graph data are classified into the fifth node 621, the sixth node 622, and the seventh node 623, so that the nodes are divided into three clusters more subdivided than the correct answer sheet. . When the clustering result shown in FIG. 6C is used as an input value of the verification calculator 23 (FIG. 2 ), the clustering result by the algorithm and the NMI of the correct answer sheet in FIG. 5 are about 0.5928.

Therefore, when the user compares the three community detection algorithms corresponding to each of FIGS. 6A to 6C with the correct answer sheet, the community detection algorithm of FIG. 6A having the highest NMI value of 1 is selected as the community detection algorithm that obtains the result closest to the correct answer sheet. The clustering result by the community detection algorithm of FIG. 6A may be finally selected as the output value of the system.

The operation by the graph data clustering method according to the above-described embodiments may be at least partially implemented by a computer program and recorded on a computer-readable recording medium. A program for implementing an operation by the graph data clustering method according to embodiments is recorded, and a computer-readable recording medium includes all types of recording devices in which data that can be read by a computer is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. In addition, the computer readable recording medium may be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment will be readily understood by those skilled in the art to which this embodiment belongs.

The present invention described above has been described with reference to the embodiments shown in the drawings, but this is only exemplary and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, it should be considered that such modifications are within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (9)

A graph database storage configured to store graph data including one or more nodes and one or more edges between the nodes;
An event processing unit configured to receive a user request, an event occurrence, or a clustering request according to a preset call cycle, and provide clustering results for the graph data; And
A clustering processor configured to generate the clustering result from the graph data by using any one community detection algorithm selected from a plurality of preset community detection algorithms, which is called by the event processing unit,
The clustering processing unit,
A clustering correct answer sheet processor configured to receive and store a preset correct answer sheet;
A community detection algorithm performer configured to generate a cluster result by each of the plurality of community detection algorithms;
It is configured to generate a normalized cross-information value between the cluster result and the correct answer sheet, A for a first cluster result corresponding to the correct answer sheet, B for a second cluster result by the community detection algorithm, and B for the first cluster result. The number of clusters c _A , the number of clusters of the second cluster result c _B , the number of nodes of the graph data N, and the number of nodes belonging to the i th cluster, which is an arbitrary natural number in the first cluster result, N _i , The number of nodes belonging to the j th cluster, which is an arbitrary natural number in the second cluster result, is N _j , of the nodes belonging to the j th cluster of the second cluster result among nodes belonging to the i th cluster of the first cluster result. If the number is N _ij ,

A verification calculator for determining I(A, B) as the normalized mutual information value by the equation of And
And an algorithm selector configured to determine a cluster result using the community detection algorithm having the largest normalized mutual information value among the plurality of community detection algorithms as the clustering result,
The cluster result by each of the plurality of community detection algorithms is compared with a preset correct answer sheet to calculate normalized mutual information values between the cluster result by each of the plurality of community detection algorithms and the correct answer sheet, and the plurality of community detection algorithms Graph data clustering system further configured to determine the community detection algorithm having the largest normalized mutual information value as the one community detection algorithm.
delete delete delete Storing graph data comprising one or more nodes and one or more edges between the nodes in a graph data clustering system;
Receiving, by the graph data clustering system, a user request, an event occurrence, or a clustering request according to a preset call cycle;
The graph data clustering system compares a cluster result by each of a plurality of preset community detection algorithms with a preset correct answer sheet, and compares a first cluster result corresponding to the correct answer sheet with A and a second cluster result by the community detection algorithm. B, the number of clusters of the first cluster result c _A , the number of clusters of the second cluster result c _B , the number of nodes in the graph data N, the i th cluster, which is an arbitrary natural number in the first cluster result The number of nodes belonging to N _i , the number of nodes belonging to the j th cluster, which is an arbitrary natural number in the second cluster result, is N _j , the result of the second cluster of nodes belonging to the i th cluster of the first cluster result If the number of nodes belonging to the j th cluster of N is _ij ,

The I(A, B) is determined as the normalized mutual information value according to the equation, and the normalized mutual information value between the cluster result by each of the plurality of community detection algorithms and the correct answer sheet is calculated, and the plurality of Determining a community detection algorithm having the largest normalized mutual information value among community detection algorithms as one of the plurality of community detection algorithms;
Generating, by the graph data clustering system, a clustering result from the graph data using the determined community detection algorithm; And
And the graph data clustering system providing the clustering result.
delete delete delete A computer program stored in a computer-readable recording medium to execute the graph data clustering method according to claim 5 in combination with hardware.

Images