JP6021111B2 - Flow aggregation apparatus and method - Google Patents

Description

  The present invention relates to a flow aggregation apparatus and method, and more particularly, to a flow aggregation apparatus and method for improving the efficiency of flow aggregation based on flow data analysis results in a flow data analysis apparatus and flow aggregation apparatus.

  Monitoring the traffic on the network is a technology that is indispensable for the appropriate design of network resources and the detection and control of abnormal traffic. This requires a fine-grained traffic monitoring technique.

  In NetFlow technology, the source attribute (SrcIP), destination address (DstIP), source port (SrcPort), destination port (DstPort), and protocol (Proto) 5-tuple are added as the flow attribute information. It includes flow attribute information such as an AS (Autonomous System) number, a destination AS number, an input / output number of a router interface used for transfer, a ToS (Type of Service) value, and a TCP (Transmission Control Protocol) _flag value.

  In addition, the amount of traffic to be transferred is increasing day by day, and the number of flows of NetFlow data observed is increasing explosively. For this reason, it is becoming difficult to freely access NetFlow data using conventional methods.

  As a data structure to hold multidimensional flow data, create a one-dimensional hash table that holds FlowID (5-touple flow information) as a key or a hash table for each Tuple, and use the Linked List as the hash table key. There is a multi-dimensional hash table having a data structure connected by use (for example, Non-Patent Document 1).

  However, since the one-dimensional hash table holds the 5-tuple information as it is, it is not possible to make a query by combining Tuples freely. In addition, the multidimensional hash table has a problem that the search time is long when a specific tuple is designated as a wild card to search for an aggregate flow.

  In addition, there is a multidimensional database based on a multidimensional logical data space bitmap (see, for example, Patent Document 1). This is to create a bitmap in which data indicates coordinates in a multidimensional logical data space in a specific combination of dimensions.

  By performing a flow search (slice search) specifying a combination of specific dimensions in a multi-dimensional flow database, it is possible to extract a characteristic flow such as abnormal traffic. For example, by searching the multidimensional database by specifying the pair of the source IP address of the host suspected of worm infection and the destination port number used for spreading the worm infection, it will be transmitted along with the spread of the worm infection. You can get the flow and IP address information of the infected host.

JP 09-265479 A

Yan Hu, Dah-Ming Chiu, John C. S. Lui, "Entropy Based Adaptive Flow Aggregation", IEEE / ACM TRANSACTIONS ON NETWORKING, VOL., 17. NO. 3, (2009), pp. 698-711.

  The above multi-dimensional database can be accessed at high speed, but the storage capacity required to hold the bitmap increases according to the number of dimension members, the number of dimensions, and the number of data. The amount of memory becomes huge. In addition, accessible dimension combination patterns are limited, and access in any combination of dimensions is difficult. When the above database is constructed in any combination every time there is a query in the on-the-fly method, the database construction time and the amount of memory required for the database become enormous, and the access overhead is large.

  If the characteristics of abnormal traffic to be analyzed are known in advance, and if the number of dimensions / scale data is below a certain level, a multi-dimensional logical data space bitmap can be created at high speed as in Patent Document 1. Access is possible. However, if the characteristics of the abnormal traffic to be analyzed are unknown, a task such as searching for the characteristics of the abnormal traffic while performing a slice search of multiple patterns in any combination of dimensions in the multidimensional database is performed. Necessary. In the prior art, it is difficult to search a slice of a free dimension combination for a multidimensional database.

  As described above, in a one-dimensional hash table and a multi-dimensional hash table, it is difficult to access an aggregated flow that freely combines flow attribute information. In addition, the multidimensional logical data space bitmap of the multidimensional flow database can be accessed in a short time, but the number of combinations explodes when considering all the multidimensional combinations. Is too large to implement.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a flow aggregation apparatus and method capable of speeding up a search function (slice function) of an arbitrary dimension combination of large-scale multidimensional data.

According to one aspect, for a high-dimensional and large-scale flow data, a flow aggregation device that constructs a database for searching an aggregation flow that combines multidimensional data,
A multi-dimensional attribute table for storing multi-dimensional attributes of the read multi-dimensional data;
A data index table storing index numbers for the multidimensional data;
A data table for storing flow information of the multidimensional data;
Database construction means for constructing a database for the search;
Have
The database construction means includes
Means for storing a unique member for each dimension in the multi-dimensional attribute table in the dimension member table for each dimension;
Means for assigning an index number to the member of the dimension member table, assigning a unique index number to members across dimensions, centralizing multi-dimensional member information, and storing in a member index conversion table;
The attribute of the multi-dimensional attribute table is converted into a member index, a member index for the member is obtained by referring to the member index conversion table, and a data index of the data table and a member index / data index from the acquired member index Means for creating a conversion table;
Means for converting the member index of the member index / data index conversion table into an index in a root column of a wavelet tree, and creating a wavelet tree index conversion table;
The member index and the data index are acquired from the member index / data index conversion table, converted from the data index into a single data string, a wavelet tree is created from the data string, the member index and the data index And a means for creating a search wavelet tree DB comprising a combination of the wavelet trees.

  According to one aspect, large-scale high-dimensional flow data can be managed with a realistic amount of memory space. Further, when searching for a target flow, high speed with a time calculation amount that is logarithmically proportional to the number of flows. Search becomes possible.

It is a structural example of the multidimensional database structure apparatus in one embodiment of this invention. It is an example of the data index table in one embodiment of this invention, a multidimensional attribute table, and a data table. It is an example of the dimension member table in one embodiment of this invention. It is an example of the member index / data index conversion table in one embodiment of the present invention. It is an example of the member index conversion table in one embodiment of the present invention. It is a figure which shows the conversion method to the wavelet tree for a search in one embodiment of this invention. It is an example of the wavelet tree in one embodiment of this invention. It is an example of the wavelet tree index conversion table in one embodiment of the present invention. It is a flowchart (the 1) of a process of the database access part in one embodiment of this invention. It is an example of the common subset operation using the wavelet tree in one embodiment of this invention. It is a flowchart (the 2) of a process of the database access part in one embodiment of this invention. It is a flowchart (the 3) of a process of the database access part in one embodiment of this invention. It is an example of Patricia tree DB in one embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention provides a flow aggregation device (multi-dimensional database configuration device) that enables high-speed access to any combination of aggregate flows for high-dimensional and large-scale flow data.

  FIG. 1 is a configuration example of a multidimensional database configuration apparatus according to an embodiment of the present invention.

  The multidimensional database configuration apparatus 100 shown in the figure includes a NetFlow data reading unit 110, a table creation unit 120, a DB construction unit 130, a user query analysis unit 140, a database access unit 150, a data output unit 160, an intermediate DB 170, a DB 180, data A table 101, a multidimensional attribute table 102, and a data index table 103;

  The intermediate DB 170 is set on a memory and includes a dimension member table 171 and a member index / data index conversion table 172. When the DB 180 is constructed, the memory of the intermediate DB 170 is released.

  The DB 180 includes a search wavelet tree DB 181, a wavelet tree index conversion table 182, a member index conversion table 183, and a Patricia tree DB 184.

  The multidimensional data reading unit 110 reads multidimensional data such as flow data and passes the data to the table creation unit 120 and the DB construction unit 130. In this example, it is assumed that time-series traffic information of five-dimensional IP flow information is read.

  The table creation unit 120 stores the read multidimensional data in the tables 101, 102, and 103 on the memory as shown in FIG. The data index table 101 holds an index number assigned to each read five-dimensional information. The multidimensional attribute table 102 stores SrcIP, DstIP, SrcPort, DstPort, and Proto corresponding to each index number. The data table 103 holds flow information (traffic amount, number of packets, etc.) corresponding to the index number in time series. The data table 103 is divided into lines for each flow, and the multidimensional attribute table 102 returns a pointer of each line of the data table 103. Therefore, the data table 103 is obtained by acquiring the pointer via the multidimensional attribute table 102. Information on the corresponding flow can be obtained.

  The DB constructing unit 130 constructs the intermediate DB 170 and DB 180 as follows.

  First, when reading multidimensional data, the DB construction unit 130 stores, in the dimension member table 171, unique members that appear in the read data for each dimension, as shown in FIG. 3. However, members are stored in ascending order for SrcPort, DstPort, and Proto. For SrcIP and DstIP, members are stored in ascending order of IP addresses using the constructed Patricia tree DB 184.

  The DB construction unit 130 generates a member index / data index conversion table 172.

  The member index / data index conversion table 172 includes a member index of the dimension member table 171 and a data index obtained from the data table 103, as shown in FIG.

  The DB constructing unit 130 generates the member index / data index conversion table 172 by the following procedure.

(1) Using the multidimensional attribute table 102 and the member index conversion table 183 shown in FIG. 5, the attributes of the multidimensional attribute table 102 are converted into member indexes. For example, the first line of the multidimensional table 102 in FIG.
"SrcIP: 10.0.01, DstIP: 20.0.0.1, SrcPort: 10, DstPort: 20, Proto: 6"
About the member index conversion table 183 in FIG.
"SrcIP: 0, DstIP: 1000, SrcPort: 2000, DstPort: 3000, Proto: 4000"
(This is assumed to be multi-dimensional attribute table-2).

(2) A specific row in the multi-dimensional attribute table-2 can be obtained from an appropriate Data Index in the data index table 101. For example, when accessing Data Index 1,
"SrcIP: 0, DstIP: 1000, SrcPort: 2000, DstPort: 3000, Proto: 4000"
The member index / data index conversion table 172 is created as follows.

Since member index: 0 is Data Index 1, 0 → 1;
Since member index: 1000 is Data Index 1, 1000 → 1;
Since member index: 2000 is Data Index 1, 2000 → 1;
Since member index: 3000 is Data Index 1, 3000 → 1;
Since member index: 4000 is Data Index 1, 4000 → 1;
The member index / data index conversion table 172 is created by accessing and repeating all the above-described processes.

  The Patricia tree DB 184 holds a Patricia tree constructed using members that appear in the read data when reading multi-dimensional data regarding the SrcIP and DstIP dimensions. As shown in FIG. 13 to be described later, the Patricia tree DB 184 is configured for each of SrcIP and DstIP, and stores all SrcIP addresses and DstIP addresses existing in the read multidimensional data.

  The member index conversion table 183 holds a member index corresponding to member information. The DB construction unit 130 refers to the dimension member table 171 and assigns index numbers to the members. At the same time, as shown in FIG. 5, a table for converting member information into index numbers is created. At this time, by assigning unique index numbers to members that cross dimensions, one-dimensionalization of multidimensional member information is performed (because a wavelet tree must be constructed with a one-dimensional list).

  The wavelet tree DB 181 for search performs one-dimensional mapping of multidimensional data using the member index (Member index in FIG. 4) and the data index (Data Index in FIG. 4) of the member index / data index conversion table 172, and the wavelet The tree index conversion table 182 is referenced to generate a multi-index structure (wavelet tree) as shown in FIG.

  Specifically, the member index / data index conversion table 172 is referred to in order from the top, and the Data Index (FIG. 4) is converted into a single data string. In the example of FIG. 4, it becomes 1,3,2,4,6,5,..., 1,2,3,5,4,6. A wavelet tree is created from the data sequence. Note that an existing technique (for example, Non-Patent Document 2: http://www.slideshare.net/pfi/ss-15916040) can be used for generating a wavelet sequence from a data sequence. FIG. 7 shows an example of a wavelet tree. The wavelet tree is a complete binary tree, and each node is accompanied by a bit string. A leaf corresponds to each value, and an internal node corresponds to a range of descendant leaves.

  By using this wavelet tree, a subset of data indexes including a common member index can be obtained. For this purpose, for example, a simple dictionary that can be constructed with low memory, called the Rank dictionary, is built, and at high speed (time complexity O (log n, where n is the number of flows) that is logarithmically proportional to the number of data) It is possible to obtain a common subset (for example, Non-Patent Document 3: T. Gagie, G. Navarro, SJ Puglisi, New algorithms on wavelet trees and applications to information retrieval, Theoretical Computer Science 426-427 (2012) pp See 25-41. The Rank dictionary has an index structure, and a dictionary having the following operations for the bit string B [0 ... n] is called a complete dictionary (FID).

Rankb (B, pos): returns the number of occurrences of b in B [0 ... pos];
Selectb (B, ind): returns the (ind + 1) th occurrence position of b;
For example, in the example of FIG. 7, when rank1 (6) = 2, “1” is returned twice in B [0,6), and when select0 (4) = 8, (4 The +1) th "0" returns to appear at 8.

  The wavelet tree index conversion table 182 is used at the time of data search, and is a table indicating where the search target data exists in the wavelet tree. This is a table for converting the member index (Member index) of the member index conversion table 183 into an index in the root column of the wavelet tree, and is created simultaneously with the creation of the member index conversion table 183. The DB construction unit 130 creates the wavelet tree index conversion table 182 from the member index / data index conversion table 172. Specifically, in the example of FIG. 6, information that the member index “0” exists in the 0th to 1st positions in the data string, and the member index “3” exists in the 4th to 6th positions in the data string. As shown in FIG. 8, a wavelet start index (Wavelet_start_Index) and an end index (Wavelet_last_Index) are set for each member index.

  The query analysis unit 140 analyzes a user query including each input dimension name and its member information, and generates a database access query for accessing the database 180.

  The database access unit 150 converts the database access query into a data index according to the following procedure.

  (1) The database access query is converted into a set of indexes in the wavelet tree root column with reference to the wavelet tree index conversion table 182.

  (2) A process for obtaining a common subset is performed on the search wavelet tree DB 181.

  In this example, since it is five-dimensional data, a function for obtaining a maximum of five common subsets is prepared. Specifically, an algorithm for obtaining two common subsets described in Chapter 3 (3. New algorithms) of Non-Patent Document 3 is used. In section 3.3 (3.3 Range intersection) of Non-Patent Document 3, it is described that a function extension for obtaining three or more common subsets is possible.

  The specific operation of the database access unit 150 will be described below.

  First, a case where all of SrcIP, DstIP, SrcPort, DstPort, and Proto are specified as the user query X will be described.

  FIG. 9 is a flowchart (No. 1) of the process of the database access unit in the embodiment of the present invention.

In this example, as user query X,
"SrcIP = 10.0.0.1, DstIP = 20.0.0.5, SrcPort = 10, DstPort = 20, Proto = 6"
Is entered (step 101), the member index table conversion table 183 (FIG. 5) is referred to by the user query X, and the member index X corresponding to the user query X is entered.
"0, 1001, 2000, 3000, 4001"
Is acquired (step 102).

Next, based on the member index X, the wavelet tree index conversion table 182 (FIG. 8) is referred to, and the wavelet tree index X for the member index X is referred to.
"[0,1], [40,46], [60,72], [89,94], [130,132])"
Is acquired (step 103).

  A common subset operation is performed on the wavelet tree index X acquired above by applying the technique of Non-Patent Document 3, and a data index (1) is acquired and output to the data output unit 160 (step 104).

  The common subset operation using the wavelet tree in step 104 is to calculate the common subset of the two sets [(214) and (250)] using the wavelet tree when obtaining the common subset for “0721436725047263” in FIG. And You can move to the left node with the Rank0 operation (solid line) and move to the right node with the Rank1 operation (dashed line). Rank operation is performed at the start point and end point of the set (regardless of the set length). Rank operation is a constant time. If two sets point to the same value pointer, it becomes the value of the common subset.

  Next, a case where only DstIP and Proto are specified as the user query Y will be described.

  FIG. 11 is a flowchart (part 2) of the process of the database access unit in the embodiment of the present invention.

In this example, as user query Y,
"SrcIP = *, DstIP = 20.0.0.1, SrcPort = *, DstPort = *, Proto = 6"
Is entered (step 201), the member index table conversion table 183 (FIG. 5) is referred to by the user query Y, and the member index Y corresponding to the user query Y is entered.
"1000, 4000"
Is acquired (step 202).

Next, based on the member index Y, the wavelet tree index conversion table 182 (FIG. 8) is referred to, and the wavelet tree index Y for the member index Y is referred to.
"[30,40], [120,130])"
Is acquired (step 203).

  The common subset operation is performed on the wavelet tree index Y acquired above by applying the technique of Non-Patent Document 3, the data index (1, 2, 5) is acquired and output to the data output unit 160 ( Step 204). The common subset operation is the same as described above.

  Next, a case where only SrcIP and DstIP are specified as the user query Z will be described.

  FIG. 12 is a flowchart (No. 3) of the process of the database access unit in the embodiment of the present invention.

In this example, as user query Z,
"SrcIP = *, DstIP = 10.0.0.4 / 30, DstIP = 20.0.0.1"
Is input (step 301), since the address included in the member index conversion table 183 is 10.0.0.1 to 10.0.0.6 in the range of DstIP [10.0.0.4/30] for SrcIP in the user query Z [ 10.0.0.1, 10.0.0.6] and refer to the Patricia tree DB 184 as shown in FIG. (Step 302).

  Based on SrcIP = [10.0.0.1] and SrcIP = [10.0.0.6] of the user query Z, the member index conversion table 183 is referred to, the member index [0, 3] is obtained, and based on DstIP = 20.0.0.1 Get member index "1000". Here, the pair of the head and tail of the member index of SrcIP_first is used as a query (step 303).

  Based on the user query Z ([0, 3], 1000) obtained in step 303, the wavelet tree index conversion table 182 (FIG. 8) is referred to, and the wavelet tree index Z ([0, 6], [40, 46]) is acquired (step 304). In the example of the wavelet tree index conversion table 182 of FIG. 8, the Wavelet_start_Index of the member index “0” that is the head in the first set is “0”, and the Wavelet_last_Index of the member index “3” that is the end in the set. Is "6".

  The common subset operation is performed on the wavelet tree index Z acquired above by applying the technique of Non-Patent Document 3, and the data index (1, 2, 3, 5) is acquired and output to the data output unit 160 (Step 305). The common subset operation is the same as described above.

  As described above, in the present invention, the multi-index structure allows access to any combination of tuples in a uniform time. Also, by combining the wavelet tree structure, the time complexity of access time can be reduced from O (n) to O (log n).

  The operation of each component of the multidimensional database configuration apparatus shown in the above embodiment is constructed as a program and installed in a computer used as the multidimensional database configuration apparatus to be executed or distributed via a network. It is possible.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

DESCRIPTION OF SYMBOLS 100 Multidimensional database structure apparatus 101 Data index table 102 Multidimensional attribute table 103 Data table 110 Multidimensional data reading part 120 Table creation part 130 DB construction part 140 User query analysis part 150 Database access part 160 Data output part 170 Intermediate DB
171 Dimension member table 172 Member index / data index conversion table 180 DB
181 Wavelet tree DB for search
182 Wavelet tree index conversion table 183 Member index conversion table 184 Patricia tree

Claims (6)

A flow aggregation device that constructs a database for searching aggregate flows combining multidimensional data for high-dimensional and large-scale flow data,
A multi-dimensional attribute table for storing multi-dimensional attributes of the read multi-dimensional data;
A data index table storing index numbers for the multidimensional data;
A data table for storing flow information of the multidimensional data;
Database construction means for constructing a database for the search;
Have
The database construction means includes
Means for storing a unique member for each dimension in the multi-dimensional attribute table in the dimension member table for each dimension;
Means for assigning an index number to the member of the dimension member table, assigning a unique index number to members across dimensions, centralizing multi-dimensional member information, and storing in a member index conversion table;
The attribute of the multi-dimensional attribute table is converted into a member index, a member index for the member is obtained by referring to the member index conversion table, and a data index of the data table and a member index / data index from the acquired member index Means for creating a conversion table;
Means for converting the member index of the member index / data index conversion table into an index in a root column of a wavelet tree, and creating a wavelet tree index conversion table;
The member index and the data index are acquired from the member index / data index conversion table, converted from the data index into a single data string, a wavelet tree is created from the data string, the member index and the data index And means for creating a search wavelet tree DB comprising a combination of the wavelet trees;
A flow aggregating apparatus comprising: The database construction means includes
When creating the dimension member table, the dimension is a source port (SrcPort), a destination port (DstPort), and a protocol (Proto) for storing in the dimension member table in ascending order;
For the source address (SrcIP) and the destination address (DstIP), means for storing in the dimension member table in ascending order of IP addresses using the Patricia tree;
The flow aggregation device according to claim 1, comprising: When a user query consisting of a dimension name and member information is input, query analysis means for analyzing the user query and generating a database access query for accessing the database;
A database that performs processing for converting the database access query to a set of indexes in a wavelet tree root sequence with reference to the wavelet tree index conversion table, and for obtaining a common subset by referring to the wavelet tree DB for search Access means;
The flow aggregation device according to claim 1, further comprising: A flow aggregation method for constructing a database for searching aggregate flows combining multidimensional data for high-dimensional and large-scale flow data,
A multi-dimensional attribute table for storing multi-dimensional attributes of the read multi-dimensional data;
A data index table storing index numbers for the multidimensional data;
A data table for storing flow information of the multidimensional data;
Database construction means for constructing a database for the search;
In a device having
The database construction means
Storing a unique member for each dimension in the multidimensional attribute table in the dimension member table for each dimension;
Assigning an index number to the member of the dimension member table, assigning a unique index number to members across dimensions, centralizing multi-dimensional member information, and storing in a member index conversion table;
The attribute of the multi-dimensional attribute table is converted into a member index, a member index for the member is obtained by referring to the member index conversion table, and a data index of the data table and a member index / data index from the acquired member index Creating a translation table;
Converting the member index of the member index / data index conversion table into an index in a root column of a wavelet tree to create a wavelet tree index conversion table;
The member index and the data index are acquired from the member index / data index conversion table, converted from the data index into a single data string, a wavelet tree is created from the data string, the member index and the data index Creating a search wavelet tree DB comprising a combination of the wavelet trees;
The flow aggregation method characterized by performing. When creating the dimension member table,
For the dimension, the source port (SrcPort), the destination port (DstPort), and the protocol (Proto) are stored in the dimension member table in ascending order; and
For the source address (SrcIP) and the destination address (DstIP), using the Patricia tree, storing in the dimension member table in ascending order of IP addresses;
The flow aggregation method according to claim 4, comprising: In an apparatus further comprising query analysis means and database access means,
When the query analysis means receives a user query consisting of a dimension name and member information, the query analysis means analyzes the user query and generates a database access query for accessing the database;
The database access means converts the database access query into a set of indexes in a wavelet tree root sequence with reference to the wavelet tree index conversion table, refers to the search wavelet tree DB, and sets a common subset. Performing a process for obtaining
The flow aggregation method according to claim 4 or 5, further comprising:

Images