JP2007228112A - Traffic data relay system, distributed traffic data allocation system, and distributed traffic data allocation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of flexibly extending aggregation/storage processing of traffic data depending on a network scale by mounting a traffic data aggregation/selection function to a traffic data relay system located between traffic data storage devices and network apparatuses. <P>SOLUTION: In the traffic data relay system located between the traffic data storage devices and the network apparatuses, a collection process means receives the traffic data distributed from the network apparatuses or other traffic data relay systems, a storage process means stores the traffic data received by the collection process means, a selection process means judges whether or not an aggregation process means is to take over the traffic data received by the collection process means, the aggregation process means aggregates the traffic data within a period by using prescribed field information of the traffic data for key information, and a distribution process means distributes the traffic data aggregated by the aggregation process means. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a traffic data relay system, a traffic data distribution / arrangement system, and a traffic data distribution / arrangement method, and more particularly, to monitor traffic flowing on a network represented by the Internet, and The present invention relates to a method for efficiently monitoring traffic.

In recent years, attention has been focused on functions (sFlow, netFlow, IPFIX) for monitoring traffic flowing on a network for the purpose of detecting abnormal traffic such as DDoS attack traffic.
In order to reduce the load on the traffic data storage device, the traffic data aggregation function is implemented in the network device (hereinafter referred to as NW device). In this case, refer to the traffic data before aggregation. There is a problem that it becomes impossible.
Moreover, in a certain NW device, it is possible to distribute the same traffic data to a plurality of traffic data storage devices using a traffic data distribution protocol. However, this is for redundantly arranging traffic data, and is not intended to reduce the load of the storage process itself.
At present, a function for distributing data to be distributed according to the characteristics of traffic data has not been studied.
Further, a system that receives traffic data distributed from NW devices, performs aggregation processing, and distributes aggregate traffic data again using a traffic data distribution protocol has been proposed in IPFIX (IPFIX-aggregation). (See Non-Patent Document 1 below)

As prior art documents related to the invention of the present application, there are the following.
http://www.ietf.org/internet-drafts/draft-dressler-ipfix-aggregation-02.txt RFC3954 Cisco NetFlow version9

However, the IPFIX-aggregation method is specialized only for the aggregation processing function. Therefore, traffic data before aggregation cannot be accumulated, and data from a high aggregation level to a small data cannot be referenced or analyzed. Also, in terms of performance, the load distribution function related to the storage function is not realized because it specializes only in the aggregation processing.
Assuming that the NW scale is increasing explosively in recent years, it is difficult to implement both aggregation processing and storage processing with one device. For this reason, there is a need for a load distribution method for aggregation processing and storage processing corresponding to all NW scales.
The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to provide a traffic data relay system by interposing a traffic data relay system between a traffic data storage device and a network device. It is an object of the present invention to provide a technology capable of flexibly expanding traffic data aggregation / storage processing according to the scale of a network by implementing a traffic data aggregation / selection function.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
(1) A traffic data relay system disposed between a traffic data storage device and a network device, wherein a collection process means, a selection process means, an aggregation process means, a distribution process means, and a storage process means The collection process means receives traffic data distributed from the network device or another traffic data relay system, and the accumulation process means accumulates the traffic data received by the collection process means The selection process means determines whether to take over the traffic data received by the collection process means to the aggregation process means, and the aggregation process means uses predetermined field information of the traffic data as key information. Traffic traffic within a certain amount of time Aggregating data, the delivery process means, said traffic data storage device, or, wherein the distributing other traffic data relay system F, the traffic data aggregated by the aggregation process means.

(2) In (1), the collection process means is generated for each traffic data distribution protocol unit.
(3) In (1) or (2), a plurality of the selection process means, and the aggregation process means and the distribution process means provided for each of the selection process means in the plurality of selection process means are provided. In each selection process means, the traffic data is selected based on predetermined field information of the traffic data, and the aggregation process means provided for each selection process means is selected by each selection process means. The distribution process means provided for each of the selection process means outputs the traffic data aggregated by the aggregation process means.

(4) Between the network device and the traffic data storage device, the traffic data relay system of the means of (1) or (2), the network device side is the first stage, and the traffic data storage device side is n (2 ≦ n) A traffic data distribution / arrangement system (or a traffic data distribution / arrangement method) arranged in n stages and multiple stages as the stage, and the storage process means of the traffic data relay system in the first stage Accumulates the traffic data distributed from the network device, and the aggregation process means aggregates the traffic data distributed from the network device and relays the traffic data in the k (2 ≦ k ≦ n) stage. The accumulation process means of the system is the traffic at the (k−1) th stage. The traffic data distributed from the data relay system is accumulated, and the traffic data distributed from the traffic data relay system at the (k−1) -th stage is aggregated by the aggregation process means. The traffic data distributed from the traffic data at the nth stage is stored.
(5) In (4), the traffic data stored in the traffic data relay system in the first to nth stages and the traffic data storage device are traffic data having different aggregation levels.
(6) In (4) or (5), referring to the collection process means of the traffic data relay system in the first to nth stages and the management information held in the aggregation process means, The traffic data relay system in which traffic data is stored or the traffic data storage device is specified.

(7) A method for distributing and distributing traffic data in a system configured by arranging the traffic data relay system of the means of (3) between a network device and a plurality of traffic data storage devices, The data relay system selects the distribution process means provided for each of the selection process means by selecting the traffic data selected by the selection process means and aggregated by the aggregation process means provided for each of the selection process means. The traffic data storage device to the corresponding traffic data storage device, and each traffic data storage device stores the traffic data distributed from each distribution process means of the traffic data relay system To do.
(8) In (7), referring to information set by the selection process means of the traffic data relay system, a traffic data storage device in which predetermined traffic data is stored is specified.

In the present invention, a traffic data relay system is interposed between a traffic data storage device (traffic collector) and a network device (hereinafter referred to as NW device) (such as a router), and the traffic data relay system has a traffic data aggregation / selection function. By implementing it, load distribution of processing related to aggregation and accumulation of traffic data is performed. This realizes efficient aggregation of traffic data information and distributed arrangement of traffic data.
The traffic data relay system of the present invention receives traffic data output from an NW device, accumulates, selects, and aggregates traffic data, and distributes traffic data using a traffic data distribution protocol.
By using the traffic data relay system of the present invention, it becomes possible to store and control traffic data according to the NW scale. For example, by connecting traffic data relay systems in multiple stages, it becomes possible to distribute and arrange traffic data with different aggregation levels, and in the subsequent system, the aggregation level is increased in order and processing is performed. It becomes possible to distribute the load itself.
In addition, when the NW scale becomes large, it is possible to increase the number of traffic data relay systems so that traffic data can be aggregated and stored according to the network scale.
Further, the traffic data relay system defines setting information that can be browsed from other nodes such as SNMP (Simple Network Management Protocol), so that when the other nodes search for traffic data, the data arrangement is recognized. Is possible.

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
According to the present invention, the traffic data relay system is interposed between the traffic data storage device and the network device, and the traffic data relay system is provided with the traffic data aggregation / selection function. Processing can be flexibly expanded according to the scale of the network.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In all the drawings for explaining the embodiments, parts having the same functions are given the same reference numerals, and repeated explanation thereof is omitted.
FIG. 1 is a block diagram showing a system configuration to which a traffic data relay system of the present invention is applied.
In FIG. 1, 10-1 to 10-8 are network devices such as routers, 11-1 to 11-6 are traffic data relay systems, and 12 is a traffic data storage device (traffic collector).
The traffic data relay system (11-1 to 11-6) of the present invention is arranged between the traffic data storage device 12 and the NW devices (10-1 to 10-8).
The traffic data output from the NW devices (10-1 to 10-8) includes various information such as mirrored packets and traffic data distribution protocols (sFlow, netFlow, IPFIX).
The traffic data relay system (11-1 to 11-6) of this embodiment receives traffic data output from the NW devices (10-1 to 10-8), and accumulates, selects, and aggregates traffic data. The traffic data is distributed using a traffic data distribution protocol.
The traffic data received from the NW device (10-1 to 10-8) is a counter such as the number of bytes and the number of packets using the transmission source IP address, transmission source port number, destination IP address, destination port number, protocol, etc. as key information. Information is stored and distributed in units of flows. This single flow information is called a flow record, and each information element is defined as a field.

FIG. 2 is a block diagram showing a schematic configuration of the traffic data relay system of this embodiment.
As shown in FIG. 2, the traffic data relay system 11 of this embodiment includes a collection process means 1, a selection process means 2, an aggregation process means 3, a distribution process means 4, a storage process means 5, and a database 6. And have.
The collection process means 1 has a function of receiving traffic data distributed from the NW device (10-1 to 10-8) or another traffic data relay system 11. The collection process means 1 is generated for each traffic data distribution protocol unit and manages session information such as a destination IP address and a port number. In addition, in order to manage the information elements included in the flow record constituting the traffic data to be distributed, the data format corresponding to the field value included in the flow record is managed.
The traffic data received by the collection process means 1 can be taken over by a plurality of selection process means 2 or storage process means 5.
The accumulation process means 5 has a function of accumulating received traffic data in the database 6. At this time, the accumulation process means 5 determines whether or not to accumulate in the database 6 in field units. These pieces of information are set in advance by the user. The accumulation process means 5 stores traffic data including only specified field information in the database 6.

The selection process means 2 determines whether or not the distributed traffic data is taken over by the aggregation process means 3. These conditions are set in advance by the user. In the determination, a conditional statement is specified for each field unit, and only the traffic data that meets all the conditions is taken over by the aggregation process means 3.
The aggregation process means 3 aggregates traffic data within a certain time by using values such as the source IP address as key information. At that time, counter information such as the number of bytes is added together to generate aggregated traffic data (ie, flow record). Which field in the traffic data is used as key information or counter information is designated in advance by the user. The aggregated traffic data is taken over by the next distribution process means 4.
The distribution process means 4 has a function of distributing traffic data to the traffic data storage device 12 or other traffic data relay systems (11-1 to 11-6). The distribution process means 4 manages field information in the traffic data to be distributed separately from the session information such as the destination IP address and the port number.

FIG. 3 is a diagram for explaining a distribution method of traffic data using the traffic data relay system of the present embodiment. Hereinafter, a traffic data distribution and arrangement method using the traffic data relay system of this embodiment will be described.
As shown in FIG. 3, traffic data can be distributed in a distributed manner by interposing the traffic data relay system 11 between three traffic data storage devices (12-1 to 12-3). In FIG. 3, 10 is a network device, and 7-1 to 7-3 are databases of traffic data storage devices (12-1 to 12-3).
In FIG. 3, three selection process means (2-1 to 2-3), aggregation process means (3-1 to 3-3), and distribution process means (4-1 to 4) of the traffic data relay system 11 are shown. -3) is used to distribute the traffic data to be distributed to the three subsequent traffic data storage devices (12-1 to 12-3).
As an information element used for selection, information included in a flow record constituting traffic data can be arbitrarily designated. For example, when the INDEX (INPUT IF-INDEX) of the interface into which the packet has flowed is assigned, the INPUT IF-INDEX designated in advance is assigned to each selection process means (2-1 to 2-3). The traffic data received by the collection process means 1 is taken over by a plurality of selection process means (2-1 to 2-3), and each selection process means (2-1 to 2-3) receives IF- Only INDEX is selected.

For example, in FIG. 3, the selection process means (2-1) selects traffic data including INPUT IF-INDEX 1 to 10, and the selection process means (2-2) includes INPUT IF-INDEX 11 to 20. The traffic data is selected, and the selection process means (2-2) selects the traffic data including INPUT IF-INDEX 21-30.
By the operation of the selection process means (2-1 to 2-3), traffic data is distributed and arranged in the traffic data storage devices (12-1 to 12-3).
When searching for distributed traffic data, the traffic data storage devices (12-1 to 12-12) are referred to by referring to the information set by the selection process means (2-1 to 2-3). -3) makes it possible to recognize what kind of traffic data is stored.
By defining the information set by each selection process means (2-1 to 2-3) as an SNMP MIB object, it is possible to refer from other nodes and to specify the storage destination of traffic data. Become.

FIG. 4 is a diagram for explaining a load distribution arrangement method of traffic data aggregation and accumulation processing using the traffic data relay system of this embodiment. In the following, a description will be given of a load distribution arrangement method of traffic data aggregation and storage processing using the traffic data relay system of the present embodiment.
As shown in FIG. 4, the load distribution of the aggregation processing is performed by connecting the traffic data relay systems (11-1 to 11-3) in multiple stages.
Based on the traffic data distributed from the NW device (10-1 to 10-2), the traffic data relay system (11-1) in the previous stage performs primary traffic aggregation, and the NW device (10-3 to 10-3) Based on the traffic data distributed from 10-4), the traffic data relay system (11-2) at the previous stage performs primary traffic aggregation. Then, secondary aggregation is performed by the traffic data relay system (11-3) at the subsequent stage.
For example, in FIG. 4, in the preceding traffic data relay system (11-1 to 11-2), prefix values are designated by masks for the source IP address and destination IP address, and the values are aggregated as key information. .
Thereafter, in the subsequent traffic data relay system (11-3), aggregation processing is performed using the transmission source AS (Autonomous System) and the destination AS as key information. As a result, load distribution of the aggregation processing is realized.
Eventually, the traffic data storage device 12 can receive traffic data using the transmission source AS and the destination AS as key information.
Accordingly, the flow record indicated by 8-1 in FIG. 4 is aggregated into the flow record indicated by 8-2 in FIG. 4 by the traffic data relay system (11-1), and further, the traffic data relay system (11-3) ) Is collected into a flow record indicated by 8-3 in FIG.

FIG. 5 illustrates a method for distributing and storing traffic data of different aggregation levels by using the traffic data relay system of this embodiment and storing the traffic data received by the traffic data relay system in a database. It is a figure for doing. Hereinafter, a method for distributing and accumulating traffic data of different aggregation levels by using the traffic data relay system of the present embodiment and accumulating traffic data received by the traffic data relay system in a database will be described.
The traffic data received from the network devices (10-1 to 10-4) is accumulated in the preceding traffic relay system (11-1 to 11-2), and after the primary aggregation, the subsequent traffic relay system (11- 3).
In the subsequent traffic relay system (11-3), the primary aggregation result is accumulated, and after the secondary aggregation, it is distributed to the traffic accumulation device 4. Thereby, traffic data with different aggregation levels are distributed and arranged.
In FIG. 5, 9-1 indicates key information and counter information of traffic data distributed from the network device (10-1), and 9-2 indicates traffic data in the traffic relay system (11-1). 9-3 indicates key information and counter information when aggregating traffic data, and 9-3 indicates key information and counter information when traffic data is aggregated in the traffic relay system (11-3).
When searching for these data, it is possible to refer to the management information held in the collection process means 1 and the aggregation process means 3 and identify what traffic data is stored in which system. It becomes.

By defining information managed by each process means in an SNMP MIB (Management Information Base) object, reference from an external node becomes possible.
As a result, it is possible to search for data by tracing back to more detailed traffic data based on the traffic data referred to by the traffic storage device 12.
As described above, in this embodiment, the traffic data relay system 11 is interposed between the traffic data storage device 12 and the NW device 10 and the traffic data relay system 11 is provided with a traffic data aggregation / selection function. Thus, the system can be flexibly expanded according to the scale of the network.
Further, when the network scale becomes large, it is possible to increase the number of traffic data relay systems so that traffic data can be aggregated and stored according to the network scale. Therefore, it is effective in the field of monitoring IP traffic whose network scale tends to increase.
As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

1 is a block diagram showing a system configuration to which a traffic data relay system of the present invention is applied. It is a block diagram which shows schematic structure of the traffic data relay system of the Example of this invention. It is a figure for demonstrating the distribution arrangement method of the traffic data using the traffic data relay system of the Example of this invention. It is a figure for demonstrating the load distribution arrangement | positioning system of the aggregation of the traffic data using the traffic data relay system of the Example of this invention, and an accumulation | storage process. To describe a method of distributing and storing traffic data having different aggregation levels by using the traffic data relay system according to the embodiment of the present invention and storing the traffic data received by the traffic data relay system in a database. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Collection process means 2,2-1 to 2-3 Selection process means 3,3-1 to 3-3 Aggregation process means 4,4-1 to 4-3 Distribution process means 5 Accumulation process means 6,7,7- 1-7-3 Database 8-1-8-3 Flow Record 9-1-9-3 Key Information and Counter Information 10, 10-1 to 10-8 Network Equipment 11, 11-1 to 11-6 Traffic Data Relay System 12, 12-1 to 12-6 Traffic data storage device

Claims (9)

A traffic data relay system arranged between a traffic data storage device and a network device,
A collection process means, a selection process means, an aggregation process means, a distribution process means, and a storage process means,
The collection process means receives traffic data distributed from the network device or other traffic data relay system,
The accumulation process means accumulates traffic data received by the collection process means;
The selection process means determines whether to take over the traffic data received by the collection process means to the aggregation process means;
The aggregation process means aggregates traffic data within a certain time using predetermined field information of the traffic data as key information,
The traffic data relay system, wherein the distribution process means distributes the traffic data aggregated by the aggregation process means to the traffic data storage device or another traffic data relay system.   2. The traffic data relay system according to claim 1, wherein the collection process means is generated for each traffic data distribution protocol unit. A plurality of said selection process means;
The aggregation process means and the distribution process means provided for each selection process means of the plurality of selection process means,
In each of the selection process means, the traffic data is selected based on predetermined field information of the traffic data,
The aggregation process means provided for each selection process means aggregates the traffic data selected by each selection process means,
The traffic data relay system according to claim 1 or 2, wherein the distribution process means provided for each selection process means outputs the traffic data aggregated by the aggregation process means. 3. The traffic data relay system according to claim 1 or 2 between the network device and the traffic data storage device, wherein the network device side is the first stage and the traffic data storage device side is the n (2 ≦ n) stage. Is a traffic data distributed arrangement system configured to be arranged in n stages and multiple stages,
The accumulation process means of the traffic data relay system in the first stage accumulates traffic data distributed from the network device, and the aggregation process means aggregates traffic data distributed from the network device,
The storage process means of the traffic data relay system in the k (2 ≦ k ≦ n) stage stores the traffic data distributed from the traffic data relay system in the (k−1) stage, and the aggregation process Means to aggregate the traffic data distributed from the traffic data relay system in the (k-1) stage,
The traffic data accumulating device accumulates traffic data distributed from the traffic data at the nth stage. 3. The traffic data relay system according to claim 1 or 2 between the network device and the traffic data storage device, wherein the network device side is the first stage and the traffic data storage device side is the n (2 ≦ n) stage. As a method for distributing and distributing traffic data in a system configured with n stages and multiple stages,
The traffic data relay system in the first stage accumulates the traffic data distributed from the network device by the accumulation process means, and aggregates the traffic data distributed from the network device by the aggregation process means. ,
The traffic data relay system in the k (2 ≦ k ≦ n) stage stores the traffic data distributed from the traffic data relay system in the (k−1) stage by the storage process means, and Aggregating the traffic data distributed from the traffic data relay system at the (k-1) stage by the aggregation process means,
The traffic data storage device stores traffic data distributed from the traffic data at the nth stage, and distributes and distributes traffic data.   6. The traffic data according to claim 5, wherein the traffic data stored in the traffic data relay system in the first to nth stages and the traffic data storage device are traffic data having different aggregation levels. Distributed distribution method.   The traffic data relay system in which predetermined traffic data is stored with reference to management information held in the collection process means and the aggregation process means of the traffic data relay system in the first to nth stages Alternatively, the traffic data storage device is specified, and the traffic data distribution method according to claim 5 or 6. A method for distributing and arranging traffic data in a system configured by arranging the traffic data relay system according to claim 3 between a network device and a plurality of traffic data storage devices,
The traffic data relay system selects the traffic data selected by the selection process means and aggregates the traffic data collected by the aggregation process means provided for each of the selection process means. The distribution process provided for each of the selection process means Via the means, to the corresponding traffic data storage device among the plurality of traffic data storage devices,
The traffic data storage device stores traffic data distributed from the distribution process means of the traffic data relay system, and distributes and distributes traffic data.   9. The distribution of traffic data according to claim 8, wherein information set by the selection process means of the traffic data relay system is referenced to identify a traffic data storage device in which predetermined traffic data is stored. Placement method.

Images