JP4346096B2 - Traffic data aggregating apparatus and traffic data aggregating method - Google Patents

Description

  The present invention relates to a traffic data aggregating apparatus and a traffic data aggregating method, and in particular, when monitoring traffic data flowing on a network typified by the Internet, the traffic data on a large-capacity network is efficiently and flexibly changed. It relates to the method of aggregation.

In recent years, attention has been focused on functions (sFlow, netFlow, IPFIX) for monitoring traffic flowing on a network for the purpose of detecting abnormal traffic such as DDoS attack traffic.
Conventionally, the aggregation of traffic data has been performed by the system independently in a network device (hereinafter referred to as NW device) or a traffic data information collection device. In this case, the information used as a key used for aggregation is determined depending on the system, and cannot be changed in the system in advance.
Currently, a function for distributing aggregated traffic data using Net Flow or the like is provided, but this is fixed by the system.
Moreover, in the aggregation method (IPFIX-aggregation) proposed by IPFIX, it is proposed that the aggregation method can be changed by the user. (See Non-Patent Document 1 below)

As prior art documents related to the invention of the present application, there are the following.
http://www.ietf.org/internet-drafts/draft-dressler-ipfix-aggregation-02.txt RFC3954 Cisco NetFlow version9

However, the aggregation method (IPFIX-aggregation) proposed in IPFIX is not sufficiently flexible in the following points.
(A) Since information that can be specified as key information is limited in advance, statistical data based on the BYTE value cannot be calculated. The fields that can be specified in the key information cannot be specified other than those specified in Flow-Key.
(B) A method of collecting additional information such as external information as key information has not been considered.
(C) A method of generating additional information for correcting information lost during aggregation has not been studied.
The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to aggregate traffic data flexibly in accordance with a user's application in a traffic data aggregation device and a traffic data aggregation method. It is to provide a technology that makes it possible.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
(1) In accordance with an aggregation instruction template for instructing an aggregation method for each flow record of traffic data, a correction information addition instruction template for instructing information to be added to each flow record of aggregated traffic data, and the correction information addition instruction template, Based on the activated application process and the aggregation instruction template, the flow records of each distributed traffic data are aggregated, and based on the correction information addition instruction template, the application process is activated, and each of the aggregated traffic data And an aggregation process for obtaining correction information to be added to the flow record .

(2) In (1) , the aggregation instruction template includes first to third operation instruction words described in units of fields, and the first operation instruction word includes the first operation instruction word. The described field represents a field that becomes key information when the flow records are aggregated, and the second operation instruction word is the field in which the second operation instruction word is described. The third operation instruction word is a field in which the field in which the third operation instruction word is described is discarded when the flow records are aggregated. Represents that.
(3) In (1) or (2), the correction information addition instruction template includes a correction information addition operation instruction word described in each field unit, and the field in which the correction information addition operation instruction word is described Information specifying the application process to be started and information specifying the field to be cited are also described.

(4) An information addition instruction template for instructing information to be added to each flow record of traffic data, an aggregation instruction template for instructing an aggregation method of each flow record of traffic data, and an addition to each flow record of aggregated traffic data A correction information addition instruction template for instructing correction information, a first application process activated in accordance with the information addition instruction template, a second application process activated in accordance with the correction information addition instruction template, and the information addition Based on the instruction template, the first application process is started, and an information addition process for acquiring information to be added to each flow record of the distributed traffic data, and each of the distributed traffic based on the aggregation instruction template With aggregating Flow Records Kkudeta, based on the correction information addition instruction template, activating the second application process, it is provided with a aggregation process of obtaining correction information to be added to each flow record traffic data aggregation It is characterized by.

(5) Based on the aggregation instruction template for instructing the aggregation method of each flow record of traffic data, the correction information addition instruction template for instructing correction information to be added to each flow record of the aggregated traffic data, and the aggregation instruction template, A traffic data aggregation method of a traffic data aggregation device, comprising: an aggregation process for aggregating flow records of each distributed traffic data, wherein the aggregation process is based on the correction information addition instruction template, Starts an application process, acquires correction information to be added to each flow record of the aggregated traffic data, adds the acquired correction information to a new field, and keys the field indicated in the aggregation instruction template As broadcast, combined field information instructed by the aggregation instruction template, or by destroying the indicated fields by the aggregation instruction template, characterized by aggregating Flow Records distributed each traffic data were .

In the present invention, traffic data output from a network device (hereinafter referred to as NW device) such as a router is received, flow records are added and aggregated, and traffic data is generalized.
A general method for aggregating traffic data aggregates traffic data within a certain period of time using values such as a source IP address as key information. As for the counter value at the time of aggregation, the amount of BYTE, the number of flows, and the like correspond to this.
These key information and counter values are determined in advance as fixed to the system, and are not versatile. For example, the distribution of the number of flows cannot be calculated using the BYTE amount or the like as key information.
In the present invention, a general aggregation function for flow information is provided in two processes, an information addition process for adding a new field value to flow information based on external information and the like and an aggregation process for aggregating a plurality of flow records. Realize.
Each process operates based on an instruction template indicating an execution procedure, and realizes a flexible aggregation method according to the instruction template.
In addition, information for correcting information lost during aggregation is also added. For example, during aggregation, information other than the counter value is often lost. That is, the total number of BYTEs within a certain time can be calculated due to aggregation, but information such as the number of BYTEs per flow is lost. In order to correct such lost information, for example, an average BYTE amount per flow or a minimum BYTE amount per flow can be used. Also in this case, the aggregation process operates according to the instruction template, so that this additional information can be generated.

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
According to the traffic data aggregating apparatus and the traffic data aggregating method of the present invention, it is possible to aggregate traffic data flexibly according to a user's application.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In all the drawings for explaining the embodiments, parts having the same functions are given the same reference numerals, and repeated explanation thereof is omitted.
FIG. 1 is a block diagram showing a schematic configuration of a traffic data aggregating apparatus according to an embodiment of the present invention.
In FIG. 1, 1-1 and 1-2 are application processes, 2 is an addition instruction template, 3 is an aggregation instruction template, 5 is an information addition process, 6 is an aggregation process, 10-1 to 10-4 are NW devices, 11 Is a traffic data aggregating device, and 12 is a database or external traffic information collecting device.
The traffic data aggregating apparatus 11 of the present embodiment is premised on being mainly mounted on a traffic information collecting device, but may be mounted on an NW device having an aggregation function.
The traffic data distributed from the NW devices (10-1 to 10-4) includes various information such as mirrored packets and traffic flow statistics distribution protocols (sFlow, netFbw, IPFIX).
The traffic data distributed from the NW devices (10-1 to 10-4) is aggregated in the traffic data aggregating apparatus 11, and the aggregated information is held in a database in its own system indicated by 12 in FIG. Alternatively, it is distributed to an external traffic information collecting device by a traffic flow statistics distribution protocol.

As used herein, a flow refers to data having information elements common to traffic data.
The flow information received from the NW device (10-1 to 10-4) is a counter such as the number of BYTEs and the number of packets using the transmission source IP address, transmission source port number, destination IP address, destination port number, protocol, etc. as key information. Information is stored. This single flow information is called a flow record, and each information element is defined as a field.
The information addition process 5 that has received the flow information adds a required field to each flow record based on the information addition instruction template 2.
In the information addition instruction template 2, an operation instruction word is set for each field. The operation instruction words of the information addition instruction template 2 are three, “use”, “nouse”, and “add”.
When “use” is specified in the information addition instruction template 2, it means that the value set in this field is quoted when a new field is added.
If “nouse” is designated, this means that this field is not referred to in this process.
Further, “add” is designated for the information element to be added. In addition to the field for which “add” is specified, an identifier of another application process to be started is included, and a field ID cited in the application process is also set.

The information addition process 5 can acquire new additional information by passing the designated field value as an argument to the application process (1-1).
When the field record to be added is already included in the flow record received from the NW device (10-1 to 10-4), the designated application is activated according to the information addition instruction template 2, and the activation result Thus, the information is overwritten.
For example, in the information addition instruction template 2 shown in FIG. 2, “transmission source BGP NEXT-HOP address” is added, and the content of this addition instruction is based on the transmission source IP address described in the field ID 02. This means that an application process whose application identifier is BGPSEARCH is started, and a BGP (Border Gateway Protocol) NEXT-HOP address of the corresponding route is searched for and obtained.
Similarly, a “destination BGP NEXT-HOP address” is added to the information addition instruction template 2 shown in FIG. 2, and the content of this addition instruction is based on the destination IP address described in the field ID 03. This means that the BGP NEXT-HOP address of the corresponding route is searched and obtained.

FIG. 3 shows flow information when information is added by the information addition instruction template 2 shown in FIG. In FIG. 3, reference numeral 10 denotes an NW device.
In the example shown in FIG. 3, the information addition process 5 starts an application process whose application identifier is BGPSEARCH according to the information addition instruction template 2 of FIG. At that time, the source BGP NEXT-HOP address = “30.30.30.30” can be acquired by passing the source IP address = “10.10.10.10” described in the field ID of 02 as an argument. The same applies to the destination IP address.
As a result, fields of the source BGP NEXT-HOP address and the destination BGP NEXT-HOP address are added to the flow information 7-1 distributed from the NW device 10, and the flow information 7-2 is obtained.
Further, when the field value of the BYTE number is already set in the information transmitted from the NW device 10 even though it is set to “add”, this value is overwritten.
For example, FIG. 4 shows an information addition instruction template 2 in the case where the number of BYTEs to be transmitted is set to “kilo-BYTE” and rounded off after the decimal point. FIG. 5 shows flow information when information is added by the information addition instruction template 2 shown in FIG.
In the example shown in FIG. 4, the information addition process 5 starts an application process whose application identifier is KBYTE according to the information addition instruction template 2 of FIG. At this time, the result is acquired as “46” kilobytes by passing the BYTE number = “45907” described in the field ID of 05 as an argument. Then, the flow information 7-4 is obtained by overwriting the already set BYTE number field to the flow information 7-3 distributed from the NW device 10.

In this specification, a new record generated by aggregating a plurality of flow records is referred to as an aggregated flow record.
The aggregation process 6 that has inherited the flow information aggregates each flow record based on the aggregation instruction template 3.
As shown in FIG. 6, in the aggregation instruction template 3, operation instruction words are set for each field. The operation instruction words of the aggregation instruction template 3 are three, “key”, “keep”, and “discard”.
When “key” is designated in the aggregation instruction template 3, it means that the value set in this field becomes key information of the aggregation flow record.
For example, when both the IP address field and the prefix field are designated as “key”, the prefix is masked on the IP address, and the masked value is used as key information.
When “keep” is designated, the field value is inherited as it is in the aggregated flow record.

If the field value specified by “keep” is a counter, it is added and inherited. In the case of additional information, the information is inherited as it is. If the information to be taken over does not match in the aggregated flow record, an invalid value is set.
If “discard” is specified, the field is discarded in the aggregation process 6.
FIG. 6 shows a setting example of the aggregation instruction template 3 when the transmission source IP address is masked with a prefix value and this value is used as the aggregation key.
FIG. 7 shows aggregation flow information when information is aggregated by the aggregation instruction template 3 shown in FIG.
In FIG. 7, the aggregation process 6 uses the source IP address, the destination IP address, the source prefix value, and the destination prefix value as keys for the flow information 7-5 and 7-6 distributed from the NW device 10. As information, the number of BYTEs and the number of PACKETs are added together. Thereby, the flow information 7-8 is obtained.

The present embodiment has a function of adding a field to correct lost information when performing aggregation. This function adds a field required for each flow record based on the correction information addition instruction template 4 during the aggregation process of the aggregation process 6.
In the correction information addition instruction template 4, an operation instruction word is set for each field. The operation instruction words of the correction information addition instruction template 4 are “use”, “nouse”, and “add”.
When “use” is designated in the correction information addition instruction template 4, it means that the value set in this field is quoted when a new field is added.
If “nouse” is designated, this means that this field is not referred to in this process.
Further, “add” is designated for the information element to be added. In addition to the field for which “add” is specified, another application identifier to be activated is included, and a field ID referred to by the application is also set.
The information specified as an argument includes key information of the aggregated flow record in addition to the field ID specified here.

Examples of fields added for the purpose of correction include the following.
(1) Flow count (number of aggregated flows in the aggregated flow record)
(2) Average ACTIVE TIME (Average ACTIVE time of aggregated flows in the aggregated flow record)
(3) Minimum ACTIVE TIME (Minimum ACTIVE time of aggregated flows in the aggregated flow record)
(4) Maximum ACTIVE TIME (Maximum active time of aggregated flows in the aggregated flow record)
(5) SYN count (number of flows for which SYN flag was set based on information such as TCP Flag)
(6) ACK count (number of flows for which ACK flag was set based on information such as TCP Flag)
(7) PSH count (the number of flows for which the PSH flag was set based on information such as TCP Flag)
(8) FIN count (the number of flows for which the FIN flag was set based on information such as TCP Flag)
(9) RST count (number of flows for which the RST flag was set based on information such as TCP Flag)
FIG. 8 is a diagram illustrating an example of the correction information addition instruction template 4.
In the example shown in FIG. 8, the aggregation process 6 starts an application process whose application identifier is AVE-ACTIVE, for example. At that time, by passing the flow start time and flow end time described in the field IDs of 12 and 13 as arguments, the average in the flow records targeted for aggregation based on the flow start time and flow end time You can ask for ACTIVE-TIME. The same applies to the determination of the minimum ACTIVE-TIME in the flow record that is the aggregation target.

In order to grasp the communication amount between individual hosts, it is necessary to aggregate the source IP address and the destination IP address as key information. However, when aggregation is performed under these conditions, the number of flow records cannot be effectively reduced because the combination of the source IP address and the destination P address reaches a considerable amount.
In this case, if the hash values of the source IP address and the destination IP address are obtained and aggregation is performed using the hash values as key information, the purpose of reducing the number of flow records and grasping the communication amount between individual hosts can be achieved. .
In addition, a case where an average ACTIVE TIME and a minimum ACTIVE TIME are added for the purpose of correcting individual flow information by aggregation will be described.
FIG. 9 shows an information addition instruction template 2 in this case, FIG. 10 shows an aggregation instruction template 3 in this case, and FIG. 11 shows a correction information addition instruction template 4 in this case.
FIG. 12 shows flow information generated by the information addition instruction template 2 shown in FIG. 9, the aggregation instruction template 3 shown in FIG. 10, and the correction information addition instruction template 4 shown in FIG.

In the information addition process 5, the information addition process 5 starts the application process (1-5) whose application identifier is HASH according to the information addition instruction template 2 of FIG. As a result, the fields of hash (source IP address) and hash (destination IP address) are added to the flow information 7-9 distributed from the network device 10, and the flow information 7-10 is obtained.
Further, the aggregation process 6 adds the BYTE number and the PACKET number using the hash (source IP address) and the hash (destination IP address) as key information according to the aggregation instruction template 3 of FIG.
Furthermore, according to the correction information addition instruction template 4 of FIG. 11, an application process (1-6) with an application identifier AVE-ACTIVE and an application process (1-7) with an application identifier MIN-ACTIVE are started. Thereby, the average ACTIVE TIME and the minimum ACTIVE TIME in the flow records to be aggregated are obtained, and the flow information 7-11 is obtained.

As described above, according to the present embodiment, each process operates based on the instruction template (2, 3, 4), so that it is possible to construct an aggregation method arbitrarily thought by the user. .
This instruction template is defined in advance by the user, such as MIB used in SNMP or configuration information that can be changed by the system.
By implementing an efficient and flexible aggregation method according to this embodiment, it is possible to dynamically change the aggregation method, and it can be used for various purposes such as grasping equipment design and detecting abnormal traffic. This is particularly useful for traffic monitoring functions.
As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

It is a block diagram which shows schematic structure of the traffic data aggregation apparatus of the Example of this invention. It is a figure which shows an example of the information addition instruction | indication template of the Example of this invention. It is a figure which shows the flow information when information is added by the information addition instruction | indication template shown in FIG. It is a figure which shows the other example of the information addition instruction | indication template of the Example of this invention. It is a figure which shows the flow information when information is added by the information addition instruction | indication template shown in FIG. It is a figure which shows an example of the aggregation instruction | indication template of the Example of this invention. FIG. 7 is a diagram illustrating aggregation flow information when information is aggregated according to the aggregation instruction template illustrated in FIG. 6. It is a figure which shows an example of the correction information addition instruction | indication template of the Example of this invention. It is a figure which shows the information addition instruction | indication template of an example of the traffic data aggregation apparatus of the Example of this invention. It is a figure which shows the aggregation instruction | indication template of an example of the traffic data aggregation apparatus of the Example of this invention. It is a figure which shows the correction information addition instruction | indication template of an example of the traffic data aggregation apparatus of the Example of this invention. FIG. 11 is a diagram illustrating flow information generated by the information addition instruction template illustrated in FIG. 9, the aggregation instruction template illustrated in FIG. 10, and the correction information addition instruction template illustrated in FIG. 11.

Explanation of symbols

1-1 to 1-7 Application Process 2 Addition Instruction Template 3 Aggregation Instruction Template 5 Information Addition Process 6 Aggregation Process 7-1 to 7-11 Flow Record 10, 10-1 to 10-4 Network Device 11 Traffic Data Aggregation Device 12 Database or external traffic information collection device

Claims (3)

An aggregation instruction template for instructing how to aggregate each flow record of traffic data, a correction information addition instruction template for instructing information to be added to each flow record of aggregated traffic data,
In accordance with the correction information addition instruction template, an application process that is started,
Based on the aggregation instruction template, the flow records of the distributed traffic data are aggregated, and the application process is activated based on the correction information addition instruction template to add to the flow records of the aggregated traffic data. and a aggregation process of obtaining,
The aggregation instruction template includes first to third operation instruction words described in each field unit,
The first operation instruction word represents that the field in which the first operation instruction word is described is a field that becomes key information when the flow records are aggregated,
The second operation instruction word represents that the field in which the second operation instruction word is described is a field that is directly inherited when the flow records are aggregated,
The third operation instruction word represents that the field in which the third operation instruction word is described is a field that is discarded when the flow records are aggregated.
The correction information addition instruction template includes a correction information addition operation instruction word described in each field unit,
In the field in which the correction information addition operation instruction word is described, information specifying an application process to be started and information specifying a field to be cited are described;
A traffic data aggregating device. An information addition instruction template for instructing information to be added to each flow record of traffic data;
An aggregation instruction template that indicates how to aggregate each flow record of traffic data;
A correction information addition instruction template for instructing correction information to be added to each flow record of the aggregated traffic data;
A first application process to be started in accordance with the information addition instruction template;
A second application process to be started in accordance with the correction information addition instruction template;
An information addition process for starting the first application process based on the information addition instruction template and acquiring information to be added to each flow record of the distributed traffic data;
Based on the aggregation instruction template, the flow records of each distributed traffic data are aggregated, and based on the correction information addition instruction template, the second application process is started and added to each flow record of the aggregated traffic data. and a aggregation process of obtaining correction information,
The aggregation instruction template includes first to third operation instruction words described in each field unit,
The first operation instruction word represents that the field in which the first operation instruction word is described is a field that becomes key information when the flow records are aggregated,
The second operation instruction word represents that the field in which the second operation instruction word is described is a field that is directly inherited when the flow records are aggregated,
The third operation instruction word represents that the field in which the third operation instruction word is described is a field that is discarded when the flow records are aggregated.
The correction information addition instruction template includes a correction information addition operation instruction word described in each field unit,
In the field in which the correction information addition operation instruction word is described, information specifying an application process to be started and information specifying a field to be cited are described;
A traffic data aggregating device. An aggregation instruction template that indicates how to aggregate each flow record of traffic data;
A correction information addition instruction template for instructing correction information to be added to each flow record of the aggregated traffic data;
A traffic data aggregation method for a traffic data aggregation device, comprising: an aggregation process for aggregating flow records of each distributed traffic data based on the aggregation instruction template,
The aggregation instruction template includes first to third operation instruction words described in each field unit,
The first operation instruction word represents that the field in which the first operation instruction word is described is a field that becomes key information when the flow records are aggregated,
The second operation instruction word represents that the field in which the second operation instruction word is described is a field that is directly inherited when the flow records are aggregated,
The third operation instruction word represents that the field in which the third operation instruction word is described is a field that is discarded when the flow records are aggregated.
The correction information addition instruction template includes a correction information addition operation instruction word described in each field unit,
In the field in which the correction information addition operation instruction word is described, information specifying an application process to be started and information specifying a field to be cited are described,
The aggregation process starts the application process based on the correction information addition instruction template, acquires correction information to be added to each flow record of the aggregated traffic data, and adds the acquired correction information to a new field. In addition, the field indicated by the aggregation instruction template is used as key information, the field information indicated by the aggregation instruction template is added together, or the field indicated by the aggregation instruction template is discarded. A traffic data aggregation method characterized by aggregating flow records of each traffic data.

Images