KR101345095B1 - Method and system for bgp routing data processing based on cluster - Google Patents

Abstract

The present invention relates to a method and a system for analyzing cluster environment based BGP routing information and for realizing routing information in each cluster node. According to the present invention, the massive BGP routing information can be stored using low cost computer. [Reference numerals] (100) Node management unit;(200) Query processing unit;(310,AA,DD) BGP routing information collecting unit;(320,BB,EE) BGP routing information storage unit;(330,CC,FF) BGP routing information analysis unit

Description

JP Routing Information Analysis Method and System Based on Cluster Environments {Method and System for BGP Routing Data Processing Based on Cluster}

The present invention relates to a cluster-based BGP routing information analysis system and method for implementing routing information in parallel in a distributed system.

By detecting and analyzing functional performance factors that can determine the performance of a network, you can not only address the physical or logical problems that exist in the network today, but also identify potential problems that can cause serious network failures in the future. It can be revealed and solved. In addition, the analysis data on network performance factors can be secured over a long period of time, thus providing direction for network improvement.

In order to optimize the network, it can be divided into methods of measuring network performance such as measuring, monitoring, and filtering traffic flowing through the network, and analyzing and managing routing tables to efficiently manage routing.

As the Internet expands worldwide, the importance of inter-network routing is growing. Border Gateway Protocol (BGP) is an external routing protocol based on path-vector routing that can support large networks, and is widely used as a routing protocol between AS (Autonomous System). Therefore, measuring and analyzing BGP routing data is an important measure to determine the stability of the Internet environment, and it is necessary to develop a tool that can analyze the measurement results more efficiently and set advanced routing policies.

BGP routing information generated in real time may be collected through peering with a BGP router, or information provided in a file format through a BGP archiving project such as RouteView or RIPE. Since BGP routing information exchanged between routers of the Internet can be collected for each router and includes routing information between a large number of ASs constituting the Internet, it takes a lot of time and effort to determine the stability of the Internet using BGP routing information. .

The method for stably analyzing BGP routing information may be performed using a single cluster node, but there is a limitation in processing power. Therefore, in order to solve this problem, a large-scale analysis method of a method in which a plurality of distributed computers analyze and tag separate BGP routing information separately, put a result of analyzing the public queue, put it in the queue, and distribute it from the queue and process it again. This has been proposed. The problem with this approach is that each cluster node does not exist under one consistent setup or control, making it difficult to switch the entire computing task as needed, and if one cluster node fails, it affects the overall outcome. Is going crazy.

Hadoop was developed to support Nutch's distributed processing, providing the foundation for building and operating applications that can handle hundreds of gigabytes to terabytes or petabytes of data. Processing platform. Because Hadoop's data is typically at least several hundred gigabytes in size, data is not stored on a single computer, but divided into blocks and distributed across multiple computers. Therefore, Hadoop includes the Hadoop Distributed File System (HDFS), which allows the processing of incoming data separately. The distributed data is processed by the MapReduce process developed for parallel processing of large data in a cluster environment. It is done. Hadoop is widely used in various fields requiring large data processing. However, there has not been developed a system and method for analyzing BGP routing information using Hadoop.

An object of the present invention for solving the problems of the prior art as described above is to use a distributed system as a framework that can process a large amount of BGP routing information, distributed BGP routing information to a plurality of servers and stores a plurality of servers To provide a system and method that can be analyzed by parallel operation in.

The present invention for achieving the above object (A) a collecting unit for collecting BGP routing information; A storage unit which stores the collected routing information in a file system; At least one cluster node comprising: an analyzer configured to distribute and store the stored routing information in parallel; (B) a node manager for managing each of the cluster nodes; And (C) a query processor for transmitting a query from the user to the node manager and delivering the processed result to the user. The present invention relates to a cluster environment-based BGP routing information analysis system.

Another aspect of the present invention provides a method comprising the steps of: (A) receiving, by a cluster system configured with one or more computing devices, routing information from a server storing BGP routing information or a server responsible for BGP routing and storing the routing information in a file system; (B) distributing and processing the routing information among cluster nodes of the cluster system; (C) storing the routing information processed in the step in a storage device shared by one or more computing devices constituting the cluster system; And (D) processing the routing information stored in the storage device and transmitting the routing information stored in the storage device to the query for routing information from the outside.

According to the BGP routing information analysis system of the present invention, large-scale BGP routing information can be distributed, collected, purified, and stored using a low cost computer.

In addition, the BGP routing information analysis method of the present invention makes it possible to stably analyze BGP routing information continuously generated by using a cluster file system and a computing device for forensic analysis and real-time analysis by adding nodes.

1 is a block diagram showing the configuration of a BGP routing information analysis system according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of a single node in a cluster-based BGP routing information analysis system according to an embodiment of the present invention using Hadoop and MapReduce.
Figure 3 is a structural diagram showing the configuration of a cluster-based BGP routing information analysis system according to an embodiment of the present invention using Hadoop and MapReduce.
4 is a flowchart illustrating an embodiment of a processing procedure of an input unit for processing a BGP routing information file in MRT binary form.
5 is a flowchart illustrating an embodiment of a record identification procedure of an input unit that processes a BGP routing information file in an MRT binary form.
6 is a process flow diagram showing an embodiment of the BGP routing information input unit.
7 is a processing flowchart showing an embodiment of information processing of a BGP routing information analyzer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings. However, these drawings are only examples for easily describing the content and scope of the technical idea of the present invention, and thus the technical scope of the present invention is not limited or changed. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the technical idea of the present invention based on these examples.

The present invention relates to a BGP routing information analysis system implemented to enable parallel processing of a large amount of BGP routing information in each cluster node in a distributed system based on a cluster environment. FIG. 1 is a BGP according to an embodiment of the present invention. A block diagram showing the configuration of a routing information analysis system.

1 is an embodiment of a BGP routing information analysis system of the present invention constructed based on a cluster environment including one or more cluster nodes, and includes one or more cluster nodes 300 and nodes for managing and controlling each node of the cluster. The management unit 100 is configured to receive a query from a user, hand over to the node management unit 100, and receive a processed result and transmit the result to a user. Each cluster node 300 includes a collection unit 310 for collecting BGP routing information, a storage unit 320 for storing the collected information in a file system in a file system, and an analysis unit for distributing and parallel processing the stored BGP routing information ( 330).

Cluster-based BGP routing information analysis system of the present invention can be configured based on the Hadoop framework. 2 is a block diagram illustrating in more detail the configuration of a single node of the cluster-based BGP routing information analysis system according to an embodiment of the present invention using Hadoop and MapReduce, and FIG. 3 is a view using Hadoop and MapReduce. A structure diagram showing a configuration of a cluster-based BGP routing information analysis system according to an embodiment of the present invention.

As illustrated in FIGS. 2 and 3, the storage unit 320 stores the collected routing information in a HDFS (Hadoop Distributed File System) in each cluster node of the BGP routing information analysis system using Hadoop and MapReduce. The storage unit 330 distributes and stores a large amount of BGP routing information on the stored HDFS in a map reduce method in each cluster node of Hadoop. More specifically, the BGP routing information input unit 330 receives the input processing unit 331 and the routing information transferred from the input processing unit to the distributed processing unit 332 to process a large amount of BGP routing information stored in the HDFS in a map reduce method. It is composed of a distributed processing unit 332 distributed by the map reduce method, the query processing unit 200 receives a query from the user to configure the MapReduce Job, which is a distributed job form that each node can understand, the node management unit 100 After receiving the result of executing the cluster node, the delivered MapReduce Job is delivered to the user. Although only one cluster node is shown in FIG. 2 for convenience, the cluster-based BGP routing information analysis system of the present invention includes one or more cluster nodes, which may include a plurality of cluster nodes as shown in FIG. 1. .

The present invention also relates to a method for analyzing BGP routing information using the system. More specifically, in the cluster-based BGP routing information analysis method of the present invention, (A) a cluster system including one or more computing devices is routed from a server storing BGP routing information or a server in charge of Border Gateway Protocol (BGP) routing. Receiving and distributing information in a file system; (B) distributing and processing the routing information among cluster nodes of the cluster system; (C) storing the routing information processed in the step in a storage device shared by one or more computing devices constituting the cluster system; And (D) processing and forwarding routing information stored in the storage device in response to a routing information query from the outside.

In particular, the file system of step (A) and the storage device of step (C) are HDFS (Hadoop Distributed File System), and step (B) distributes routing information by a map reduce method to process the Hadoop framework. Can be configured based on.

The BGP routing information may collect routing information delivered in real-time streaming form through peering with a BGP router in operation by running a routing software module such as Quagga. You can also collect from routing information that exists in the form of a file created in. The collected routing information may be RIB information or may be information such as a BGP update message exchanged between BGP peering nodes. In addition, the BGP routing information in the form of a file may be a binary file in the form of a multi-threaded routing tooling (MRT), a file in the form of a text, or a file existing in both compression forms.

4 is a flowchart illustrating an embodiment of a processing procedure of an input unit for processing a BGP routing information file in MRT binary format. When a BGP routing information file in MRT binary format exists, the packet is read by Hadoop MapReduce. In order to analyze the data, each cluster node reads and processes data blocks by input format. In FIG. 4, it is assumed that the time for collecting BGP routing information before the job execution is obtained from a configuration property or a storage that can be shared by cluster nodes.

When you open a block for data processing, make sure the start of the block is the start of the MRT record. First, if the block is the first block of the packet MRT record, the starting point will be the starting point of the MRT record, so the point is defined as the starting point of the InputSplit. If the starting point of the block is not the first block of the BGP routing information file, since the starting point of the block and the starting point of the routing information may not coincide, the process of finding the starting point of the MRT record having the routing information must be performed (201).

5 is a flowchart illustrating an embodiment of a record identification procedure of an input unit that processes a BGP routing information file in an MRT binary form.

First, (A) Assuming that the start byte of the block is the start of the MRT record, the header information including the timestamp and the length of the MRT record is extracted from the MRT header. Hereinafter, each is described as TS1 and Lengh1, respectively. Normally, the header r of an MRT record records the timestamp in the first 4 bytes, the record type, the 2 bytes of SubType, and the 4-byte length in the next 2 bytes. Can be.

(B) When extracting the data for the first MRT record, the header information for the second MRT record is extracted in the same manner from where it is assumed to be the starting point of the second record. Hereinafter, TS2, Type2, SubType2, and Length2 will be described respectively. The starting point of the second MRT record will be the point shifted by adding the length of the MRT header (usually 12 bytes) of the first record plus the Length value recorded in the MRT header.

(C) From the header information of the first record and the second record obtained in (A) and (B), respectively, verify whether the first byte of the data block is the starting point of the first MRT record. More specifically, a) it is checked whether TS1 and TS2 are valid values within the range from the collection start time to the end time of the record acquired from the configuration property. b) Check that the Type and SubType of the two records have valid identification values. It also assumes valid values for the record length information (that is, checking whether they are positive or not too large). c) Verify that TS1 and TS2 can be identified as incoming records in succession. To do this, the difference between TS1 and TS2 is determined, and the delta time, which is considered to be a continuous record, is determined and checked to be within the range of the value. In general, the delta time is preferably within 5 seconds, but can be properly adjusted in consideration of the network environment or other variables. If all the above steps a), b) and c) are satisfied, the start byte of the currently assumed record is regarded as the byte of the actual MRT record. On the other hand, if any one of a), b) and c) is not satisfied, go to the next byte and assume the beginning of the record and repeat the process of a), b) and c) to verify the starting point again. Search for the start of the first packet in the data block.

In FIG. 5, a), b), and c) are all used to verify the starting point of the record. However, this is related to an example, and the starting point of the record may be verified based on only one or two conditions. In addition, you can verify the starting point of a record. Naturally, the larger the number of conditions used for the verification, the more accurate the starting point verification of the record can be.

When moving to the start of the first record in the data block by the method illustrated by Figs. 4 and 5, it is defined as the start of the InputSplit. That is, InputSplit of a data block defines InputSplit for the data block from the start of the first record until the start of InputSplit for the next data block.

6 is an embodiment of an internal module of the input unit 331 which extracts one record from a file existing in a file system recognizable by Hadoop HDFS or Hadoop and delivers it to the analysis unit 332. An internal system module for identifying a start position of a record from a BGP routing information file of an MRT binary format or an uncompressed MRT binary format, and extracting and delivering each record one by one is shown.

If the file is compressed by reading the MRT BGP routing information file, the MRT record length information provided by the MRT format is read from the file read through the compression codec provided by Hadoop, and each record is extracted and transmitted to the map. In this case, the extracted binary MRT record may be converted into text form in the input unit and transferred to the analysis unit.

If the routing information is a file in a compressed or uncompressed text format, the record may be identified using a string that can identify one record instead of the MRT record length.

7 is an embodiment of an internal module of the BGP routing information analyzer. The analysis unit reads routing information (for example, RIB information and BGP update message) received from the input unit, extracts text fields in the map step, and delivers them in group form by key values from the fields extracted in the reduce step. Processes records The key may use a combination of one or more of the field values constituting the BGP RIB or the update message, and as a result, the total number of announcements, the number of withdrawal, the number of different AS route information for the same prefix destination, the same prefix destination For example, we can extract the result of the same number of AS path information. Also, from the extracted result, it converts into a table form through Hive, which is a Hadoop-based data warehousing solution, and processes and delivers a routing degree for a user's query delivered in a query form.

The analysis of the routing information using the map reduce is an example of parallel BGP routing information processing, and this job may be performed by implementing a necessary map and a reduce function according to a target to be analyzed. You can also configure more than one job to link the results of the previous job to the input of the next job to get more complex and refined analysis results.

For example, to find the number of messages per prefix for the BGP update message file, use the prefix value extracted from the MRT record in the map step as the key and the value '1' as the value in the reduce step as the reduce step. In the reduce step, the number of messages per prefix can be obtained by summing and outputting the message number '1', which is grouped and delivered according to the key value, which is a prefix. This method can be applied similarly to obtaining the number of Withdrawal or Announcement per prefix, or the like. This calculated value can be used as input for the second job to get information grouped by AS or country. At this time, the mapping process of AS to Prefix and the country's mapping information file to AS can be received as two inputs. In this case, if you output the record with the key that can be shared while going through the map phase from two kinds of files, that is, Prefix as a key, in the reduce step, the AS information, the country information, and the number of messages for the prefix are grouped and delivered. You can sum the numbers to get the number of countries, ASs, and messages for the prefix. In order to group these results by country or AS, it is passed to the input of the next job to send the result of the map with the value to be grouped, that is, the AS number or the country name, and when the reduce step is grouped as the key value of the map result. By adding the values, you can get the total number of messages based on the AS number or country name, the number of prefixes, and the size of the managed Internet address space using the prefix values.

Claims (10)

(A) a collection unit for collecting Border Gateway Protocol (BGP) routing information; a storage unit for storing the collected routing information in HDFS (Hadoop Distributed File System); At least one cluster node comprising: and an analysis unit configured to receive routing information from the input processing unit and distribute the data in parallel with the map reduce method in each cluster node.
(B) a node manager for managing each of the cluster nodes; And
(C) a query processing unit for transmitting a query from the user to the node management unit and delivering the processed result to the user;
Cluster environment-based BGP routing information analysis system, characterized in that comprises a.
delete (A) distributedly receiving and storing routing information from a server in which BGP routing information is stored or a server in charge of Border Gateway Protocol (BGP) routing in a cluster system including one or more computing devices and storing them in a file system;
(B) distributing and processing the routing information among cluster nodes of the cluster system;
(C) storing the routing information processed in the step in a storage device shared by one or more computing devices constituting the cluster system; And
(D) processing and forwarding routing information stored in the storage device for querying routing information from outside;
Cluster environment-based BGP routing information analysis method comprising a.
The method of claim 3, wherein
The BGP routing information of step (A) is routing information existing in the form of a file in the BGP routing information archiving server or routing information delivered in real-time streaming through peering with the BGP router. How to analyze routing information.
The method according to claim 3 or 4,
The file system of step (A) and the storage device of step (C) are HDFS (Hadoop Distributed File System),
Step (B) is a cluster environment-based BGP routing information analysis method, characterized in that for distributing and processing the routing information by the map reduce method.
The method of claim 5, wherein
The routing information of step (A) is a cluster environment-based BGP routing information analysis method, characterized in that the form of text, compressed text, MRT binary or compressed MRT binary.
The method according to claim 6,
When the routing information is in the form of MRT or compressed MRT, step (B) may include:
Identifying one record from the block of the HDFS with the length information of the MRT record and passing it to the map;
A map step of generating and outputting (Key, Value) from the record transferred to the map; And
A reduce step of processing a value based on a key from the output (Key, Value);
Cluster environment-based BGP routing information analysis method comprising a.
The method of claim 7, wherein
The key is a cluster environment-based BGP routing information analysis method characterized in that the combination of one or more of the field values constituting the BGP RIB or UPDATE message.
The method according to claim 6,
When the routing information is in the form of text or compressed text, the step (B) may include:
Identifying one record by string from the block of HDFS and passing it to the map;
A map step of generating and outputting (Key, Value) from the record transferred to the map; And
A reduce step of processing a value based on a key from the output (Key, Value);
Cluster environment-based BGP routing information analysis method comprising a.
The method of claim 9,
The key is a cluster environment-based BGP routing information analysis method characterized in that the combination of one or more of the field values constituting the BGP RIB or UPDATE message.

Images