KR20120084880A - Inputformat for handling network packet data on hadoop mapreduce - Google Patents

Abstract

PURPOSE: An input format for analyzing a network packet in Hadoop map reduce is provided to define InputSpilt for each distributed node when performing distributed processing of binary packet data having a variable length record in a Hadoop environment, thereby simultaneously accessing and processing. CONSTITUTION: Information about the beginning and termination time of packet capture is obtained from configuration property. The beginning point of a first packet is searched in a data block required to be processed among data blocks stored in HDFS(Hadoop Distributed File System). InputSplit sets up a boundary between previous InputSplit and its InputSplit with the beginning point of the first packet in order to be defined.

Description

InputFormat for handling network packet data on Hadoop MapReduce}

The present invention relates to a new input format for processing binary data packet data having variable length records in Hadoop MapReduce.

Hadoop was developed to support Nutch's distributed processing, providing the foundation for building and operating applications that can handle hundreds of gigabytes to terabytes or petabytes of data. Processing platform. Because Hadoop's data is typically at least several hundred gigabytes in size, data is not stored on a single computer, but divided into blocks and distributed across multiple computers. Therefore, Hadoop includes the Hadoop Distributed File System (HDFS), which allows the processing of incoming data separately. The distributed data is processed by the MapReduce process developed for parallel processing of large data in a cluster environment. It is done.

1 is a conceptual diagram illustrating the flow of data during job processing in Hadoop MapReduce. An input file stores data for map reduction, which is typically stored in HDFS. Hadoop supports a variety of data formats, as well as textual data.

When a Job is started at the client's request, the InputFormat (101) determines how to divide and read the input file. In other words, inputSplit is returned by dividing the input file with respect to the data of the block, while generating and returning a RecordReader 102 that converts the InputSplit into a mapper-readable (key, value) form. InputSplit is the unit of data processed by a single map job in MapReduce. Hadoop has several types of input formats: TextInputFormat, KeyValueInputFormat, and SequenceInputFormat. The typical input format is TextInputFormat, which divides input files stored in block units based on each line to form InputSplit, which is a logical input unit, and extracts a record of the form (LongWritable, Text) from this InputSplit. Returns.

The returned RecordReader is responsible for reading the key-value pair record from InputSplit and passing it to the mapper during normal Map process. The mapper creates this record as a record of new keys and values through the process defined in the Map. OutputFormat (103) is a format for outputting data generated during the MapReduce process to a file in HDFS. The output format is a record composed of key and value pairs received as a result of MapReduce processing through RecordWriter 104, a subclass. Writing to HDFS terminates the data processing process.

Hadoop provides various types of input and output formats for processing text data according to the characteristics of web crawling. Among them, sequence file format provides input and output for data formats other than text. It also supports input and output of compressed files such as deflate, gzip, ZIP, bzip2, and LZO, and the compressed file format can increase the storage space efficiency. However, in order to process the input file by compressed file format, it is necessary to decompress and recompress the processing result before the MapReduce operation starts. Therefore, the processing speed becomes low. The sequence file format provides a framework for storing data in a variety of formats, including binary, but requires another conversion process that requires converting the raw data to be contained in a sequence of sequences.

For this reason, in order to process a large amount of binary data such as images and communication packets in a Hadoop distributed environment, it is necessary to convert it to text or data that can be recognized by other Hadoop. This conversion process consists of reading, converting, and rewriting files for conversion by a single system, which is contrary to the fundamental purpose of improving processing performance by using Hadoop's distributed system. Therefore, in order to process binary data in Hadoop distributed environment, a more effective method needs to be developed.

An object of the present invention for solving the problems of the prior art as described above is to provide an input format that can effectively process binary packet data having a variable length data record block in the Hadoop distributed processing system.

The present invention for achieving the above object relates to an input format in Hadoop MapReduce for distributing binary packet data having a variable-length record, (A) at the start time and end time to capture the packet Obtaining information about the configuration property; (B) searching for a start point of a first packet in a data block to be processed among data blocks stored in the Hadoop Distributed File System (HDFS); (C) defining an InputSplit by setting a boundary between a previous InputSplit and its own InputSplit with the starting point of the first packet as the starting point of the InputSplit; (D) Create and return a RecordReader that reads the captured packet length (capLen) recorded in the captured packet header of each packet from the starting point for its entire InpuSplit region defined above. Making; And (E) extracting records in the form of (LongWritable, BytesWritable) from (Key, Value) through the RecordReader.

The start point of the first packet assumes the start byte of the block as the start of the first packet, and (a) the timestamp, the captured packet length (CapLen), and the actual incoming packet length from the pcap header of the packet at the assumed point. extracting header information including (wired length, WiredLen); (b) moving from the start byte of the first packet by (length of pcap header + CapLen) obtained in (a); (c) Assuming that the point moved in (b) is the start of the second packet, the timestamp from the pcap header of the second packet, the captured packet length (CapLen), and the actual packet length (wired length, WiredLen) Extracting header information including; And (d) verifying that the starting point of the first packet corresponds to the point that assumes the beginning of the first packet from the pcap header information of the first packet and the second packet obtained in (a) and (c), respectively; (e) If the result of the verification of (d) is not the start point of the first packet, the steps of (a) to (d) are assumed to be the start of the first packet at the point where one byte is moved from the position assumed as the start of the first packet. Repeating to retrieve the first packet starting point; It may be made, including.

As described above, according to the input format of the present invention, when distributing binary packet data having a variable-length record in a Hadoop environment, an InputSplit is defined for each distributed node to simultaneously access and process the binary packet data in the InputSplit. By extracting the packet data and delivering it to the mapper, it can be processed without converting the data format, which requires less storage space and enables faster processing speed than other types of data.

The input format for binary packet data having a variable-length record of the present invention can be utilized to construct an intrusion detection system through an application such as pattern matching of a packet using a Hadoop system.

1 is a conceptual diagram showing the flow of data during Job processing in Hadoop.
2 is a flowchart illustrating a procedure in which each cluster node of Hadoop reads and processes a data block.
3 is a flow chart illustrating a method for finding the start byte of the first packet in 201 of FIG. 2 according to an embodiment of the present invention.
4 is a schematic diagram for explaining packet processing by the method of FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings. However, these drawings are only examples for easily describing the content and scope of the technical idea of the present invention, and thus the technical scope of the present invention is not limited or changed. In addition, it will be apparent to those skilled in the art that various modifications and changes can be made within the scope of the present invention based on these examples.

The present invention relates to an input format in Hadoop MapReduce for distributing packet data, which is binary data having a variable-length record, (A) configuration property of information about a start time and an end time when a packet is captured. Obtaining from; (B) searching for a start point of a first packet in a data block to be processed among data blocks stored in the Hadoop Distributed File System (HDFS); (C) defining an InputSplit by setting a boundary between a previous InputSplit and its InputSplit with the start of the first packet as the starting point of the InputSplit; (D) Create and return a RecordReader that reads the captured packet length (capLen) recorded in the captured packet header of each packet from the starting point for its entire InpuSplit region defined above. Making; And (E) extracting records in the form of (LongWritable, BytesWritable) by (Key, Value) through the RecordReader; for distributing binary packet data having a variable-length record, characterized in that it comprises a. It is about input format in Hadoop MapReduce.

FIG. 2 is a flowchart illustrating a procedure in which each cluster node reads and processes data blocks according to an input format of the present invention in order to read a large packet trace file and analyze packets by Hadoop MapReduce. In FIG. 2, it is assumed that information on a start time and an end time of capturing a packet before executing a job is previously obtained through a configuration property.

When you open a block for data processing, make sure that the start of the block is the start of the packet. First, if the block is the first block of the packet trace file, the starting point will be the starting point of the packet, so define that point as the starting point of the InputSplit. If the starting point of the block is not the first block of the packet trace file, the starting point of the block and the starting point of the packet may not coincide, and thus, the process of finding a starting point for actual packet processing should be performed (201).

3 illustrates an embodiment for finding a starting point of a first packet in a data block. First, (A) Assuming that the start byte of the block is the start of the packet, the timestamp from the packet's pcap header, the captured packet length (CapLen), and the actual packet length (wired length, WiredLen) are included. Extract header information. Hereinafter, each is described as TS1, CapLen1, and WiredLen1. In general, the pcap header records a timestamp in the first 8 bytes, CapLen in the next 4 bytes, and WireLen in the next 4 bytes, so that the information can be extracted by reading 16 bytes from the start byte of the block. In this case, since the timestamp can obtain timestamp information in seconds even if only the first 4 bytes are used, only 4 bytes may be used, or 8 bytes may be used to further increase accuracy. (B) When extracting the data for the first packet, the header information for the second packet is extracted in the same manner from where it is assumed to be the starting point of the second packet. Hereinafter, each is described as TS2, CapLen2, and WiredLen2. The starting point of the second packet will be the point where the length of the pcap header of the first packet (usually 16 bytes) plus the CapLen recorded in the pcap header is moved. (C) From the header information of the first packet and the second packet obtained in (A) and (B), respectively, verify whether the first byte of the data block is the start point of the first packet.

A method of verifying the starting point of a packet will be described with reference to FIG. 3. a) Check whether TS1 and TS2 are valid values within the range from the capture start time to the end time of the packet obtained from the configuration property. b) Check that the difference between Wiredlen1 and CapLen1 is less than the difference between the maximum packet length and the minimum length of the packet. Similarly, check wiredlen2 and caplen2. The maximum and minimum lengths of packets are assumed to be 1,518 bytes and 64 bytes, respectively, according to the Ethernet frame definition. c) Verify that TS1 and TS2 can be identified as packets coming in consecutively. To do this, the difference between TS1 and TS2 is determined, and the delta time, which is recognized as a continuous packet, is determined and checked to be within the range of the value. In general, the delta time is preferably within 5 seconds, but can be properly adjusted in consideration of the network environment or other variables. If all the above steps a), b) and c) are satisfied, the start byte of the currently assumed packet is recognized as the actual packet byte of the packet. On the other hand, if any one of a), b) and c) is not satisfied, go to the next byte and assume the start point of the packet and repeat the process of a), b) and c) to verify the starting point again. Search for the start of the first packet in the data block.

In FIG. 3, all of a), b), and c) are used to verify the packet starting point. However, this is related to an example. The packet starting point may be verified based on only one or two conditions, and additional information may be added. You can verify the packet start point. Naturally, the larger the number of conditions used for the verification, the more accurate the packet starting point verification can be.

Moving to the start of the first packet in the data block by the method illustrated in FIG. 3, this is defined as the start of the InputSplit. That is, InputSplit of a data block defines InputSplit for the data block from the start of the first packet to the start of InputSplit for the next data block.

If an InputSplit is defined, it reads the capLen recorded in the pcap header from the inputSplit's starting point to perform the map operation from the InputSplit, and creates and returns a RecordReader that reads the actual packet by the length. At this time, (Key, Value) that is passed to Map by RecordReader is Hadoop's Writable class type of (LongWritable, BytesWritable) type, and Key can use offset from file start point. As a value, a packet corresponding to a specific protocol on an OSI 7 layer, such as an Ethernet frame, an IP packet, a TCP segment, a UDP segment, and an http payload corresponding to the entire byte of a packet record, may be extracted and transmitted. Similarly, a packet that does not remove the Pcap header, that is, a whole byte including the Pcap header and the Ethernet frame may be used as the value. In addition, a packet corresponding to all protocols on the OSI 7 Layer such as ICMP, ARP, RIP, and SSL may be used as a value, but is not limited thereto. It will be easy for those skilled in the art to properly select and use Value according to the data to be analyzed.

If you define an InputSplit with the start point of the first packet in the block as the previous InputSplit and return RecordReader by the above method, Mapper performs Map Function by reading from InputSplit one record by using RecordReader. At this point, the RecordReader ensures that the offset of the start point of the record to be passed out of the block to be processed is not invaded by the area of the InputSplit of the subsequent block to determine whether all records of the InputSplit for the data block have been processed. If the offset does not exceed the area of the block, RecordReader repeats the operation of reading and creating records until the offset exceeds the area of the block. If the last packet is stored in the next block, a part of the next block is read and the packet record is completed and returned.

Such an input format including the RecordReader of the present invention is called a packet input format (PcapInputFormat).

Figure 4 shows an embodiment to help understand the data processing method by the PcapInputFormat 105 of the present invention. First, the general process for each cluster node to process a large packet trace file is as follows. 2) When the job starts, the configuration information is sent to each task. The nodes in the cluster access the block that needs to be processed, move one byte from the beginning of the block, and find the starting point of the packet in the block that needs to be processed. 3) Read the packet from the location using the packet's capture length information in the packet header and pass each packet to the map. 4) It repeats the process of reading and forwarding all the packets in the block in this way. In this process, by repeating until the offset of the start point of a new packet does not exceed the block size, the split processing is completed, which is a data block that a single map task must perform.

Claims (5)

In input format in Hadoop MapReduce,
(A) obtaining information about a start time and an end time for capturing a packet from a configuration property;
(B) searching for a start point of a first packet in a data block to be processed among data blocks stored in the Hadoop Distributed File System (HDFS);
(C) defining an InputSplit by setting a boundary between a previous InputSplit and its InputSplit with the start of the first packet as the starting point of the InputSplit;
(D) Generate and return a RecordReader that reads the captured packet length (capLen) recorded in the captured Pcap header of each packet from the starting point for its entire InpuSplit region defined above. Doing; And
(E) extracting records in the form of (LongWritable, BytesWritable) with (Key, Value) through the RecordReader;
Input format in Hadoop MapReduce for distributing binary packet data having a variable-length record comprising a.
The method of claim 1,
The starting point of the first packet is
Assume the start byte of the block is the start of the first packet,
(a) extracting header information including a timestamp, a captured packet length (CapLen), and an actual incoming packet length (wiredLen) from a pcap header of a packet at an assumed point;
(b) moving from the start byte of the first packet by (length of pcap header + CapLen) obtained in (a);
(c) A header including the timestamp from the pcap header, the captured packet length (CapLen), and the actual incoming packet length (wired length, WiredLen) from the pcap header, assuming that the point moved in (b) is the start of the second packet. Extracting information; And
(d) verifying whether a starting point of the first packet is correct based on the assumption that the starting point of the first packet is correct from the pcap header information of the first packet and the second packet obtained in (a) and (c), respectively;
(e) If the result of the verification of (d) is not the start point of the first packet, the steps of (a) to (d) are assumed to be the start of the first packet at the point where one byte is moved from the position assumed as the start of the first packet. Repeating to retrieve the first packet starting point;
Input format in Hadoop MapReduce for distributed processing of the packet data comprising a.
The method of claim 2,
The step (d)
Both the timestamp of the first packet and the timestamp of the second packet obtained in steps (a) and (c), respectively, are valid within the range from the capture start time to the end time of the packet obtained from the configuration property in step (A). Value,
(Difference between WiredLen and CapLen) of the first packet obtained in step (a) is less than (difference between maximum packet length and minimum packet length),
If (difference between WireLen and CapLen) of the second packet obtained in step (c) is less than (difference between maximum packet length and minimum packet length),
Input format in Hadoop MapReduce for distributed processing of packet data, characterized in that the point assumed the start of the first packet is determined as the start point of the first packet.
The method of claim 3, wherein
In the step (d)
Further processing whether the difference between the timestamp of the first packet and the timestamp of the second packet obtained in step (a) and (c), respectively, falls within the delta time range recognized as a continuous packet. Input format in Hadoop MapReduce to use.
The method according to any one of claims 1 to 4,
The Key is an offset value from the file start point,
Value is one selected from the group consisting of Ethernet frames, IP packets, TCP segments, UDP segments, http Payload Pcap, ICMP, ARP, RIP, SSL, and Pcap headers and full packets including Ethernet frames corresponding to all bytes of the packet record. Input format in Hadoop MapReduce for distributed processing of packet data.

Images