KR102655234B1 - Method and apparatus for retrieving packet at high-speed - Google Patents

Abstract

A high-speed packet search method according to an embodiment of the present disclosure includes the steps of: generating data specialized for the search condition and storing it in a database; and when the search condition is entered, searching for a packet corresponding to the entered search condition based on the stored data.

Description

{Method and apparatus for retrieving packet at high-speed}

This disclosure relates to a high-speed packet search method and apparatus. More specifically, it relates to a method and device that can quickly search for packets matching search conditions in an environment where network detection and response technology based on full packet analysis is applied.

Network Detection and Response, based on full packet collection and analysis, is a technology that can detect and respond to threats by collecting and analyzing all packets coming in and out of a company's network at high speed.

According to network detection and response technology based on full packet collection and analysis, it is possible to analyze abnormal behavior by a company's internal employees as well as analyze security threats that may occur in services operated by the company.

Despite these advantages, it still takes a considerable amount of time to search for a desired packet from a large amount of data called a full packet.

Republic of Korea Patent No. 10-2080478 (Title of invention: Pattern-based index processing system and pattern-based index processing method using the same; Registration date: February 18, 2020)

The problem that the present disclosure aims to solve is to provide a high-speed packet search method and device that can quickly search for packets meeting search conditions in a large amount of data called full packets.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the above-described problem, a high-speed packet search method according to an embodiment of the present disclosure is a method of high-speed searching for packets corresponding to search conditions among packets collected in a network, and includes data specialized for the search conditions. creating and storing it in a database; and searching for packets corresponding to the entered search conditions based on the stored data.

Specific details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present disclosure, packets matching search conditions can be searched at high speed from a large amount of data called full packets, thereby reducing the time required for a company's security manager to recognize and respond to a cyber attack.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a diagram illustrating the configuration of a high-speed packet search device according to an embodiment of the present disclosure.
Figure 2 is a diagram illustrating a meta table according to an embodiment of the present disclosure.
Figure 3 is a diagram illustrating a meta description table according to an embodiment of the present disclosure.
Figure 4 is a diagram illustrating an IP index table according to an embodiment of the present disclosure.
FIG. 5 is a diagram for explaining a search process using the IP index table of FIG. 4.
Figure 6 is a diagram illustrating an LPM table according to an embodiment of the present disclosure.
Figure 7 is a diagram illustrating a conversion table including a URL and a constant value of the URL according to an embodiment of the present disclosure.
Figure 8 is a diagram illustrating a hash table according to an embodiment of the present disclosure.
Figure 9 is a flow chart illustrating a high-speed packet search method according to an embodiment of the present disclosure.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments posted below and may be implemented in various different forms. These embodiments are merely provided to ensure that the present disclosure is complete and to fully inform those skilled in the art of the invention of the scope of the invention, and that the present disclosure is only defined by the scope of the claims. .

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

The terms used herein are for describing embodiments and are not intended to limit the disclosure. In this specification, the singular form also includes the plural form unless specifically mentioned in the text. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements.

Hereinafter, embodiments of the present disclosure will be described with reference to the attached drawings. In the drawings, like reference numerals represent like components.

FIG. 1 is a diagram illustrating the configuration of a high-speed packet search device 100 according to an embodiment of the present disclosure.

The high-speed packet search device 100 according to an embodiment of the present disclosure uses a database specialized for the packet search function instead of a relational database (RDBMS) with various functions. In particular, among various search conditions such as IP, port, URL, application, and file, a database specialized for search conditions such as IP and URL is used.

Referring to FIG. 1, the high-speed packet search device 100 includes a meta table generator 110, a meta description table generator 120, an IP index table generator 130, an LPM table generator 140, and a conversion table. It includes a creation unit 150, a hash table creation unit 160, a database 170, and a control unit 180.

Among the components shown in FIG. 1, the meta table creation unit 110, meta description table creation unit 120, and IP index table creation unit 130 may operate as a first search engine. When IPs less than the standard number are input as search conditions, the first search engine generates data necessary to quickly search packets corresponding to the input IPs and stores them in the database 170.

Among the components shown in FIG. 1, the meta table creation unit 110 and the LPM table creation unit 140 may operate as a second search engine. The second search engine generates the data necessary to determine at high speed whether the IP of the packet collected from the network is included in the input IP list when a large number of IP lists exceeding the standard number are input as search conditions and enters the database. Save.

Among the components shown in FIG. 1, the conversion table generator 150 and the hash table generator 160 may operate as a third search engine. When a URL is entered as a search condition, the third search engine generates data necessary to quickly search packets corresponding to the entered URL and stores it in the database.

The control unit 180 determines the type and/or number of search conditions entered. For example, it is determined whether the entered search condition is IP or URL. If the entered search condition is IP, it is determined whether IPs less than the standard number have been entered or whether a large IP list has been entered. Based on this determination result, the control unit 180 can select data to be used for search from among the data stored in the database 170.

Hereinafter, the components shown in FIG. 1 will be described in more detail.

The meta table generator 110 sorts the collected packets based on IP and creates a meta table as illustrated in FIG. 2. Referring to Figure 2, it can be seen that the IPs are sorted in ascending order, and the sorted IPs are assigned a primary index, which is a number indicating the location in the meta table. Additionally, since the meta table sorts all collected packets based on IP, you can see that the same IP appears repeatedly. Hereinafter, for convenience of explanation, a group consisting of identical IPs is referred to as 'same IP section'.

The meta description table generator 120 extracts the IP located first and the primary index of the IP from each identical IP section of the meta table and generates a meta description table as illustrated in FIG. 3. . Referring to Figure 3, it can be seen that the first IPs extracted from each identical IP section are sorted based on the primary index. In addition, it can be seen that the sorted IPs are assigned a secondary index, which is a number indicating the location in the meta description table.

The IP index table generator 130 creates an IP index table as illustrated in FIG. 4 based on the meta description table. The IP index table is used to quickly search the secondary index within the meta description table in relation to the IP to be searched.

Before describing the IP index table in detail with reference to FIG. 4, IP will be briefly described. IP is divided into IPv4 system and IPv6 system. According to the IPv4 system, one IP is expressed as a 3-digit number in 4 words. Each measure is called an octet, and is internally processed as 32 bits (8 bits for each measure). For example, 192.168.123.123 is displayed as 11000000.10101000.1111011.1111011.

IP is divided into classes A, B, C, D, and E depending on the band. Class A is used in large-scale network environments, where 8 bits of the first node represent the network ID, and 24 bits of the second to fourth nodes represent the host ID. Class B is used in a medium-scale network environment, where 16 bits of the first and second nodes represent the network ID, and 16 bits of the third and second nodes represent the host ID. C class is used in small network environments, where 24 bits of the first to third nodes represent the network ID, and 8 bits of the fourth node represent the host ID. D class is used for multicasting. The E class is a class reserved for research/development or future use and is not used for general purposes. The IPv4 class classification described above is shown in Table 1.

[Table 1]

Referring to FIG. 4, the basic unit of the IP index table consists of four fields. Specifically, the basic unit is configured to include a class field, value field, index field, and jump field, and these basic units are gathered to form an IP index table. Each basic unit may be assigned a tertiary index, which is a number indicating the position in the IP index table.

In the class field, an index indicating the location of the octet of the IP is recorded. That is, an index indicating the location of each node is recorded. For example, in the case of the C class, the first to third nodes among the four nodes represent the network ID. Therefore, an index is needed to distinguish the first to third bars. According to an embodiment, the index for the first measure may be set to '0', the index for the second measure may be set to '1', and the index for the third measure may be set to '2'. In other words, it can be understood that Octet 1=0, Octet 2=1, and Octet 3=2.

The value of the node corresponding to the index of the class field is recorded in the value field. For example, if 0 is recorded in the class field, the value of the first node of the IP is recorded in the value field. If 1 is recorded in the class field, the value of the second node of the IP is recorded in the value field.

In the index field, the secondary index related to the value recorded in the value field among the secondary indexes of the meta description table is recorded. If there is no related secondary index, '0' may be recorded in the index field.

In the jump field, if the value recorded in the value field does not match the value of the IP to be searched, the tertiary index of the basic unit where the value to be compared next is recorded is recorded.

Figure 4 shows the case where each basic unit is aligned vertically, but the basic units may be aligned horizontally or in a matrix form. The created IP index table can be stored in the database.

FIG. 5 is a diagram for explaining a process of searching for a specific IP using the IP index table illustrated in FIG. 4.

Referring to FIG. 5, it can be seen that the basic units of the IP index table illustrated in FIG. 4 are arranged horizontally, and some basic units are divided into steps.

As a search condition, assume that the IP address 1.2.1.1 is entered. In this case, the basic unit whose tertiary index is '0' in the IP index table is referenced first. When interpreting the values recorded in the class field and value field of the basic unit in the IP index table, the answer is 'Is the value of the first node 1?' Since the value of the first node in the IP entered as a search condition is '1', it matches the value '1' recorded in the value field. However, you can see that '0' is recorded in the index field of the referenced basic unit. This means that there is no secondary index in the meta description table. Therefore, the basic unit to which the tertiary index of the following order is assigned is referred to. In other words, the basic unit whose tertiary index is '1' is referred to.

When interpreting the values recorded in the class field and value field of the referenced basic unit, the answer is 'Is the value of the second node 1?' Since the value of the second node in the IP entered as the search condition is '2', it does not match the value '1' recorded in the value field. If the values do not match like this, check the jump field of the referenced basic unit. You can see that '4' is written in the jump field. This means that the basic units with a tertiary index of '2' and '3' should be skipped, and the basic unit with a tertiary index of '4' should be skipped. As a result, the basic unit with the tertiary index '4' is referenced.

When interpreting the values recorded in the class field and value field of the referenced basic unit, the answer is 'Is the value of the second node 2?' Since the value of the second node in the IP entered as a search condition is '2', it matches the value '2' recorded in the value field. However, you can see that '0' is recorded in the index field of the referenced basic unit. This means that there is no secondary index in the meta description table. Therefore, the basic unit to which the tertiary index of the following order is assigned is referred to. In other words, the basic unit whose tertiary index is '5' is referenced.

When interpreting the values recorded in the class field and value field of the referenced basic unit, the answer is 'Is the value of the third node 1?' Since the value of the third node in the IP entered as the search condition is '1', it matches the value '1' recorded in the value field. However, you can see that '30' is recorded in the index field of the referenced basic unit. In this way, if there is a value recorded in the index field, the corresponding value is obtained.

Afterwards, the secondary index equal to '30', which is the value obtained from the index field, is searched within the meta description table. As a result, the primary index corresponding to the secondary index is obtained. Referring to FIG. 3, it can be seen that the primary index '200' corresponds to the secondary index '30'. Once the primary index is obtained, IP addresses corresponding to the primary index can be searched in the previously stored meta table.

Referring again to FIG. 1, the LPM table generator 140 converts the IP list exceeding the standard number into an LPM (Longest Prefix Matching) table. The LPM table has 256 cells arranged along the horizontal direction. Hereinafter, the 256 cells arranged along the horizontal direction are referred to as 'standard rows'. The 256 cells included in the standard row are assigned indices of 0 to 255 (hereinafter referred to as 'horizontal indices'). Each cell can have one value between 0 and 1. Specifically, among the cells in the standard row, the value of the cell having the same horizontal index as the value of the first node of the IP included in the IP list is set to '1'. For example, let's say the IP list includes the following IPs:

192.168.10.232

193.168.10.232

In this case, the values of the first measure are 192 and 193. Accordingly, among the cells in the reference row, the values of a cell with a horizontal index of '192' and a cell with a horizontal index of '193' may each be set to 1. The values of all remaining cells can be set to 0.

At the bottom of the standard row, rows containing the same number of cells as the standard row are arranged one after another. The cells arranged along the vertical direction are sequentially assigned indices corresponding to the value range of the first node, indexes corresponding to the value range of the second node, and indices corresponding to the value range of the third node ( Hereinafter referred to as 'vertical index'). If the IPs included in the IP list correspond to the C class, the value of the first node ranges from 192 to 223. And the range of the second and third node values is 0 to 255, respectively. Therefore, cells arranged along the vertical direction have vertical indices of 192 to 223 corresponding to the value range of the first node, vertical indices of 0 to 255 corresponding to the value range of the second node, and vertical indexes of 0 to 255 corresponding to the value range of the second node. Vertical indices from 0 to 255 corresponding to the value range are sequentially assigned.

For example, if the IP list includes only the IP “192.168.10.232”, an LPM table as shown in FIG. 6 is created. Since the value of the first node of the IP is 192, the value of the cell with the horizontal index of 192 among the cells in the standard row is set to 1 (cells with a value of 1 are displayed in a different color from the remaining cells). And when the value of the first node is 192, the value of the second node is 168, so the row with a vertical index of 192 is selected among the vertical indices corresponding to the value range of the first node, and the cells belonging to the selected row are selected. Among them, the value of the cell with horizontal index 168 is set to 1. Additionally, when the value of the second node is 168, the value of the third node is 10, so the row with a vertical index of 168 is selected among the vertical indices corresponding to the value range of the second node, and the row belonging to the selected row is selected. Among the cells, the value of the cell with a horizontal index of 10 is set to 1. Additionally, when the value of the third node is 10, the value of the fourth node is 232, so the row with a vertical index of 10 is selected among the vertical indices corresponding to the value range of the third node, and the row belonging to the selected row is selected. Among the cells, the value of the cell with a horizontal index of 232 is set to 1.

After the LPM table is created in this way, if the IP of the actual collected packet is "192.168.10.232", the process of searching the IP of the collected packet in the IP list is explained as follows.

First, the value of the first node of the IP is searched. Since the value of the cell with horizontal index 192 in the standard row of the LPM table is 1, the value of the first node is determined to exist in the LPM table. Therefore, a query is made for the value of the second node. Since the value of the cell with a vertical index of 192 and a horizontal index of 168 in the LPM table is 1, the value of the second node is determined to exist in the LPM table. Therefore, a query is made for the value of the third node. In the LPM table, since the value of the cell with the vertical index is 168 and the horizontal index is 10 is 1, the value of the third node is determined to exist in the LPM table. Therefore, a query is made for the value of the fourth node. In the LPM table, the vertical index is 10 and the value of the cell with the horizontal index 232 is 1, so the value of the fourth node is determined to indicate that the LPM table exists. In this way, whether the IP is included in the IP list can be determined through a total of four comparison processes, so the number of comparisons can be significantly reduced compared to the conventional method. Specifically, if there are n IPs obtained from the collected packets and the black IP list already held by the company is m, conventionally, n x m comparisons must be performed. However, when using the LPM table according to an embodiment of the present disclosure, only a minimum of nx1 to a maximum of nx4 comparisons need to be performed, so the number of comparisons is greatly reduced. And search speed improves as the number of comparisons decreases.

Referring again to FIG. 1, the conversion table generator 150 converts the URL of the collected packets into a constant value. URLs have a text format with an unspecified length, and converting these URLs into constant values means assigning an ID to the URL. Here, ID may mean Primary Key (PK). An example of a conversion table including URL and constant values of URL is shown in Figure 7.

The hash table generator 160 generates a hash table of a standard size based on the constant value. The reference size may be, for example, 65536. However, the reference size is not limited to the illustrated size, and may be smaller or larger than the illustrated size. Hereinafter, for convenience of explanation, the case where the standard size is 65536 will be used as an example.

Figure 8 is a diagram illustrating a hash table according to an embodiment of the present disclosure.

Referring to Figure 8, the hash table includes a hash index and cumulative value. The hash index is determined by the remainder when the URL ID is divided by 65536. Therefore, the hash index ranges from 0 to 65535. IDs with the same hash index are sequentially recorded in the hash index.

The cumulative value is recorded by sequentially accumulating the number of IDs recorded in the hash index. For example, it can be seen that a total of 3 IDs are recorded in hash index 0, a total of 4 IDs are recorded in hash index 1, and a total of 2 IDs are recorded in hatch index 2. In this case, the cumulative value for hash index 1 is 3+4=7. And the cumulative value for hash index 2 is 7+2=9.

These cumulative values can be used to narrow your search. For example, let's say the URL 'naver.com' is entered as a search condition. Referring to Figure 7, the ID for the URL 'naver.com' is 2, so the hash index is also 2. In Figure 8, it can be seen that the cumulative value for hash index 2 is 9, and the immediately previous cumulative value is 7. This result can be obtained by searching only 7 to 9 metadata.

Figure 9 is a flow chart illustrating a high-speed packet search method according to an embodiment of the present disclosure.

First, data specialized for the search conditions is created and stored in the database (S910). Here, examples of search conditions include IP and URL. In the case of IPs, IPs less than the standard number may be used as search conditions, or IP lists greater than the standard number may be used as search conditions. For example, a black IP list containing more than 1 million IP addresses already secured by a company may be used as a search condition. Examples of data specialized for IP search below the standard number include meta tables, meta description tables, and IP index tables. An example of data specialized for IP list search is the LPM table. Examples of data specialized for URL search include a conversion table and hash table containing URLs and constant values of URLs.

Afterwards, based on the data stored in the database, packets corresponding to the entered search conditions are searched (S920).

In step S920, if the input search condition is less than the standard number of IPs, the IP index table, meta description table, and meta table may be selected from data stored in the database. The value of each node of the IP entered as a search condition is compared with the value determined by the class field and value field of the standard unit of the IP index table. If the two values match as a result of the comparison, the secondary index and primary index are searched from the meta description table using the value of the index field of the standard unit. If the two values do not match as a result of the comparison, it jumps to the standard unit indicated by the value of the jump field of the standard unit. Using the value of the jump field like this reduces the number of comparisons, and the search speed improves as the number of comparisons decreases.

In step S920, if the input search condition is a large IP list, the LPM table may be selected from data stored in the database. Then, the value of each node of the large IP list is compared with the value of each cell of the LPM table. During the comparison process, if it is determined that the value of each node of the IP list is not recorded in the LPM table, the comparison process is immediately stopped.

In step S920, if the input search condition is URL, the conversion table and hash table stored in the database may be selected. Referring to the conversion table, the ID corresponding to the URL entered as the search condition is searched. Afterwards, when the remainder when the searched ID is divided by 65536 is calculated, a hash index with the same value as the calculated value and the corresponding cumulative value can be obtained from the hash table.

Above, embodiments of the present disclosure have been described. In the above description, the components that make up the high-speed packet search device shown in FIG. 1 may be implemented as modules. A module refers to software or hardware components such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and the module performs certain roles. However, module is not limited to software or hardware. A module may be configured to reside on an addressable storage medium and may be configured to execute on one or more processors.

Thus, as an example, a module may contain components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, properties, procedures, and subroutines. , segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided by components and modules may be combined into smaller numbers of components and modules or may be further separated into additional components and modules.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory 200, optical data storage device, etc.

Additionally, computer-readable recording media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable recording medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play StoreTM) or on two user devices (e.g. It may be distributed directly between smartphones (e.g. smartphones) or distributed online (e.g. downloaded or uploaded). In the case of online distribution, at least a portion of a computer program product (e.g., a downloadable app) is stored at least temporarily on a machine-readable recording medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be stored or created temporarily.

Embodiments according to the present disclosure have been described with reference to the above and the attached drawings. A person skilled in the art to which this disclosure pertains will understand that this disclosure can be implemented in other specific forms without changing its technical idea or essential features. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: High-speed packet search device

Claims (10)

In a method of quickly searching for packets matching search conditions among packets collected from a network,
generating data specialized for the search conditions and storing it in a database; and
Based on the stored data, searching for packets corresponding to the entered search conditions,
The saving step is,
Sorting the packets collected from the network based on IP and assigning a primary index to the sorted packets to create a meta table;
Extracting a first IP and a primary index of the first IP for each same IP section of the meta table, and assigning secondary indexes to the extracted IPs to create a meta description table; and
Generating an IP index table for high-speed access to a secondary index related to the IP entered as the search condition among the secondary indexes of the meta description table,
The IP index table includes a plurality of basic units,
The basic unit includes a class field, value field, index field, and jump field,
A tertiary index is assigned to the plurality of basic units,
High-speed packet search method.
delete delete According to paragraph 1,
In the class field, an index indicating the location of each node of the IP included in the meta description table is recorded,
The value of the node corresponding to the index of the class field is recorded in the value field,
In the index field, a secondary index related to the value recorded in the value field among secondary indexes of the meta description table is recorded,
In the jump field, if the value recorded in the value field and the value of the IP entered as the search condition do not match, a tertiary index of the basic unit in which the value to be compared next is recorded is recorded,
High-speed packet search method. According to paragraph 4,
The above search steps are
Comparing the value of each node of the IP input as the search condition with a value determined by a class field and a value field of a standard unit of the IP index table;
If the two values match as a result of the comparison, searching for a secondary index in the meta description table using the value of the index field of the reference unit;
As a result of the comparison, if the two values do not match, jumping to the reference unit indicated by the value of the jump field of the reference unit,
High-speed packet search method. According to paragraph 1,
The saving step is,
generating a meta table by sorting packets collected from the network based on IP; and
Including the step of creating an LPM (Longest Prefix Matching) table for the already secured IP list,
High-speed packet search method. According to clause 6,
The LPM table is
Base row containing 256 cells arranged along the horizontal direction; and
It includes the same number of cells as the reference row, and includes a plurality of rows arranged along the vertical direction at the bottom of the reference row,
Horizontal indices are assigned to cells in the reference row,
Cells arranged along the vertical direction include indices corresponding to the value range of the first node of the IP included in the IP list, indexes corresponding to the value range of the second node, and the value range of the third node. Indexes corresponding to are assigned sequentially,
High-speed packet search method. In clause 7,
The above search step is,
Comparing the value of each node of the IP list input as the search condition with the value of each cell of the LPM table; and
If the value of each node of the input IP list is not recorded in the LPM table, stopping the comparison,
High-speed packet search method. According to paragraph 1,
The saving step is,
Sorting packets collected from the network based on URLs and assigning IDs that are constants of the URLs to the sorted URLs;
Creating a hash table using the remainder when dividing the ID by the standard size as a hash index;
sequentially recording IDs having the same hash index in the corresponding hash index of the hash table; and
Including recording a cumulative value for the number of IDs recorded in the hash index,
High-speed packet search method. A computer-readable recording medium on which a program including instructions for performing each step according to the high-speed packet search method according to any one of claims 1, 4 to 9 is recorded.