JP4717106B2 - Flow information processing apparatus and network system - Google Patents

Description

  The present invention relates to processing of flow information which is a log of processing contents performed by a router device, and particularly relates to a flow information processing device for collecting flow information and a network system including the flow information processing device.

  In a router apparatus constituting a network system, the contents processed by the router apparatus are notified to an external apparatus as flow information, and NetFlow is known as a protocol for notifying flow information (for example, Non-Patent Document 1). In the router device, the notification content, notification target, and transmission cycle of the flow information are set in advance, and a NetFlow packet as flow information is created according to the set content and transmitted to an external device. The flow information is information for notifying the packet transfer content. For example, in the case of IPv4, the source IP address, destination IP address, source port number, destination port number, and protocol are included.

  As a successor to NetFlow, standardization of IPFIX (IP Flow Information eXport) is being promoted by IETF (Internet Engineering Task Force) (for example, Non-Patent Document 2). Since IPFIX was formulated in response to NetFlow technology, NetFlow technology can also be applied to IPFIX.

A flow information processing apparatus is known as an apparatus that collects the above flow information and provides flow information in response to a search request from a server that monitors a network system (for example, Patent Document 1).
JP 2007-228513 A RFC 3954 Cisco Systems NetFlow Services Export Version 9. B. Claise, October 2004, [online], searched June 10, 2008, Internet <URL <http://www.faqs.org/rfcs/rfc3954.html> RFC 5101 Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information, [online], searched on June 10, 2008, Internet <http://tools.ietf.org/html/rfc5101 > JP-A-8-191955

  The conventional flow information processing apparatus stores received flow information in a database or the like, and performs statistical processing or the like on a query from a server. When the flow information processing apparatus is operated in a large-scale network having a large number of router apparatuses, a large amount of flow information is received and stored in a database. When a flow monitoring apparatus requests flow information from a server that monitors the network system, the flow information processing apparatus retrieves the requested flow information from a large amount of flow information by conventional database processing, and responds to the server. . As the scale of the network system increases, the amount of flow information stored in the database also increases, and there is a problem that the time from receiving a search request from the server to responding increases. In particular, when flow information is stored as a database in a storage device such as a hard disk device, it takes time to extract flow information that satisfies the search conditions from a large amount of flow information, and the access time of the hard disk device increases flow information. Accompanying this, the processing load of the flow information processing apparatus also increases.

  Therefore, the present invention has been made in view of the above problems, and is capable of quickly performing a search process even when a large amount of flow information is accumulated by applying a flow information processing apparatus to a large-scale network. Objective.

The present invention relates to a receiving unit that receives flow information or a search request, a storage device that stores the received flow information, and a search unit that searches for flow information stored in the storage device based on the received search request. And a transmission unit that transmits the search result of the search unit, a flow information storage unit that temporarily stores the flow information received by the reception unit, and a predetermined condition: A search information generation unit that generates search information from the flow information stored in the flow information storage unit when the search request is established, and the search unit is included in the search request when the search request is received searches the search information in the search condition, and acquires the flow information from the search for information that matches the search criteria, the search information generating unit, every time the predetermined condition is satisfied, Serial generates the search information for each item included in the flow information.

  The search information generation unit extracts the contents of items included in the flow information stored in the flow information storage unit, calculates the appearance frequency of the extracted contents, and assigns an index to the flow information. The contents of the items are set in the order of the calculated appearance frequencies, and bit information is generated as the search information from the relationship of the flow information indicated by the index.

  Therefore, according to the present invention, even when a large amount of flow information is accumulated, access to the storage device can be reduced, and the speed of search processing can be improved. Further, the processing time for acquiring flow information from the storage device does not need to search all the flow information as in the conventional example, so that processing can be performed in a very short time, and high-speed search is possible.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram of a network system including a flow information processing apparatus 101 according to an embodiment of the present invention.

  The flow information processing apparatus 101 is mainly configured by a computer including a CPU (processor) 2, a memory 3, a storage device (storage device) 4, an interface 5, and a communication interface 104.

  The CPU 2 executes the program (flow processing unit 106) stored in the memory 3 to perform storage of the flow information 200, generation of search information, storage, and search processing as will be described later. The CPU 2 executes the flow processing unit 106 loaded in the memory 3, stores the received flow information 200 in the cache area 30 once set in the memory 3, and rank as search information when a predetermined condition is satisfied. The attached bitmap index 700 is created, and the flow-added information 200A in which the ranked bitmap index 700 and the flow information 200 are collected in a file as will be described later is stored in the storage device 109. The flow processing unit 106 refers to the generation condition stored in the setting table 31 set on the memory 3 when generating the ranked bitmap index 700.

  The storage device 109 includes a disk device and the like. The file format flow information 200A transferred from the cache area 30 of the memory 3, the ranked bitmap index 700 generated by the flow processing unit 106, and the flow processing unit 106. Etc. are stored. Note that the storage medium for storing the program such as the flow processing unit 106 is not limited to the storage device 109 configured by an HDD or the like, and can be stored in an optical disk, a nonvolatile semiconductor memory, or the like.

  The interface 5 is connected to the storage device 109, the input device 6, the display 7, the communication interface 104, and the like. The input device 15 transmits information input by the person in charge to the CPU 2 via the interface 5. The display 7 displays information instructed by the CPU 2. The communication interface 104 is connected to a network 8 such as a LAN and transmits and receives signals (packets).

  A plurality of router devices 102 and a plurality of server devices 103 are connected to the network 8. The router apparatus 102 communicates with the server apparatus 103 and an external network, and transmits the contents processed by the router apparatus 102 to the flow information processing apparatus 101 as flow information 200. Also, any of the plurality of server apparatuses 103 monitors the network system and issues a query requesting the flow information 200 to the flow information processing apparatus 101.

  FIG. 2 is a functional block diagram of the network system according to the present invention.

  The flow information processing apparatus 101 is connected to one or more router apparatuses 102. When the router apparatus 102 transfers the packet, it transmits the flow information 200 to the flow information processing apparatus 101. In the flow information processing apparatus 101, the flow information 200 from the router apparatus 102 is received by the communication interface 104, the protocol processing unit 105 performs preprocessing for each protocol, and the flow processing unit 106 accumulates and ranks the flow information 200. A bitmap index 700 is generated and searched. The protocol processing unit 105 is included in the OS executed by the CPU 2 of the flow information processing apparatus 101, and performs, for example, transport layer processing. The protocol processing unit 105 determines whether the received information is flow information 200, a query, or other information from the protocol of the information (packet) received by the communication interface 104. The information is subjected to a preset process and then transferred to the flow information receiving unit 107 of the flow processing unit 106. If it is a query, the received query is subjected to a preset process and then sent to the query receiving unit 111 of the flow processing unit 106. Send.

  The flow information 200 issued by the router apparatus 102 is first sent to the flow information receiving unit 107. The received flow information 200 is stored in the storage device 109 through the flow information storage unit 108.

  The flow information storage unit 108 is set in the cache area 30 of the memory 3 illustrated in FIG. 1 and temporarily stores the received flow information 200. The search information generation unit 110 generates a ranked bitmap index 700 from the flow information 200 stored in the flow information storage unit 108 of the cache area 30 every time a predetermined condition is satisfied, and the flow information storage unit 108 The flow information 200A in which the flow information 200 is in a file format and the ranked bitmap index 700 are stored in the storage device 109. The flow information storage unit 108 clears the flow information 200 in the cache area 30 when the flow information 200 in the cache area 30 is written in the storage device 109.

  As a predetermined condition for starting the generation of search information, the search information generation unit 110, for example, when a preset period is reached, or when the cache area 30 is filled with the flow information 200 and there is no more free space (The free capacity of the cache area 30 is a predetermined value = 0%) or the like can be satisfied.

  When the predetermined condition is set to a preset cycle, the search information generation unit 110 monitors the time with a timer or the like (not shown), and when the predetermined cycle is reached, all the flow information 200 in the flow information storage unit 108 in the cache area 30. To get. Then, the search information generation unit 110 generates a ranked bitmap index 700 from the flow information 200 as described later, stores it in the storage device 109, and further collects the acquired flow information 200 into one file as flow information. 200A is stored in the storage device 109. That is, the flow information 200A as a file including the ranked bitmap index 700 and the flow information 200 is written in the storage device 109 at every predetermined period.

  The flow information processing apparatus 101 is connected to one or more server apparatuses 103 as described above, and the server apparatus 103 describes a search condition when referring to the flow information 200 stored in the flow information processing apparatus 101. A request (hereinafter referred to as a query) is transmitted to the flow information processing apparatus 101.

  The flow information processing apparatus 101 receives a query from the server apparatus 103 by the communication interface 104, performs preprocessing for each protocol by the protocol processing unit 105, and transmits the query to the query reception unit 111.

  The query processing unit 112 extracts search conditions from the received query and transfers them to the search unit 113. The search unit 113 retrieves the search information from the storage device 109 according to the search condition and performs a search process.

  The search unit 113 searches the flow information 200 requested by the query using the ranked bitmap index 700 which is search information, and extracts the flow information corresponding to the search processing result from the storage device 109. At this time, in addition to the result of searching with the ranked bitmap index 700, whether or not the flow information 200 matches the search request may be searched with the flow information 200 of the storage device 109.

  The flow information 200 extracted from the storage device 109 by the search unit 113 is sent to the transmission unit 114 via the query processing unit 112.

  Flow information 200 as a search result is shaped into a predetermined protocol from the transmission unit 114 via the protocol processing unit 105, and then transmitted from the communication interface 104 to the server device 103 via the network 8.

  This transmission process may be executed each time a search result is generated, or may be transmitted to the server apparatus 103 after storing the search result for a certain period or a certain amount.

  Setting values in these series of processes (for example, predetermined conditions for determining the start of creation of search information, parameters for creating the ranked bitmap index 700, etc.) are set by the administrator or user as described later. Setting is made from the input device 6 through the user interface 115.

  FIG. 3 is an explanatory diagram illustrating an example of the flow information 200.

  The flow information 200 received from the router apparatus 102 includes information on packets processed by the router apparatus 102 in accordance with a format determined by NetFlow or IPFIX as described in the conventional example. As an example of the flow information 200, in addition to the source IP address 201, the destination IP address 202, and the protocol number 203 in the IP packet, the number 204 of packets transmitted and received between IP addresses and the total number of bytes (205) An example in which fields such as are provided for each entry is shown.

  FIG. 4 is an example of search conditions included in a query transmitted from the server apparatus 103 to the flow information processing apparatus 101.

  The search condition includes the protocol number 301 existing in FIG. 2 and the IP address 302 with the prefix specified, and the minimum value 303 of the time when the flow information processing apparatus 101 receives the flow information 200 from the router apparatus 102. It is also possible to specify the maximum time 304. The search unit 113 that has received this search condition includes a flow including the specified protocol number 301 with the specified IP address + prefix 302 within the specified time range (between the minimum time value 303 and the maximum time value 304). The information 200 is searched from the ranked bitmap index 700.

  FIG. 5 is a flowchart showing an example of a search process according to a conventional example.

  The flow information 200 to be processed is all the flow information 200 received by the flow information processing apparatus 101. At this time, as an example, if the flow information 200 is recorded on the hard disk device constituting the storage device 109 and the recording time is written in the file name, it can be narrowed down in advance. You may narrow down the files.

  When performing a search, all the flow information 200 recorded in the storage device 109 is (401) retrieved one by one (402), and the extracted flow information 200 is included in the conditions (query included in FIG. 3). The search condition is satisfied (403). If it is determined that the flow information 200 satisfies the search condition, a process (404) of transmitting the flow information 200 to the server apparatus 103 is performed. If the search condition is not satisfied, the flow information 200 is moved to the next flow information 200.

  When the flow information 200 is recorded on a medium having a large input / output overhead such as a hard disk device in such a conventional search method, the waiting time from the start of the search to the output of the search result increases. Not only increases the time until the result is returned, but also reduces the overall performance of the flow information processing apparatus 101.

  As a solution to such a brute force search method, there is a method using search information. In the present invention, a bitmap index is used as search information.

  FIG. 6 is an explanatory diagram showing an example of a bitmap index. The bitmap indexes 501 and 502 are created when the search information generating unit 110 generates the ranked bitmap index 700.

  The search information generation unit 110 is configured by a single file as shown in FIG. 6 from a plurality of flow information 200 stored in the flow information storage unit 108 of the cache area 30 shown in FIG. 3 at a predetermined cycle. Generated flow information 200A is generated.

  The search information generation unit 110 adds a row 211 to the index for identifying the flow information 200, stores a column 210 storing the item name (element) of the field of the flow information 200, and a column 212 storing the contents of each flow information 200. Generate a file containing For example, the element A in FIG. 6 is assigned to the protocol number shown in FIG. 3, and the protocol number included in each flow information 200 is sequentially stored as “4”, “17”, “6” in the row direction. Similarly, the source IP address 201 shown in FIG. 3 is assigned to element B, the contents of each flow information 200 are stored in the row direction, and the destination IP address 202 shown in FIG. 3 is assigned to element C. Thus, the flow information 200A is generated by storing the contents of each flow information 200 in the row direction.

  The search information generation unit 110 includes the time when the flow information 200 is received in the file name of the flow information 200A. As this time, for example, the reception time of the flow information 200 is a pair of the latest time and the oldest time among the plurality of flow information 200 stored in the flow information storage unit 108 of the cache area 30. Can do.

  In this way, the flow information 200A sets the index for identifying the flow information 200 in the row 211 in the row direction, and sets the elements (IP address, protocol number, number of bytes or number of packets) of the flow information 200 in the column 210 direction. Then, the element value of the flow information 200 corresponding to the index row 211 is stored in the column 212.

  Next, the search information generation unit 110 generates bitmap indexes 501 and 502 for each element (item name) of the flow information 200, respectively. The bitmap index 501 is a bitmap index related to the element A of the flow information 200A, and the search information generation unit 110 sets a value that the element A of the flow information 200A can take in the column 510, and the flow information 200A in the row 511. The index set in the row 211 is stored as an identifier. A unique serial value or the like can be used as an identifier assigned to the row 211 as an index. The search information generation unit 110 sets the column 512 for each identifier of the row 511 and sets “1” (bit information is on) to the row of the value of the corresponding element A. For example, since the value of the element A is “4” in the flow information with the index = 0 in the flow information 200A, the column A in the column 512 with the identifier = 0 of the bitmap index 501 has a value of “4” in the element A. “1” is set, and “0” (bit information is off) is set in the other rows. The search information generation unit 110 creates a bitmap index 501 corresponding to the value of the element A for each index of the flow information 200A.

  Similarly, the search information generation unit 110 sets values that can be taken by the element B of the flow information 200A in the column 520, and stores the index set in the row 211 of the flow information 200A as an identifier in the row 521. The search information generation unit 110 sets the column 522 for each identifier of the row 521 and sets “1” to the row of the value of the corresponding element A.

  Through the above processing, the search information generation unit 110 sets the bitmap indexes 501 and 502 for each element of the flow information 200. In the above example, the bitmap index of the elements A and B of the flow information 200A is created. However, the bitmap index may be created only for preset elements. At this time, the bitmap indexes 501 and 502 are held on the memory 3.

  In the bitmap index 501A of FIG. 10, when the element identifiers 0 to 99 exist for the element value A of the flow information 200A, and each element takes a value from 0 to 4, a binary of 5 × 100 Create a table. This is a bitmap index. For this bitmap index, if the value of identifier 1 in row 511 is 1, the value in the row with value 1 and identifier 1 is 1, and the other values in identifier 1 are 0. This is done for all elements.

  When it is desired to search for an element whose value A is X among these elements, the row whose column 510 value is X in the bitmap index is examined, and the element whose value is 1 is applicable.

  When the bitmap indexes 501 and 502 in FIG. 6 are used, as shown in FIGS. 7 and 8, when the search unit 113 uses a plurality of bitmap indexes 501 and 502 as shown in FIGS. The calculation tables 503 and 504 can be created on the memory 3 according to the search conditions. For example, when bitmap indexes 501 and 502 are created for the values A and B of each element of the flow information 200A, it is desired to search for an identifier of an element whose value A is 1 or value B is 2. Is obtained by performing an OR operation on the values of the column 1 of the value A and the column 2 of the value B in the bitmap indexes 501 and 502 and extracting the information of the identifier whose result is 1. This is held on the memory 3 as a calculation table 503 as shown in FIG.

  Similarly, when it is desired to search for an identifier of an element whose value A is 1 and value B is 2, an AND operation is performed on the values of column 1 of value A and column 2 of value B in bitmap indexes 501 and 502, and the result is 1 What is necessary is just to take out the information of the identifier which is. The calculation result is held on the memory 3 as a calculation table 504 as shown in FIG.

  Then, the search unit 113 can execute a bitmap search using the calculation tables 503 and 504 in FIG. 7 or FIG. 8 for a plurality of search conditions.

  FIG. 9 is a flowchart showing an example of a search using the bitmap indexes 501 and 502 for comparison with the ranked bitmap index of the present invention. This flowchart can be executed by the search unit 113 of the flow processing unit 106.

  The search unit 113 acquires the search condition received from the query processing unit 112 (601). Then, the search information generation unit 110 reads the bitmap indexes 501 and 502 stored in the storage device 109 from the period (time range) included in the search condition into the memory 3. The values and bits (rows) of elements satisfying the search conditions are searched on the bitmap indexes 501 and 502. If there is a row satisfying the search condition in the bitmap indexes 501 and 502, the search unit 113 acquires an identifier whose value is 1 as the index of the flow information 200A (602). When searching the bitmap index 501, the search unit 113 may create the calculation tables 503 and 504 as shown in FIGS. 7 and 8 and perform a search using these tables.

  Next, the search unit 113 acquires the flow information 200A corresponding to the period included in the search condition from the storage device 109, and acquires the flow information 200 from the column 212 corresponding to the index (604, 605). Then, the acquired flow information 200 is returned to the server apparatus 103 via the query processing unit 112 and the transmission unit 114 (605). Note that the processing in steps 603 to 605 is repeated for the number of identifiers acquired in step 602.

  The difference between the processing using the bitmap index shown in FIG. 9 and the conventional processing not using the search information shown in FIG. 5 is that the bitmap indexes 501 and 502 which are search information are used. The flow information 200A that matches the search condition is calculated in advance (601), and only the corresponding flow information 200A is extracted (602). As a result, the number of times the entire flow information 200A is read from the storage device 109 can be greatly reduced, and a reduction in query response time and an increase in the speed of the flow information processing device 101 can be expected.

  The advantages of the bitmap indexes 501 and 502 are the ease and speed of operations such as AND and OR between a plurality of bitmap indexes by generating the calculation tables 503 and 504. Disadvantages are the complexity of new data insertion and the increased size of the search data.

  However, when new data is inserted into the bitmap index, the data behind the newly inserted data must be shifted bit by bit. The size of the search information is “value range taken by the element × number of flow information” for each element, and when the bitmap index is applied to all elements, the capacity equivalent to the received flow information 200 is obtained. Necessary for search information.

  In the present invention, a “ranked bitmap index” is proposed by taking advantage of the bitmap index and reducing the disadvantages.

  FIG. 10 shows an example of a ranked bitmap index 700 in which the range of values is 0 to 255 and the number of flows is 0 to 99. In the bitmap indexes 501 and 502 shown in FIG. 6, the values taken by the elements of the flow information 200 </ b> A correspond to the bitmap index rows on a one-to-one basis, whereas in the ranked bitmap index 700 shown in FIG. 10. The “other” line (703) is newly added, and elements having low appearance frequency (appearance frequency is less than a predetermined value) are combined into one line, thereby greatly reducing the data amount of the index and the search unit 113. Is a fast search.

  After creating the same bitmap index 501A as shown in FIG. 6 above, the search information generation unit 110 calculates “rank number” and “ranking” from the statistics and set values of the flow information 200A. Then, the upper rank “rank number” of “ranking” is stored in the ranked bitmap index 700, and the rest is stored in the “other” column.

  The bitmap index 501A in FIG. 10 indicates a case where the flow information 200 of 100 (0 to 99) has an element value of 100 (0 to 99). The bitmap index 501A includes a column 512 for storing the values of 100 elements and a row 511 for storing the index of the flow information 200A as an identifier, and the column 512 corresponding to the identifier has the flow information similar to FIG. “1” is set in the row corresponding to the value of the 200A element.

  Next, the search information generation unit 110 sorts the element values in the column 510 in the row direction in descending order of the number in which “1” is set. Then, the search information generating unit 110 adds the rows having ranks up to a preset rank number R to the ranked bitmap index 700. That is, the rank number R is used as the bitmap index in descending order of the appearance frequency of the element values.

  Next, the bits from the rank number R to the lowest row are combined into one row, and this row is added as the other 703 of the ranked bitmap index 700. Thereby, the data amount of the ranked bitmap index 700 becomes (the number of ranks R + 1) × the number of flow information 200. In this example, the bit area where “1” is set can be expressed by 5 × 100 bits. . On the other hand, in the bitmap index 501A, the bit area for which “1” is set is the number of element values × the number of flow information 200. In this example, the number of bits of 100 × 100 is required.

  As described above, in the ranked bitmap index 700, the amount of data can be significantly reduced compared to the bitmap index 501A, and the row storing the element values to be searched becomes the rank number R. Processing can be greatly speeded up.

  Here, the protocol number, the port number, the IP address, and the like, which are elements of the flow information 200, can have a wide range of element values such as 0h to FFh or 0h to FFFFh. The element values of the flow information 200 rarely appear evenly. For example, the protocol numbers that appear in the network 8 include commonly used protocols and extremely rare protocols. Furthermore, when the flow information 200 is searched from the server device 103, a failure or the like occurs in the network 8 or the router device 102, and therefore, the frequency of appearance of a specific protocol tends to increase due to retry or the like.

  Therefore, the ranked bitmap index 700 is generated as the bitmap index 501A for the value of the element having a high appearance frequency and as the other 703 for the value of the element having a low appearance frequency.

  When the search unit 113 executes a query from the server device 103 and the search condition corresponds to other 703 (the frequency of appearance of element values is low), as shown in FIG. Since the search is performed from 200A, the time required for the search process is the same as the conventional example. However, if the search unit 113 executes a query from the server device 103 and the search condition matches the value of the rank rank element, the index of the flow information 200A can be acquired from the ranked bitmap index 700. By obtaining the target flow information 200A from the storage device 109 using this index, it becomes possible to access the information at an extremely high speed as compared with the conventional example.

  As described above, since the value of the element A of the flow information 200A is generally not uniform and has a bias, the search condition does not frequently become other 703, and most of the search conditions In the range of the number of ranks R, the search process can be performed very quickly.

  Furthermore, since it is necessary to generate the bitmap index 501A for each element of the flow information 200A, when the number of element values is large and there is a large amount of the flow information 200, the bitmap index 501A for one flow information 200A. Therefore, the time required for the search unit 113 to read the bitmap index 501A from the storage device 109 also increases.

  On the other hand, in the ranked bitmap index 700 in which the number of element values is limited, even when the number of element values is large and the flow information 200 is large, the number of rows maintains a predetermined rank number R + 1. Therefore, it is possible to prevent the amount of data from becoming excessive, prevent the search unit 113 from reading the ranked bitmap index 700 from the storage device 109, and contribute to speeding up the search process. It can be done.

  FIG. 11 shows an example of the user interface 115 shown in FIG. 2, and is a screen image displayed on the display 7 by a user interface for setting conditions for generating the ranked bitmap index 700.

  In the user interface 115, among the elements included in the flow information 200, the bitmap index creation 1401 for determining the element (accumulated item in the figure) 1400 for generating the ranked bitmap index 700, and the rank or ratio of the appearance frequency of the elements And rank setting 1402 for designating any one of the above.

  In the illustrated example, the bitmap index creation 1401 is set to a circle for the source IP address, the destination IP address, the protocol number, and the destination port number among the elements of the flow information 200, and the bitmap index Select to generate. For the source IP address and the destination IP address, the rank number R of the rank setting 1402 is set to “5”, the elements with the highest appearance frequency up to the 4th are stored in the bitmap index, and the 5th and below are the others It shows that it is summarized in 703. As for the protocol number, among the elements of the flow information 200, the elements up to the top 95% of the appearance frequencies are stored in the bitmap index, and the remaining elements having the appearance frequency of 5% are stored in the other 703. Indicates. The same applies to the destination port number, which indicates that the upper 90% of the appearance frequencies are stored in the bitmap index. These settings are input by a manager or the like using a mouse or a keyboard constituting the input device 6.

  The flow processing unit 106 stores the generation condition of the ranked bitmap index 700 set by the user interface 115 in the setting table 31 on the memory 3 and sets it in the setting table 31 when the search information generation unit 110 is executed. The ranked bitmap index 700 is generated with reference to the generated generation conditions.

  FIG. 12 is a flowchart illustrating an example of processing for generating the ranked bitmap index 700. This process is a program executed when the search information generating unit 110 of the flow processing unit 106 satisfies a predetermined condition. As described above, the search information generation unit 110 determines that the predetermined condition is satisfied when the predetermined period (time) is reached, or when the free space in the cache area 30 is exhausted. Start processing.

  First, in step 1001, the search information generation unit 110 starts the loop processing up to step 1004 for all the flow information 200 stored in the flow information storage unit 108 in the cache area 30 as a processing target.

  In step 1002, the search information generation unit 110 reads the setting table 31 on the memory 3, and among the fields 201 to 205 of the flow information 200 shown in FIG. 3, the bitmap index creation 1401 of the user interface 115 in FIG. Get the selected item. The search information generation unit 110 sets the acquired item as an element for which the ranked bitmap index 700 is to be created.

  In step 1003, the search information generation unit 110 reads one flow information 200 and extracts the values of the fields 201 to 205 of the creation target element acquired in step 1002.

  Next, in step 1004, 1 is added to the appearance frequency (number of appearances) of the field value extracted in step 1003 by the search information generation unit 110. That is, “1” is set to the value of the column 512 for each identifier of the flow information 200 corresponding to the column 510 storing the value of the bitmap index 501 in FIG.

  When the search information generation unit 110 obtains the appearance frequency of the element to be created of one flow information 200 in the loop processing of steps 1002 to 1004, the search information generation unit 110 returns to step 1001 and performs the same processing on the next flow information 200. Do. When the calculation of the appearance frequency of the value of each element for which the ranked bitmap index 700 is created for all the flow information 200 in the flow information storage unit 108 is completed, the process proceeds to step 1005.

  In step 1005, the search information generation unit 110 performs the loop processing of steps 1006 and 1007 for each field (element) for which the ranked bitmap index 700 is to be created. In step 1005, the bitmap indexes 501 and 502 shown in FIG. 6 are generated for each element for which the ranked bitmap index 700 is to be created.

  In step 1006, the search information generation unit 110 creates a bitmap index from the appearance frequency of the value obtained in step 1004 for each item of the fields 201 to 205. That is, in step 1004, the search information generation unit 110 creates the column 512 shown in FIG. 6 for each index (identifier) of the flow information 200, so that the flow information 200 for the bitmap index 501 for each element. Add column 512 for each identifier. By this process, the bitmap index 501 of the field (element) currently focused on is generated.

  Next, in step 1007, the search information generation unit 110 reads the rank number R from the setting table 31, and sorts the bitmap index 501A in the column direction in descending order of appearance frequency as shown in FIG. The bitmap index 501A up to the upper rank “rank number R” of the appearance frequency is held, and the values of the rows with the rank of the appearance frequency after the rank number R + 1 are put together in the other 703 rows, and the rank-added bitmap for each element An index 700 is generated.

  When the search information generating unit 110 generates the ranked bitmap index 700 for one field, the search information generating unit 110 returns to step 1005 and repeats the process of generating the ranked bitmap index 700 for the next field, and enters all the fields. Then, a ranked bitmap index 700 is generated.

  As a result of the processing described above, for each field for which the bitmap index creation 1401 is selected in the user interface 115 of FIG. 11 among the fields 201 to 205 of the flow information 200 shown in FIG. A ranked bitmap index 700 is generated. Although the case where the number of ranks R is designated is shown above, the case where the number of ranks is designated by a ratio from the top will be described later.

  When the creation of the ranked bitmap index 700 for every field (element) is completed, the search information generation unit 110 stores the bitmap index 700 of each element held on the memory 3 in step 1008. Write to 109. At this time, the search information generation unit 110 adds the index 211 to the flow information 200 of the flow information storage unit 108 in the cache area 30, and then collects the time stamp as one file as described above, This file is stored in the storage device 109 as flow information 200A.

  Further, the search information generation unit 110 generates index information 1200 and stores it in the storage device 109 in order to speed up the search process. As shown in FIG. 13, the index information 1200 includes a start time 1202 indicating the start time of the collection range of the flow information 200 in one entry 1201, a flow number 1203 indicating the number of flow information included in the flow information 200A, It includes the rank number R of the ranked bitmap indexes 700A and 700B corresponding to the flow information 200A and the index 1205 indicating the storage position of the bitmap indexes 700A and 700B.

  Through the above processing, when the flow information processing apparatus 101 collects the flow information 200 received from the router apparatus 102 and a predetermined condition is satisfied, the search information generating unit 110 uses the element (field) designated by the user interface 115. A ranked bitmap index 700 is generated every time. Then, as shown in FIG. 13, the search information generation unit 110 generates and stores flow information 200 </ b> A in which the flow information 200 is combined into one file and index information 1200 as an index of the ranked bitmap index 700. Store in device 109.

  Next, a search process using the ranked bitmap index 700 will be described.

  FIG. 14 shows an example of a flowchart of search processing using a ranked bitmap index. In this process, the query receiving unit 111 receives a query including a search condition, the query processing unit 112 notifies the search unit 113 of the search condition, and the search unit 113 executes the process.

  First, in step 801, when the search unit 113 receives a query from the server apparatus 103, the search unit 113 extracts a search condition from the query. Note that the search condition includes time, an element, and an element value.

  In step 802, the search unit 113 refers to the index information 1200 stored in the storage device 109 and identifies an entry that includes the search condition time within a predetermined time from the start time 1202. Then, the index 1205 is searched from the elements included in the search condition for the specified entry, and the ranked bitmap index 700 to be searched is read into the memory 3. Next, as shown in FIG. 15 to be described later, the search unit 113 searches for flow information that matches the search condition using the ranked bitmap index 700, and if the bit that matches the search condition is in a row within the rank number R. The identifier corresponding to the index of the flow information 200A is acquired. On the other hand, if the bit matching the search condition is “other”, the search result is stored as “other”. This process is performed using the ranked bitmap index 700 for each element of the flow information 200 included in the search condition.

  The search unit 113 repeats the processing in steps 803 to 807 for the number of identifiers acquired in step 802 (the number of flow information to be acquired).

  First, in step 804, the flow information 200A including the time included in the search condition is specified from the storage device 109, and the flow information 200A corresponding to the acquired identifier is read into the memory 3.

  In step 805, it is determined whether or not the flow information 200 </ b> A read into the memory 3 by the search unit 113 corresponds to “others” in the ranked bitmap index 700. If the read flow information 200A does not correspond to “others”, the process proceeds to the transmission process in step 807. On the other hand, if the result of the search of the ranked bitmap index 700 in Step 802 corresponds to “Other”, there is no guarantee that the target flow information 200A matches the search condition, so the confirmation process in Step 806 is performed. Do.

  In step 806, the search unit 113 searches the flow information 200A stored in the storage device 109 for the flow information 200A that matches the search condition by the conventional search process shown in FIG. If there is flow information 200A that matches the search condition, the process proceeds to step 807. If there is no flow information 200A that matches the search condition, the process returns to step 803 and the above processing is repeated for the next identifier.

  In step 807, the flow information 200A read in step 804 or step 806 is returned to the server apparatus 103 as a response to the query.

  As a result of the above search processing, if there is a bit that matches the search condition in the ranked bitmap index 700 read on the memory 3, the identifier in the flow information 200A can be obtained by acquiring the identifier, so step 804 is obtained. The processing time for acquiring the flow information 200A from the storage device 109 is not required to search all the flow information 200A as in the conventional example, so that the processing can be performed in a very short time. Then, only when the search result of the ranked bitmap index 700 is “other”, the flow information 200A in the storage device 109 is searched as in the conventional example. However, as described above, due to the characteristics of the flow information 200A, there are few opportunities for the search result of the ranked bitmap index 700 to be “other”, and therefore the search process as a whole is processed at a very high speed compared to the conventional example. Is possible.

  FIG. 15 shows a flowchart of search processing using the ranked bitmap index 700 shown in step 802 of FIG.

  In step 901, the search unit 113 first creates a calculation table corresponding to the search condition in the memory 3 in the same manner as the calculation table 503 shown in FIG. This calculation table corresponds to the rank number R of the ranked bitmap index 700 shown in FIG. 7 when the search condition is a single search condition for a single element. Part.

  Next, in step 902, the search unit 113 creates the other table on the memory 3. The other table counts the number of times that the search condition does not match the element value of the rank number R in order to manage the search result of the ranked bitmap index 700. Any table that can be stored is acceptable.

  Next, in Steps 903 to 906, the calculation using the ranked bitmap index 700 is repeated by the number of search condition parameters (for example, elements). In step 904, the search unit 113 extracts a bit string that matches the search condition parameter (a line that matches the value of the element hit in the column 710 in FIG. 10). In step 905, the extracted bit string is stored in the calculation table generated in step 901, and compared with the search condition parameter. The identifier corresponding to the value of the element that is the result of the comparison operation is stored in the calculation table.

  Next, in step 906, it is determined whether or not the line hit as the element value in the processing of step 904 is “other”. If “other”, the process proceeds to step 907, and the rank within the rank number R is determined. If the element is hit, the process returns to step 903 to search for the next parameter.

  In step 907, “1” is added to the other table created in step 902 and updated.

  In the present invention, the ratio other than “other” in the flow information 200 (ranked bitmap index 700) retrieved by the search unit 113 from the storage device 109 is “hit rate”, and the ratio of “other” is “miss rate”. Call. As the hit rate is higher, the overhead of reading the flow information 200A from the storage device 109 is reduced, and the performance of the entire flow information processing apparatus 101 is improved.

  As a result of performing the search using the ranked bitmap index 700 as many as the number of search condition parameters, the bit string that matches the search condition is stored in the calculation table, and the value of the element in the rank number R is used. The number of times of no hit is recorded in the other table. The search unit 113 obtains the hit rate by dividing the number of element values stored in the calculation table by the sum of the value of the element stored in the calculation table and the value of the other table. it can. In addition, the search unit 113 can obtain the miss rate by dividing the value of the other table by the sum of the number of element values stored in the calculation table and the value of the other table.

  Further, by recording whether or not the “other” line has been referenced, it is possible to know whether or not an identifier search from the storage device 109, which is an additional process, is necessary in step 904.

  FIG. 16 shows an example of a query input to the flow information processing apparatus 101, and FIG. 17 is an explanatory diagram showing an example of a search processing result for the query of FIG.

  FIG. 16 shows a query 1301 requested by the server apparatus 103 to the flow information processing apparatus 101. In the query 1301, a transmission source IP address and a transmission destination IP address are set as information (extraction information) requested by the server apparatus 103. The flow information 200A is designated as the extraction information read destination. As a retrieval condition (retrieval information) for the extracted information, a protocol number is specified as a start time, an end time, and an element value.

  The flow information processing apparatus 101 that has received the query 1301 performs the search processing of FIGS. 14 and 15 and outputs the result to the server apparatus 103 as a query output 1302 shown in FIG. The query output of FIG. 17 is a period of the search condition start time and end time, and the destination IP address and the source IP address are output for each identifier (flow ID) from the flow information 200A whose element value is protocol number = 6. Is done.

  In the examples of FIGS. 16 and 17, the element (protocol number) value (= 6) hits in the first row of the ranked bitmap index 700 of FIG. The search unit 113 acquires the identifier (= 1, 2, 4) of the hit line. The search unit 113 reads the extracted information from the index of the flow information 200A corresponding to the acquired identifier and responds to the server device 103.

  As described above, in the ranked bitmap index 700 read on the memory 3, if there is a bit that matches the search condition, the identifier can be acquired, and the index in the flow information 200A can be obtained from this identifier. Since it is not necessary to search all the flow information 200A as in the above-described conventional example, the processing time for acquiring the flow information 200A from the flow information can be performed in a very short time, and the flow information processing apparatus 101 can obtain a large amount of flow information. Even when the information 200A is managed, a high-speed search is possible.

  18 shows a case where “rank number” or “ranking” is determined from the appearance frequency of each element in the flow information 200 when the rank setting is designated by the ratio in the user interface 115 shown in FIG. is there.

  The “number of ranks” and “ranking” in the bitmap index 700 with rank may be performed every time the flow information 200 is received, and statistics on the appearance frequency of each element in the flow information 200 may be collected at a specific time or a specific number. You may do it.

  An example of determining “number of ranks” or “ranking” based on a preset ratio is shown below.

  FIG. 18 is a flowchart illustrating an example of processing for generating a ranked bitmap index 700. This process is a program executed when the search information generation unit 110 of the flow processing unit 106 satisfies a predetermined condition, as in FIG. As described above, the search information generating unit 110 determines that the predetermined condition is satisfied when the predetermined period (time) is reached, or when the free space in the cache area 30 is exhausted, as shown in FIG. Start processing.

  First, in step 1101, the search information generation unit 110 starts a loop process up to step 1104 for all the flow information 200 stored in the flow information storage unit 108 in the cache area 30 as a processing target.

  In step 1102, the search information generation unit 110 reads the setting table 31 on the memory 3, and among the fields 201 to 205 of the flow information 200 shown in FIG. 3, creates a bitmap index 1401 of the user interface 115 in FIG. 11. Is selected, and the item whose rank setting 1402 is specified by the ratio from the top is acquired. The search information generation unit 110 sets the acquired item as an element for which the ranked bitmap index 700 is to be created.

  In step 1103, the search information generation unit 110 reads one flow information 200 from the cache area 30 and extracts the values of the fields 201 to 205 of the creation target element acquired in step 1102.

  In step 1104, the search information generation unit 110 adds 1 to the appearance frequency (number of appearances) of the field value extracted in step 1103. That is, “1” is set to the value of the column 512 for each identifier of the flow information 200 corresponding to the column 510 storing the value of the bitmap index 501 in FIG.

  When the search information generation unit 110 obtains the appearance frequency of the creation target element of one flow information 200 in the loop processing of steps 1102 to 1004, the search information generation unit 110 returns to step 1101 and performs the same processing on the next flow information 200. Do. When the calculation of the appearance frequency of the value of each element for which the ranked bitmap index 700 is created for all the flow information 200 in the flow information storage unit 108 is completed, the process proceeds to step 1105.

  In step 1105, “1” is set as an initial value in a variable N indicating the rank order (ranking).

  In step 1106, the search information generation unit 110 sorts the values obtained in step 1104 for each item in the fields 201 to 205 in descending order of appearance frequency, and sets the variable N to the value of the field having the highest appearance rank. Then, similarly to step 1006 in FIG. 12, a bitmap index 501 of the field (element) currently focused on is generated.

  Next, in step 1107, the frequency of appearance of the variable N of the field value currently focused on is the number from 1 to N and the total number of the flow information 200 to be created. Find the ratio of the number of values up to the place.

  In step 1108, the ratio specified in the user interface 115 is compared with the ratio obtained in step 1107, and it is determined whether or not this ratio exceeds the specified ratio. If the ratio exceeds the specified ratio, the process proceeds to step 1109 to generate a ranked bitmap index 700. If the ratio is equal to or less than the specified ratio, the process proceeds to step 1110 and 1 is added to the variable N. Returning to step 1106, a bitmap index of the next rank N is created.

  In step 1109, since the ratio of the number of the field value variable N to 1 to N with respect to the total number of the flow information 200 to be created has reached the specified ratio, the variable N = 1 to N A ranked bitmap index 700 is generated for the element values of the rank number, and the element values of the rank after the variable N = N + 1 are collected in other rows 703 to complete the ranked bitmap index 700.

  As described above, when the rank setting 1402 is a ratio from the top in the user interface 115, the ratio of the value of the field value variable N to the number of 1 to N with respect to the total number of flow information 200 to be created The number of ranks R having a specified ratio is automatically determined. As a result, when the search process is performed, the probability that the search condition matches with the value of the element having a high appearance frequency is improved, and the chance of searching the index of the flow information 200A from the storage device 109 due to a search miss hit is reduced. Can do. Therefore, even when the flow information processing apparatus 101 is installed in a large-scale network and a large amount of flow information 200 is collected, the search processing speed is prevented from decreasing, and the load of the flow information processing apparatus 101 becomes excessive. Can be prevented.

  In the above embodiment, an example in which a router device is used as a device that transmits flow information has been described. However, any device that can transfer a packet and output the transfer result as flow information may be used. For example, L3 (Layer 3) Communication devices such as switches and load balancers can be used.

  As described above, the present invention can be applied to a management apparatus that is installed in a network system, collects flow information, and searches the collected flow information. In addition to storage and retrieval of flow information, it is possible to reduce the capacity of the storage device and speed up retrieval in all devices that process a large amount of log information every hour.

The block diagram of the network system provided with the flow information processing apparatus of this invention. 1 is a block diagram showing functional elements of a flow information processing apparatus and a network system according to the present invention. Explanatory drawing which shows an example of the flow information concerning this invention. Explanatory drawing which shows an example of the query transmitted from a server apparatus. The flowchart which shows an example of the search process by a prior art example. Explanatory drawing which shows the process which produces | generates a bitmap index from flow information. Explanatory drawing which shows an example of the table for a calculation produced | generated according to search conditions from the bitmap index. Explanatory drawing which shows an example of the table for a calculation produced | generated according to search conditions from the bitmap index. The flowchart which shows an example of the search process using a bitmap index. Explanatory drawing which shows the process of producing | generating a ranked bitmap index from a bitmap index. A screen image showing an example of a user interface. The flowchart which shows an example of the production | generation process of a bitmap index with a rank. Explanatory drawing which shows the process which produces | generates a bitmap index with a rank from flow information, and also produces | generates index information. The flowchart which shows an example of the search process using a bitmap index with a rank. The subroutine which shows an example of the search process of the ranked bitmap index performed at step 802 of FIG. Explanatory drawing which shows an example of the query input into the flow information processing apparatus. Explanatory drawing which shows an example of the response with respect to the query output from the flow information processing apparatus. The flowchart which shows the production | generation process of a ranked bitmap index when the rank setting of a user interface is a ratio.

Explanation of symbols

101 Flow processing device 102 Router device 103 Server device 104 Communication interface 105 Protocol processing unit 106 Flow processing unit 107 Flow information receiving unit 108 Flow information storage unit 109 Storage device 110 Search information generation unit 111 Query reception unit 112 Query processing unit 113 Search Unit 114 transmitting unit 115 user interface 1200 index information 200, 200A flow information 501 bitmap index 700 ranked bitmap index

Claims (10)

A receiver for receiving flow information or a search request;
A storage device for storing the received flow information;
A search unit for searching for flow information stored in the storage device based on the received search request;
A transmission unit for transmitting the search result of the search unit;
In a flow information processing apparatus comprising:
A flow information storage unit for temporarily storing the flow information received by the reception unit;
A search information generation unit that generates search information from the flow information stored in the flow information storage unit when a predetermined condition is satisfied,
The search unit
When the search request is accepted, the search information is searched with a search condition included in the search request, and the flow information is acquired from the search information that matches the search condition ,
The search information generating unit
A flow information processing apparatus that generates the search information for each item included in the flow information each time the predetermined condition is satisfied . The flow information processing apparatus according to claim 1,
The search information generating unit
A flow information processing apparatus that extracts the contents of flow information from items included in the flow information stored in the flow information storage unit and generates information including the contents of the extracted items as search information. The flow information processing apparatus according to claim 2,
The search information generating unit
Extract the contents of the items included in the flow information stored in the flow information storage unit,
Calculate the appearance frequency of the extracted content,
Index the flow information,
A flow information processing apparatus, wherein the contents of the items are set in the order of the calculated appearance frequencies, and bit information is generated as the search information from the relationship of the flow information indicated by the index. The flow information processing apparatus according to claim 3,
The search information generating unit
Among the bit information, for the contents of items after the order of appearance frequency set in advance, the contents of a plurality of items are set to the contents set in advance, and correspond to the contents of the items after the order set in advance. A flow information processing apparatus characterized in that a plurality of indexes are collected into the preset content. The flow information processing apparatus according to claim 3 ,
The search unit
When the search request is received, the bit information is searched with a search condition included in the search request,
An index is acquired from bit information that matches the search condition,
A flow information processing apparatus that acquires the flow information from the index . The flow information processing apparatus according to claim 4 ,
The search unit
When the search request is received, the bit information is searched with a search condition included in the search request,
A flow information processing apparatus that searches the storage device for flow information that matches the search condition when the content of the item that matches the search condition is the preset content . The flow information processing apparatus according to claim 1 ,
The search information generating unit
A flow information processing apparatus that determines whether a predetermined condition is satisfied each time a preset period elapses . The flow information processing apparatus according to claim 1,
The search information generating unit
A flow information processing apparatus for determining whether a predetermined condition is satisfied when a free capacity of the flow information storage unit reaches a predetermined value . The flow information processing apparatus according to claim 1,
The flow information processing apparatus is characterized in that a NetFlow or IPFIX protocol is used for the flow information. A communication device for transmitting flow information;
  A computer that sends a search request;
  A receiving unit that receives the flow information or search request; a storage device that stores the received flow information; a search unit that searches for flow information stored in the storage device based on the received search request; A flow information processing apparatus comprising: a transmission unit that transmits a search result of the search unit to the server device;
In a network system with
  The flow information processing apparatus includes:
  A flow information storage unit for temporarily storing the flow information received by the reception unit;
  A search information generation unit that generates search information from the flow information stored in the flow information storage unit when a predetermined condition is satisfied,
  The search unit
  When the search request is received, the search information is searched with a search condition included in the search request, and flow information is acquired from the search information that matches the search condition.
  The search information generating unit
  The network system, wherein the search information is generated for each item included in the flow information every time the predetermined condition is satisfied.