WO2024111088A1

WO2024111088A1 - Analysis device, analysis method, and analysis program

Info

Publication number: WO2024111088A1
Application number: PCT/JP2022/043380
Authority: WO
Inventors: 里美井上; 裕平林; 篤史須藤
Original assignee: 日本電信電話株式会社
Priority date: 2022-11-24
Filing date: 2022-11-24
Publication date: 2024-05-30

Abstract

This analysis device (10) stores definition information, which defines processing content, and association information, which is information that associates the definition information with processing source data. The analysis device (10) also has a receiving unit (14) and a processing unit (13). The receiving unit (14) receives key-value format data (e.g., xFlow packets) indicating information about the network as processing source data. The processing unit (13) specifies the processing content of the processing source data received by the receiving unit (14) on the basis of the association information, and processes the processing source data according to the processing content.

Description

Analytical device, analytical method, and analytical program

The present invention relates to an analysis device, an analysis method, and an analysis program.

　xFlow is known as a technology for network monitoring and traffic trend analysis.

xFlow is a method for collecting and analyzing traffic by transferring statistical information calculated from the header information of sampled packets, or the header portion itself (header sample).

In addition, a packet encapsulation technique is known that embeds a packet in the payload of another packet on a network and transfers the other packet.

In addition, a format conversion technique is known that enables extraction and analysis of raw packets and packets inside capsules (hereafter, inner packets) in header samples.

In addition, there is known technology for registering the header of an inner packet and the header of the packet outside the capsule (hereinafter, the outer packet) in a database in association with each other for encapsulated packets (see, for example, Patent Document 1).

Also, Elastic's Ingest Pipelines are known as a function for linking information, as they process documents containing information in a specified manner to format them.

JP 2020-174257 A

However, the problem with conventional technology is that it is not easy to change the processing content of the flow information.

Here, it is possible to process the flow information contained in xFlow appropriately depending on the analysis method. The flow information includes data in a key-value format such as 5-tuple that identifies the flow.

On the other hand, in order to add or change keys in processed flow information, it may be necessary to develop a new program for the processing. In such cases, it is not easy to change the processing content of the flow information.

In order to solve the above-mentioned problems and achieve the objective, the analysis device is characterized by having a storage unit that stores association information, which is information that associates definition information that defines the processing content with the source data, the definition information, a receiving unit that receives data in a key-value format that indicates information related to the network as the source data, and a processing unit that identifies the processing content of the source data received by the receiving unit based on the association information and processes the source data in accordance with the processing content.

According to the present invention, the processing content of flow information can be easily changed.

FIG. 1 is a diagram illustrating an example of the configuration of an analysis apparatus according to the first embodiment. FIG. 2 is a diagram for explaining a process for changing the processing contents. FIG. 3 is a diagram for explaining a process for changing the processing contents. FIG. 4 is a diagram showing a change in flow statistical information accompanying a change in processing content. FIG. 5 is a diagram showing an example of a process for adding a VPN user. FIG. 6 is a diagram for explaining a method for speeding up the process of changing the processing contents. FIG. 7 is a diagram for explaining a method for speeding up the process of changing the processing contents. FIG. 8 is a diagram for explaining a method for speeding up the process of changing the processing contents. FIG. 9 is a diagram for explaining a method for speeding up the process of changing the processing contents. FIG. 10 is a diagram illustrating optimization of association information. FIG. 11 is a diagram illustrating parallel processing. FIG. 12 is a flow chart illustrating the flow of processing performed by the analysis device. FIG. 13 is a flow chart illustrating the flow of processing performed by the analysis device. FIG. 14 illustrates an example of a computer that executes an analysis program.

Below, the embodiments of the analysis device, analysis method, and analysis program according to the present application are described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[Configuration of the first embodiment]
First, the configuration of an analysis device according to a first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an analysis device according to a first embodiment.

As shown in FIG. 1, the analysis device 10 is connected to an OpS (Operation System) 20, a conversion device 30, and a terminal device 40.

OpS20 provides the analysis device 10 with information that associates VPN users with outer information.

The conversion device 30 obtains xFlow packets from a network (e.g., a core network). In the network, tunnels, which are virtual communication paths, are configured. In addition, in the network, communication is performed using a VPN, and encapsulated packets are sent and received. In addition, a user who communicates using a VPN is called a VPN user.

For example, the conversion device 30 acquires xFlow packets via a network device that samples packets transmitted and received over a network.

The network device extracts the outer header (header of the outer packet) and inner header (header of the inner packet) of the sampled packet, and transfers the xFlow packet encapsulating each of the extracted headers to the conversion device 30.

Note that encapsulation here means embedding data in the payload section of the xFlow packet.

Furthermore, the network device transfers xFlow packets encapsulating statistical information about the sampled packets to the conversion device 30.

Here, the statistical information is calculated based on the inner header or the outer header. For example, the statistical information is the number of packets for each flow (inner flow or outer flow) based on the inner header or the outer header, the amount of communication data (example unit: Mbps), etc.

For example, statistical information calculated based on the Outer header is encapsulated in the xFlow packet along with the Outer header.

As a result, the conversion device 30 can obtain an xFlow packet that includes the Outer header and Inner header of the sampled packet, and an xFlow packet that includes statistical information.

The conversion device 30 converts the format of the acquired xFlow packets and transfers the xFlow packets obtained by the conversion to the analysis device 10.

For example, in the example of FIG. 1, the conversion device 30 extracts outer flow statistical information (outer statistics) from the acquired xFlow, and transfers the xFlow packet encapsulating the extracted statistical information to the analysis device 10.

Here, each device on the network forwards packets in the tunnel based on the packet's outer header.

The analysis device 10 then identifies information about the VPN of the flow based on the outer statistics contained in the xFlow packet received from the conversion device 30. For example, the analysis device 10 can identify the user of the VPN that is the source of the flow.

For example, the analysis device 10 can store a 5-tuple (source IP address, source port number, destination IP address, destination port number, protocol) in association with a VPN user, and identify the VPN user by comparing the stored 5-tuple with a 5-tuple included in the outer statistics. The analysis device 10 can obtain information such as the 5-tuple for identifying a VPN from the OpS 20.

The analysis device 10 processes the outer statistics appropriately (e.g., adds a key) based on the information contained in the xFlow packet and the identified information, and transmits them to the terminal device 40. The terminal device 40 is used by a user who analyzes the network. Note that the subject of processing by the analysis device 10 is not limited to outer statistics, and may be, for example, inner statistics (statistical information of the inner flow).

As shown in FIG. 1, the analysis device 10 includes a collection unit 11, an association information DB 12, a processing unit 13, a receiving unit 14, a flow statistics information DB 15, and a definition information DB 16.

The association information DB12, the flow statistics information DB15, and the definition information DB16 are stored in a memory unit provided in the analysis device 10. In other words, the memory unit stores the association information DB12, the flow statistics information DB15, and the definition information DB16.

The memory unit is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. The memory unit may also be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), flash memory, or non-volatile static random access memory (NVSRAM). The memory unit stores the operating system (OS) and various programs executed by the analysis device 10.

The collection unit 11, processing unit 13, and receiving unit 14 are realized by a control unit provided in the analysis device 10.

The control unit controls the entire analysis device 10. The control unit is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

The process of changing the processing content by the analysis device 10 will be explained using Figures 2 and 3, and each part of the analysis device 10 will be explained in detail. Figures 2 and 3 are diagrams explaining the process of changing the processing content.

First, the receiver 14 receives an xFlow packet from the conversion device 30 as source data. The xFlow packet received by the receiver 14 is flow statistical information in key-value format. In the example of FIG. 2, the source data "nw_data" contains "key1", "key2", and "key3" as keys.

The source data received by the receiver 14 is not limited to xFlow packets sent by the conversion device 30. The receiver 14 receives data in a key-value format indicating information related to the network as source data. For example, the receiver 14 may receive data provided by Telemetry (Reference: https://www.janog.gr.jp/wg/telemetry-wg/wp-content/uploads/2018/06/20180518_Cisco_Telemetry_WG01.pdf) as source data instead of xFlow packets.

The processing unit 13 acquires the original data from the receiving unit 14. Then, the processing unit 13 identifies the processing content of the original data received by the receiving unit 14 based on the association information, and processes the original data in accordance with the processing content.

The definition information DB16 stores definition information that defines the processing content. The association information DB12 stores association information that associates the definition information with the source data.

In the example of FIG. 2, the processing unit 13 first reads a processing content definition named "VPN user specific" from the definition information DB 16. Then, the processing unit 13 reads association information associated with the processing content definition "VPN user specific" from the association information DB 12.

Here, the association information read by the processing unit 13 shows "key2:yyy" as the condition. In this way, the association information shows the condition in key-value format. Note that "key2" is an example of a key, and "yyy" is an example of a value.

Then, the processing unit 13 checks the condition "key2:yyy" against the source data and determines whether the source data contains "key2:yyy". In the example of FIG. 2, since the source data contains "key2:yyy", the processing unit 13 processes the source data according to the processing content indicated by the processing content definition "VPN user identification".

The "input" in the processing content definition is information that identifies the data to be processed. Also, the "output" in the processing content definition is information that indicates the data to be processed or the data to be output as the result of processing.

As shown in Figure 3, the processing content definition "VPN user identification" defines that a new key "hoge" is added to the original data "nw_data" according to "add new_key:hoge" and output as "nw_data_result". As shown in Figure 3, the processing unit 13 outputs "nw_data_result" as the processing result.

Note that "add" is a command that means adding a key. In addition to "add," other commands include "delete," which deletes a key, "link," which adds a value, and "calculate," which performs calculations using a numerical value.

In the examples of Figures 2 and 3, if the source data includes "key2", the processing unit 13 adds "new_key" to the processing result. "new_key" corresponds to the first key whose addition is defined by the definition information. Also, "key2" corresponds to the second key included in the source data.

Figure 4 shows the change in flow statistics information associated with a change in processing content. The processing unit 13 stores the processed xFlow packets in the flow statistics information DB 15.

Flow statistical information DB15a is the flow statistical information DB15 before the processing described in Figures 2 and 3 is performed. Flow statistical information DB15b is the flow statistical information DB15 after the processing described in Figures 2 and 3 is performed. As shown in Figure 4, a record to which "new_key" has been added is registered in the flow statistical information DB15 by the processing unit 13.

Another example of processing by the processing unit 13 will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of processing when adding a VPN user.

As shown in FIG. 5, the xFlow packet received by the receiver 14 contains the flow statistics information "ExporterIP:10.0.0.1", "outer srcIP:10.1.0.1", "outer dstIP:10.2.0.1", "tunnelID:AAAAA", and "sessionID:aaa".

The processing unit 13 reads the processing content definition named "VPN user specific" from the definition information DB 16. The processing unit 13 then reads the association information associated with the processing content definition "VPN user specific" from the association information DB 12.

In the example of FIG. 5, the processing content definition includes information that defines the addition of a key that identifies a VPN user. In addition, the association information includes information that associates the processing content definition with a key that identifies the VPN.

In this case, the "outer srcIP", "outer dstIP", "tunnelID", and "sessionID" following "input nw_data" in the processing content definition represent the keys for the association conditions.

In other words, the processing unit 13 checks the value "10.1.0.1&10.2.0.1&AAAAA&aaa" separated by "&" in the association information against the values of the keys "outer srcIP", "outer dstIP", "tunnelID", and "sessionID" in the source data.

In the example of FIG. 5, the matching result shows that the condition indicated by the association information matches the source data, so the processing unit 13 executes the processing content "link" and performs processing to add the value "User A" to the key "vpn_user".

The association information stored in the association information DB 12 and the processing content definition stored in the definition information DB 16 can be set by the user as desired. This allows for flexible definition of the processing content. For example, the user can set the key and value in the range from "{" to "}" in the association information. In addition, for example, the user can set the "input" and "output" of the processing content definition and the processing content.

[Speed up by specifying conditions]
A method for speeding up the process of changing the processing contents is explained using Figures 6, 7, 8, and 9. Figures 6, 7, 8, and 9 are diagrams for explaining a method for speeding up the process of changing the processing contents.

As the number of key:values (key and value combinations) in the source data increases, it takes more time to match them with the associated information. Here, we will explain a method to shorten the time required for such matching by searching the source data for the presence or absence of the associated key, and only performing a match if the key is found in the source data.

First, as shown in Figure 6, it is assumed that the search target key (condition: key name) is set in advance in the processing content definition. In addition, it is assumed that the search target value (condition: value) and assigned value (equivalent to "{hoge}") are set in advance in the association information.

As shown in FIG. 7, first, the receiving unit 14 obtains the original data "nw_data" (e.g., flow statistical information) from the conversion device 30. "nw_data" includes "key1", "key2", and "key3" as keys.

Next, the processing unit 13 reads "input nw_data" and the search target key "key2" from the definition information DB 16. At this point, the processing unit 13 does not read any information (processing contents, etc.) other than the search target key. Note that "key2" is an example of a third key.

As shown in FIG. 8, the processing unit 13 searches for the key "key2" among the keys of the source data. If the search is a hit, i.e., if the key "key2" is among the keys of the source data, the processing unit 13 reads information other than the key to be searched for in the processing content definition from the definition information DB 16.

Then, the processing unit 13 reads the value of the search target and the assigned value associated with the processing content definition "VPN user identification" from the association information DB 12. Note that the assigned value is the value of the key added by the command "add".

As shown in FIG. 9, based on the read information, the processing unit 13 adds the key "new_key" to the source data "nw_data" in accordance with the processing content definition "VPN user identification", sets the value of the key to "hoge", and outputs the processing result "nw_data_result".

As shown in Figure 9, there may be two or more "outputs" for one processing operation.

[Improved speed through optimization of association information]
Furthermore, when multiple pieces of association information exist, the processing unit 13 refers to each piece of association information and performs matching. In contrast, by consolidating multiple pieces of association information into one as shown in Fig. 10, the speed of referring to and matching the association information by the processing unit 13 can be increased. Fig. 10 is a diagram for explaining optimization of association information.

In this case, the analysis device 10 extracts the necessary key information from the multiple pieces of association information that have been input, optimizes the association information by combining the information based on the key, and stores the association information in the association information DB 12.

In this way, the analysis device 10 can speed up processing by storing association information that consolidates multiple pieces of information that associate definition information that defines the processing content with the source data.

[High speed through parallel processing]
As shown in Fig. 11, the processing unit 13 can execute data processing processes in parallel for multiple source data. Fig. 11 is a diagram for explaining parallel processing. The data processing process includes a series of processes for each source data, such as referencing association information, matching, processing, and registering the processing results. For example, the processing unit 13 performs parallel processing using a multi-core processor. The number of parallel processes can be set in advance by the user.

In this way, the receiving unit 14 receives multiple xFlow packets as source data. At that time, the processing unit 13 identifies the processing content for each of the multiple source data received by the receiving unit 14, and executes in parallel the process of processing the source data in accordance with the processing content.

[Processing flow of the first embodiment]
The flow of processing by the analysis device will be described with reference to Fig. 12. Fig. 12 is a flow chart for explaining the flow of processing by the analysis device. As shown in Fig. 12, first, the analysis device 10 reads the target association information and source data from the processing content definition (step S101).

For example, when the analysis device 10 acquires the source data, it refers to the definition information DB 16, acquires association information corresponding to the processing content definition from the association information DB 12, and further reads the source data (e.g., "input") to be matched as indicated in the processing content definition.

The analysis device 10 then compares the key:value of the original data with that of the association information (step S102). If the key:value of the original data and the association information do not match (step S103, No), the analysis device 10 ends the process.

On the other hand, if the key:value of the original data matches the key:value of the association information (step S103, Yes), the analysis device 10 executes the processing content of the processing content definition (step S104). Then, the analysis device 10 outputs the result of the processing content (step S105).

The process flow of the analysis device when speeding up processing by specifying conditions is explained using FIG. 13. FIG. 13 is a flow chart that explains the process flow of the analysis device.

As shown in FIG. 13, first, the analysis device 10 reads the condition:key name and source data (e.g., "input") from the processing content definition (step S201).

The analysis device 10 then searches the source data, and if the data contains the condition:key name, it reads the association information of the processing content definition and the processing content (step S202). Furthermore, the analysis device 10 obtains the condition:value corresponding to the condition:key name from the association information (step S203).

Then, the analysis device 10 compares the conditions with the key:value of the source data, and if they match, executes the processing content of the processing content definition (step S204).

[Effects of the First Embodiment]
As described above, the analysis device 10 stores definition information and association information that associates definition information that defines processing contents with source data. The analysis device 10 also has a receiving unit 14 and a processing unit 13. The receiving unit 14 receives data in a key-value format that indicates information related to the network as source data. The processing unit 13 identifies the processing contents of the source data received by the receiving unit 14 based on the association information, and processes the source data in accordance with the processing contents. As a result, according to the first embodiment, the user can easily change the processing contents of the flow information by setting the definition information and association information.

The analysis device 10 stores association information that associates definition information that defines the addition of a first key with a second key included in the source data. Furthermore, when the source data includes a second key, the processing unit 13 adds the first key to the source data. This makes it possible to easily execute processing that adds a key.

The analysis device 10 stores definition information that defines the addition of a first key that identifies a VPN user, and association information that associates a second key that identifies the VPN. This makes it possible to easily perform processing to add a key that identifies a VPN user.

The analysis device 10 stores information identifying the third key and the processing content as definition information. Furthermore, when the source data received by the receiving unit 14 contains the third key, the processing unit 13 reads the processing content associated with the third key from the definition information and processes the source data according to the read processing content. In this case, the analysis device 10 only needs to read the processing content when the source data contains the third key, which speeds up processing.

The analysis device 10 stores association information that consolidates multiple pieces of information that associate definition information that defines the processing content with the source data. This reduces the number of times the association information is referenced, and speeds up the processing of the analysis device 10.

The receiving unit 14 receives multiple pieces of source data. The processing unit 13 identifies the processing content for each of the multiple pieces of source data received by the receiving unit 14, and executes in parallel the process of processing the source data according to the processing content. This makes it possible to speed up the processing of the multiple pieces of source data by the analysis device 10.

[System configuration, etc.]
In addition, each component of each device shown in the figure is functionally conceptual, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
In one embodiment, the analysis device 10 can be implemented by installing an analysis program that executes the above-mentioned analysis process as package software or online software on a desired computer. For example, the above-mentioned analysis program can be executed by an information processing device, causing the information processing device to function as the analysis device 10. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

The analysis device 10 can also be implemented as an analysis server device that provides services related to the above-mentioned analysis processing to a client, the client being a terminal device used by a user. For example, the analysis server device is implemented as a server device that provides an analysis service that takes as input key-value format data (e.g., xFlow packets) that indicates information about the network, and outputs the analysis results. In this case, the analysis server device may be implemented as a web server, or may be implemented as a cloud that provides services related to the above-mentioned analysis processing through outsourcing.

FIG. 14 is a diagram showing an example of a computer that executes an analysis program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process of the analysis device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the analysis device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Furthermore, the setting data used in the processing of the above-mentioned embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the processing of the above-mentioned embodiment.

The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

10 Analysis device 11 Collection unit 12 Association information DB
13 Processing unit 14 Receiving unit 15 Flow statistics information DB
16 Definition information DB
20 OpS
30 Conversion device 40 Terminal device

Claims

A storage unit that stores definition information that defines processing content and association information that associates the processing source data with each other, and the definition information;
A receiving unit that receives data in a key-value format indicating information related to a network as source data;
a processing unit that specifies processing details of the original data received by the receiving unit based on the association information, and processes the original data in accordance with the processing details;
An analytical device comprising:
the storage unit stores definition information defining the addition of a first key and association information associating a second key included in the original data;
The analysis device according to claim 1 , wherein the processing unit adds the first key to the original data when the original data includes the second key.
The analysis device according to claim 2, characterized in that the storage unit stores definition information that defines the addition of the first key that identifies a user of a VPN and association information that associates the second key that identifies the VPN.
the storage unit stores information for identifying a third key and a processing content as the definition information;
The analysis device according to claim 1, characterized in that, when the original data received by the receiving unit includes the third key, the processing unit reads processing content associated with the third key from the definition information, and processes the original data in accordance with the read processing content.
The analysis device according to claim 1, characterized in that the storage unit stores association information that consolidates multiple pieces of information that associate definition information that defines processing content with the source data.
The receiving unit receives a plurality of original data,
The analysis device according to claim 1 , characterized in that the processing unit identifies processing content for each of the plurality of original data received by the receiving unit, and performs a process in parallel to process the original data in accordance with the processing content.
An analytical method performed by an analytical device, comprising:
A receiving step of receiving data in a key-value format indicating information related to a network as source data;
a processing step of referring to a storage unit that stores association information, which is information that associates definition information that defines processing contents with source data, and the definition information, specifying processing contents of the source data received by the receiving step based on the association information, and processing the source data in accordance with the processing contents;
An analytical method comprising the steps of:
An analytical method performed by an analytical device, comprising:
A receiving step of receiving data in a key-value format indicating information related to a network as source data;
a processing step of referring to a storage unit that stores association information, which is information that associates definition information that defines processing contents with original data, and the definition information, specifying processing contents of the original data received in the receiving step based on the association information, and processing the original data in accordance with the processing contents;
An analysis program characterized by causing a computer to execute the above.