KR20220074638A - A method and apparatus for determining sampling point and sampling rate for multiple traffic analyzers using reinforcement learning on software-defined networks - Google Patents

Abstract

The present invention relates to a method and apparatus for determining sampling points and sampling rates for multiple traffic analyzers using reinforcement learning in an SDN supported network. An operating method of SDN using reinforcement learning according to an embodiment of the present invention comprises the steps of: (a) receiving a test result of sampled traffic for each sampling point from a traffic analyzer; and (b) determining at least one traffic analyzer, at least one sampling point, and sampling rate-related information based on the inspection result of the sampled traffic.

Description

A method and apparatus for determining sampling point and sampling rate for multiple traffic analyzers using reinforcement learning on software-defined networks}

The present invention relates to a software-defined networking (SDN) supported network, and more particularly, to a method and apparatus for determining a sampling point and a sampling rate for a multi-traffic analyzer using reinforcement learning in an SDN supported network.

Over the past few decades, the Internet has become an integral part of many people's lives, resulting in a massive increase in network traffic.

With the growth of Internet users and network traffic, the importance of cybersecurity in providing defense against multiple network attacks and anomalies such as denial of service attacks, man-in-the-middle attacks, and malware has steadily increased.

One of the serious threats to the Internet is advanced persistent threat (APT). APTs use a variety of attack techniques and operate silently, allowing attackers to remain undetected and control over a target system for extended periods of time.

In other words, these attacks are difficult to detect because they steal sensitive information from specific targeted facilities such as government agencies, corporations or the military through a covert and continuous attack process.

The purpose of an APT attack is not to paralyze the target's network, but to steal and seize the target's data and create a monetary demand for the release of the confiscated data.

Therefore, the attacker first breaks into the target network, installs the APT malware on the victim's computer, and transforms the computer into an abnormal state. This malware is used by attackers to remotely take control of compromised systems and steal sensitive data, such as confidential information, from the target over an extended period of time.

Accordingly, various studies are being conducted to detect network anomalies such as APT attacks, but improvement is insufficient.

[Patent Document 1] Korean Patent No. 10-2033169

The present invention was created to solve the above problems, and an object of the present invention is to provide a method and apparatus for determining a sampling point and a sampling rate for a multi-traffic analyzer using reinforcement learning in an SDN supported network.

Another object of the present invention is to provide a method and apparatus for determining a traffic analyzer, a sampling point and a sampling rate using a DDPG model.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the description below.

In order to achieve the above objects, an SDN operating method using reinforcement learning according to an embodiment of the present invention includes the steps of: (a) receiving a test result of sampled traffic for each sampling point from a traffic analyzer; and (b) determining at least one traffic analyzer, at least one sampling point, and sampling rate-related information based on the inspection result of the sampled traffic.

In an embodiment, the step (b) comprises: calculating a reward for the inspection result of the sampled traffic; and determining at least one traffic analyzer, at least one sampling point, and sampling rate related information by applying the reward to the reinforcement learning model.

In an embodiment, the step (b) may include determining a sampling policy by applying the reward to the reinforcement learning model; and determining, according to the sampling policy, the at least one traffic analyzer, at least one sampling point, and sampling rate related information for maximizing the reward.

In an embodiment, the traffic sampled by the sampling point may be forwarded to the traffic analyzer.

In an embodiment, the reinforcement learning model may include a Deep Deterministic Policy Gradient (DDPG) model.

In an embodiment, the step (b) may include: determining state space information about the at least one traffic analyzer and the at least one sampling point based on the inspection result of the sampled traffic; and determining action space information for the at least one traffic analyzer and the at least one sampling point and sampling rate related information based on the state space information.

In an embodiment, the step (b) includes, when the action space information is executed at a first time step, from the state space information for the first time step to the relative spatial information for the second time step. determining transition probability information for a change of ; determining a reward according to a reinforcement learning model based on the conversion probability information; and determining at least one traffic analyzer, at least one sampling point, and sampling rate related information by applying the reward to the reinforcement learning model.

In an embodiment, the SDN device may include: a communication unit configured to receive an inspection result of sampled traffic for each sampling point from a traffic analyzer; and a control unit configured to determine at least one traffic analyzer, at least one sampling point, and sampling rate related information, based on a test result of the sampled traffic.

In an embodiment, the control unit calculates a reward for the inspection result of the sampled traffic, and applies the reward to a reinforcement learning model to at least one traffic analyzer, at least one sampling point, and sampling rate related information can be decided

In an embodiment, the control unit determines a sampling policy by applying the reward to the reinforcement learning model, and according to the sampling policy, the at least one traffic analyzer for maximizing the reward, at least one sampling point, and sampling Rate-related information may be determined.

In an embodiment, the traffic sampled by the sampling point may be forwarded to the traffic analyzer.

In an embodiment, the reinforcement learning model may include a Deep Deterministic Policy Gradient (DDPG) model.

In an embodiment, the control unit determines state space information for the at least one traffic analyzer and the at least one sampling point based on a test result of the sampled traffic, and based on the state space information Thus, action space information for the at least one traffic analyzer and the at least one sampling point and sampling rate related information may be determined.

In an embodiment, when the action space information is executed at a first time step, the controller is configured to change from state space information for the first time step to relative spatial information for a second time step. to determine transition probability information for the service, determine a reward according to the reinforcement learning model based on the transition probability information, and apply the reward to the reinforcement learning model to include at least one traffic analyzer, at least one sampling point, and Sampling rate related information may be determined.

Specific details for achieving the above objects will become clear with reference to the embodiments to be described in detail below in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but may be configured in various different forms, and those of ordinary skill in the art to which the present invention pertains ( Hereinafter, "a person skilled in the art") is provided to fully inform the scope of the invention.

According to an embodiment of the present invention, by determining a traffic analyzer, a sampling point, and a sampling rate using a DDPG model, the probability of sampling a malicious flow during traffic sampling can be increased, and processing overhead of SDN can be reduced. .

Effects of the present invention are not limited to the above-described effects, and potential effects expected by the technical features of the present invention will be clearly understood from the following description.

1 is a diagram illustrating an SDN support network according to an embodiment of the present invention.
2 is a diagram illustrating a DDPG model according to an embodiment of the present invention.
3A is a diagram illustrating a reward performance graph according to an embodiment of the present invention.
3B is a diagram illustrating a probability graph of sampling a malicious flow according to an embodiment of the present invention.
3C is a diagram illustrating a load balancing performance graph of multiple traffic analyzers according to an embodiment of the present invention.
3D is a diagram illustrating a processing overhead performance graph of SDN according to an embodiment of the present invention.
4 is a diagram illustrating a method of operating an SDN controller according to an embodiment of the present invention.
5 is a diagram illustrating a functional configuration of an SDN controller according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

Various features of the invention disclosed in the claims may be better understood upon consideration of the drawings and detailed description. The apparatus, methods, preparations, and various embodiments disclosed herein are provided for purposes of illustration. The disclosed structural and functional features are intended to enable those skilled in the art to specifically practice the various embodiments, and are not intended to limit the scope of the invention. The disclosed terms and sentences are for the purpose of easy-to-understand descriptions of various features of the disclosed invention, and are not intended to limit the scope of the invention.

In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, a method and apparatus for determining a sampling point and a sampling rate for a multi-traffic analyzer using reinforcement learning in an SDN supported network according to an embodiment of the present invention will be described.

1 is a diagram illustrating an SDN support network 100 according to an embodiment of the present invention.

Referring to FIG. 1 , the SDN support network 100 may include an SDN controller 110 , at least one traffic analyzer 120 , and at least one switch 130 .

Since the traffic analyzer 130 such as an Intrusion Detection System (IDS) has a limited capacity to perform deep packet inspection (DPI) on a fixed location and a huge amount of network traffic, it samples network traffic and analyzes the traffic analyzer. Adjusting the sampled traffic to 130 can be important for cyber security.

In an SDN-enabled network 100 that provides flexibility, resiliency and programmability by separating the control plane and data plane from the network ( SDN controller 110 The captured and sampled traffic can be directed to any destination, such as the traffic analyzer 120 .

For example, the switch 130 may be implemented as an OF-enabled switch (OF-enabled switch).

However, because of the potential loss of useful information in discarded traffic, it is necessary to determine sampling points and sampling rates for traffic monitoring. Here, the sampling point may mean at least one switch for performing sampling of traffic among the plurality of switches 130 .

Therefore, according to the present invention, it is possible to determine the sampling point and sampling rate for the multi-traffic analyzer 120 in the SDN support network 100 based on a DRL (deep reinforcement learning) algorithm.

The DRL algorithm according to the present invention may learn a sampling resource allocation policy under uncertainty of flow distribution according to the inspection result of the sampled traffic obtained by the multi-traffic analyzer 120 .

The SDN controller 110 may provide flexibility, flexibility and programmability by separating the control plane and data plane.

The problem of determining the sampling point and sampling rate for the multiple traffic analyzer 120 in the SDN support network 100 may be formulated as a Markov decision process (MDP). MDP may consider the occurrence of network flow uncertainty by maximizing a long-term objective function in order to improve the monitoring performance of the network flow. In this case, the current decision may be calculated in consideration of future rewards.

Then, a DDPG-based DRL algorithm that automatically determines a sampling rate according to the inspection result obtained from the multi-traffic analyzer 120 may be used.

In an embodiment, a sampling approach for each sampling point constituting a sampling rate for each sampling point rather than for each flow may be used. In this case, the reason for selecting the sampling method for each sampling point may be that scalability is higher.

As the size of the network increases, in general, the rate at which the size of the number of flows increases can be much faster than the rate of the number of sampling points.

In addition, when sampling by selecting a specific switch 130, that is, a specific sampling point, instead of sampling from all switches 130, the processing overhead of the SDN controller 110 for updating the flow table of all switches 130 can be reduced. can

Since it is impossible to know which malicious flows will occur in a real network environment, it is necessary to determine where to sample in order to sample as many flows as possible.

In one embodiment, the sampling process may sample the traffic at each sampling point and pass the sampled traffic to the multi-traffic analyzer 120 .

In one embodiment, through the inspection process, the traffic analyzer 130 performs an inspection on the forwarded sampled traffic, and the inspection includes information about the number of non-redundant sampled flows. The result may be transmitted to the SDN controller 110 .

In an embodiment, through the reconfiguration process, the SDN controller 110 may adjust the sampling rate of the sampling point according to the sampling policy determined using the DRL model of the SDN controller 110 .

The whole process can be performed periodically over time.

및 D=

로 표시될 수 있다. In terms of the system model, when there are n switches 130 and l traffic analyzers 120 in the SDN supported network 100, the set of switches 130 and traffic analyzers 120 is 0=

and D=

can be displayed as

The SDN controller 110 may allocate sampling rates by periodically selecting m sampling points among the n switches 130 . In this case, T may represent a sampling resource allocation period.

In one embodiment, the processing capacity of the traffic analyzer (d _k ) 130 is expressed as c _k (packets/T), l The total capacity of the traffic analyzer 130 can be calculated as in Equation 1 below .

동안 네트워크의 총 플로우 집합이고,

는 T 동안 네트워크의 총 비 중복 샘플링 플로우 집합을 나타낸다. 이 경우,

는 네트워크 플로우의 불확실성 특성 때문에 동적으로 변경될 수 있다. 또한,

는 샘플링 자원 할당 정책에 의해 주기적으로 결정될 수 있다. In one embodiment,

is the total set of flows in the network during

denotes the total set of non-redundant sampling flows of the network during T. in this case,

can be dynamically changed due to the uncertainty nature of the network flow. In addition,

may be periodically determined by the sampling resource allocation policy.

로 나타낼 수 있다. In one embodiment, to determine the sampling point and sampling rate, an ordered m-tuple of sampling points during T is taken as P=(p ₁ , p ₂ ,..., p _m ) and

can be expressed as

When the sampling point is determined, the volume of traffic sampled at each sampling point may be determined according to the order of the m-tuple.

.The sampling rate difference of each sampling point may be reduced by r. where r is a positive small number between 0 and 1. in other words,

.

The number of sampling points m and the sampling rate reduction ratio r of the sampling points may vary depending on network conditions.

로 나타낼 수 있다. 여기서, x_j는 샘플링 포인트 k에서의 샘플링 레이트를 나타내고 x_j의 단위는 T 당 패킷일 수 있다.Then the sampling rate vector is x=

can be expressed as Here, x _j represents a sampling rate at the sampling point k, and the unit of x _j may be a packet per T.

Since the total amount of sampled traffic must not exceed the total processing capacity of the traffic analyzer 130, the constraint on the relationship between C and x can be expressed as Equation 2 below.

Also, in an embodiment, the amount of each sampling rate x _j may be expressed as in Equation 3 below.

는 m개의 샘플링 포인트가 l개의 그룹으로 클러스터링되는 트래픽 분석기(120)를 선택하기 위한 그룹 세트를 나타낼 수 있다. In one embodiment, according to the traffic analyzer selection G, it is possible to obtain the resource utilization rate _uk of the traffic analyzer _ck during T as shown in Equation 4 below. where G=

may represent a group set for selecting the traffic analyzer 120 in which m sampling points are clustered into l groups.

where w _j is the sampling cost of p _j during T, in this case, w _j may be equal to x _{j if x j is} less than or equal to the data rate of p _j during T. Otherwise w _j may be equal to the data rate of p _j during T.

After determining the sampling point and traffic analyzer selection, it is possible to steer the sampled traffic to the selected traffic analyzer 130 , which is an overhead of the SDN support network 100 .

는 트래픽 분석기의 스티어링 오버헤드(steering overhead)의 집합이며, v_k는 하기 <수학식 5>와 같이 나타낼 수 있다.V=

is a set of steering overhead of the traffic analyzer, and v _k may be expressed as in Equation 5 below.

는 j번째 샘플링 포인트에서 트래픽 분석기 d_k까지의 홉의 수를 나타낼 수 있다. here,

may represent the number of hops from the j-th sampling point to the traffic analyzer d _k .

SDN-enabled network 100 operates on a discrete-time basis based on time steps, and uses reinforcement learning over continuous time steps with uncertainty in future information, resulting in a better sampling policy This can be learned gradually.

The length of a time step is in units of 1, the time it takes for a single state to change. According to the present invention, by setting the time step to the sampling period P, two uncertainties can be taken into account: the occurrence of a network flow in future information and the data rate of the flow. There is information about these two uncertainties from the initial state to the present state (t).

Therefore, the system can be modeled to find the optimal policy using the Markov Decision Process (MDP). In this case, the DDPG model may be used as an optimization method.

In one embodiment, in order to apply the DDPG model to the SDN supported network 100 having the multiple traffic analyzer 120 and the SDN controller 110 , a state space, an action space of the system by assuming uncertainty ), transition probabilities and reward functions can be formulated as MDP.

In one embodiment, in terms of state space and action space, the state space of the SDN support network 100 may consist of a currently allocated sampling resource state and a selected traffic analyzer state.

The state s _t represents the state of the time step t, and may be represented by the traffic analyzer 120 selected for the pair set and p _j arranged for m sampling points as shown in Equation 6 below.

Here, state s _t may represent an m-order selected sampling point with a traffic analyzer d _k for each sampling point p _j .

Further, in one embodiment, the action space of the SDN support network 100 may be composed of three actions (action).

를 결정하는 행동이고, 두 번째는 각 샘플링 포인트를 할당할 트래픽 분석기를 결정하는 행동이고, 세 번째는 샘플링 레이트가 얼마나 감소하는지(r)를 결정하는 행동입니다. 시간 스텝 t에서의 일련의 행동 a_t는 하기 <수학식 7>과 같이 나타낼 수 있다.The first is the sampling point.

The second action determines the traffic analyzer to assign each sampling point to, and the third action determines how much the sampling rate decreases (r). A series of actions a _t at time step t can be expressed as in Equation 7 below.

Here, the action a _t represents selecting one of the n switches 130 using one of the traffic analyzers 120 for the selected switch 130 and selecting the sampling rate reduction ratio r at time step t. can

If the action is performed in the state s _t , the switch 130 selected by the action a _t may be selected as the first element of the next state s _t+1 .

Since the sampling rate vector x is determined according to the order of the sampling points, the switch 130 selected in the action a _t may have the largest sampling rate of the sampling points. Also, the sampling rate of the sampling point may be changed according to the selected sampling reduction rate.

로 나타낼 수 있다.In one embodiment, in terms of a transition probability and a reward function, when the action a _t is performed, the transition probability of the SDN support network 100 from the state s _t to the state s _t+1 is

can be expressed as

는 정책

에 의해 정의된 상태가 s_t일 때 행동을 선택할 확률에 의해 결정될 수 있다.Since the sampling rate of each switch 130 can be dynamically changed using the SDN controller 110, the switching probability

is the policy

It can be determined by the probability of choosing an action when the state defined by s _t .

Comprehensive rewards of SDN-enabled network 100 include (1) fair-share flow monitoring, (2) load balancing to multiple traffic analyzers 120, and (3) sampled traffic. It can be designed to achieve three main goals: reducing sampled traffic steering overhead with multiple traffic analyzers.

In one embodiment, in order to achieve process sharing flow monitoring, the flow process sharing reward r ^f may be expressed as in Equation 8 below.

동안 네트워크의 총 플로우 집합이고,

는 T 동안 네트워크의 총 비 중복 샘플링 플로우 집합을 나타낸다. 이 경우,

는 네트워크 플로우의 불확실성 특성 때문에 동적으로 변경될 수 있다. 또한,

는 샘플링 자원 할당 정책에 의해 주기적으로 결정될 수 있다. here,

is the total set of flows in the network during

denotes the total set of non-redundant sampling flows of the network during T. in this case,

can be dynamically changed due to the uncertainty nature of the network flow. In addition,

may be periodically determined by the sampling resource allocation policy.

In an embodiment, the load balancing reward r ^u of the traffic analyzer 120 is defined for balanced utilization of the traffic analyzer 120 , and the fairness index r ^u may be expressed as in Equation 9 below.

In an embodiment, a normalized average steering overhead penalty r ^v is defined as in Equation 10 below to consider the sampled traffic steering overhead.

은 다중 트래픽 분석기(120)를 동등하게 활용하여 샘플링된 트래픽 스티어링 오버헤드를 고려하면서 네트워크 플로우를 공정하게 샘플링하는지 여부를 나타내는 비용 값을 반환하는 리워드 함수를 나타낼 수 있다. 상태 s_t에서 행동 a_t이 수행되면 리워드 함수는 하기 <수학식 11>과 같이 나타낼 수 있다. In one embodiment,

may represent a reward function that returns a cost value indicating whether to fairly sample network flows while equally utilizing multiple traffic analyzers 120 to account for sampled traffic steering overhead. When the action a _t is performed in the state s _t , the reward function can be expressed as in Equation 11 below.

에 따라 예상되는 총 할인 리워드(total expected discounted reward)을 하기 <수학식 12>와 같이 나타낼 수 있다. Here, the range of R(s _t , a _t ) is [-1, 1], indicating the sampling performance of the SDN support network 100 . A given policy to take into account the impact of current actions on future rewards

The total expected discounted reward can be expressed as in Equation 12 below.

≤ 1은 다음 시간 스텝 t+1에서 무한 시간 스텝까지 미래 리워드의 중요성을 결정하는 할인 계수를 나타낸다.where 0 ≤

≤ 1 represents the discount coefficient that determines the importance of the future reward from the next time step t+1 to an infinite time step.

는 시간 스텝 t에서의 순간 리워드와 다음 시간 스텝 t+1에서 할인된 리워드의 합으로 정의될 수 있다. 예를 들어,

=0은 SDN 지원 네트워크(100)가 현재 리워드만 고려함을 의미하고,

=1은 시스템이 현재 리워드와 동일한 가중치로 무한 시간 스텝에서 장기 리워드를 고려함을 의미할 수 있다.in other words,

can be defined as the sum of the instantaneous reward at time step t and the discounted reward at the next time step t+1. for example,

=0 means that the SDN support network 100 only considers the current reward,

=1 may mean that the system considers the long-term reward in an infinite time step with the same weight as the current reward.

를 증가시키는 행동 선택 정책(action-selection policy)을 결정할 수 있다. That is, to increase the sampling performance of the SDN support network 100, the total expected discounted reward (total expected discounted reward)

It is possible to determine an action-selection policy that increases .

2 is a diagram illustrating a DDPG model according to an embodiment of the present invention.

Referring to FIG. 2 , a reinforcement learning (RL) algorithm may determine a policy defined as a series of actions that maximize rewards to be accumulated in the future.

In one embodiment, the optimal value function and optimal policy of MDP may be determined through Bellman's expectation equation and Bellman's optimal equation. Bellman equations can be basically solved through dynamic programming advanced with SARSA and Q-learning.

Q-learning, which is a representative RL algorithm, can be used because of the model-free property that operates without prior knowledge of the system's future reward or conversion potential, and the out-of-policy method of learning about the optimal policy while following the behavioral policy.

Therefore, Q-learning can be used for real-time system operation considering the uncertainty of future information. In real applications, namely, network environment management and operation, RL algorithms such as Q-learning may face complexity problems due to the enormous space and behavior of MDPs.

A combination of RL and deep learning, recently known as Deep Reinforcement Learning (DRL), can be used to solve the curse of dimensionality problems caused by Q-learning-based table value updates.

The purpose of DRL may be to learn the optimal policy by utilizing a multilayer perceptron (MLP)-based nonlinear function approximator that represents the probability distribution of the DRL agent action strategy that can maximize the expected long-term reward.

Deep Q-network (DQN), a common DRL algorithm, can be widely used to solve complex state space problems in Q-learning.

However, since DQN has a discrete action space, it may be difficult to apply in an environment that requires control over a continuous action space.

To solve this problem, as shown in FIG. 2 , a deep deterministic policy gradient (DDPG) model using an actor-critic technique based on a deterministic policy gradient (DPG) algorithm may be used.

A critical network 220 that updates an action-value function to maximize long-term reward and an actor network that determines a policy according to the crit model ( 210) may be used.

DPG can determine the optimal deterministic policy using a policy gradient method that optimizes a parameterized policy for nonlinear function approximations such as MLP through a gradient escalation algorithm.

The DDPG model 200 is a model-free, off-policy, actor-critic algorithm for solving MDP, and a huge state space and action space may exist.

In one embodiment, a sampling policy update algorithm using the DDPG model 200 may be used.

In order to find an action selection policy that increases the total expected discount reward defined in Equation (12) above, the DDPG model 200 having model-free, off-policy, and actor-critic characteristics may be used.

In an embodiment, it is possible to determine an optimal behavior selection policy to increase sampling accuracy and reduce sampled traffic steering overhead by balanced resource utilization of the multi-traffic analyzer 120 .

에 따라 상태 s_t에서 행동 a_t를 취할 때, 총 예상 할인 리워드를 반환하는 행동-가치 함수(action-value function) Q(s_t, a_t)를 하기 <수학식 13>과 같이 나타낼 수 있다. A given policy, denoted by Q(st, at), to achieve the goal

When an action a _t is taken in the state _{s t according} to .

The optimal behavior-value function Q ^* (s, a) defined as follows can be expressed as in Equation 14 below for approximation.

이 사용될 수 있다. Q-learning, an algorithm outside the commonly used policy, can be approximated using both behavioral and target policies. For the behavior policy, the epsilon greedy method, i.e.,

this can be used

However, in the continuous action space, it may be difficult to find the optimal action in the action-value function because it is necessary to explore the exponentially increasing complexity according to all Q values and states and action spaces.

In one embodiment, in terms of the DDPG model for sampling resource allocation, in order to process the continuous action space of the DRL, the DDPG model 200 uses two neural networks as shown in FIG. 2 to obtain an optimal action-value function. can be approximated.

One may be a crit network 220 that approximates a behavior-value function. The input may be action and observation, and the output may be the value of a state-action pair, that is, Q(s _t , a _t ).

Another neural network is used to approximate a policy function called the actor network 210 , where the input may be an observation value and the output may be an action value.

및 크리틱 네트워크(220)를 위한

를 사용할 수 있다. The DDPG model 200 is a method for the actor network 210 to parameterize the non-linear function approximation.

and for the crit network 220 .

can be used

및

는 정책 기울기(policy gradient) 및 손실 함수(loss function)에 따라 반복적으로 업데이트될 수 있다.

and

may be iteratively updated according to a policy gradient and a loss function.

는 손실을 최소화하기 위해

방향으로 업데이트하여 하기 <수학식 15>와 같이 최적화될 수 있다.crit network

to minimize the loss

direction and can be optimized as in Equation 15 below.

이다. here,

to be.

방향으로 정책 기울기 방법의 목적 함수인 업데이트 J(

)를 통해 액터 네트워크

는 하기 <수학식 16>과 같이 최적화될 수 있다.In one embodiment,

Update J(

) via Actor Network

can be optimized as in Equation 16 below.

및

는 각각 하기 <수학식 17> 및 <수학식 18>과 같이 업데이트될 수 있다. To smoothly update the target network

and

may be updated as in <Equation 17> and <Equation 18>, respectively.

는 업데이트된 크리틱 네트워크를 나타낸다. here,

represents the updated critique network.

는 업데이트된 액터 네트워크를 나타낸다. 또한,

및

는 1보다 작은 양(positive)의 작은 숫자를 나타낸다. here,

represents the updated actor network. In addition,

and

represents a positive small number less than 1.

In an embodiment, a DDPG model-based sampling policy update algorithm may be represented as shown in Table 1 below.

and actor network

with weight

and

3: Set target network

and

with weights

and

4: Initialize experience replay buffer B
5: Set initial state s₀ according to the initial sampling policy
6: for each time step do
7: Select action

following the parameter noise for exploration
8: Take action at and observe R(s_t, a_t), s_t+1
9: Store transition {s_t, a_t, R(s_t, a_t), s_t+1} in B
10: Randomly Sample a batch of N transitions{s_i, a_i, R(s_i, a_i), s_i+1} from B
11: Set

12: Update critic

by minimizing loss in (15)
13: Update actor

using the sampled policy gradient in (16)
14: Update the targets softly in (17) and (18)
15: end for1: // Initialization
2: Set critic

and actor network

with weight

and

3: Set target network

and

with weights

and

4: Initialize experience replay buffer B
5: Set initial state s ₀ according to the initial sampling policy
6: for each time step do
7: Select action

following the parameter noise for exploration
8: Take action at and observe R(s _t , a _t ), s _t+1
9: Store transition {s _t , a _t , R(s _t , a _t ), s _t+1 } in B
10: Randomly Sample a batch of N transitions{s _i , a _i , R(s _i , a _i ), s _i+1 } from B
11: Set

12: Update critic

by minimizing loss in (15)
13: Update actor

using the sampled policy gradient in (16)
14: Update the targets softly in (17) and (18)
15: end for

3A is a diagram illustrating a reward performance graph according to an embodiment of the present invention.

Referring to FIG. 3A , a fat-tree topology in which 10 switches 130 and the number of flows are set to 500 may be considered, and the data rate of the flows may be set to 1 to 20 Mbps. In order to take into account the uncertainty of the network flow, the routing and data rate of the flow can be changed randomly, and 2% of the total flows are malicious flows.

The number of traffic analyzers 120 may be set to three with the same processing capacity of 1 Gbps. In the simulation, the sampling algorithm according to the present invention can operate in a discrete time manner whenever the time step is set to 1 second.

The initial state s0 can be arbitrarily chosen in the state space S. The performance of the sampling algorithm according to the present invention can be compared with that of the conventional random sampling and flow betweenness centrality (FBC) sampling methods.

Conventional random sampling randomly selects a sampling point, a sampling rate, and a traffic analyzer at each time step.

The conventional FBC sampling determines the sampling rate of the sampling switch according to rate proportional sampling, which selects a sampling point among switches based on the centrality measurement of graph theory and samples traffic proportional to the data rate of each flow.

The conventional FBC sampling performs greedy selection according to a sampling rate of a sampling point selected for traffic analyzer selection. That is, a sampling switch with a high sampling rate selects a traffic analyzer with many available resources.

According to the present invention, the number of sampling points is set to 5, and all sampling methods can perform sampling at each iteration under the condition that the aggregated amount of sampled traffic is equal to the IDS capacity.

Thus, the total rate of sampled data traffic can be kept below the IDS capacity.

Referring to FIG. 3A , rewards according to the number of time steps can be checked. The DDPG model 200 can be used to evaluate how the sampling algorithm according to the present invention improves performance in terms of rewards.

3B is a diagram illustrating a probability graph of sampling a malicious flow according to an embodiment of the present invention. 3C is a diagram illustrating a load balancing performance graph of multiple traffic analyzers according to an embodiment of the present invention. 3D is a diagram illustrating a processing overhead performance graph of SDN according to an embodiment of the present invention.

가 증가함에 따라 제안된 방법이 다른 방법보다 악의적인 플로우를 샘플링하지 못할 확률이 낮음을 보여준다.Referring to FIG. 3B , the ratio of switches excluded when routing malicious flows

It shows that the proposed method has a lower probability of failing to sample a malicious flow than other methods as .

가 0.1인 경우 악의적인 플로우들을 중심성 측정 값 상위 10%의 스위치를 제외한 나머지 스위치들을 지나도록 라우팅한다. 즉 제안된 방법은 높은 중심성 측정 값을 갖지 않는 스위치들을 지나는 악의적인 플로우들에 대한 모니터링 성능이 다른 방법보다 높음을 보여준다.for example,

If is 0.1, malicious flows are routed through the switches except for the switches with the top 10% of the centrality measurement value. That is, the proposed method shows that the monitoring performance for malicious flows passing through switches that do not have a high centrality measurement value is higher than that of other methods.

3c and 3d show that the proposed method can reduce the overhead of steering the sampled traffic to multiple traffic analyzers compared to other methods when the number of traffic analyzers is two or more, while maintaining a balanced load for multiple traffic analyzers. show

It can be seen that the algorithm using the DDPG model 200 according to the present invention increases the reward more than the conventional method as the time step increases. This may be because the algorithm according to the present invention can select a better behavior when it experiences repetition.

According to the present invention, network traffic monitoring can play an important role in cyber security to detect network anomalies. It is important to sample the traffic so that it can be selectively monitored, as monitoring massive amounts of total network traffic is not possible due to the limited capacity of the traffic analyzer. Using the SDN controller 110 may enable network traffic sampling and adjustment for multiple traffic analyzers 120 such as IDS performing DPI.

According to the present invention, it is possible to determine the sampling point and the sampling rate of the multi-traffic analyzer 120 under the network flow uncertainty. The sampling point and rate determination for the multi-traffic analyzer 120 may be formulated as a discrete time MDP where uncertainty is a dynamic characteristic of network flows.

Using the available real-time traffic analyzer monitoring results, a deep reinforcement learning approach can be used.

Algorithms according to the present invention may correct for network flow uncertainty in order to make sampling rate decisions.

4 is a diagram illustrating an operation method of the SDN controller 110 according to an embodiment of the present invention.

Referring to FIG. 4 , step S401 is a step of receiving a test result of sampled traffic for each sampling point 130 from the traffic analyzer 120 .

In one embodiment, traffic sampled by each sampling point 130 may be passed to a traffic analyzer 120 .

Step S403 is a step of determining at least one traffic analyzer 120 , at least one sampling point 130 , and sampling rate related information, based on the inspection result of the sampled traffic.

In one embodiment, at least one traffic analyzer 120, at least one sampling point 130, and a sampling rate are calculated by calculating a reward for the inspection result of the sampled traffic, and applying the reward to the reinforcement learning model. information can be determined.

Specifically, in one embodiment, at least one traffic analyzer 120, at least one sampling point 130 for determining a sampling policy by applying a reward to the reinforcement learning model, and maximizing a reward according to the sampling policy. and sampling rate related information.

In an embodiment, the reinforcement learning model may include a Deep Deterministic Policy Gradient (DDPG) model.

In this case, according to an embodiment, state space information about the at least one traffic analyzer 120 and the at least one sampling point 130 may be determined based on the inspection result of the sampled traffic.

Thereafter, action space information for at least one traffic analyzer 120 and at least one sampling point 130 and sampling rate related information may be determined based on the state space information.

Further, in one embodiment, when the action space information is executed at a first time step, the transition probability for a change from the state space information for the first time step to the relative spatial information for the second time step (transition probability) information can be determined.

In addition, a reward according to the reinforcement learning model is determined based on the conversion probability information, and the reward is applied to the reinforcement learning model to determine at least one traffic analyzer 120 , at least one sampling point 130 , and sampling rate related information can

5 is a diagram illustrating a functional configuration of the SDN controller 110 according to an embodiment of the present invention. In an embodiment, the SDN controller 110 may be referred to as an SDN, an SDN device, or a term having an equivalent technical meaning.

Referring to FIG. 5 , the SDN controller 110 may include a communication unit 510 , a control unit 520 , and a storage unit 530 .

The communication unit 510 may receive a test result of sampled traffic for each sampling point 130 from the traffic analyzer 120 .

In an embodiment, the communication unit 510 may include at least one of a wired communication module and a wireless communication module. All or part of the communication unit 510 may be referred to as a 'transmitter', 'receiver', or 'transceiver'.

The control unit 520 may determine at least one traffic analyzer 120 , at least one sampling point 130 , and sampling rate related information based on a test result of the sampled traffic.

In one embodiment, the control unit 520 may be implemented as a centralized SDN controller that remotely controls the sampling point 120 (eg, a switch or router) using the OpenFlow (OF) protocol through the control plane.

In an embodiment, the controller 520 may include at least one processor or microprocessor, or may be a part of the processor. Also, the controller 520 may be referred to as a communication processor (CP). The controller 520 may control the operation of the SDN controller 110 according to various embodiments of the present invention.

The storage 530 may store the inspection result of the sampled traffic. In one embodiment, the storage unit 530 stores the at least one traffic analyzer 120 , the at least one sampling point 130 , and state space information, action space information, conversion probability information, and rewards for the sampling rate related information. can

In an embodiment, the storage unit 530 may be configured as a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. In addition, the storage unit 530 may provide stored data according to the request of the control unit 520 .

Referring to FIG. 5 , the SDN controller 110 may include a communication unit 510 , a control unit 520 , and a storage unit 530 . In various embodiments of the present invention, the SDN controller 110 is not essential to the configurations described in FIG. 5, so it may be implemented as having more configurations than the configurations described in FIG. 5, or having fewer configurations. .

The above description is merely illustrative of the technical spirit of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

The various embodiments disclosed herein may be performed out of order, and may be performed simultaneously or separately.

In an embodiment, at least one step may be omitted or added in each figure described herein, may be performed in the reverse order, or may be performed simultaneously.

The embodiments disclosed in the present specification are not intended to limit the technical spirit of the present invention, but to illustrate, and the scope of the present invention is not limited by these embodiments.

The protection scope of the present invention should be construed by the claims, and all technical ideas within the scope equivalent thereto should be understood to be included in the scope of the present invention.

100: SDN-enabled network
110: SDN controller
120: traffic analyzer
130: switch
200: DDPG model
210: Actor Network
220: Critic Network
510: communication department
520: control unit
530: storage

Claims (14)

(a) receiving an inspection result of the sampled traffic for each sampling point from the traffic analyzer; and
(b) determining at least one traffic analyzer, at least one sampling point, and sampling rate-related information based on a test result of the sampled traffic;
containing,
How SDN works using reinforcement learning.
According to claim 1,
The step (b) is,
calculating a reward for an inspection result of the sampled traffic; and
determining at least one traffic analyzer, at least one sampling point, and sampling rate related information by applying the reward to a reinforcement learning model;
containing,
How SDN works using reinforcement learning.
3. The method of claim 2,
The step (b) is,
determining a sampling policy by applying the reward to the reinforcement learning model; and
determining, according to the sampling policy, the at least one traffic analyzer, at least one sampling point, and sampling rate related information for maximizing the reward;
containing,
How SDN works using reinforcement learning.
According to claim 1,
Traffic sampled by the sampling point is passed to the traffic analyzer,
How SDN works using reinforcement learning.
3. The method of claim 2,
The reinforcement learning model, including a DDPG (Deep Deterministic Policy Gradient) model,
How SDN works using reinforcement learning.
6. The method of claim 5,
The step (b) is,
determining state space information about the at least one traffic analyzer and the at least one sampling point based on a test result of the sampled traffic; and
determining action space information for the at least one traffic analyzer and at least one sampling point and sampling rate related information based on the state space information;
containing,
How SDN works using reinforcement learning.
7. The method of claim 6,
The step (b) is,
When the action space information is executed at a first time step, transition probability information for a change from the state space information for the first time step to the relative spatial information for a second time step determining;
determining a reward according to a reinforcement learning model based on the conversion probability information; and
determining at least one traffic analyzer, at least one sampling point, and sampling rate related information by applying the reward to a reinforcement learning model;
containing,
How SDN works using reinforcement learning.
a communication unit configured to receive an inspection result of sampled traffic for each sampling point from the traffic analyzer; and
a control unit configured to determine at least one traffic analyzer, at least one sampling point, and sampling rate-related information based on a test result of the sampled traffic;
containing,
SDN device using reinforcement learning.
9. The method of claim 8,
The control unit is
Calculate a reward for the inspection result of the sampled traffic,
Applying the reward to a reinforcement learning model to determine at least one traffic analyzer, at least one sampling point, and sampling rate related information,
SDN device using reinforcement learning.
10. The method of claim 9,
The control unit is
Apply the reward to the reinforcement learning model to determine a sampling policy,
determining, according to the sampling policy, the at least one traffic analyzer for maximizing the reward, at least one sampling point, and sampling rate related information;
SDN device using reinforcement learning.
9. The method of claim 8,
Traffic sampled by the sampling point is passed to the traffic analyzer,
SDN device using reinforcement learning.
10. The method of claim 9,
The reinforcement learning model, including a DDPG (Deep Deterministic Policy Gradient) model,
SDN device using reinforcement learning.
13. The method of claim 12,
The control unit is
determining state space information for the at least one traffic analyzer and at least one sampling point based on the inspection result of the sampled traffic;
determining action space information for the at least one traffic analyzer and at least one sampling point and sampling rate related information based on the state space information;
SDN device using reinforcement learning.
14. The method of claim 13,
The control unit is
When the action space information is executed at a first time step, transition probability information for a change from the state space information for the first time step to the relative spatial information for a second time step to decide,
Determine a reward according to the reinforcement learning model based on the conversion probability information,
Applying the reward to a reinforcement learning model to determine at least one traffic analyzer, at least one sampling point, and sampling rate related information,
SDN device using reinforcement learning.

Images