JP2010263442A - Optical path setting method, and optical information communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an optical path setting method for a plurality of on-demands that attains differentiation of service for optical path setting and effective utilization of wavelength (optical path) and can be used in actual environment, and to provide an optical information communication system. <P>SOLUTION: An optical path setting table or link information sets a reward to obtain when an optical path is set, for each service class, as well as a cost when the setting of the optical path fails. They determine the acceptance or refusal of the optical path setting request of each class so that a reward function calculated from the reward and cost may be a maximum, and find out an optimal action for next link status from the present link status in transmission node by using a reinforcement learning algorithm based on the reward function. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique of an optical path setting method for a WDM (Wavelength Division Multiplexing) network that performs data transmission using an optical path.

  In an optical network using the WDM technology, data is transmitted using a wavelength (optical path) set between transmitting and receiving nodes. In recent years, the amount of data transmitted by each user over the Internet has increased dramatically. In the future, it is assumed that each user will transmit data of about 1 gigabit / second, and an optical path is set on demand. As a result, studies on utilization methods for transmitting a large amount of data are in progress.

In on-demand optical path setting, research has been conducted on a method for determining in advance the maximum number of optical paths that can be set for each service class in order to provide users with multiple optical path setting services according to usage charges ( For example, see Patent Document 1 and Patent Document 2.)
In the method of determining the maximum number of optical paths in advance, a method is employed in which the maximum number of optical paths set for the high priority class is made larger than the maximum number of optical paths set for the low priority class. However, in this method, even when there are many free wavelengths, the low priority class cannot set more optical paths than the set number of optical paths, so the optical path setting request frequency of the high priority class is low priority. When the frequency is less than the optical path setting request frequency of the class, there is a problem that the wavelength utilization rate is lowered and wavelength resources cannot be effectively used.

  In order to solve this problem, a method using a Markov Decision Process (MDP) has been proposed. This method using the Markov decision process determines in advance which class of optical path setting request should be accepted in accordance with the setting condition of the optical path. That is, the optimum behavior to be taken by each node is derived by the Markov decision process, and each node decides whether or not to accept the optical path setting according to the derived optimum behavior. Here, the optimum behavior depends on the reward function in the Markov decision process. However, in the method using the Markov decision process, when the number of wavelengths (optical paths) set between the transmitting and receiving nodes increases, the reward function does not converge, so the use in the real environment, that is, the network scale increases. There is a problem that it is difficult to use in some cases. In addition, information such as the topology information of the entire network and the number of wavelengths in each link is necessary and cannot be used in a distributed environment.

JP 2006-279767 A JP 2008-17256 A

As described above, in a usage method in which a user sets an optical path on demand to transmit a large amount of data, the conventional technology uses wavelength resources from the point of placing too much importance on service differentiation and flexibility. There is a problem that (optical channel channel resources) cannot be effectively used, and a calculation method used when the network scale becomes large cannot be used.
Further, in the conventional method, it is not possible to set an optical path by utilizing information that can be collected such as network topology, the number of usable wavelengths, and traffic information.

  In view of the above situation, the present invention provides a plurality of on-demand optical path setting methods and an optical information communication system that can be used in a real environment and can differentiate services of optical path setting and effectively use wavelengths (optical paths). The purpose is to provide.

It is another object of the present invention to provide an optical path setting method that can utilize collectable information such as network topology and traffic information. Moreover, it aims at providing the same effect even in a distributed environment where network topology and traffic information cannot be obtained.
Furthermore, the present invention can also be applied to cases where the wavelength conversion function of each node on the network is limited.

  In order to achieve the above object, an optical path setting method according to a first aspect of the present invention is an optical path setting method having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network. Sets the reward for each service class when the optical path is set, sets the cost when the optical path setting fails, and uses the reward function derived from the reward and cost to calculate the Q function Update, determine whether to accept the optical path setting request of each class so that the Q value of each action for the current state is maximized, and calculate the Q function from the current link state to the next link state in the transmitting node It is set as the structure updated based on a reward function.

According to such a configuration, in an optical network using the WDM technology, a plurality of on-demand optical path setting services having different service qualities can be provided to the user, and limited wavelength resources can be effectively used.
The present invention is characterized in that an optical path is set using reinforcement learning in an optical network using WDM technology. Specifically, the reward obtained when an optical path is set for each node is set for each service class, the cost when the optical path setting fails is also set, and the reward function derived from these rewards To determine whether or not to accept the optical path setting request of each class. In addition, this optimum optical path setting action (optimum action) can be obtained by performing a Monte Carlo simulation or the like in advance in a pseudo network environment in a centralized management environment where network information can be obtained. In a distributed environment where no network information is available and each node knows only the number of wavelengths of adjacent links and the number of service classes to be provided, it gradually learns and acquires optimal behavior while actually operating and providing services. To do. The acquired optimum behavior enables each node to effectively use wavelengths while providing a plurality of optical path setting services without using global optical path setting information.
Further, by using reinforcement learning, there is no problem that the reward function does not converge when the number of wavelengths (optical paths) set between the transmitting and receiving nodes increases as in the Markov decision process (MDP), and the usable wavelength. The number can be used in an actual environment such as 10 or more.

Here, the above-mentioned link state specifically refers to the number N of optical paths set for the service class i of the number M of classes at time (t) when considering a discrete time with an optical path setting request arrival event as a unit. _{Using i} (t) and the acceptance result I (t−1) for the optical path setting request at the previous time (t−1), the following equation 1 is used.

(Equation 1)
_{s t = (N 1 (t} ), N 2 (t), ···, N M (t), I (t-1))

In addition, the above-mentioned reward function, specifically, from the current link state s _t taking the action a _t, the next link state s (t + 1) = ( N 1 (t + 1), N 2 (t + 1), · .., N _M (t + 1), I (t)) is defined by the following formula 2 when transiting. The parameter variable R _i takes a value between 0 and 1, and when i <j, R _j <R _i , the parameter C takes a value greater than 0, and the parameter variable β takes a value between 0 and 1 inclusive. .

(Equation 2)
r _{t + 1} = β (R ₁ N ₁ (t) + R ₂ N ₂ (t) +... + R _M N _M (t)) − (1-β) CI (t) (where 0 <R _M <R _M-1 <... <R _i <... <R ₂ <R ₁ <= 1, 0 <= β <= 1, 0 <C)

Then, based on reward function, action value function action a _t for the state s _t (Q function) is defined by the following Expression 3, the action a _t with the highest Q value for state s _t as optimal action Whether to accept the optical path setting is determined.

(Equation 3)
Q (s _t , a _t ) ← Q (s _t , a _t ) + α [r _{t + 1} + γmaxQ (s _{t + 1} , a) −Q (s _t , a _t )] (where 0 <α <= 1, 0 <= γ <1)

By defining the link state and the reward function as described above, the ease of setting the optical path can be changed according to the service class, and the free wavelength can be used as effectively as possible.
Here, the parameter variable R _i is a reward set for each service class when an optical path is set in each node, and the parameter variable C is set when the optical path setting fails. Cost. By using the parameter variable β, optical path setting service differentiation and wavelength (optical path) effective use can be achieved.

  Further, the parameter α is a learning rate. When α = 0, learning is not possible because it cannot be learned. When α = 1, only the latest information is considered. The parameter γ is a discount rate and indicates how much past information affects. When the parameter γ = 0, only the current reward is considered, and when γ = 1, the past information is excessively considered for a long time and is excluded.

In addition, the above reinforcement learning algorithm is used in a centralized management environment where the network topology, number of nodes, number of usable wavelengths, traffic information, etc. can be grasped.
1) constructing a pseudo network environment given at least the network topology, the number of nodes, the number of usable wavelengths, the average optical path setting request interval of each class, and the average optical path usage time of each class;
2) A step of generating an optical path setting request event and an optical path use end event of each class in a pseudo manner in a pseudo network environment;
3) while taking the action a _t in which each node is in accordance with the state s _t, and the step to continue to update the Q value,
4) determining whether or not to accept the optical path setting request according to the optimal action pair (s _t , a _t ) changed each time;
It is set as the structure provided with.

In a pseudo network environment, an optical path setting request event and an optical path use end event are actually generated based on the optical path setting request event interval and the average optical path usage time of each class. In this case, the initial value s ₀ of the link state of each node is set to (0, 0, 0, 0,..., 0), and the action a ₀ for each link state s ₀ is given at random. By doing so, each node, the arrival event occurs, the link state _{s 0 = (0,0,0, ...,} 0) action a ₀ with respect to the optical path setting request generated according action a ₀ for the actual To run. Sufficient optimum action a _t learning and is performed for each link state s _t is derived, so that the effective use of differentiation and the wavelength of the service is achieved.

Next, an optical path setting method according to a second aspect of the present invention is an optical path setting method for setting an optical path having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
1) A step of setting a reward obtained when an optical path is set for each node for each service class, and setting a cost when the optical path setting fails;
2) setting the number of wavelengths and the number of service classes of links adjacent to the own node;
3) a step of updating a Q value of an action value function (Q function) while taking an action according to a link state when an optical path setting request event or an optical path use end event occurs;
4) The step of determining whether or not to accept the optical path setting request event according to the optimum behavior pair of the link state and the action when an optical path setting request event occurs is provided.

The optical path setting program of the present invention is
A program for setting an optical path having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
On each node's computer,
1) setting a reward obtained when an optical path is set for each service class, and setting a cost when the optical path setting fails;
2) setting the number of wavelengths and the number of service classes of links adjacent to the own node;
3) a step of updating a Q value of an action value function (Q function) while taking an action according to a link state when an optical path setting request event or an optical path use end event occurs;
4) When an optical path setting request event occurs, determining whether or not to accept the optical path setting request event according to the optimal behavior pair of the link state and the action;
Is to execute.

According to the second aspect of the optical path setting method or the optical path setting program, without performing a pseudo simulation at all, it is possible to find the final optimal action a _t.
In other words, without performing pseudo simulation at all, the optical switch having the above-described optical path setting program of the present invention is placed in a wavelength division multiplexing network, and only the number of wavelengths of adjacent links and the number of classes to be differentiated are known. (i.e. a state in which only a range that can be taken in the link state s _t and action a _t is known), the actual When the optical path setting request event and a light path using end event occurs, the action a _t in accordance with the state s _t performing update the Q value, that will also change action a _t to be taken with respect to the state s _t accordingly, it become that ultimately optimal action a _t is found.
Since the link state s _t and Q values, are as described for optical path setting method of the first aspect of the present invention described above, description thereof will be omitted.

The optical switch of the present invention is equipped with the above-described program of the present invention, and performs optical path setting having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network.
According to the optical path setting program of the present invention and the optical switch equipped with the present program, there is an advantage that no information of other nodes is required and it can be used in a distributed environment.

  If the network topology or the like is known for the optical path setting program or optical switch of the present invention, a value that has been learned to some extent may be used as initial value data by using a pseudo simulation. In such a case, the optical path setting program and the optical switch of the present invention are additionally learned while being introduced into the network and used.

  In the optical information communication system of the present invention, each node in the wavelength division multiplexing network having a plurality of service classes includes an optical transmission unit, an optical reception unit, an optical / electrical conversion unit, an electric / optical conversion unit, and an arithmetic unit. And the storage means stores the table information of the link state and the optimum behavior set by the above-described optical path setting method, and the calculation means receives the optical path setting request of the service class i In this case, the table information is used to determine whether or not to reject the optical path setting request according to the optimum behavior corresponding to the current link state, and the Q value of each behavior for each state using the Q function is updated. It is said.

  Here, the computing means generates table information online based on the network topology, the number of nodes, the number of usable wavelengths, the average optical path setting request interval of each class, and the average optical path usage time of each class. It is characterized by that.

  In addition, the optical information communication system described above further includes a parameter adjustment unit that can adjust a parameter of a reward function that balances service differentiation and wavelength utilization efficiency.

  According to the optical path setting method and the optical information communication system of the present invention, there is an effect that service differentiation of optical path setting and effective use of wavelength (optical path) can be achieved, and it can be used in an actual environment.

Illustration of WDM network Illustration of optical path setting in WDM network Schematic configuration diagram of each node in the WDM network Processing flow diagram of reinforcement learning algorithm Explanatory drawing of optical path setting of Example 1 (1) Explanatory drawing of optical path setting of Example 1 (2) The graph which shows the rejection rate in the case of the optical path setting of Example 1. The graph which shows the rejection rate in the case of the optical path setting of Example 1 (parameter γ is abscissa) The graph which shows the rejection rate in the case of the optical path setting of Example 1 (parameter α is a horizontal axis) The graph which shows the rejection rate in the case of the optical path setting of Example 1 (the number of wavelengths is a horizontal axis)

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The scope of the present invention is not limited to the following examples and illustrated examples, and various changes and modifications can be made.

  As shown in FIG. 1, a conventional optical network transmits data using one wavelength using one optical fiber, whereas a WDM network multiplexes a plurality of wavelengths using one optical fiber. By assigning one or more wavelengths that are not used between other transmitting / receiving nodes among a plurality of wavelengths between transmitting / receiving nodes (between the transmitting node and the receiving node) that can transmit several times the amount of data and perform communication, It occupies the transmission wavelength between the transmitting and receiving nodes and realizes large-capacity data transmission, and makes effective use of the optical fiber bandwidth by allowing other nodes to use it immediately after the use of the wavelength.

  In a WDM network, each node autonomously selects a wavelength (optical path) and uses a physical path between the transmitting and receiving nodes in order to efficiently use the wavelength (optical path) and to ensure fairness in securing the optical path. It is desirable to secure an optical path that is a communication channel.

  In the optical path setting in the WDM network as shown in FIG. 2, when transmitting a large amount of data from the transmitting computer to the receiving computer, first, an optical path setting request is made from the transmitting computer (user) and the WDM network is set up. Data transmission is performed after setting the optical path using the available wavelengths. If there is no free wavelength, the optical path setting will fail and data transmission will not be performed. When data transmission is performed, one wavelength band is occupied between the transmitting computer and the receiving computer.

  Here, a schematic configuration of each node illustrated in FIG. 2 will be described with reference to FIG. A signal passing through the optical fiber 11 of the WDM network is demultiplexed or multiplexed by a demultiplexer / multiplexer (12, 13). The demultiplexed light is switched by the optical switch 14 in the physical channel through which the optical signal propagates. The optical switch 14 is controlled by the controller 15. The controller 15 receives a node control signal sent via the optical or electrical channel 8 using the transceiver 18. Further, the controller 15 performs optical path setting with reference to the optical path setting table and link information stored in the memory 16.

Next, an optical path setting table and link information stored in the memory 16 referred to by the controller 15 of each node will be described below.
In the present invention, the optical path setting table and link information set the reward obtained when the optical path is set for each service class, and sets the cost when the optical path setting fails, and the reward and Decide whether to accept the optical path setting request of each class so that the reward function derived from cost is maximized, and strengthen the optimal action from the current link state to the next link state at the sending node based on the reward function Derived using a learning algorithm.

Link state _{s t} at time t at each _node, s _t = is defined as _{(N 1 (t), N} 2 (t), ···, N M (t), I (t-1)).
Here, M is the number of service classes, N _i (t) is the number of optical path settings for service class i at time t, and I (t−1) is the acceptance result for the previous optical path setting request (if it fails) 1. If successful, 0). This I is a cost when the optical path setting fails. That is, in a reward function to be described later, a reward is obtained by subtracting I multiplied by a weighting factor, and the reward is reduced if the optical path setting request fails.
In addition, the current link status s _t acted _{a t,} if a transition is made to the next time t + 1 to the link state _{s t + 1,} the link state _{s t + 1 is, s t + 1 = (N} 1 (t + 1), N 2 (t + 1 ,..., N _M (t + 1), I (t)).

Here, the action a _t, among the class of service M, which service class to set the optical path request acceptance light path, indicates whether not set rejected by optical paths of light path request any service class is there.
If the service class is M, the possible actions _{a t} ^exists as ⁽² M -1). For example, if the service class is 3, the possible actions a _t exists ways following 7.

1) Only service class 1 optical path setting requests are accepted.
2) Only service class 2 optical path setting requests are accepted.
3) Only an optical path setting request for service class 3 is accepted.
4) Accepts only optical class setting requests for service class 1 and service class 2.
5) Accepts only optical path setup requests for service class 2 and service class 3.
6) Accepts only optical class setting requests for service class 1 and service class 3.
7) Accept all service class 1, 2, and 3 optical path setting requests.

The reward function is defined by the following formula 4. Using this reward function, takes action a _r a state s _t, compensation can be calculated when the state changes to s _{t + 1.}
Here, the parameter variable R _i is a reward set for each service class when an optical path is set in each node, and the parameter variable C is set when the optical path setting fails. Cost. By using the parameter variable β, optical path setting service differentiation and wavelength (optical path) effective use can be achieved. As the value of the parameter variable β decreases, the weight for effective wavelength use increases, making it difficult to provide service differentiation, while enabling effective use of wavelengths. The parameter variable R _i and the parameter variable β take a positive value of 1 or less, and C takes a positive value.

(Equation 4)
r _{t + 1} = β (R ₁ N ₁ (t) + R ₂ N ₂ (t) +... + R _M N _M (t)) − (1-β) CI (t)

And the reinforcement learning algorithm, as shown in the flow diagram of FIG.
(Step 1) Step of constructing a pseudo network environment given at least the network topology, the number of nodes, the number of usable wavelengths, the average optical path setting request interval of each class, and the average optical path usage time of each class (Step 2) on pseudo network environment, while pseudo-step of generating a light path setting request event and a light path using end event of each class (step 3) each node takes action a _t in accordance with the link state s _t, Q value (Step 4) is a step of determining whether or not to accept the optical path setting request according to the optimum behavior pair (s _t , a _t ) of the link state s and behavior a that is changed each time.

  The network topology of the optical network, the number of nodes, the number of usable wavelengths, the average optical path setting request interval for each class, and the average optical path usage time for each class on a computer independent of the controller or node of each node A given simulated network environment is constructed, a Monte Carlo simulation is executed, and a reward function value is calculated.

Finally, as step 5, based on the optimal action pair (s _t , a _t ) with the maximum reward function value, the optical path setting table information of the link state and the optimal action is generated, and the link state and the optimal action It is stored in the memory 16 of each node as optical path setting table information.
When each node receives an optical path setup request for service class i, it rejects the optical path setup request according to the optimum behavior corresponding to the current link state using the optical path setup table information stored in the memory 16. It will be determined whether or not.

  According to the above optical path setting method and optical information communication system, service differentiation of optical path setting and effective use of wavelength (optical path) can be achieved, and it can be used in an actual environment. Below, it demonstrates through a specific Example.

FIG. 5A and FIG. 5B illustrate how an optical path is set in an optical network topology with three nodes. Here, it is assumed that the number of service classes is 3 and the number of wavelengths is 8.
In FIG. 5A, the state s _t of the node B at time t is (2, 3, 1, 0), and there are two free wavelengths. Consider a case where a service class 2 data transmission request is generated from node A to node C in this state. If state _s t = (2,3,1,0) optimal action _{a t} for is actions like "Accept only service class 1 and service class 2 of the optical path setting request", the service class 2 optical path setting request Is accepted and the state s _t transitions to the state s _{t + 1} = ( _{2, 4, 1,} 0) (the optical path setting is successful).

Next, in Figure 5-2, in the state _{s t + 1,} if the data transmission request for service class 2 to node C is generated from the node A, the state _s t + 1 = optimal action _{a t + 1} for the (2,4,1,0) Is an action such as “accept only service class 1 optical path setup request”, the service class 2 optical path setup request is not accepted, and the state transits to the state s _{t + 1} = ( _{2, 4, 1, 1} ). (Optical path setting failed).

Next, when performing optical path setting from node A to node C, the conventional optical path setting method and the optical path setting method of the present invention are used for the rejection rate for the optical path setting request from node A in relay node B. Compare. A conventional optical path setting method uses a method in which the number of optical paths that can be set for each class is fixed. The number of wavelengths is 8, and the number of service classes is 3.
Also, α of the Q function is 0.1, γ is 0.95, C of the reward function is 1.0, and R ₁ , R ₂ , and R ₃ are changed from 0.05 to 1.0 in increments of 0.05. However, R ₃ <R ₂ <R ₁ and β = 0.1.
Further, as a user request, the failure probability of service class i is set to ζi or less. ξ1 = 0.4, ξ2 = 0.9, ξ3 = 1.0.

  FIG. 6 shows a graph showing the rejection rate in the case of setting an optical path. Compared with the conventional optical path setting method, in the optical path setting method of the present invention, the rejection rate is reduced in both service class 1 and service class 2, and service differentiation and effective use of wavelengths are further improved. I can understand that.

FIG. 7 also shows that the number of wavelengths is 6, the number of classes is 3, the Q function α is 0.1, the reward function C is 1.0, R ₁ = 0.8, R ₂ = 0.4, R ₃ = 0.2, and β = 0.1. At this time, the rejection rate of each class when the optical path setting method of the present invention is used is shown with the parameter γ on the horizontal axis.
As shown in FIG. 7, when γ is larger than 0.65, service differentiation can be realized by learning. From this, it can be understood that the performance of the optical path setting method of the present invention is affected by the parameter γ.

FIG. 8 shows that the number of wavelengths is 6, the number of classes is 3, the γ of the Q function is 0.95, the C of the reward function is 1.0, R ₁ = 0.8, R ₂ = 0.4, R ₃ = 0.2, and β = 0.1. At this time, the rejection rate of each class when the optical path setting method of the present invention is used is shown with the parameter α as the horizontal axis.
As shown in FIG. 8, when α changes from 0 to 0.1, service differentiation becomes more effective. Also, as α approaches 1.0, the difference in rejection rate becomes smaller. Therefore, it can be understood that the performance of the optical path setting method of the present invention is affected by the parameter α.

FIG. 9 shows that the number of classes is 3, the γ of the Q function is 0.95, α is 0.1, the reward function C is 1.0, R ₁ = 0.8, R ₂ = 0.4, R ₃ = 0.2, and β = When 0.1, the horizontal axis indicates the rejection rate of each class when the optical path setting method of the present invention is used.
As shown in FIG. 9, it can be seen that the optical path setting method of the present invention can be used even when the number of wavelengths is 100, which is more effective than the Markov decision process method of the previous research.

  The present invention is expected to be used for telemedicine, high-definition video streaming, and grid computing.

DESCRIPTION OF SYMBOLS 1 Optical fiber 2 Transmission side computer 3 Reception side computer 4 Data 5 Optical path 7 Optical network 8 Node 11 Optical fiber 12, 13 Demultiplexer / Multiplexer 14 Optical switch 15 Controller 16 Memory 17 Transceiver 18 Optical or electrical channel 21 Optical path Setting information table 22 Link status information table

Claims (12)

An optical path setting method having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
Remuneration obtained when an optical path is set for each node is set for each service class, and the cost when the optical path setting fails is set,
Deciding whether to accept the optical path setting request for each class so that the reward function derived from the reward and cost is maximized,
An optical path setting method, wherein an optimal behavior from the current link state to the next link state in the transmitting node is derived using a reinforcement learning algorithm based on a reward function. The link state is
Introduce a discrete time t in units of optical path setting request arrival events,
By using the optical path setting number N _i (t) of the service class i of the class number M at the time (t) and the acceptance result I (t−1) for the optical path setting request at the previous time (t−1), Defined by the number 1,
The reward function is defined by the following _{equation 2} when taking the action a _t from the current link state s _t and transitioning to the next link state s _{t + 1} :
Based the reward function, using the Q value of each behavior a _t to the state s _t obtained from the action-value function defined by the following Expression 3 (Q function), the largest Q value for state s _t optical path setting method according to claim 1, receiving availability of optical path setting is being determined action a _t as optimal action with.
(Equation 1)
_{s t = (N 1 (t} ), N 2 (t), ···, N M (t), I (t-1))
(Equation 2)
r _{t + 1} = β (R ₁ N ₁ (t) + R ₂ N ₂ (t) +... + R _M N _M (t)) − (1-β) CI (t) (where 0 <R _M <R _M-1 <... <R _i <... <R ₂ <R ₁ <= 1, 0 <= β <= 1, 0 <C)
(Equation 3)
Q (s _t , a _t ) ← Q (s _t , a _t ) + α [r _{t + 1} + γmaxQ (s _{t + 1} , a) −Q (s _t , a _t )] (where 0 <α <= 1, 0 <= γ <1) The reinforcement learning algorithm is:
Constructing a pseudo network environment given at least the network topology, the number of nodes, the number of usable wavelengths, the average optical path setting request interval of each class, and the average optical path usage time of each class;
On the pseudo network environment, generating an optical path setting request event and an optical path use end event for each class in a pseudo manner;
While taking the action a _t each node according to the link state s _t, and the step to continue to update the Q value,
A step in accordance with the link state s _t and optimal action pairs of the actions a _t are each time changed (s _{t, a t),} will determine the acceptance possibility of the optical path setting request,
The optical path setting method according to claim 2, further comprising: An optical path setting method for setting an optical path having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
For each node, setting a reward obtained when an optical path is set for each service class, and setting a cost when the optical path setting fails,
Setting the number of wavelengths and the number of service classes of links adjacent to the own node;
A step of updating a Q value of an action value function (Q function) while taking an action according to a link state when an optical path setting request event or an optical path use end event occurs;
Determining whether to accept the optical path setting request event according to the optimal behavior pair of the link state and the action when the optical path setting request event occurs; and
An optical path setting method comprising: The link state is
Introduce a discrete time t in units of optical path setting request arrival events,
By using the optical path setting number N _i (t) of the service class i of the class number M at the time (t) and the acceptance result I (t−1) for the optical path setting request at the previous time (t−1), Defined by equation (4)
The reward function is defined by the following _{equation 5} when taking the action a _t from the current link state s _t and transitioning to the next link state s _{t + 1} :
On the basis of the compensation function, with the Q value of each behavior a _t to the state s _t obtained from the Q function defined by the following Expression 6, a behavior a _t with the highest Q value for state s _t 5. The optical path setting method according to claim 4, wherein acceptability of the optical path setting is determined as the optimum behavior.
(Equation 4)
_{s t = (N 1 (t} ), N 2 (t), ···, N M (t), I (t-1))
(Equation 5)
r _{t + 1} = β (R ₁ N ₁ (t) + R ₂ N ₂ (t) +... + R _M N _M (t)) − (1-β) CI (t) (where 0 <R _M <R _M-1 <... <R _i <... <R ₂ <R ₁ <= 1, 0 <= β <= 1, 0 <C)
(Equation 6)
Q (s _t , a _t ) ← Q (s _t , a _t ) + α [r _{t + 1} + γmaxQ (s _{t + 1} , a) −Q (s _t , a _t )] (where 0 <α <= 1, 0 <= γ <1) A program for setting an optical path having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
On each node's computer,
A step of setting a reward obtained when an optical path is set for each service class, and setting a cost when the optical path setting fails;
Setting the number of wavelengths and the number of service classes of links adjacent to the own node;
A step of updating a Q value of an action value function (Q function) while taking an action according to a link state when an optical path setting request event or an optical path use end event occurs;
Determining whether to accept the optical path setting request event according to the optimal behavior pair of the link state and the action when the optical path setting request event occurs; and
Optical path setting program to execute. The link state is
Introduce a discrete time t in units of optical path setting request arrival events,
By using the optical path setting number N _i (t) of the service class i of the class number M at the time (t) and the acceptance result I (t−1) for the optical path setting request at the previous time (t−1), Defined by Equation 7,
The reward function is defined by the following _{equation 8} when taking the action at from the current link state s _t and transitioning to the next link state s _{t + 1} :
On the basis of the compensation function, with the Q value of each behavior a _t to the state s _t obtained from the Q function defined by the following Expression 9, the action a _t with the highest Q value for state s _t The optical path setting program according to claim 6, wherein acceptability of the optical path setting is determined as the optimum action.
(Equation 7)
_{s t = (N 1 (t} ), N 2 (t), ···, N M (t), I (t-1))
(Equation 8)
r _{t + 1} = β (R ₁ N ₁ (t) + R ₂ N ₂ (t) +... + R _M N _M (t)) − (1-β) CI (t) (where 0 <R _M <R _M-1 <... <R _i <... <R ₂ <R ₁ <= 1, 0 <= β <= 1, 0 <C)
(Equation 9)
Q (s _t , a _t ) ← Q (s _t , a _t ) + α [r _{t + 1} + γmaxQ (s _{t + 1} , a) −Q (s _t , a _t )] (where 0 <α <= 1, 0 <= γ <1) The step of setting the optimal action pair (s _t , a _t ) obtained by the optical path setting method of the reinforcement learning algorithm according to claim 3 as an initial value is further provided. Optical path setting program. An optical switch for performing optical path setting having a plurality of service classes from a transmission node to a reception node in a wavelength division multiplexing network,
An optical switch equipped with the program according to claim 6. Each node in the wavelength division multiplexing network having a plurality of service classes includes an optical transmission means, an optical reception means, an optical / electrical conversion means, an electric / optical conversion means, an arithmetic means, and a storage means,
The storage unit stores table information of the link state and the optimum behavior set by the optical path setting method according to any one of claims 1 to 5,
When receiving the service class i optical path setup request, the computing means determines whether to reject the optical path setup request according to the optimum behavior corresponding to the current link state using the table information. Updating the Q value of each action for each state using the Q function;
An optical information communication system.   The computing means generates the table information online based on the network topology, the number of nodes, the number of usable wavelengths, the average optical path setting request interval of each class, and the average optical path usage time of each class. The optical information communication system according to claim 10. 11. The optical information communication system according to claim 10, further comprising parameter adjusting means capable of adjusting a parameter of the reward function for achieving a balance between service differentiation and wavelength utilization efficiency.