CN115714937A - All-optical switching distributed reinforcement learning system and method based on array waveguide grating - Google Patents

Abstract

The invention provides a full optical switching distributed reinforcement learning system and method based on arrayed waveguide grating, the system includes the arrayed waveguide grating router and multiple clusters among the clusters; each cluster comprises a parameter server rack, an in-cluster arrayed waveguide grating router and a plurality of working server racks; the parameter server rack comprises a first top switch and a plurality of parameter servers, and the network ports of the parameter servers are connected to the first top switch; the work server rack comprises a second top switch and a plurality of work servers, and the network ports of the work servers are connected to the second top switch; the intra-group arrayed waveguide grating router is interconnected with the first top-rack switch and each second top-rack switch in a full-connection mode; the inter-cluster arrayed waveguide grating router is interconnected in a fully connected manner with a first top-of-rack switch within each intra-cluster parameter server chassis. The distributed reinforcement learning method is suitable for distributed reinforcement learning, only needs one network hop for data transmission, and has the characteristics of low time delay and low loss.

Description

All-optical switching distributed reinforcement learning system and method based on arrayed waveguide grating

technical field

The invention relates to the field of optical switching technology, in particular to an arrayed waveguide grating-based all-optical switching distributed reinforcement learning system and method.

Background technique

With the rapid development and application of big data in various fields, machine learning has gradually become an important tool for processing big data. The core idea of machine learning is to train a model to fit the input training data, deploy the trained model in the corresponding application, and expect the model to accurately classify or predict the data generated during the running of the application. Today's machine learning applications often require complex models to handle very large datasets. Single-machine deployment can no longer meet the needs of such large-scale machine learning training, and distributed machine learning has emerged as the times require. It uses multiple computing nodes for collaborative training through distributed deployment, which greatly improves the speed of training. However, in the face of huge data, there is still a problem that the communication frequency between nodes increases in order to transmit more data per unit time between distributed machine learning nodes. If some computing nodes do not receive due to network congestion Data, as a whole, cannot enter the next iteration in time, and finally the bottleneck of the completion time of the training task is shifted from calculation to network communication.

Reinforcement learning is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of maximizing the report or achieving specific goals through learning strategies during the interaction between the agent and the environment. Compared with the commonly used algorithms in distributed machine learning, such as deep convolutional neural network, recurrent neural network, deep graph convolutional neural network, etc., distributed reinforcement learning training needs to use smaller gradient aggregation to generate a larger order of magnitude iterate. A typical reinforcement learning algorithm will generate 150,000 to 20 million iterations, so the latency of gradient communication in each iteration is a key factor affecting the performance of distributed reinforcement learning training.

One of the existing technologies is a switch architecture using a centralized parameter server, and the parameter server aggregates the gradient parameters of each working server through the electric switch to update the weights. This scheme limits the scalability of the distributed reinforcement learning system. At the same time, considering the iterations on the order of tens of millions, the limited bandwidth and energy-consuming electrical exchange will bring a delay of hundreds of microseconds to ten milliseconds. The other is to use an All-reduce-based architecture. Each worker server divides the task into N subtasks, and the same sequence of subtasks in each worker server will be cyclically aggregated sequentially. Although the gradient aggregation of this scheme is performed in a non-centralized manner, the number of network hops in the communication process increases linearly with the increase of the network size, and there will also be a millisecond-level delay.

Contents of the invention

In view of this, the embodiment of the present invention provides an all-optical switching distributed reinforcement learning system and method based on arrayed waveguide gratings, to eliminate or improve one or more defects existing in the prior art, and solve the limitations of the prior art. The scalability of the traditional reinforcement learning system and the high iteration delay of model training parameters.

On the one hand, the present invention provides an all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings, which is characterized in that it includes:

A plurality of clusters, each cluster includes a parameter server rack, an arrayed waveguide grating router in the group, and a plurality of working server racks; the parameter server rack includes a first top-of-rack switch and a plurality of parameter servers, and the network of each parameter server ports are all connected to the first top-of-rack switch; the working server rack includes a second top-of-rack switch and a plurality of working servers, and the network ports of each working server are connected to the second top-of-rack switch; the group The inner arrayed waveguide grating router is interconnected with the first top-of-rack switch and each second top-of-rack switch in a fully connected manner; in each cluster, each parameter server and each working server pass through the first top-of-rack switch, The second top-of-rack switch communicates with the arrayed waveguide grating router in the group;

An arrayed waveguide grating router between groups, the arrayed waveguide grating router between groups is interconnected with the first top-of-rack switch in the parameter server rack in each cluster in a fully connected manner; the parameter servers in each cluster are connected through the corresponding first rack The top switch and the arrayed waveguide grating router between the groups are connected for communication;

Among them, in each cluster, the parameter server in the parameter server rack sends the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks, and the parameters in each working server rack The working server feeds back the parameters obtained by the preset reinforcement learning model to the parameter server according to the corresponding subtask training, and performs gradient aggregation; between each cluster, the parameter servers in each parameter server rack pass through the inter-cluster array waveguide The grating router exchanges the parameters of the preset reinforcement learning model obtained after the gradient aggregation of each cluster.

In some embodiments of the present invention, the arrayed waveguide grating routers within the group and the arrayed waveguide grating routers between groups use wavelength division multiplexing technology to establish multi-channel communication links.

In some embodiments of the present invention, both the intra-group arrayed waveguide grating routers and the inter-group arrayed waveguide grating routers route optical signals to corresponding output ports in a circular wavelength routing manner.

In some embodiments of the present invention, both the first top-of-rack switch and the second top-of-rack switch include a switching module, multiple receiving modules, and multiple sending modules, and the switching module includes a data packet processor, a scheduling device, a broadcast module and a selector, and the switch module also constructs and records a flow table for mapping LAN addresses and sending ports.

In some embodiments of the present invention, the data packet processor determines the destination of the data packet according to the header of the data packet to be forwarded;

When the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to the server in the corresponding rack;

When the destination of the data packet points to the inter-rack, the scheduler extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination of the data packet; The selector forwards the data packet to the sending port according to the obtained sending port.

In some embodiments of the present invention, the selector forwards the data packet to the sending port according to the obtained sending port, and further includes:

When the destination and the data packet are in the same cluster, forward the data packet to a corresponding destination server via the arrayed waveguide grating router in the cluster;

When the destination and the data packet are in different clusters, the data packet is forwarded to a corresponding destination server via the inter-cluster ARWG router.

In some embodiments of the present invention, the scheduler extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination of the data packet, and further includes:

When the LAN address and its corresponding sending port do not exist in the flow table, flooding is performed and an alarm is issued.

On the other hand, the present invention provides an all-optical switching distributed reinforcement learning method based on arrayed waveguide gratings, which is characterized in that the method runs on the all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings as described above , in a loop, the method includes:

In each cluster, the parameter server in the parameter server rack sends the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks;

In each cluster, the working servers in each working server rack feed back the parameters obtained by training the preset reinforcement learning model according to corresponding subtasks to the parameter server, and perform gradient aggregation.

In some embodiments of the present invention, the working server in each working server rack feeds back the parameters obtained by the preset reinforcement learning model according to the corresponding subtask training to the parameter server, and performs gradient aggregation, and further includes:

In each cluster, the parameter server in the parameter server rack performs gradient aggregation synchronously, and after updating the parameters of the preset reinforcement learning model; between each cluster, exchange each cluster through the inter-cluster array waveguide grating router The parameters of the preset reinforcement learning model obtained after gradient aggregation;

In each cluster, the parameter server in the parameter server rack re-establishes subtasks, sends new parameters to the working servers in each working server rack, and executes the next cycle.

On the other hand, the present invention also provides a computer-readable storage medium on which a computer program is stored, which is characterized in that, when the program is executed by a processor, the AWG-based all-optical switching as described in any one of the above Steps of a distributed reinforcement learning method.

The beneficial effects of the present invention are at least:

The present invention provides an all-optical switching distributed reinforcement learning system and method based on arrayed waveguide gratings. Each cluster of the system is provided with a parameter server rack and a working server rack, and each parameter server is loaded into the parameter server rack. Each working server is loaded into the working server rack, and the concept of parameter server pool and working server pool is constructed, and the first top-of-rack switch, the second top-of-rack switch and the array waveguide grating router in the group are used for communication, so as to solve the problems caused by centralized servers. The scalability problem greatly improves the flexibility of network expansion.

The arrayed waveguide grating router in the group is interconnected with the first top-of-rack switch and each second top-of-rack switch in a fully connected manner; the inter-group arrayed waveguide grating router is fully connected with the first Top-of-rack switch interconnect. Arrayed waveguide grating routers within a group and arrayed waveguide grating routers between groups cooperate with lasers to realize all-optical switching, avoiding additional delay and power consumption caused by photoelectric-optical conversion. At the same time, the fully connected topology architecture greatly reduces the number of network hops. Only one network hop is needed to meet gradient communication and nanosecond-level low-latency communication. while increasing linearly.

Additional advantages, objects, and features of the present invention will be set forth in part in the following description, and will be partly apparent to those of ordinary skill in the art after studying the following text, or can be learned from the practice of the present invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and appended drawings.

It will be understood by those skilled in the art that the objects and advantages that can be achieved by the present invention are not limited to the above specific ones, and the above and other objects that can be achieved by the present invention will be more clearly understood from the following detailed description.

Description of drawings

The drawings described here are used to provide further understanding of the present invention, constitute a part of the application, and do not limit the present invention. In the attached picture:

FIG. 1 is a schematic diagram of the overall structure of an all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings in an embodiment of the present invention.

Fig. 2 is a schematic diagram of the intra-group structure of the all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings in an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of an arrayed waveguide grating router within a group and an arrayed waveguide grating router between groups in an embodiment of the present invention.

Fig. 4 is a schematic structural diagram of a first top-of-rack switch and a second top-of-rack switch in an embodiment of the present invention.

FIG. 5 is a schematic diagram of the steps of an arrayed waveguide grating-based distributed reinforcement learning method for all-optical switching in an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the embodiments and accompanying drawings. Here, the exemplary embodiments and descriptions of the present invention are used to explain the present invention, but not to limit the present invention.

Here, it should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only the structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, and the related Other details are not relevant to the invention.

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

Here, it should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection, but also to an indirect connection with an intermediate.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

What needs to be emphasized here is that the labels of the steps mentioned below do not limit the sequence of the steps, but should be understood as the steps can be performed in the order mentioned in the embodiment, or can be different from the order in the embodiment, Or several steps are performed simultaneously.

In order to solve the problem that the existing technology limits the scalability of the distributed reinforcement learning system and the high iteration delay, the present invention provides an all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings, as shown in Figure 1, the system Including multiple clusters and an inter-cluster arrayed waveguide grating router, specifically:

As shown in Figure 2, each cluster includes parameter server racks, array waveguide grating routers within the cluster, and multiple working server racks. Among them, the parameter server rack includes the first top-of-rack switch and multiple parameter servers, and the network ports of each parameter server are connected to the first top-of-rack switch; the working server rack includes the second top-of-rack switch and multiple working servers, each The network ports of the working servers are all connected to the second top-of-rack switch; the arrayed waveguide grating routers in the group are interconnected with the first top-of-rack switch and each second top-of-rack switch in a fully connected manner; in each cluster, each parameter server 1. Each working server is communicatively connected through the first top-of-rack switch, the second top-of-rack switch and the array waveguide grating router in the group.

The arrayed waveguide grating router between groups is interconnected with the first top-of-rack switch in the parameter server rack in each cluster in a fully connected manner; the parameter servers in each cluster are connected through the corresponding first top-of-rack switch and the inter-group arrayed waveguide grating router Make a communication connection.

In each cluster, the parameter server in the parameter server rack sends the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks; the working servers in each working server rack The parameters obtained by the preset reinforcement learning model are fed back to the parameter server according to the corresponding subtask training, and the gradient aggregation is performed; between each cluster, the parameter servers in each parameter server rack exchange the gradients of each cluster through the inter-cluster array waveguide grating router. The parameters of the preset reinforcement learning model obtained after aggregation.

The invention utilizes the intra-group array waveguide grating router and the inter-group array waveguide grating router to realize flow gradient switching and weight switching within the cluster and between the clusters.

Arrayed waveguide grating routers within a group have the same structure as arrayed waveguide grating routers between groups. The full English name of arrayed waveguide grating routers is ArrayedWaveguide Grating Router.

Arrayed Waveguide Grating (AWG for short) is the preferred technology in Dense Wavelength Division Multiplexing (DWDM). AWG is a passive planar waveguide device, which is an arrayed waveguide grating fabricated on a chip substrate using programmable logic controller (PLC) technology. Compared with fiber Bragg grating (FBG) and dielectric film filter (TTF), AWG has the advantages of high integration, large number of channels, small insertion loss, and easy batch automation. AWG is usually used in optical multiplexers in wavelength division multiplexing systems. Through these devices, light of many wavelengths can be multiplexed into a single optical fiber, thereby improving the propagation efficiency of the optical fiber network.

In some embodiments, the arrayed waveguide grating routers within a group and the arrayed waveguide grating routers between groups use 5×5 ports, and the specific structure is shown in FIG. 3 . On the left are input port 1, input port 2, input port 3, input port 4, and input port 5; on the right are output port a, output port b, output port c, output port d, and output port e; in the middle are arrayed waveguide gratings . Optical signals of different wavelengths are input to each input port, and corresponding optical signals are routed to corresponding output ports according to wavelength routing. Wherein, the wavelength routing refers to selecting a route according to the wavelength of an optical signal when passing through a network node. Exemplarily, as shown in FIG. 3 , λ ₁ , λ ₂ , λ ₃ , λ ₄ , and λ ₅ represent optical signals of five wavelengths respectively. Taking input port 3 as an example, input λ ₁ , λ ₂ , λ ₃ , λ ₄ , and λ ₅ optical signals of five wavelengths, according to their respective wavelengths, the output port of the optical signal with a wavelength of λ ₁ is e; the output port of an optical signal with a wavelength of λ ₂ is d; the output port of an optical signal with a wavelength of λ ₃ The output port of the optical signal is c; the output port of the optical signal with a wavelength of _λ4 is b; the output port of the optical signal with a wavelength of _λ5 is a. The optical signals of other input ports are the same.

In some embodiments, both the arrayed waveguide grating routers within the group and the arrayed waveguide grating routers between the groups route the optical signals to the corresponding output ports in a circular wavelength routing manner. As shown in Figure 3, take the optical signal of wavelength λ ₃ as an example, the wavelength of the input port 1 input is the optical signal of λ ₃ , the corresponding output port is a; the wavelength of the input port 2 input is the optical signal of λ ₃ , The corresponding output port is b; the input port 3 is an optical signal with a wavelength of λ3, and the corresponding output port is c; the input port 4 is an optical _signal with a wavelength of _λ3 , and the corresponding output port is d; the input port 5 The input optical signal with a wavelength of _λ3 corresponds to an output port e, forming a loop. The same applies to optical signals of other wavelengths.

In some embodiments, the arrayed waveguide grating routers within the group and the arrayed waveguide grating routers between the groups use wavelength division multiplexing technology to establish multi-channel communication links. Exemplarily, as shown in FIG. 3 , 25 channel communication links are established based on 5 wavelengths.

The structure of the first top-of-rack switch and the second top-of-rack switch are the same, and both are realized by Field Programmable Gate Array (Field Programmable GateArray, FPGA). routing and forwarding.

FPGA is a product of further development on the basis of programmable devices such as programmable array logic (PAL) and general array logic (GAL). It emerged as a semi-custom circuit in the field of application-specific integrated circuits (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of original programmable device gates.

Both the first top-of-rack switch and the second top-of-rack switch adopt a top-of-rack (Top of Rank, TOR) connection mode. TOR is an extension of the EOR (End of Row)/MOR (Middle of Row) method, all of which are an architectural design method of the data center. Traditional racks are mainly based on EOR and MOR, and adopt similar centralized wiring. The difference between EOR and MOR mainly lies in the position of the switch in the network rack. In the TOR mode, the switch is set on the top of the rack, and all servers in the rack are directly connected to the top switch through short jumpers, and then connected to the core switch from the uplink port of the switch through optical fibers. The TOR method changes the centralized wiring into a point-to-point wiring method, which greatly reduces the amount of wiring used.

At the same time, in the present invention, the TOR mode is adopted to set the first top-of-rack switch on the top of the parameter server rack, and the network ports of each parameter server are connected to the first top-of-rack switch; the second top-of-rack switch is set to work On the top of the server rack, the network ports of each working server are connected to the second top-of-rack switch. Realize the full connection between the first top-of-rack switch and each parameter server, and the full connection between the second top-of-rack switch and each working server. Install all the parameter servers in the cluster into the parameter server rack, and install all the working servers into the working server rack, which is completed by the first top-of-rack switch, the second top-of-rack switch, intra-group array waveguide grating routers, and inter-group array waveguide grating routers Data communication, that is, the concept of parameter server pool and working server pool is proposed to effectively improve the scalability of the network.

As shown in FIG. 4 , it is a structure diagram of the first top-of-rack switch and the second top-of-rack switch. Since both of them have the same structure, they are collectively referred to as top-of-rack switches when describing the structure later. The top-of-rack switch includes a switching module, a plurality of receiving modules and a plurality of sending modules, wherein the switching module includes a data packet processor, a scheduler, a broadcast module and a selector, which are shared by each receiving module and each sending module. The switching module also builds and records a flow table for mapping LAN addresses and sending ports.

After receiving the data packet to be forwarded by the receiving module, it is stored in the data packet buffer area, wherein, the data packet buffer area adopts a FIFO (First In First Out) data buffer, and FIFO means that when reading and writing data, it can only be written sequentially read out in sequence. In some embodiments, the data packets are Ethernet data packets.

In some embodiments, the data packet processor determines the destination of the data packet according to the Ethernet packet header of the data packet. If the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to the server in the corresponding rack. Exemplarily, the first top-of-rack switch in the parameter server rack collects data packets from the data packet buffer area, and the data packet processor judges that the destination of the data packets is just the parameter server 3 in the parameter server rack, Then the selector of the first top-of-rack switch directly forwards the data packet to the parameter server 3 in the parameter server rack.

If the destination of the data packet points to the inter-rack, the scheduler extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination of the data packet; The port forwards the packet to the sender. Exemplarily, the first top-of-rack switch in the parameter server rack collects data packets from the data packet buffer, and the data packet processor determines that the destination of the data packets is the working For the working server 3 in the server rack 1, the scheduler extracts the LAN address from the header of the data packet, compares the extracted LAN address with the address recorded in the flow table, and obtains the sending port corresponding to the LAN address. The selector of a top-of-rack switch sends the data packet to the corresponding sending port, and then transmits the data packet to the working server 3 in the working server rack 1 via the arrayed waveguide grating router in the group.

In some embodiments, when the destination and the data packet are in the same cluster, the data packet is forwarded to the corresponding destination server via the array waveguide grating router in the cluster; when the destination and the data packet are in different clusters, the data packet is forwarded to It is forwarded to the corresponding destination server via the intergroup array waveguide grating router.

In the initialization phase, the first top-of-rack switch and the second top-of-rack switch have just started up, and there are no entries in their corresponding flow tables. After accessing each parameter server or working server, the corresponding first top-of-rack switch or second top-of-rack switch starts to learn the LAN address. Exemplarily, the first top-of-rack switch associates the source address MAC _A in the data packet sent by the parameter server 1 in the parameter server rack with the port A that received the frame, and records the MAC _A in the flow table associatively - a. After the first top-of-rack switch and the second top-of-rack switch associate all data packet source addresses with corresponding ports, the flow table learning of the first top-of-rack switch and the second top-of-rack switch is completed, and data packet forwarding can begin.

In some embodiments, when the destination of the data packet points to the inter-rack, the scheduler extracts the LAN address from the header of the data packet, and the LAN address and its corresponding sending port are not recorded in the flow table. When the switch and/or the second top-of-rack switch does not know which port to send the packet to, the first top-of-rack switch and/or the second top-of-rack switch will send the packet to all sending ports and sound an alarm Manual review.

In some embodiments, both the first top-of-rack switch and the second top-of-rack switch are 25GbE top-of-rack switches. The 25GbE top-of-rack switch can realize nanosecond-level data distribution, speed up the flow of data to the optical switch, and avoid the disadvantages caused by the many-to-one architecture.

The arrayed waveguide grating-based all-optical switching distributed reinforcement learning system provided by the present invention will be further described below in conjunction with a specific embodiment.

Exemplarily, the arrayed waveguide grating-based all-optical switching distributed reinforcement learning system provided by the present invention can be divided into four working modes: establishing communication mode, parameter delivery mode, gradient aggregation mode and inter-group communication mode.

Establish communication mode: In each cluster, the parameter server and the working server establish a connection using optical circuit switching technology, and establish a communication channel through handshaking. Among them, the handshake refers to the process of establishing communication parameters between the receiving end and the sending end after the communication circuit is established and before the information transmission starts. Exemplarily, the parameters include information transmission rate, alphabet, parity check, interrupt process, etc. protocol properties. After the communication channel is established, the parameter server and the working server use the channel to perform parameter delivery and gradient aggregation operations in the parameter delivery mode and gradient aggregation mode.

Parameter delivery mode: In each cluster, the parameter server sends the parameters of the preset reinforcement learning model to the working server according to different subtasks. The parameter server sends the parameters to the first top-of-rack switch in the form of Ethernet data packets, and the receiving module of the first top-of-rack switch receives the data packets and stores them in the data packet buffer area; the data packet processing of the first top-of-rack switch The device judges the destination of the data packet according to the message header of the data packet, and judges that the destination of the data packet is inter-rack; within three working cycles of the FPGA, according to the address of the Ethernet message header of the data packet Switch the message to the cluster; the scheduler of the first top-of-rack switch extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination; the first top-of-rack switch sends the data The packet passes through the obtained sending port, and is converted into an optical signal of the corresponding wavelength by a fixed-wavelength laser; then the optical switching is completed by the arrayed waveguide grating router in the group according to the cyclic wavelength routing method, and finally the data packet is transmitted by the second rack to which each working server belongs After being demodulated by the top switch, it is sent to the corresponding working server.

Gradient aggregation mode: In each cluster, the working server feeds back the parameters obtained by the preset reinforcement learning model according to the corresponding subtask training to the parameter server, and performs gradient aggregation. The gradient result is sent to the second top-of-rack switch in the form of Ethernet data packet, and the receiving module of the second top-of-rack switch receives the data packet and stores it in the data packet buffer area; the data packet processor of the second top-of-rack switch according to The message header of the data packet judges the destination of the data packet, and it is judged that the destination of the data packet is between racks; within three working cycles of the FPGA, according to the address of the Ethernet message header of the data packet, the The text is switched to the cluster; the scheduler of the second top-of-rack switch extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination; the second top-of-rack switch passes the data packet through The obtained sending port is converted into an optical signal of the corresponding wavelength through a fixed-wavelength laser; then the optical switching is completed by the arrayed waveguide grating router in the group according to the cyclic wavelength routing method, and finally the data packet is transmitted by the first top-of-rack switch to which each parameter server belongs After demodulation, it is sent to the corresponding parameter server.

Inter-group communication mode: Each parameter server updates the parameters of the training model according to the gradient results fed back by each working server, and sends the updated parameters to the first top-of-rack switch in the form of Ethernet data packets, and the first top-of-rack switch The receiving module receives the data packet and stores it in the data packet buffer area; the data packet processor of the first top-of-rack switch judges the destination of the data packet according to the message header of the data packet, and determines the destination of the data packet It is between the racks; within three working cycles of the FPGA, the message is switched to the inter-cluster according to the address of the Ethernet header of the data packet; the scheduler of the first top-of-rack switch extracts from the header of the data packet LAN address, and query the flow table to obtain the sending port corresponding to the destination; the first top-of-rack switch passes the data packet through the obtained sending port, and converts the data packet into an optical signal of the corresponding wavelength through a fixed-wavelength laser; and then routes according to the cyclic wavelength The inter-group waveguide grating router completes the optical switching, and finally the data packet is demodulated by the first top-of-rack switch to which each parameter server belongs and then sent to the corresponding parameter server.

During implementation, each parameter server sends parameters to each working server, and each working server performs gradient aggregation to each parameter server. After the parameter server synchronizes the gradient aggregation, the parameter server starts the next task slice according to the training model requirements, that is, a new round The parameters are issued, and this cycle continues until the training model reaches the preset performance.

In some embodiments, when the first top-of-rack switch and the second top-of-rack switch both use 25GbE switches, it only takes 975 nanoseconds for the system to complete a parameter delivery and a gradient aggregation (that is, an iterative operation), compared with traditional An architecture that effectively accelerates the communication phase of distributed reinforcement learning.

The present invention also provides an all-optical switching distributed reinforcement learning method based on arrayed waveguide gratings. The method runs on the all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings as described above, as shown in FIG. 5 , in In one cycle, the method includes the following steps S101-S102:

Step S101: In each cluster, the parameter server in the parameter server rack delivers the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks.

Step S102: In each cluster, the working servers in each working server rack feed back the parameters obtained by the preset reinforcement learning model according to the corresponding subtask training to the parameter server, and perform gradient aggregation.

In some embodiments, the working servers in each working server rack feed back the parameters obtained by the preset reinforcement learning model according to the corresponding subtask training to the parameter server, and perform gradient aggregation, further including steps S103-S104:

Step S103: In each cluster, the parameter server in the parameter server rack performs gradient aggregation synchronously, and updates the parameters of the preset reinforcement learning model; between each cluster, the gradient aggregation of each cluster is exchanged through the inter-cluster array waveguide grating router The parameters of the preset reinforcement learning model obtained after.

Step S104: In each cluster, the parameter server in the parameter server rack re-establishes a subtask, sends new parameters to the working servers in each working server rack, and executes the next cycle.

Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the instructions in the memory. Stored computer instructions, when the computer instructions are executed by the processor, the device implements the steps of the aforementioned method.

An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method for deploying an edge computing server can be implemented. The computer readable storage medium may be a tangible storage medium such as random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

To sum up, the present invention provides an all-optical switching distributed reinforcement learning system and method based on arrayed waveguide gratings. In each cluster of the system, a parameter server rack and a working server rack are set, and each parameter server is loaded into a parameter The server rack, put each working server into the working server rack, construct the concept of parameter server pool and working server pool, and use the first top-of-rack switch, the second top-of-rack switch and the array waveguide grating router in the group to communicate, solve the problem of The scalability problem brought about by the centralized server greatly improves the flexibility of network expansion.

The arrayed waveguide grating router in the group is interconnected with the first top-of-rack switch and each second top-of-rack switch in a fully connected manner; the inter-group arrayed waveguide grating router is fully connected with the first Top-of-rack switch interconnect. Arrayed waveguide grating routers within a group and arrayed waveguide grating routers between groups cooperate with lasers to realize all-optical switching, avoiding additional delay and power consumption caused by photoelectric-optical conversion. At the same time, the fully connected topology architecture greatly reduces the number of network hops. Only one network hop is needed to meet gradient communication and nanosecond-level low-latency communication. while increasing linearly.

Those of ordinary skill in the art should understand that each exemplary component, system and method described in conjunction with the embodiments disclosed herein can be implemented by hardware, software or a combination of the two. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments employed to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves.

It is to be understood that the invention is not limited to the specific arrangements and processes described above and shown in the drawings. For conciseness, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after understanding the spirit of the present invention.

In the present invention, features described and/or exemplified for one embodiment can be used in the same or similar manner in one or more other embodiments, and/or can be combined with features of other embodiments or replace other Features of the implementation.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, various modifications and changes may be made to the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. An all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings, characterized in that it comprises: A plurality of clusters, each cluster includes a parameter server rack, an arrayed waveguide grating router in the group, and a plurality of working server racks; the parameter server rack includes a first top-of-rack switch and a plurality of parameter servers, and the network of each parameter server ports are all connected to the first top-of-rack switch; the working server rack includes a second top-of-rack switch and a plurality of working servers, and the network ports of each working server are connected to the second top-of-rack switch; the group The inner arrayed waveguide grating router is interconnected with the first top-of-rack switch and each second top-of-rack switch in a fully connected manner; in each cluster, each parameter server and each working server pass through the first top-of-rack switch, The second top-of-rack switch communicates with the arrayed waveguide grating router in the group; An arrayed waveguide grating router between groups, the arrayed waveguide grating router between groups is interconnected with the first top-of-rack switch in the parameter server rack in each cluster in a fully connected manner; the parameter servers in each cluster are connected through the corresponding first rack The top switch and the arrayed waveguide grating router between the groups are connected for communication; Among them, in each cluster, the parameter server in the parameter server rack sends the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks, and the parameters in each working server rack The working server feeds back the parameters obtained by the preset reinforcement learning model to the parameter server according to the corresponding subtask training, and performs gradient aggregation; between each cluster, the parameter servers in each parameter server rack pass through the inter-cluster array waveguide The grating router exchanges the parameters of the preset reinforcement learning model obtained after the gradient aggregation of each cluster. 2. The all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings according to claim 1, wherein the arrayed waveguide grating routers in the group and the arrayed waveguide grating routers between the groups adopt wavelength division multiplexing technology Establish a multi-channel communication link. 3. The all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings according to claim 1, wherein the arrayed waveguide grating routers in the group and the arrayed waveguide grating routers between the groups are all routed according to the cycle wavelength The way to route the optical signal to the corresponding output port. 4. The all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings according to claim 1, wherein the first top-of-rack switch and the second top-of-rack switch both include a switching module, a plurality of receiving module and a plurality of sending modules, the switching module includes a data packet processor, a scheduler, a broadcast module and a selector, and the switching module also constructs and records a flow table for mapping LAN addresses and sending ports. 5. The all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings according to claim 4, wherein the data packet processor judges the purpose of the data packet according to the header of the data packet to be forwarded land; When the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to the server in the corresponding rack; When the destination of the data packet points to the inter-rack, the scheduler extracts the LAN address from the header of the data packet, and queries the flow table to obtain the sending port corresponding to the destination of the data packet; The selector forwards the data packet to the sending port according to the obtained sending port. 6. The all-optical switching distributed reinforcement learning system based on arrayed waveguide gratings according to claim 5, wherein the selector forwards the data packet to the sending port according to the sending port obtained, further comprising: When the destination and the data packet are in the same cluster, forward the data packet to a corresponding destination server via the arrayed waveguide grating router in the cluster; When the destination and the data packet are in different clusters, the data packet is forwarded to a corresponding destination server via the inter-cluster array waveguide grating router. 7. The AWG-based all-optical switching distributed reinforcement learning system according to claim 5, wherein the scheduler extracts the LAN address from the header of the data packet, and queries the flow table to obtain The sending port corresponding to the destination of the data packet also includes: When the LAN address and its corresponding sending port do not exist in the flow table, flooding is performed and an alarm is issued. 8. An arrayed waveguide grating-based all-optical switching distributed reinforcement learning method, characterized in that the method is based on an arrayed waveguide grating-based all-optical switching distributed reinforcement learning method according to any one of claims 1 to 7. system, in a loop, the method includes: In each cluster, the parameter server in the parameter server rack sends the parameters of the preset reinforcement learning model to the working servers in each working server rack according to different subtasks; In each cluster, the working servers in each working server rack feed back the parameters obtained by training the preset reinforcement learning model according to corresponding subtasks to the parameter server, and perform gradient aggregation. 9. The all-optical switching distributed reinforcement learning method based on arrayed waveguide gratings according to claim 8, wherein the working servers in each working server rack train the preset reinforcement learning model according to corresponding subtasks to obtain The parameters of are fed back to the parameter server for gradient aggregation, including: In each cluster, the parameter server in the parameter server rack performs gradient aggregation synchronously, and after updating the parameters of the preset reinforcement learning model; between each cluster, exchange each cluster through the inter-cluster array waveguide grating router The parameters of the preset reinforcement learning model obtained after gradient aggregation; In each cluster, the parameter server in the parameter server rack re-establishes subtasks, sends new parameters to the working servers in each working server rack, and executes the next cycle. 10. A computer-readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the all-optical switching distribution based on the arrayed waveguide grating according to any one of claims 8 to 9 is realized. Steps in a reinforcement learning method.

Images