CN115714937A - All-optical switching distributed reinforcement learning system and method based on array waveguide grating - Google Patents
All-optical switching distributed reinforcement learning system and method based on array waveguide grating Download PDFInfo
- Publication number
- CN115714937A CN115714937A CN202211372521.0A CN202211372521A CN115714937A CN 115714937 A CN115714937 A CN 115714937A CN 202211372521 A CN202211372521 A CN 202211372521A CN 115714937 A CN115714937 A CN 115714937A
- Authority
- CN
- China
- Prior art keywords
- waveguide grating
- arrayed waveguide
- cluster
- rack
- reinforcement learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002787 reinforcement Effects 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000003287 optical effect Effects 0.000 claims abstract description 30
- 230000005540 biological transmission Effects 0.000 claims abstract description 7
- 230000006854 communication Effects 0.000 claims description 31
- 238000004891 communication Methods 0.000 claims description 30
- 230000002776 aggregation Effects 0.000 claims description 28
- 238000004220 aggregation Methods 0.000 claims description 28
- 238000012549 training Methods 0.000 claims description 21
- 239000000284 extract Substances 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 7
- 125000004122 cyclic group Chemical group 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000007547 defect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a full optical switching distributed reinforcement learning system and method based on arrayed waveguide grating, the system includes the arrayed waveguide grating router and multiple clusters among the clusters; each cluster comprises a parameter server rack, an in-cluster arrayed waveguide grating router and a plurality of working server racks; the parameter server rack comprises a first top switch and a plurality of parameter servers, and the network ports of the parameter servers are connected to the first top switch; the work server rack comprises a second top switch and a plurality of work servers, and the network ports of the work servers are connected to the second top switch; the intra-group arrayed waveguide grating router is interconnected with the first top-rack switch and each second top-rack switch in a full-connection mode; the inter-cluster arrayed waveguide grating router is interconnected in a fully connected manner with a first top-of-rack switch within each intra-cluster parameter server chassis. The distributed reinforcement learning method is suitable for distributed reinforcement learning, only needs one network hop for data transmission, and has the characteristics of low time delay and low loss.
Description
Technical Field
The invention relates to the technical field of optical switching, in particular to an all-optical switching distributed reinforcement learning system and method based on arrayed waveguide grating.
Background
With the rapid development and application of big data in various fields, machine learning is gradually becoming an important tool for processing big data. The core idea of machine learning is to train a model to fit input training data, deploy the trained model in corresponding applications, and expect that the model can accurately classify or predict data generated in the application running process. Today's machine learning applications often require complex models to process ultra-large scale data sets. The single machine deployment can not meet the requirement of large-scale machine learning training, and the distributed machine learning comes from the turn, and the training speed is greatly improved by utilizing a plurality of computing nodes to perform collaborative training in a distributed deployment mode. However, in the face of huge data, the problem that communication frequency between nodes is increased due to the fact that more data are transmitted between distributed machine learning nodes in unit time still exists, if some computing nodes do not receive data due to network congestion, the whole computer cannot enter next iteration in time, and finally the bottleneck of training task completion time is transferred from computing to network communication.
Reinforcement learning is one of the paradigms and methodologies of machine learning to describe and solve the problem of maximizing reports or achieving specific goals through learning strategies during interaction of an agent with the environment. Compared with algorithms commonly used in distributed machine learning, such as deep convolutional neural networks, cyclic neural networks, deep map convolutional neural networks, and the like, the distributed reinforcement learning training needs to generate a larger number of levels of iteration with smaller gradient aggregation. A typical reinforcement learning algorithm will produce 15 to 2000 million iterations, and therefore the delay of gradient communication in each iteration is a key factor affecting the performance of distributed reinforcement learning training.
One of the prior art is a switch architecture using a centralized parameter server, where the parameter server performs weight update by aggregating gradient parameters of each work server through an electrical switch. The scheme limits the expansibility of the distributed reinforcement learning system, and meanwhile, considering tens of millions of iterations, the electric exchange with limited bandwidth and energy consumption can bring about delay of hundreds of microseconds to ten milliseconds. And the other method is to adopt an All-reduce-based architecture, each work server divides the task into N subtasks, and the subtasks in the same sequence in each work server are sequentially subjected to cyclic gradient aggregation. Although the gradient aggregation of the scheme is performed in a non-centralized manner, the network hop count of the communication process increases linearly with the increase of the network size, and the delay of millisecond level also exists.
Disclosure of Invention
In view of this, embodiments of the present invention provide an all-optical switching distributed reinforcement learning system and method based on an arrayed waveguide grating, so as to eliminate or improve one or more defects in the prior art, and solve the problems that the prior art limits the expansibility of the distributed reinforcement learning system and the iteration delay of model training parameters is high.
In one aspect, the present invention provides an all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating, which is characterized by comprising:
a plurality of clusters, each cluster comprising a parameter server chassis, an intra-cluster arrayed waveguide grating router, and a plurality of working server chassis; the parameter server rack comprises a first top switch and a plurality of parameter servers, and the network ports of the parameter servers are connected to the first top switch; the working server rack comprises a second top switch and a plurality of working servers, and the network ports of the working servers are connected to the second top switch; the intra-group arrayed waveguide grating router is interconnected with the first top-of-rack switch and each second top-of-rack switch in a full-connection mode; in each cluster, each parameter server and each work server are in communication connection with the first top-of-rack switch, the second top-of-rack switch and the intra-cluster arrayed waveguide grating router;
the inter-cluster arrayed waveguide grating router is interconnected with the first top-of-rack switch in each intra-cluster parameter server rack in a full-connection mode; the parameter servers in each cluster are in communication connection through the corresponding first top-of-rack switch and the inter-cluster arrayed waveguide grating router;
in each cluster, a parameter server in a parameter server rack issues parameters of a preset reinforcement learning model to working servers in all working server racks according to different subtasks, and the working servers in all the working server racks feed back the parameters obtained by training the preset reinforcement learning model to the parameter server according to corresponding subtasks and perform gradient aggregation; and among the clusters, the parameter servers in the parameter server racks exchange the parameters of the preset reinforcement learning model obtained after gradient aggregation of the clusters through the inter-cluster arrayed waveguide grating router.
In some embodiments of the invention, the intra-group arrayed waveguide grating router and the inter-group arrayed waveguide grating router use wavelength division multiplexing to establish a multi-channel communication link.
In some embodiments of the present invention, the intra-group and inter-group arrayed waveguide grating routers each route optical signals to corresponding output ports in a cyclic wavelength routing manner.
In some embodiments of the invention, the first top-of-rack switch and the second top-of-rack switch each comprise a switching module, a plurality of receiving modules, and a plurality of sending modules, the switching module comprising a packet processor, a scheduler, a broadcasting module, and a selector, the switching module further constructing and recording a flow table for mapping local network addresses and sending ports.
In some embodiments of the invention, the packet processor determines the destination of a packet to be forwarded based on a header of the packet;
when the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to a server in the corresponding rack;
when the destination of the data packet points to the rack, the scheduler extracts a local area network address from a message header of the data packet and queries a flow table to obtain a sending port corresponding to the destination of the data packet; and the selector forwards the data packet to the sending port according to the obtained sending port.
In some embodiments of the present invention, the forwarding, by the selector, the packet to the sending port according to the obtained sending port further includes:
when the destination is in the same cluster as the data packet, forwarding the data packet to a corresponding destination server via the intra-cluster arrayed waveguide grating router;
when the destination is in a different cluster than the data packet, forwarding the data packet to a corresponding destination server via the inter-cluster arrayed waveguide grating router.
In some embodiments of the present invention, the scheduler extracts a local area network address from a header of the data packet, and queries a flow table to obtain a sending port corresponding to a destination of the data packet, further including:
and when the local area network address and the corresponding sending port are not in the flow table, flooding is carried out, and an alarm is sent out.
In another aspect, the present invention provides an all-optical switching distributed reinforcement learning method based on an arrayed waveguide grating, where the method runs on an all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating as described above, and in one cycle, the method includes:
in each cluster, a parameter server in a parameter server rack issues parameters of a preset reinforcement learning model to working servers in each working server rack according to different subtasks;
in each cluster, the working servers in the working server racks feed back the preset reinforcement learning model to the parameter server according to the parameters obtained by training the corresponding subtasks, and perform gradient aggregation.
In some embodiments of the present invention, the working servers in the respective working server racks feed back the parameters obtained by the preset reinforcement learning model according to the training of the corresponding subtasks to the parameter server, and perform gradient aggregation, further including:
in each cluster, synchronously performing gradient aggregation on parameter servers in a parameter server rack, and updating parameters of the preset reinforcement learning model; exchanging parameters of a preset reinforcement learning model obtained after gradient aggregation of the clusters among the clusters through the inter-cluster arrayed waveguide grating router;
in each cluster, the parameter server in the parameter server rack reestablishes the subtasks, issues the new parameters to the work servers in the work server racks, and executes the next cycle.
In another aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the program is executed by a processor to implement the steps of the all-optical switching distributed reinforcement learning method based on arrayed waveguide grating as described in any one of the above.
The invention has the beneficial effects that:
the invention provides an all-optical switching distributed reinforcement learning system and method based on arrayed waveguide gratings, wherein a parameter server rack and a working server rack are arranged in each cluster of the system, each parameter server is arranged in the parameter server rack, each working server is arranged in the working server rack, concepts of a parameter server pool and a working server pool are constructed, and a first overhead switch, a second overhead switch and an in-cluster arrayed waveguide grating router are utilized for communication, so that the problem of expansibility brought by a centralized server is solved, and the expansion flexibility of a network is greatly improved.
The intra-group arrayed waveguide grating router is interconnected with the first top-rack switch and each second top-rack switch in a full-connection mode; the inter-cluster arrayed waveguide grating router is interconnected in full connectivity with a first top-of-rack switch within each intra-cluster parameter server chassis. The intra-cluster arrayed waveguide grating router and the inter-cluster arrayed waveguide grating router are matched with a laser to realize all-optical switching, and extra time delay and power consumption caused by optical-to-electrical conversion are avoided. Meanwhile, the network hop count is greatly reduced by the fully-connected topology architecture, gradient communication can be achieved only by one network hop count, nanosecond-level low-delay communication is achieved, and the problem that the network hop count is linearly increased along with the increase of the network scale in the prior art is solved.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present invention are not limited to the specific details set forth above, and that these and other objects that can be achieved with the present invention will be more clearly understood from the detailed description that follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principle of the invention. In the drawings:
fig. 1 is a schematic overall structure diagram of an all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating in an embodiment of the present invention.
Fig. 2 is a schematic diagram of an intra-group structure of an all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating in an embodiment of the present invention.
Fig. 3 is a schematic structural diagram of an intra-group and inter-group arrayed waveguide grating routers according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a first top-of-rack switch and a second top-of-rack switch in an embodiment of the invention.
Fig. 5 is a schematic step diagram of an all-optical switching distributed reinforcement learning method based on an arrayed waveguide grating in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the following embodiments and the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and/or processing steps closely related to the scheme according to the present invention are shown in the drawings, and other details not so relevant to the present invention are omitted.
It should be emphasized that the term "comprises/comprising" when used herein, is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.
It is also noted that, unless otherwise specified, the term "coupled" is used herein to refer not only to a direct connection, but also to an indirect connection with an intermediate.
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similar parts, or the same or similar steps.
It should be emphasized that the step labels mentioned in the following are not limitations to the order of steps, but should be understood that the steps may be executed in the order mentioned in the embodiments, may be executed in a different order from the embodiments, or may be executed simultaneously.
In order to solve the problem that the prior art limits the expansibility of a distributed reinforcement learning system and the iteration delay is high, the invention provides an all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating, as shown in fig. 1, the system comprises a plurality of clusters and an inter-cluster arrayed waveguide grating router, specifically:
as shown in fig. 2, each cluster includes a parametric server chassis, an intra-cluster arrayed waveguide grating router, and a plurality of working server chassis. The parameter server rack comprises a first rack top switch and a plurality of parameter servers, and the network ports of the parameter servers are connected to the first rack top switch; the working server rack comprises a second top switch and a plurality of working servers, and the network ports of the working servers are connected to the second top switch; the intra-group arrayed waveguide grating router is interconnected with the first top-of-rack switch and each second top-of-rack switch in a full-connection mode; in each cluster, each parameter server and each work server are in communication connection through a first top-of-rack switch, a second top-of-rack switch and an intra-cluster arrayed waveguide grating router.
The inter-cluster arrayed waveguide grating router is interconnected with a first top-of-rack switch in each intra-cluster parameter server rack in a full-connection mode; and the parameter servers in each cluster are in communication connection through the corresponding first top-of-rack switch and the inter-cluster arrayed waveguide grating router.
In each cluster, a parameter server in a parameter server rack issues parameters of a preset reinforcement learning model to working servers in each working server rack according to different subtasks; feeding back parameters obtained by training a preset reinforcement learning model to a parameter server according to corresponding subtasks by a working server in each working server rack, and performing gradient aggregation; among clusters, parameter servers in the parameter server racks exchange parameters of a preset reinforcement learning model obtained after gradient aggregation of the clusters through an inter-cluster array waveguide grating router.
The invention realizes the flow gradient switching and the weight switching between the clusters by utilizing the intra-cluster array waveguide grating router and the inter-cluster array waveguide grating router.
The structure of the intra-cluster arrayed waveguide Grating Router is the same as that of the inter-cluster arrayed waveguide Grating Router, and the arrayed waveguide Grating Router is called an arrayed waveguiding Router in English.
Arrayed Waveguide Grating (AWG) is the first choice technology in Dense Wavelength Division Multiplexing (DWDM) systems. AWG is a passive planar waveguide device, and is an arrayed waveguide grating fabricated on a chip substrate by using Programmable Logic Controller (PLC) technology. Compared with Fiber Bragg Gratings (FBGs) and dielectric film filters (TTFs), the AWG has the advantages of high integration level, a large number of channels, small insertion loss, easiness in batch automatic production and the like. AWGs are commonly used in optical multiplexers in wavelength division multiplexing systems, by which light of many wavelengths can be combined into a single optical fiber, thereby increasing the propagation efficiency of the fiber network.
In some embodiments, the intra-group arrayed waveguide grating router and the inter-group arrayed waveguide grating router use 5 × 5 ports, and the specific structure is shown in fig. 3. The left side is provided with an input port 1, an input port 2, an input port 3, an input port 4 and an input port 5; the right side is an output port a, an output port b, an output port c, an output port d and an output port e; the middle is an arrayed waveguide grating. Each input port inputs optical signals with different wavelengths, and the corresponding optical signals are routed to the corresponding output ports according to a wavelength routing mode. The wavelength routing refers to selecting a route according to the wavelength of an optical signal when the optical signal passes through a network node. Illustratively, as shown in FIG. 3, λ 1 、λ 2 、λ 3 、λ 4 And λ 5 Optical signals of five wavelengths are respectively shown, and λ is input to the input port 3 as an example 1 、λ 2 、λ 3 、λ 4 And λ 5 Optical signals of five wavelengths, λ being the wavelength according to the respective wavelength 1 E is the output port of the optical signal; wavelength of λ 2 D is the output port of the optical signal; wavelength of λ 3 C is the output port of the optical signal; wavelength of λ 4 B is the output port of the optical signal; wavelength of λ 5 The output port of the optical signal of (2) is a. The optical signals of other input ports are the same.
In some embodiments, both the intra-group and the inter-group arrayed waveguide grating routers route optical signals to corresponding output ports in a cyclic wavelength routing manner. As shown in fig. 3, at a wavelength λ 3 For example, the input port 1 inputs the optical signal with the wavelength λ 3 The corresponding output port of the optical signal of (a); the input of the input port 2 has a wavelength λ 3 B is the corresponding output port; wavelength input from input port 3Is λ 3 C is the corresponding output port; the input port 4 inputs a wavelength λ 3 The corresponding output port of the optical signal of (1) is d; the input port 5 inputs the wavelength lambda 3 Corresponding to the output port e, forms a loop. Optical signals of other wavelengths are the same.
In some embodiments, the intra-cluster and inter-cluster arrayed waveguide grating routers establish the multi-channel communication link using wavelength division multiplexing techniques. Illustratively, as shown in FIG. 3, 25-channel communication links are established based on 5 wavelengths.
The first top-of-rack switch and the second top-of-rack switch have the same structure and are both realized by Field Programmable Gate Arrays (FPGAs), and are mainly responsible for interconnection of working servers or parameter servers in the racks and routing and forwarding in and among clusters.
The FPGA is a product developed based on programmable devices such as Programmable Array Logic (PAL) and Generic Array Logic (GAL). The circuit is a semi-custom circuit in the field of Application Specific Integrated Circuits (ASICs), not only solves the defects of custom circuits, but also overcomes the defect that the number of gate circuits of the original programmable device is limited.
Both the first Top-of-rack switch and the second Top-of-rack switch adopt a Top-of-rack wiring method (TOR). TOR is an extension of EOR (End of row)/MOR (Middle of row) mode, and all three are a structural design mode of the data center. Conventional racks are mainly based on EOR and MOR approaches, and similar centralized wiring is adopted, and the difference between EOR and MOR is mainly that the switch is located at different positions of the network rack. The TOR mode places the switch on top of the rack, and all servers in the rack are connected directly to the top switch by short jumpers and then connected from the switch's uplink ports to the core switch via optical fibers. The TOR mode changes centralized wiring into a point-to-point wiring mode, and the wiring usage amount is greatly reduced.
Meanwhile, in the invention, a TOR (time of arrival) mode is adopted to arrange a first top-rack switch at the top of a frame of a parameter server, and the network ports of all parameter servers are accessed to the first top-rack switch; and arranging a second top switch at the top of the working server rack, and accessing the net mouths of the working servers into the second top switch. The full connection between the first top rack switch and each parameter server is realized, and the full connection between the second top rack switch and each working server is realized. All the parameter servers in the cluster are installed into the parameter server rack, all the work servers are installed into the work server rack, and data communication is completed by the first top switch, the second top switch, the intra-cluster arrayed waveguide grating router and the inter-cluster arrayed waveguide grating router, so that concepts of a parameter server pool and a work server pool are provided, and the expansibility of a network is effectively improved.
As shown in fig. 4, a first top-rack switch and a second top-rack switch are configured as a structure diagram, and since the structures of the first top-rack switch and the second top-rack switch are the same, the first top-rack switch and the second top-rack switch are collectively referred to as top-rack switches when describing the structures. The top-of-rack switch comprises a switching module, a plurality of receiving modules and a plurality of sending modules, wherein the switching module comprises a data packet processor, a scheduler, a broadcasting module and a selector and is shared by each receiving module and each sending module. The switching module also constructs and records a flow table for mapping the local network address with the transmission port.
After receiving a data packet to be forwarded, a receiving module stores the data packet In a data packet buffer area, wherein the data packet buffer area adopts a data buffer of FIFO (First In First Out), and the FIFO indicates that only sequential writing and reading can be performed when data is read and written. In some embodiments, the data packet is an ethernet data packet.
In some embodiments, the packet processor determines the destination of the packet based on the ethernet header of the packet. If the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to the server in the corresponding rack. Illustratively, the first top-of-rack switch in the parameter server chassis collects the data packet from the packet buffer, and if the destination of the data packet is determined by the packet processor to be the parameter server 3 in the parameter server chassis, the selector of the first top-of-rack switch forwards the data packet directly to the parameter server 3 in the parameter server chassis.
If the destination of the data packet points to the rack, the scheduler extracts a local area network address from a message header of the data packet and queries a flow table to obtain a sending port corresponding to the destination of the data packet; and the selector forwards the data packet to the sending end according to the obtained sending port. Illustratively, a first top-of-rack switch in a parameter server rack collects a data packet from a data packet cache region, the destination of the data packet is a work server 3 in a work server rack 1 in the same cluster as the first top-of-rack switch, the scheduler extracts a local area network address from a message header of the data packet, compares the extracted local area network address with an address recorded in a flow table to obtain a sending port corresponding to the local area network address, and a selector of the first top-of-rack switch sends the data packet to the corresponding sending port and transmits the data packet to the work server 3 in the work server rack 1 through an intra-cluster arrayed waveguide grating router.
In some embodiments, when the destination is within the same cluster as the data packet, forwarding the data packet to the corresponding destination server via the intra-cluster arrayed waveguide grating router; when the destination is in a different cluster than the data packet, the data packet is forwarded to the corresponding destination server via the inter-cluster arrayed waveguide grating router.
In the initialization stage, the first top switch and the second top switch are started just, and no entry exists in the corresponding flow tables. After each parameter server or working server is accessed, the corresponding first top-of-rack switch or second top-of-rack switch starts to learn the local area network address. Illustratively, the first set top switch MAC addresses the source address in packets from the parameter server 1 within the parameter server chassis A Associates with the port A which received the frame and records the MAC in the flow table A -A. After the first top-rack switch and the second top-rack switch associate all the packet source addresses with the corresponding ports, the flow table learning of the first top-rack switch and the second top-rack switch is completed, and packet forwarding can be started.
In some embodiments, when the destination of the packet points to the inter-chassis, the scheduler extracts a local network address from a header of the packet, the local network address and a corresponding sending port are not recorded in a flow table, and the first top-of-rack switch and/or the second top-of-rack switch does not know to which port to send the packet, and the first top-of-rack switch and/or the second top-of-rack switch sends the packet to all sending ports and sends an alarm to prompt manual review.
In some embodiments, the first top-of-rack switch and the second top-of-rack switch each employ a 25GbE top-of-rack switch. The 25GbE shelf top switch can realize nanosecond-level data distribution, accelerate the flow of data to the optical switch and simultaneously avoid the defects caused by a many-to-one architecture.
The all-optical switching distributed reinforcement learning system based on the arrayed waveguide grating provided by the invention is further described below with reference to a specific embodiment.
Illustratively, the all-optical switching distributed reinforcement learning system based on the arrayed waveguide grating provided by the invention can be divided into four working modes: establishing a communication mode, a parameter issuing mode, a gradient aggregation mode and an inter-group communication mode.
Establishing a communication mode: in each cluster, the parameter server and the working server establish a connection using an optical circuit switching technique and establish a communication channel through handshaking. The handshake is a process of establishing communication parameters between the receiving end and the transmitting end after the communication circuit is established and before information transmission begins, and the parameters exemplarily include information transmission rate, alphabet, parity, interrupt process and other protocol characteristics. After a communication channel is established, the parameter server and the working server perform parameter issuing and gradient aggregation operations in a parameter issuing mode and a gradient aggregation mode by using the channel.
A parameter issuing mode: in each cluster, the parameter server issues the parameters of the preset reinforcement learning model to the work server according to different subtasks. The parameter server sends the parameters to the first top-rack switch in the form of Ethernet data packets, and a receiving module of the first top-rack switch receives the data packets and stores the data packets in a data packet cache region; a data packet processor of the first top-of-rack switch judges the destination of the data packet according to the message header of the data packet, and judges that the destination of the data packet is between racks; in three working periods of the FPGA, switching the message into a cluster according to the address of the Ethernet message header of the data packet; a dispatcher of the first top-of-rack switch extracts a local area network address from a message header of a data packet and queries a flow table to acquire a sending port corresponding to a destination; the first top-of-rack switch converts the data packet into an optical signal with a corresponding wavelength through the obtained sending port and the fixed wavelength laser; and finishing optical switching by the intra-group array waveguide grating router in a circulating wavelength routing mode, and finally sending the data packet to the corresponding working server after the data packet is demodulated by a second top-of-rack switch to which each working server belongs.
Gradient polymerization mode: in each cluster, the working server feeds back parameters obtained by training a preset reinforcement learning model according to corresponding subtasks to the parameter server, and performs gradient aggregation. The gradient result is sent to the second top-of-rack switch in the form of an Ethernet data packet, and a receiving module of the second top-of-rack switch receives the data packet and stores the data packet in a data packet cache region; the data packet processor of the second top-rack switch judges the destination of the data packet according to the message header of the data packet, and judges that the destination of the data packet is between the racks; in three working periods of the FPGA, switching the message into a cluster according to the address of the Ethernet message header of the data packet; a dispatcher of the second top-of-rack switch extracts a local area network address from a message header of the data packet and queries a flow table to acquire a sending port corresponding to a destination; the second top-of-rack switch converts the data packet into an optical signal with a corresponding wavelength through the obtained sending port and the fixed wavelength laser; and finishing optical switching by the in-group array waveguide grating router in a circulating wavelength routing mode, and finally sending the data packet to the corresponding parameter server after the data packet is demodulated by the first top-of-rack switch to which each parameter server belongs.
Inter-group communication mode: each parameter server updates the parameters of the training model according to the gradient result fed back by each working server, and sends the updated parameters to the first top-of-rack switch in the form of Ethernet data packets, and a receiving module of the first top-of-rack switch receives the data packets and stores the data packets in a data packet cache area; a data packet processor of the first top-of-rack switch judges the destination of the data packet according to the message header of the data packet, and judges that the destination of the data packet is between racks; in three working periods of the FPGA, switching the message to the cluster according to the address of the Ethernet message header of the data packet; a dispatcher of the first top-of-rack switch extracts a local area network address from a message header of a data packet and queries a flow table to acquire a sending port corresponding to a destination; the first top-of-rack switch converts the data packet into an optical signal with a corresponding wavelength through the obtained sending port and the fixed wavelength laser; and finishing optical switching by the inter-group train waveguide grating router in a circulating wavelength routing mode, and finally sending the data packet to the corresponding parameter server after demodulating the data packet by the first top-of-rack switch to which each parameter server belongs.
During implementation, each parameter server issues parameters to each working server, each working server performs gradient aggregation to each parameter server, and after the parameter servers perform synchronous gradient aggregation, the parameter servers start next task slicing according to the requirements of the training model, namely, a new round of parameter issuing, and the process is repeated until the training model reaches the preset performance.
In some embodiments, when the first top-rack switch and the second top-rack switch both employ 25GbE switches, the system only needs 975 nanoseconds to complete one parameter issue and one gradient aggregation (i.e., one iteration operation), which effectively speeds up the communication phase of distributed reinforcement learning compared to the conventional architecture.
The present invention also provides an all-optical switching distributed reinforcement learning method based on an arrayed waveguide grating, which operates on the above-mentioned all-optical switching distributed reinforcement learning system based on an arrayed waveguide grating, as shown in fig. 5, and in one cycle, the method includes the following steps S101 to S102:
step S101: in each cluster, the parameter server in the parameter server rack issues the parameters of the preset reinforcement learning model to the work servers in the work server racks according to different subtasks.
Step S102: in each cluster, the working servers in the working server racks feed back parameters obtained by training a preset reinforcement learning model according to corresponding subtasks to the parameter servers, and gradient aggregation is performed.
In some embodiments, the working servers in the respective working server racks feed back parameters obtained by training the preset reinforcement learning model according to the corresponding subtasks to the parameter server, and perform gradient aggregation, and the method further includes steps S103 to S104:
step S103: in each cluster, synchronously performing gradient aggregation on parameter servers in the parameter server racks, and updating parameters of a preset reinforcement learning model; and exchanging parameters of the preset reinforcement learning model obtained after gradient aggregation of the clusters among the clusters through the inter-cluster array waveguide grating router.
Step S104: in each cluster, the parameter server in the parameter server rack reestablishes the subtasks, issues the new parameters to the work servers in the work server racks, and executes the next cycle.
In accordance with the above method, the present invention also provides an apparatus comprising a computer device including a processor and a memory, the memory having stored therein computer instructions, the processor being configured to execute the computer instructions stored in the memory, the apparatus implementing the steps of the method as described above when the computer instructions are executed by the processor.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the foregoing edge computing server deployment method. The computer readable storage medium may be a tangible storage medium such as Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.
In summary, the invention provides an all-optical switching distributed reinforcement learning system and method based on an arrayed waveguide grating, wherein a parameter server rack and a work server rack are arranged in each cluster of the system, each parameter server is installed in the parameter server rack, each work server is installed in the work server rack, concepts of a parameter server pool and a work server pool are constructed, and a first top-of-rack switch, a second top-of-rack switch and an intra-cluster arrayed waveguide grating router are utilized for communication, so that the problem of expansibility brought by a centralized server is solved, and the expansion flexibility of a network is greatly improved.
The intra-group arrayed waveguide grating router is interconnected with the first top-rack switch and each second top-rack switch in a full-connection mode; the inter-cluster arrayed waveguide grating router is interconnected in a fully connected manner with a first top-of-rack switch within each intra-cluster parameter server chassis. The intra-cluster arrayed waveguide grating router and the inter-cluster arrayed waveguide grating router are matched with a laser to realize all-optical switching, and extra time delay and power consumption caused by optical-to-electrical conversion are avoided. Meanwhile, the network hop count is greatly reduced by the fully-connected topological structure, gradient communication can be met only by one-time network hop count, nanosecond-level low-delay communication is met, and the problem that the network hop count is linearly increased along with the increase of the network scale in the prior art is solved.
Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein may be implemented as hardware, software, or combinations of both. Whether this is done in hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link.
It is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.
Features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments in the present invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made to the embodiment of the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. An all-optical switching distributed reinforcement learning system based on arrayed waveguide grating is characterized by comprising:
a plurality of clusters, each cluster comprising a parameter server chassis, an intra-cluster arrayed waveguide grating router, and a plurality of working server chassis; the parameter server rack comprises a first top switch and a plurality of parameter servers, and the network ports of the parameter servers are connected to the first top switch; the working server rack comprises a second top switch and a plurality of working servers, and the network ports of the working servers are connected to the second top switch; the intra-group arrayed waveguide grating router is interconnected with the first top-of-rack switch and each second top-of-rack switch in a full-connection mode; in each cluster, each parameter server and each working server are in communication connection through the first top-of-rack switch, the second top-of-rack switch and the intra-cluster arrayed waveguide grating router;
the inter-cluster arrayed waveguide grating router is interconnected with the first top-of-rack switch in each intra-cluster parameter server rack in a full-connection mode; the parameter servers in each cluster are in communication connection through the corresponding first top-of-rack switch and the inter-cluster arrayed waveguide grating router;
in each cluster, a parameter server in a parameter server rack issues parameters of a preset reinforcement learning model to working servers in all working server racks according to different subtasks, and the working servers in all the working server racks feed back the parameters obtained by training the preset reinforcement learning model to the parameter server according to corresponding subtasks and perform gradient aggregation; and among the clusters, the parameter servers in the parameter server racks exchange the parameters of the preset reinforcement learning model obtained after gradient aggregation of the clusters through the inter-cluster arrayed waveguide grating router.
2. The all-optical switched distributed reinforcement learning system based on arrayed waveguide grating of claim 1, wherein the intra-group arrayed waveguide grating router and the inter-group arrayed waveguide grating router establish a multi-channel communication link using a wavelength division multiplexing technique.
3. The all-optical switching distributed reinforcement learning system based on the arrayed waveguide grating of claim 1, wherein the intra-group arrayed waveguide grating router and the inter-group arrayed waveguide grating router both route optical signals to corresponding output ports in a cyclic wavelength routing manner.
4. The arrayed waveguide grating-based all-optical switching distributed reinforcement learning system of claim 1, wherein the first top-of-rack switch and the second top-of-rack switch each comprise a switching module, a plurality of receiving modules, and a plurality of transmitting modules, the switching module comprises a packet processor, a scheduler, a broadcasting module, and a selector, and the switching module further constructs and records a flow table for mapping local area network addresses and transmitting ports.
5. The all-optical switching distributed reinforcement learning system based on arrayed waveguide grating of claim 4, wherein the packet processor determines the destination of the packet according to the header of the packet to be forwarded;
when the destination of the data packet points to the corresponding rack, the selector directly forwards the data packet to a server in the corresponding rack;
when the destination of the data packet points to the rack, the scheduler extracts a local area network address from a message header of the data packet and queries a flow table to obtain a sending port corresponding to the destination of the data packet; and the selector forwards the data packet to the sending port according to the obtained sending port.
6. The all-optical switching distributed reinforcement learning system based on arrayed waveguide grating according to claim 5, wherein the selector forwards the packet to a transmission port according to the acquired transmission port, further comprising:
when the destination is in the same cluster as the data packet, forwarding the data packet to a corresponding destination server via the intra-cluster arrayed waveguide grating router;
when the destination is in a different cluster than the data packet, forwarding the data packet to a corresponding destination server via the inter-cluster arrayed waveguide grating router.
7. The all-optical switching distributed reinforcement learning system based on arrayed waveguide grating of claim 5, wherein the scheduler extracts a local area network address from a packet header of the data packet and queries a flow table to obtain a sending port corresponding to a destination of the data packet, further comprising:
and when the local area network address and the corresponding sending port are not in the flow table, flooding and sending an alarm.
8. An all-optical-switching distributed reinforcement learning method based on an arrayed waveguide grating, which is operated on the all-optical-switching distributed reinforcement learning system based on the arrayed waveguide grating according to any one of claims 1 to 7, and in a cycle, the method comprises:
in each cluster, a parameter server in a parameter server rack issues parameters of a preset reinforcement learning model to working servers in each working server rack according to different subtasks;
in each cluster, the working servers in the working server racks feed back the preset reinforcement learning model to the parameter server according to the parameters obtained by training the corresponding subtasks, and perform gradient aggregation.
9. The all-optical switching distributed reinforcement learning method based on the arrayed waveguide grating according to claim 8, wherein the working servers in the respective working server racks feed back parameters obtained by the preset reinforcement learning model according to the training of the corresponding subtasks to the parameter servers, and perform gradient aggregation, and further comprising:
in each cluster, synchronously performing gradient aggregation on parameter servers in a parameter server rack, and updating parameters of the preset reinforcement learning model; exchanging parameters of a preset reinforcement learning model obtained after gradient aggregation of the clusters among the clusters through the inter-cluster arrayed waveguide grating router;
in each cluster, the parameter server in the parameter server rack reestablishes the subtasks, issues the new parameters to the work servers in the work server racks, and executes the next cycle.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the method for all-optical switching distributed reinforcement learning based on arrayed waveguide gratings according to any one of claims 8 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211372521.0A CN115714937A (en) | 2022-11-03 | 2022-11-03 | All-optical switching distributed reinforcement learning system and method based on array waveguide grating |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211372521.0A CN115714937A (en) | 2022-11-03 | 2022-11-03 | All-optical switching distributed reinforcement learning system and method based on array waveguide grating |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115714937A true CN115714937A (en) | 2023-02-24 |
Family
ID=85232194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211372521.0A Pending CN115714937A (en) | 2022-11-03 | 2022-11-03 | All-optical switching distributed reinforcement learning system and method based on array waveguide grating |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115714937A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803218A (en) * | 2017-03-17 | 2017-06-06 | 西安优盛信息技术有限公司 | A kind of big data tutoring system based on virtualization and cloud computing |
| CN111193971A (en) * | 2019-11-15 | 2020-05-22 | 西安电子科技大学 | Machine learning-oriented distributed computing interconnection network system and communication method |
| WO2020132169A1 (en) * | 2018-12-21 | 2020-06-25 | Nec Laboratories America, Inc. | Optical fiber sensing systems, methods, structures and applications |
| CN112188325A (en) * | 2019-07-01 | 2021-01-05 | 谷歌有限责任公司 | Reconfigurable computing platform using optical network with one-to-many optical switch |
| US20210081776A1 (en) * | 2019-09-12 | 2021-03-18 | International Business Machines Corporation | Configuration of an optical switch fabric using machine learning |
| CN115175027A (en) * | 2022-06-30 | 2022-10-11 | 苏州大学 | Ring service deployment method for all-optical switching data center based on span architecture |
-
2022
- 2022-11-03 CN CN202211372521.0A patent/CN115714937A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803218A (en) * | 2017-03-17 | 2017-06-06 | 西安优盛信息技术有限公司 | A kind of big data tutoring system based on virtualization and cloud computing |
| WO2020132169A1 (en) * | 2018-12-21 | 2020-06-25 | Nec Laboratories America, Inc. | Optical fiber sensing systems, methods, structures and applications |
| CN112188325A (en) * | 2019-07-01 | 2021-01-05 | 谷歌有限责任公司 | Reconfigurable computing platform using optical network with one-to-many optical switch |
| US20210081776A1 (en) * | 2019-09-12 | 2021-03-18 | International Business Machines Corporation | Configuration of an optical switch fabric using machine learning |
| CN111193971A (en) * | 2019-11-15 | 2020-05-22 | 西安电子科技大学 | Machine learning-oriented distributed computing interconnection network system and communication method |
| CN115175027A (en) * | 2022-06-30 | 2022-10-11 | 苏州大学 | Ring service deployment method for all-optical switching data center based on span architecture |
Non-Patent Citations (4)
| Title |
|---|
| YOUJIE LI 等: "Accelerating Distributed Reinforcement learning with In-Switch Computing", 《2019 ACM/IEEE 46TH ANNUAL INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE (ISCA)》 * |
| YUANZHI GUO 等: "Experimental Assessments of a AWGR Based Optical Switching Network for Distributed Reinforcement Learning Training", 《2022 20TH INTERNATIONAL CONFERENCE ON OPTICAL COMMUNICATIONS AND NETWORKS (ICOCN)》, pages 1 - 3 * |
| 丁京杰 等: "基于阵列波导光栅的光洗牌网络设计", 《 光通信技术》 * |
| 廖贵成 等: "《通信新技术与信息系统应用》", 电子科技大学出版社 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103441942B (en) | Based on data centre network system and the data communications method of software definition | |
| EP3224972B1 (en) | Passive optical-based data center networks | |
| Kachris et al. | A survey on optical interconnects for data centers | |
| Lu et al. | Flexible and scalable optical interconnects for data centers: Trends and challenges | |
| Bakopoulos et al. | NEPHELE: An end-to-end scalable and dynamically reconfigurable optical architecture for application-aware SDN cloud data centers | |
| US20150295756A1 (en) | Hybrid Optical/Electrical Interconnect Network Architecture for Direct-connect Data Centers and High Performance Computers | |
| Yan et al. | All-optical programmable disaggregated data centre network realized by FPGA-based switch and interface card | |
| CN102882811A (en) | Data center light interconnection network system based on array waveguide grating and communication method | |
| CN107465966A (en) | A kind of topology reconstruction control method for optical-fiber network | |
| CN114745227B (en) | Electric power business network slicing time delay calculation method and device based on FlexE and SPN technologies | |
| Sankaran et al. | A survey of hybrid optical data center network architectures | |
| KR20050020583A (en) | Multi-dimensional disconnected mesh switching network | |
| Rofoee et al. | Griffin: Programmable optical datacenter with SDN enabled function planning and virtualisation | |
| CN115714937A (en) | All-optical switching distributed reinforcement learning system and method based on array waveguide grating | |
| Yang et al. | Which can accelerate distributed machine learning faster: Hybrid optical/electrical or optical reconfigurable DCN? | |
| Shu et al. | Programmable OPS/OCS hybrid data centre network | |
| Lin et al. | Performance evaluation of wss-based all-optical spine-leaf data center network | |
| Yu et al. | Optical circuit switching enabled reconfigurable HPC network for traffic pattern | |
| Jin et al. | Sandwich Tree: A new datacenter network based on passive optical devices | |
| Hao | Research of Optical Interconnection Technology in Datacenter Networks | |
| Wang et al. | MNDM: MEMS-based N-dimensional modular hybrid data center network | |
| Yan et al. | Next Generation Exabit/s Data Center Network Supporting All Optical Circuit and Packet Switching | |
| US12114112B2 (en) | Degree switching configuration for network architecture | |
| Lone et al. | E-HOSA: Extended-Hybrid Optical Switch Architecture for Cloud Computing Data Centers | |
| Yan et al. | FOSphere: A Scalable and Modular Low Radix Fast Optical Switch Based Data Center Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230224 |