WO2021159928A1 - 逻辑节点的分布式签名决策系统及其方法 - Google Patents

逻辑节点的分布式签名决策系统及其方法 Download PDF

Info

Publication number
WO2021159928A1
WO2021159928A1 PCT/CN2021/072786 CN2021072786W WO2021159928A1 WO 2021159928 A1 WO2021159928 A1 WO 2021159928A1 CN 2021072786 W CN2021072786 W CN 2021072786W WO 2021159928 A1 WO2021159928 A1 WO 2021159928A1
Authority
WO
WIPO (PCT)
Prior art keywords
distributed
logical
logical node
node
signature
Prior art date
Application number
PCT/CN2021/072786
Other languages
English (en)
French (fr)
Inventor
李新奇
柳俊丞
袁进辉
Original Assignee
北京一流科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京一流科技有限公司 filed Critical 北京一流科技有限公司
Publication of WO2021159928A1 publication Critical patent/WO2021159928A1/zh
Priority to US18/076,171 priority Critical patent/US11818231B2/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63—Routing a service request depending on the request content or context
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/12—Discovery or management of network topologies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/12—Discovery or management of network topologies
    • H04L41/122—Discovery or management of network topologies of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/509—Offload

Definitions

  • the present disclosure relates to a data processing technology. More specifically, the present disclosure relates to a distributed signature decision system and method for logical nodes of a distributed data processing system.
  • the purpose of the present invention is to provide a solution to at least the above-mentioned problems.
  • the present disclosure provides a method for configuring a logical node in a neural network to reduce the intermediate parameters of job data in different distributed data.
  • a logical node distributed signature decision-making system for a distributed data processing system including: an initial logical node generating component, which receives task configuration data input by a user, and generates The initial logical node topology diagram of the distributed data processing system, wherein the source logical node has a designated logical distributed signature and each initial logical node is attached with a candidate logical distributed signature set based on the task configuration data, the candidate logical
  • Each logical distributed signature in the distributed signature set specifies the distributed descriptor of each input tensor of the initial logical node to which it belongs and the distributed descriptor of each output tensor; and the logical distributed signature selection component
  • the distributed descriptor of the output terminal of each upstream logical node that has uniquely determined the logical distributed signature, for each candidate logical distributed signature of the current logical node, based on the data of the device set to be distributed in parallel by each upstream logical node The amount of data in the device set that
  • the logical distributed signature selection component calculates the required data transmission cost for each input end of the logical node with multiple input ends and The sum of the costs of the data to be transmitted for all input ends is taken as the total cost of the data transmitted when the current logical node uses the corresponding candidate logical distribution signature.
  • the uniquely determined logical distributed signature determines the distribution mode of the input tensors of all input terminals of the current logical node.
  • the distributed descriptor includes a segmentation tensor descriptor, a broadcast tensor distribution descriptor, and a part and tensor descriptor.
  • the split tensor descriptor indicates a tensor split mode.
  • a logical node distributed signature decision method for a distributed data processing system including: an initial logical node generating step, receiving task configuration data input by a user, and generating a An initial logical node topology diagram of an integrated data processing system, where the source logical node has a designated logical distributed signature and each initial logical node is attached with a set of candidate logical distributed signatures based on the task configuration data, the candidate logical distributed signature
  • Each logical distributed signature in the set specifies the distributed descriptor of each input tensor of the initial logical node to which it belongs and the distributed descriptor of each output tensor; and the logical distributed signature selection step, according to each The distributed descriptor of the output terminal of the upstream logical node that has uniquely determined the logical distributed signature, for each candidate logical distributed signature of the current logical node, based on the data volume of the device set to be distributed in parallel by each upstream logical node, and the current The logical node
  • the logical distributed signature selection step includes calculating the cost of data transmission required for each input terminal of a logical node with multiple input terminals And the sum of the cost of the data required to be transmitted at all input ends is taken as the total cost of the data transmitted when the current logical node uses the corresponding candidate logical distribution signature.
  • the uniquely determined logical distributed signature determines the distribution mode of the input tensors of all input terminals of the current logical node.
  • the distributed descriptor includes a segmentation tensor descriptor, a broadcast tensor distribution descriptor, and a part and tensor descriptor.
  • the split tensor descriptor indicates a tensor split mode.
  • the amount of data exchange between different computing devices in the data processing process of the static distributed data processing system can be minimized from a global perspective, thereby reducing This effectively reduces the adverse effects of data exchange on actual computing operations.
  • Fig. 1 shows a schematic diagram of a logical node distributed signature decision system 100 for a distributed data processing system according to the present disclosure.
  • FIG. 2 shows a schematic structural diagram of the logical node distributed signature decision system 100 of the distributed data processing system according to the present disclosure to decide the SBP signature.
  • FIG. 3 illustrates a first schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 4 illustrates a second schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 5 illustrates a third schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 6 illustrates a fourth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 7 illustrates a fifth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 8 illustrates a sixth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • first, second, third, etc. may be used in this disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.
  • one of the two possible devices may be referred to as the first logical distributed signature or the second logical distributed signature.
  • the two possible devices The other one can be called the second logical distributed signature or the first logical distributed signature.
  • the word "if” as used herein can be interpreted as "when” or "when” or "in response to determination”.
  • Deep learning is essentially a type of feature learning. From this perspective, deep learning can be directly applied to extract features from raw data.
  • the autoencoder is one of the important models to realize the feature extraction function.
  • Fig. 1 shows a schematic diagram of a logical node distributed signature decision-making system 100 for a static distributed data processing system according to the present disclosure.
  • the distributed signature decision system 100 includes an initial logical node generation component 110 and a logical distributed signature selection component 120.
  • the initial logical node generating component 110 receives task configuration data input by the user, and generates an initial logical node topology diagram 111 for the static distributed data processing system.
  • the static distributed data processing system will automatically decompose the job into many tiny job tasks based on the job description input by the user. These many tiny job tasks are composed of various operation components, which are used as logic
  • the nodes are interconnected back and forth to form a preliminary tensor processing neural network topology.
  • Each layer of these neural networks contains a large number of logical nodes, and two adjacent layers of neural networks are connected to each other, thereby providing guidance for the placement of the execution body (PLACEMENT) that performs actual job processing in the distributed data processing system.
  • Fig. 1 only schematically shows a simple initial logical node topology diagram 111, in which nodes A, B, C, D, E, F, L, and K are shown. Others not shown are replaced by omission. In actual data processing, the initial node topology 111 will be more complicated.
  • the initial logical node topology diagram 111 contains basic computing nodes that implement the computing tasks described by the user. This method of generating the initial logical node topology map 111 belongs to the conventional technology in the field, and therefore will not be repeated here.
  • Each initial logical node in the initial logical node topology diagram 111 contains multiple SBP signatures.
  • the source logical node that has been configured with SBP signature by the user or the initial logical node whose unique SBP signature is determined based on the user’s task description, such as initial logical nodes A, E, and C they only have unique SBP signatures, such as initial logic SBP-5 of node A, SBP-2 of initial logical node C, and SBP-3 of initial logical node E.
  • the initial logical node usually contains some inherent candidate SBP signatures.
  • the initial logical node B in Fig. 1 has multiple candidate SBP signatures, for example three, including SBP-1, SBP-2, and SBP-3. Other initial logical nodes also have different candidate SBP signatures, which are not listed here. Different initial logical nodes will have different fixed candidate SBP signatures according to their specific operations.
  • the SBP signature according to the present disclosure is a signature applied in a distributed data processing system.
  • distributed data processing systems there are often data parallelism, model parallelism, mixed parallelism, and stream parallelism. Therefore, there are often tasks of adjacent logical nodes that will be deployed on different computing devices at the same time.
  • intermediate parameters are exchanged between various computing devices, which will cause a lot of transmission overhead. Therefore, in order to reduce the data transmission overhead, it is necessary to further generate more logical nodes on the basis of the initial logical node topology diagram 111 to improve the logical node topology diagram, especially to reduce the transmission overhead between upstream and downstream logical nodes. Minimize the changes brought about by the data distribution method of upstream and downstream logical nodes.
  • the present disclosure specifies a logical distributed signature for each logical node.
  • the logical distributed signature is the signature of the logical node using the distributed descriptor of the tensor.
  • the distributed descriptor of each tensor describes the distribution method of each tensor in the entire computing system, mainly including partitioning ( SPLIT) tensor descriptor, broadcast (BROADCAST) tensor descriptor and partial value (PARTIAL VALUE) tensor descriptor.
  • the split (SPLIT) tensor descriptor is to describe the way to split a tensor, for example, a data block is divided in a specified dimension according to the user's description, and distributed to different computing devices for specified calculations deal with. If a data block is a two-dimensional data block, when the data block is cut in its 0th dimension, the distributed descriptor of the data tensor of a batch of data formed by the data block is S(0), then each The distributed descriptors for each logical data block to obtain this data tensor at its input are all S(0).
  • a data block is a two-dimensional data block
  • the distributed descriptor of the data tensor of a batch of data formed by the data block is S(1)
  • the distributed descriptor for each logical data block to obtain this data tensor at its input is S(1).
  • the dimensions of the task data to be processed are more dimensions, there will be more distributed descriptors, such as S(2), S(3)... and so on.
  • Such mentioned data can be processed data or models. If the data itself is cut, data parallel processing is formed on the distributed data processing system, and if the model is divided, the model parallel processing is formed on the distributed data processing system.
  • the tensor descriptor in the actual data processing process, if the data size of a tensor is T, the tensor will be distributed to four computing cards for data In parallel computing, the amount of data allocated on each card is one-quarter of the data, and the amount of data on the entire four cards is T.
  • Broadcast (BROADCAST) tensor descriptor is used to describe the way a tensor is published in a distributed system by broadcasting.
  • model data is usually broadcast to various computing devices, so broadcast data input to logical nodes is described using broadcast tensor descriptors.
  • broadcast tensor descriptors In the actual data processing process, the data block size of the broadcasted data on each actual computing card is the same.
  • the PARTIAL VALUE tensor descriptor indicates that the input or output tensor of a logical node is the partial value of multiple tensors of the same type. These partial values include partial sum (Ps), partial product (Pm), partial "and" result, partial maximum, and partial minimum. Since data is usually processed in parallel for the purpose of data processing, the processing of data on different devices is the processing of part of the data. For example, if some tensors are S(0) or S(1), the result tensor obtained on some computing devices is S(0), and the result tensor on these partial computing devices is combined into a partial value tensor. Combining similar data on all devices is the final output.
  • the distributed descriptors of the various tensors above represent the distribution of these tensors in the distributed computing system, and whether these tensors are used as the input and output of logical nodes, their respective distribution methods also describe the pairs of logical nodes.
  • the distribution description of the operating data For the convenience of description, the present disclosure refers to this distributed descriptor as "SBP descriptor" for short.
  • the initial logical nodes of the present disclosure that is, some computing nodes, also have various input and output data distributed descriptors, and these input and output distributed descriptors form A kind of signature to the logical node, that is, the signature of the operation logical node using the distributed descriptor of the tensor.
  • the English initials of these three distributed descriptors are used to abbreviate this signature as "SBP signature".
  • this descriptor will include at least three types of S(0), B, and P. If there are multiple segmentation methods for data and models, each additional segmentation method will add a descriptor. For each logical node, its signature includes various combinations of these descriptors. Therefore, in the distributed system according to the present disclosure, there are at least three types of distributed descriptors, and usually there are four types of distributed descriptors, for example, the following four SBP descriptors, S(0), S(1), P, and B. Depending on the number of tensor dimensions, there can be more distributed descriptors.
  • SBP signatures can be formed according to the arrangement and combination of input and output. Some examples of SBP signatures are listed below: (S(0), B) ⁇ S(0), (S(1), B) ⁇ S(1), P ⁇ P, B ⁇ B, (S(0) ), S(1)) ⁇ P, S(0) ⁇ P, S(0) ⁇ S(0), S(0) ⁇ S(1), P ⁇ B and so on. All SBP signatures are the result of a combination of various SBP descriptors. For a matrix multiplication logic node, if its input tensor is cut on the first dimension, its output result tensor is also cut on the first dimension.
  • S, B, and P are descriptors used to describe the distribution of data blocks in the data processing system, and the SBP signature uses multiple SBP descriptors to describe the task operations of logical nodes.
  • Each data block can have multiple SBP descriptors, and the operation mode represented by each logical node can be multiple SBP signature scenarios.
  • SBP-1 shown in Figure 1 can be (S(0), B) ⁇ S(0)
  • SBP-2 can be (S(1), B) ⁇ S(1).
  • different signature forms can have different numbers. The numbers given here are only for the convenience of description, and do not mean that each signature needs to be assigned a number. There can be no number at all.
  • the different forms of signatures are different from each other. They can be distinguished from each other without a number.
  • the SBP signature as described above can be given to each initial logical node based on the task description used.
  • the usual task logic nodes are some arithmetic operation nodes, which perform specific arithmetic operations, so they have specific candidate SBP signatures. It should be pointed out that not every task logic node has the same SBP signature.
  • the input tensor of the SBP signature of the task logic node that performs the multiplication operation does not contain the part and the tensor, so the SBP description of its input tensor The symbol does not contain the distributed descriptor P.
  • the candidate SBP signature of the task logic node that performs the addition operation it can include any combination of various SBP descriptors with each other or between themselves.
  • each initial logical node is attached with a candidate logical distributed signature set based on the task configuration data.
  • Each logical distributed signature in the candidate logical distributed signature set specifies the distributed descriptor of each input tensor of the initial logical node to which it belongs and the distributed descriptor of each output tensor.
  • each logical node in the initial logical node topology diagram 111 will use which SBP signature determined tensor or which distributed tensor and input What kind of distributed tensor needs to be further determined.
  • the logical distributed signature selection component 120 of the logical node distributed signature decision system 100 starts from the source logical node in the initial logical node topology diagram 111, and starts from the source logical node in the current logical node (for example, logical node B).
  • the transmission data amount estimation unit 121 is based on the distribution of output ends corresponding to the input ends of logical node B of all upstream logical nodes of the logical node B For each candidate logical distributed signature of logical node B, the calculation transforms the distributed descriptor of the tensor at the output of each upstream logical node into one of the candidate logical distributed signatures at the corresponding input of logical node B. The cost of the transmitted data required by the tensor distributed descriptor. As shown in Figure 1, the logical node B has many candidate SBP signatures, such as SBP-1, SBP-2, and SBP-3.
  • the possible form of SBP-1 is the signature of (S(1), B) ⁇ S(1) or (S(1), P) ⁇ S(1)
  • the signature of the initial logical node A is SBP-5
  • the possible form is, for example, the signature of (S(0), B) ⁇ S(0)
  • the possible form of the signature SBP-3 of the initial logical node E is, for example, B ⁇ B or S(0) ⁇ P.
  • the left side of the arrow is the distributed descriptor of the input tensor
  • the right side of the arrow is the distributed descriptor of the output tensor.
  • tensor with distribution descriptor S(0) will be referred to as “S(0) tensor” in the following, and “tensor with distribution descriptor B” will be referred to as “B tensor”. "The tensor whose distribution descriptor is P” is simply referred to as “P tensor”, and so on.
  • the candidate signature SBP-1 of logical node B (ie "(S(0), S(1)) ⁇ P") is selected as a certain signature, it corresponds to the input tensor of the first input of the output of node E
  • the distribution descriptor of must be S(0), that is, the first input must obtain an S(1) tensor, and the distribution descriptor of the input tensor corresponding to the second input of the output of node A must be S(0), that is, the second input must obtain an S(0) tensor.
  • the P of the output tensor distribution descriptor of node A does not match the input tensor distribution descriptor S(0) of the first input of node B.
  • the output conversion is usually performed during the actual operation, and this conversion process It is usually necessary to obtain part of the data located on another computing device to form the data required by the input of the current logical node together with the locally available data, so as to conform to the distributed descriptor of the data tensor at the input of the current logical node.
  • This process of obtaining part of data from another device will generate relatively large data transmission overhead or transmission cost. Therefore, choosing different signatures for the current logical node will produce different data transmission overheads or costs.
  • the transmission data amount estimation unit 121 estimates the data transmission overhead that each candidate signature will generate for each logical node with an undetermined signature. For example, for the logical node B, the three candidate SBP signatures are respectively estimated for the data transmission cost of the logical node B when one of the SBP signatures is used. For logical node B, selecting any candidate SBP signature can achieve its operational tasks. However, when it uses different SBP signatures, the data transmission costs generated by its operation are different. Therefore, in order to minimize the cost of data transmission in the data processing process, it is necessary to select the signature with the smallest amount of data transmission from the candidate signatures of each logical node as the signature in the actual running process.
  • the logical node A may be the source node, and its SBP signature can be generated by the user configuration, or it can be based on the user’s description of the task.
  • the SBP signature of logical node A is generated, or the SBP signature of logical node A has been basically determined according to the scheme of the present disclosure.
  • the descriptor of the output tensor of the SBP signature of logical node A is S(0).
  • the logical node B in the initial logical node topology diagram 111 it has many candidate SBP signatures, which may include (S(1),B) ⁇ S(1), B ⁇ P, S(1)) ⁇ P, And P ⁇ B, etc.
  • S(0) the distribution descriptor of the output tensor of logical node A
  • P the corresponding input tensor distribution descriptor that node B can select can be S (1), B and P.
  • the SBP signatures of the downstream logical nodes are also based on the logical distributed descriptor (SBP descriptor) of the output tensor of the upstream logical node and the candidate logic of the downstream and upstream logical nodes.
  • SBP descriptor logical distributed descriptor
  • the cost of data transmission between logical distributed descriptors (SBP descriptors) corresponding to the input tensor of the distributed signature is finally selected and determined.
  • the candidate SBP signature of a logical node means that the respective SBP descriptors of each input and output data block of the logical node are also determined, so as to calculate or estimate The total cost of data transmission of the current logical node is calculated, and the candidate logical distributed signature with the smallest total cost is used as the logical distributed signature of the current logical node.
  • the logical distributed descriptor of the input terminal of which signatures in the candidate signature of the current logical node is consistent with the logical distributed descriptor of the output tensor of the upstream logical node
  • the logical distributed description can be selected first The candidate logical distributed signature of the symbol, unless the logical distributed descriptors of the other input tensors of the candidate logical distributed signature will cause the final total cost to be greater.
  • FIG. 2 shows a schematic structural diagram of the logical node distributed signature decision system 100 of the distributed data processing system according to the present disclosure to decide the SBP signature.
  • Fig. 2 is an enlarged schematic diagram of the relationship between nodes A, B and E in Fig. 1.
  • the distribution descriptor of the output tensor of the determined SBP signature SBP-3 of the logical node E is S(0)
  • the output tensor of the determined SBP signature SBP-5 of the logical node A is
  • the distribution descriptor for the input tensor is P
  • one of the candidate SBP signatures of logical node B, SBP-2 is (S(1), S(0)) ⁇ P.
  • the SBP descriptor of the input tensor corresponding to the SBP descriptor S(0) of the output tensor of logical node B of logical node B is S(1), and the difference between the output tensor of logical node B and logical node A
  • the SBP descriptor of the input tensor corresponding to the SBP descriptor P is S(0).
  • the tensor distribution of one of its inputs needs to be transformed from the SBP descriptor S(0) of the output tensor of logical node E to S(1 ) And transform the tensor distribution of its other input from the SBP descriptor P of the output tensor of logical node A to S(0). This transformation will produce data exchange in the actual data processing process.
  • FIG. 3 illustrates a first schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • SBP-2 of task node B shown in Figure 2
  • the tasks of input source task nodes A and E and the receiving sink node B are all distributed on the same device set. As shown in Figure 3, they are all distributed on the computing cards GPU 0 and GPU 1. Although only two computing cards are shown here, in fact, the source task node and the sink task node can be distributed on more cards or on different device sets.
  • Figure 3 shows that the S(0) descriptor tensor of the task of task node E in Figure 2 is distributed on two computing cards.
  • the input terminal of task node B needs to obtain the tensor of S(0) descriptor.
  • the data exchange process under the circumstances.
  • the task node of task node B distributed on GPU 1 wants to obtain S(1), it needs to directly obtain half of the tensor distributed on GPU 1 described by the S(0) descriptor of task node E. Outside (using the solid arrow to show the acquisition process of this data part), it is also necessary to add the other half of the tensor distributed on GPU 0 described by the S(0) descriptor of task node E (using the dotted arrow to show This part of the data acquisition process). If the size of the logical data block is T 1 , the amount of data transmitted from the logical data block of task node E of GPU 0 to task node B and distributed on the task node of GPU 1 is T 1 /2.
  • T 1 (T 1 /2+ T 1 /2).
  • T 1 is the size of the logical data block distributed on the source node.
  • the size of the logical data block is S(0).
  • the size of the data block distributed in the shaded part on each card is one half of the entire tensor.
  • FIG. 4 illustrates a second schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • SBP-2 of task node B shown in FIG. 2 it is assumed to be (S(1), S(0)) ⁇ P.
  • the tasks of the input source task nodes A and E and the received sink node B are all distributed on the same device set. As shown in FIG. 4, they are all distributed on the computing cards GPU 0, GPU 1, and GPU 2. Although three calculation cards are shown here, this is just for example. It can also be two cards as shown in FIG. 3. In fact, the source task node and the sink task node can be distributed to more cards or to different device sets.
  • Figure 4 shows the case where the P descriptor tensor of the task of task node A in Figure 2 is distributed on two computing cards.
  • the input of task node B wants to obtain the tensor of S(0) descriptors. Data exchange process.
  • each of the three cards is distributed with a partial value tensor P.
  • Ps is used here to represent a partial and tensor as a description example.
  • task node B distributed on GPU 0 needs to obtain the S(0) tensor and also needs to supplement the task node from GPU 1
  • the amount of data transmitted by the node is T 2 /3.
  • task node B distributed on GPU 1 wants to obtain the S(0) tensor, it also needs to supplement the amount of data transferred from the logical data block of task node A on GPU 0 to task node B distributed on GPU 1 T 2 /3 and the amount of data T 2 /3 transferred from the logical data block of task node A on GPU 2 to task nodes of task node B distributed on GPU 1.
  • task node B distributed on GPU 2 wants to obtain the S(0) tensor, it also needs to supplement the data transmission from the logical data block of task node A on GPU 1 to the task node distributed on GPU 2 of task node B.
  • the amount T 2 /3 and the data amount T 2 /3 transferred from the logical data block of task node A on GPU 0 to the task node distributed on GPU 2 of task node B. Therefore, the amount of data transmission in the actual data processing process from the P distributed tensor to the S(0) distributed tensor shown in Figure 4 is 2T 2 (T 2 /3+T 2 /3+T 2 /3+T 2 /3+T 2 /3+T 2 /3+T 2 /3).
  • the number of computing cards distributed by the task node is two.
  • the amount of data transmission (T 2 /2+T 2 /2).
  • the data transmission cost required to select the signature SBP-2 signature (for example, signature (S(1), S(0)) ⁇ P) is two inputs The sum of the transmission cost of the end.
  • the total amount of data that the task node needs to transmit is T 1 + T 2 .
  • the transmission cost estimated by the transmission data amount estimation unit 121 for the candidate signature SBP-1 of the logical node B needs to include the transmission cost of the two input ends of the candidate signature.
  • Table 1 (The distributed device set of the source task node and the sink task node are exactly the same, and the number of cards is K).
  • FIG. 5 illustrates a third schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • the device set of the source node is completely different from the device set of the sink node. That is, the source task node E is distributed on GPU 0 and GPU1, and the sink task node B is distributed on the computing card GPU 2 and GPU 3. If the size of the logical data block distributed on each computing card is T 3 , the amount of data that needs to be transmitted is 2T 3 .
  • FIG. 6 illustrates a fourth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • the device set of the source node is completely different from the device set of the sink node. That is, the source task node A is distributed on GPU 0, GPU1, and GPU 2, and the sink task node B is distributed on the computing cards GPU 3, GPU 4, and GPU 5.
  • each of the three cards has a partial value tensor P.
  • Ps is used to represent a part and a tensor as a description example.
  • the amount of data that needs to be transmitted is 9 1/3 T 4 , that is, 3T 4 . If the number of computing cards in the task set distributed by the source task node is 2, the amount of data that needs to be transmitted is 2T 4 . If the number of computing cards in the task set distributed by the source task node A is Ks, then the amount of data transmission is Ks ⁇ T 4 .
  • FIG. 7 illustrates a fifth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • FIG. 8 illustrates a sixth schematic diagram of the transmission data amount estimation unit 121 according to the present disclosure in which the data transmission amount generated between the tensors of different distributed descriptors is estimated.
  • the device set of the source node is not exactly the same as the device set of the sink node. That is, the source task node A is distributed on GPU 0, GPU1 and GPU 2, and the sink task node B is distributed on the computing cards GPU 1, GPU 2 and GPU 3.
  • each of the three cards has a partial value tensor P.
  • Ps is used to represent a part and a tensor as a description example.
  • the amount of data that needs to be transmitted is 7 1/3 T 4 , that is, 7/3 T 4 .
  • the transmission data volume estimation unit 121 traverses all candidate signatures SBP-1, SBP-2, and SBP-3 of the logical node B in the above-mentioned manner, and obtains the transmission cost for each signature. Subsequently, the total transmission data comparison unit 122 compares the transmission cost under each candidate signature, and obtains the minimum transmission cost of the logical node to be determined, such as logical node B. Finally, the SBP signature determining unit 123 determines the candidate SBP signature corresponding to the minimum transmission cost as the final SBP signature of the logical node B.
  • the logical node topology map output component 130 outputs the final logical node topology map 131 based on the SBP signature determined by the SBP signature determining unit 123 for each logical node.
  • Each logical node that constitutes the logical node topology map 131 is accompanied by only one SBP signature, or each logical node clearly specifies the distribution mode or distribution descriptor of each input tensor, and uniquely determines the distribution mode or distribution descriptor of its input tensor.
  • the amount of data exchange between different computing devices in the process of processing data in the distributed data processing system can be minimized from a global perspective, thereby reducing
  • the overhead incurred in the data interaction process effectively reduces the adverse effects of data exchange on actual computing operations, reduces the waiting time of computing operations, and speeds up data processing.
  • the purpose of the present disclosure can also be realized by running a program or a group of programs on any computing device.
  • the computing device may be a well-known general-purpose device. Therefore, the purpose of the present disclosure can also be achieved only by providing a program product containing program code for implementing the method or device. That is, such a program product also constitutes the present disclosure, and a storage medium storing such a program product also constitutes the present disclosure.
  • the storage medium may be any well-known storage medium or any storage medium developed in the future.
  • each component or each step can be decomposed and/or recombined.
  • These decomposition and/or recombination should be regarded as equivalent solutions of the present disclosure.
  • the steps of executing the above-mentioned series of processing can naturally be executed in chronological order in the order of description, but they do not necessarily need to be executed in chronological order. Some steps can be performed in parallel or independently of each other.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本发明公开了一种用于分布式数据处理系统的逻辑节点分布式签名决策系统，包括：初始逻辑节点生成组件，接收用户输入的任务配置数据，生成用于初始逻辑节点拓扑图，其中源逻辑节点具有指定的逻辑分布式签名以及每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合；以及逻辑分布式签名选择组件，根据已经确定逻辑分布式签名的上游逻辑节点的输出端的分布式描述符，针对当前逻辑节点的每一个候选逻辑分布式签名，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为当前逻辑节点的对应输入端的张量的分布式描述符所需的传输数据的代价，并选择与代价最小值所对应候选逻辑分布式签名作为所述当前逻辑节点的确定逻辑分布式签名。

Description

逻辑节点的分布式签名决策系统及其方法 技术领域
本公开涉及一种数据处理技术。更具体地说，本公开涉及一种用于分布式数据处理系统的逻辑节点的分布式签名决策系统及其方法。
 背景技术
随着分布式计算的普及，大型的作业或大张量会通过分割而将不同部分的数据部署到不同的分布式数据处理系统的各个计算设备上进行处理，并在各个部分计算过程需要进行中间参数的交互。这样，在具体作业的处理过程中，部署在一个计算设备上的计算中间参数或结果会称为另一个计算设备上的计算任务的输入数据，这会引起计算设备之间的数据传输开销。在作业数据很大的情况下，这种不同计算设备之间的传输开销对于分布式数据处理系统而言将造成极大的计算负担。
 技术问题
，在支持数据并行、模型并行和混合并行、流式并行的分布式数据处理系统中，如何降低这种不同计算设备之间的数据传输开销是人们所需要面对的一个问题。
 技术解决方案
本发明的目的在于提供一种本发明的一个目的是解决至少上述问题，具体而言，本公开提供一种在神经网络中为配置一种逻辑节点以减少作业数据的中间参数在不同分布式数据处理系统的不同计算设备之间传输的数据量提供指引的解决方案。具体而言，根据本公开的一个方面，提供了一种用于分布式数据处理系统的逻辑节点分布式签名决策系统，包括：初始逻辑节点生成组件，接收用户输入的任务配置数据，生成用于所述分布式数据处理系统的初始逻辑节点拓扑图，其中源逻辑节点具有指定的逻辑分布式签名以及每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合，所述候选逻辑分布式签名集合中的每个逻辑分布式签名指定了其所属的初始逻辑节点的每个输入张量的分布式描述符以及每个输出张量的分布式描述符；以及逻辑分布式签名选择组件，根据每个已经唯一确定逻辑分布式签名的上游逻辑节点的输出端的分布式描述符，针对当前逻辑节点的每一个候选逻辑分布式签名，基于每个上游逻辑节点将要并行分布的设备集的数据量、当前逻辑节点将要并行分布的设备集的数据量以及每个上游逻辑节点分布在每个设备上的逻辑数据块的大小，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为当前逻辑节点的对应输入端的张量的分布式描述符所需的传输数据的代价，并选择与代价最小值所对应候选逻辑分布式签名作为所述当前逻辑节点的唯一确定逻辑分布式签名。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述逻辑分布式签名选择组件对具有多个输入端的逻辑节点的每个输入端计算所需传输数据的代价并将所有输入端的所需传输数据的代价的总和作为该当前逻辑节点使用对应候选逻辑分布签名时的传输数据的总代价。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所唯一确定的逻辑分布式签名确定了当前逻辑节点的所有输入端的输入张量的分布方式。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述分布式描述符包括分割张量描述符、广播张量分布描述符以及部分和张量描述符。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述分割张量描述符指明了张量的分割方式。
根据本公开的另一个方面，提供了一种用于分布式数据处理系统的逻辑节点分布式签名决策方法，包括：初始逻辑节点生成步骤，接收用户输入的任务配置数据，生成用于所述分布式数据处理系统的初始逻辑节点拓扑图，其中源逻辑节点具有指定的逻辑分布式签名以及每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合，所述候选逻辑分布式签名集合中的每个逻辑分布式签名指定了其所属的初始逻辑节点的每个输入张量的分布式描述符以及每个输出张量的分布式描述符；以及逻辑分布式签名选择步骤，根据每个已经唯一确定逻辑分布式签名的上游逻辑节点的输出端的分布式描述符，针对当前逻辑节点的每一个候选逻辑分布式签名，基于每个上游逻辑节点将要并行分布的设备集的数据量、当前逻辑节点将要并行分布的设备集的数据量以及每个上游逻辑节点分布在每个设备上的逻辑数据块的大小，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为当前逻辑节点的对应输入端的张量的分布式描述符所需的传输数据的代价，并选择与代价最小值所对应候选逻辑分布式签名作为所述当前逻辑节点的唯一确定逻辑分布式签名。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述逻辑分布式签名选择步骤包括对具有多个输入端的逻辑节点的每个输入端计算所需传输数据的代价并将所有输入端的所需传输数据的代价的总和作为该当前逻辑节点使用对应候选逻辑分布签名时的传输数据的总代价。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所唯一确定的逻辑分布式签名确定了当前逻辑节点的所有输入端的输入张量的分布方式。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述分布式描述符包括分割张量描述符、广播张量分布描述符以及部分和张量描述符。
根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述分割张量描述符指明了张量的分割方式。
 有益效果
通过根据本公开的分布式数据处理系统的逻辑节点分布式签名决策系统，能够从全局角度讲静态分布式数据处理系统在处理数据过程中的不同计算设备之间的数据交换量最小化，从而减少的数据交互过程产生的开销，这样有效地降低了数据交换对实际运算操作带来的不良影响。
本发明的其它优点、目标和特征将部分通过下面的说明体现，部分还将通过对本发明的研究和实践而为本领域的技术人员所理解。
 附图说明
图1所示的是根据本公开的用于分布式数据处理系统的逻辑节点分布式签名决策系统100的原理示意图。
图2所示的是根据本公开的分布式数据处理系统的逻辑节点分布式签名决策系统100决策SBP签名的一种示意结构图。
图3图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第一示意图。
图4图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第二示意图。
图5图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第三示意图。
图6图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第四示意图。
图7图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第五示意图。
图8图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第六示意图。
 本发明的实施方式
下面结合实施例和附图对本发明做进一步的详细说明，以令本领域技术人员参照说明书文字能够据以实施。
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。
在本公开使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本开。在本公开和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本公开可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本公开范围的情况下，在下文中，两个可能设备之一可以被称为第一逻辑分布式签名也可以被称为第二逻辑分布式签名，类似地，两个可能设备的另一个可以被称为第二逻辑分布式签名也可以被称为第一逻辑分布式签名。取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
为了使本领域技术人员更好地理解本公开，下面结合附图和具体实施方式对本公开作进一步详细说明。
深度学习本质上属于特征学习的一种，从这个角度考虑，可以将深度学习直接应用于从原始数据提取特征。而自动编码器是实现特征提取这一功能的重要模型之一。
图1所示的是根据本公开的用于静态分布式数据处理系统的逻辑节点分布式签名决策系统100的原理示意图。如图1所示，所述分布式签名决策系统100包括初始逻辑节点生成组件110和逻辑分布式签名选择组件120。所述初始逻辑节点生成组件110接收用户输入的任务配置数据，生成用于所述静态分布式数据处理系统的初始逻辑节点拓扑图111。在作业输入以后，静态分布式数据处理系统会基于用户输入的作业描述，自动将作业分解成众多微小的作业任务，这些众多的微小作业任务由各种操作组元构成，这些操作组元作为逻辑节点彼此前后互联形成初步张量处理神经网络拓扑图。这些神经网络的每层网络包含了众多的逻辑节点，相邻两层神经网络之间彼此连接，从而为分布式数据处理系统中执行实际作业处理的执行体的布置（PLACEMENT）提供了指引。图1中仅仅示意性地给出了一个简单的初始逻辑节点拓扑图111，其中显示了节点A、B、C、D、E、F、L以及K。其他未显示的采用省略方式替代。在实际的数据处理中，初始节点拓扑图111会更复杂。初始逻辑节点拓扑图111包含实现用户所描述的计算任务的基本运算节点。这种初始逻辑节点拓扑图111的生成方式属于本领域常规技术，因此不在此赘述。
在初始逻辑节点拓扑图111的各个初始逻辑节点每个包含多个SBP签名。作为已经由用户配置了SBP签名的源逻辑节点或基于用户的任务描述而确定了唯一SBP签名的初始逻辑节点，例如初始逻辑节点A、E以及C，其仅仅具有唯一的SBP签名，例如初始逻辑节点A的SBP-5，初始逻辑节点C的SBP-2以及初始逻辑节点E的SBP-3。在未确定唯一SBP签名的情况下，初始逻辑节点通常包含有其固有的一些候选SBP签名。如图1中的初始逻辑节点B，其具有多个候选SBP签名，例如三个，包括SBP-1、SBP-2以及SBP-3。其他初始逻辑节点也各自具有不同的候选SBP签名，在此不一一列出。不同的初始逻辑节点根据其具体执行的运算操作不同，会有不同的固定的候选SBP签名。
根据本公开的SBP签名是应用在一种分布式数据处理系统中的签名。分布式数据处理系统中，由于经常存在数据并行、模型并行以及混合并行以及流式并行等的情形，因此，经常会存在相邻的逻辑节点的任务将被同时部署到不同的计算设备上，因此在实际数据处理过程中，各个计算设备之间会对中间参数进行交换，会导致大量的传输开销。为此，为了减少数据传输开销，需要在初始逻辑节点拓扑图111的基础上，进一步生成更多的逻辑节点，以便完善逻辑节点拓扑图，尤其是减少上下游逻辑节点之间的传输开销，需要使得上下游逻辑节点的数据分布方式所带来的变化最小。为此，本公开为了获得比较好的下游逻辑节点，针对每个逻辑节点指定了逻辑分布式签名。所述逻辑分布式签名是采用张量的分布式描述符对逻辑节点的签名，每个张量的分布式描述符描述了每个张量的在整个计算系统中的分布方式，主要包括分割（SPLIT）张量描述符、广播（BROADCAST）张量描述符以及部分值（PARTIAL VALUE）张量描述符。
具体而言，分割（SPLIT）张量描述符就是描述一个张量的分割方式，例如将一个数据块根据用户的描述在指定的维度上进行分割，并分布到不同的计算设备上进行指定的计算处理。如果一个数据块为二维数据块，则该数据块在其第0维被切割时，则该数据块所形成的一批数据的数据张量的分布式描述符为S(0)，则每个逻辑数据块在其输入端获得这种数据张量的分布式描述符都为S(0)。同样，如果一个数据块为二维数据块，则该数据块在其第1维被切割时，则该数据块所形成的一批数据的数据张量的分布式描述符为S(1)，则每个逻辑数据块在其输入端获得这种数据张量的分布式描述符都为S(1)。类似地，如果待处理的任务数据的维度为更多维度，则会有更多的分布式描述符，例如S（2）、S（3）…等等。这类所提到的数据可以是被处理的数据或模型。如果数据本身被切割，则在分布式数据处理系统上形成数据并行处理，如果模型被分割，则在分布式数据处理系统上会形成模型并行处理。如果逻辑节点的输入为这种分割（SPLIT）张量描述符，则在实际数据处理过程中，如果一个张量的数据大小为T，而该张量将被分布到四张计算卡上进行数据并行计算，则每张卡上分配到的数据量为四分之一的数据，整个四张卡上的数据量则为T。
广播（BROADCAST）张量描述符是用来描述一个张量以广播方式在分布式系统中进行发布的方式。通常，对于仅仅进行数据并行的数据处理系统，模型数据通常被广播到各个计算设备，因此对于被输入到逻辑节点的广播数据采用广播张量描述符进行描述。在实际数据处理过程中，被广播的数据，在每张实际计算卡上的数据块大小都是相同的。
部分值（PARTIAL VALUE）张量描述符表示一个逻辑节点的输入或输出张量为多个同类张量的部分值。这些部分值包括部分和（Ps）、部分积(Pm)、部分“与”结果、部分最大以及部分最小。由于通常会为了对数据进行数据并行处理，因此，在不同设备上对数据的处理是对部分数据的处理。例如有些张量为S(0)或S(1)，则在一些计算设备上获得结果张量为S(0)，这些部分计算设备上的结果张量合并起来就是部分值张量。将所有设备上的同类数据合并起来才是最后的输出结果。
上述各种张量的分布式描述符代表了这些张量在分布式计算系统中的分布方式，而这些张量无论是作为逻辑节点的输入和输出，其各自的分布方式也描述了逻辑节点对操作数据的分布描述。为了描述方便，本公开将这种分布式描述符简称为“SBP描述符”。
为此，随着初始逻辑节点拓扑图111的生成，本公开的初始逻辑节点，也就是一些运算节点也具备了各个输入和输出的数据分布式描述符，这些输入和输出分布式描述符形成了对逻辑节点的一种签名，即采用张量的分布式描述符对运算逻辑节点的签名。为了方便表述，采用这三种分布式描述符的英文首字母来简称这种签名为“SBP签名”。
根据每个分布式计算系统中用户对计算任务的描述和数据并行的要求，这种描述符会包括至少三种S(0)、B以及P。如果对数据和模型存在多种分割方式，则每增加一种分割方式，则增加一种描述符。针对每个逻辑节点，其签名都包含了这些描述符的各种组合方式。因此，在根据本公开分布系统中，至少有三种分布式描述符，通常为有四种分布式描述符，例如如下四种SBP描述符，S(0)、S(1)、P以及B。根据张量维度数量不同，可以有更多分布式描述符。如果为四种SBP描述符，则可以按照输入输出的排列组合方式形成多种SBP签名。下面列出了一些SBP签名的实例： (S(0), B)→S(0)，(S(1), B)→S(1)，P→P，B→B，(S(0), S(1))→P，S(0)→P，S(0)→S(0)，S(0)→S(1)，P→B等等。所有SBP签名是各种SBP描述符组合结果。对于矩阵乘法逻辑节点，如果其输入张量是在第一维上面切割，其输出结果张量也是得到第一维上切割。综上所述，S、B、P是用于描述数据块在数据处理系统中的分布的描述符，而SBP签名利用多个SBP描述符描述逻辑节点的任务操作。每个数据块可以有多种SBP描述符，而每个逻辑节点所代表的运算方式可以多种SBP 签名的情形。例如，图1所示的SBP-1可以是(S(0), B)→S(0)这种签名形式，而SBP-2可以是(S(1), B)→S(1)这种签名形式。实际应用中，不同签名形式可以具有不同的编号，这里给出的编号仅仅是为了描述的方便，并不意味着需要对每个签名都赋予一个编号，可以完全没有编号，签名的不同形式彼此之间不需要编号就可以彼此区分。
可以基于用于的任务描述赋予每个初始逻辑节点如上所述的SBP签名。通常的任务逻辑节点是一些运算操作节点，其执行特定的运算操作，因此其具有特定的候选SBP签名。需要指出的是，并不是每个任务逻辑节点所具备的SBP签名都一样，通常进行乘法操作的任务逻辑节点其SBP签名的输入张量不包含部分和张量，因此其输入张量的SBP描述符不包含分布式描述符P。对于执行加法操作的任务逻辑节点的候选SBP签名则可以包括各种SBP描述符彼此之间或自己之间的任意组合。例如执行矩阵乘法的任务逻辑节点，在仅有数据并行的情况下，其候选SBP签名通常为(S(0), B)→S(0)，(S(1), B)→S(1)， (S(0), S(1))→P等，但不仅此这些，随着技术的发展，以前一些不适合矩阵乘法的签名也可以应用到矩阵乘法，此处仅仅是举例。因此，每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合。所述候选逻辑分布式签名集合中的每个逻辑分布式签名指定了其所属的初始逻辑节点的每个输入张量的分布式描述符以及每个输出的张量的分布式描述符。
尽管初始逻辑节点生成组件110生成了初始逻辑节点拓扑图111，但是初始逻辑节点拓扑图111中每个逻辑节点将使用哪种SBP签名所确定的张量或者说使用哪种分布式张量以及输入何种分布式张量，需要进一步确定。
因此，根据本公开的逻辑节点分布式签名决策系统100的逻辑分布式签名选择组件120从初始逻辑节点拓扑图111中的源逻辑节点开始，在当前逻辑节点（例如逻辑节点B）的所有上游逻辑节点（例如逻辑节点A和E）的逻辑标签或SBP标签已经被确定时，传输数据量估算单元121基于所述逻辑节点B的所有上游逻辑节点的与逻辑节点B的输入端对应的输出端的分布式描述符，针对逻辑节点B的每一个候选逻辑分布式签名，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为逻辑节点B的对应输入端的候选逻辑分布式签名之一的张量的分布式描述符所需的传输的数据的代价。如图1所示，逻辑节点B，其具有很多候选SBP签名，例如SBP-1、SBP-2以及SBP-3。举例而言， SBP-1其可能形式为 (S(1), B)→S(1)或(S(1), P)→S(1)的签名，初始逻辑节点A的签名SBP-5其可能形式举例而言为 (S(0), B)→S(0)的签名，初始逻辑节点E的签名SBP-3可能形式举例而言为B→B或S(0)→P。每个签名形式中，箭头左侧为输入张量的分布式描述符，箭头右侧为输出张量的分布式描述符。为了描述方便，下面将“分布描述符为S(0)的张量”简称为“S(0)张量”，将“分布描述符为B的张量”简称为“B张量”，将“分布描述符为P的张量”简称为“P张量”，以此类推。
如图1所示，初始逻辑节点拓扑图111中逻辑节点E的标签SBP-3的形式如果为“S(0)→S(0)”，则其输出张量分布描述符则为S(0)，因此其输出张量为S(0)张量。如果逻辑节点E的签名SBP-3的形式为“B→B”或 “P→P”，则其输出的张量的分布描述符为B或P，因此其输出张量为B张量或P张量。如果逻辑节点B的候选签名SBP-1（即“(S(0), S(1))→P”）被选择为确定的签名，则其对应节点E的输出端的第一输入端的输入张量的分布描述符则必须是S(0)，即第一输入端必须获得一个S(1)张量，而其对应于节点A的输出端的第二输入端的输入张量的分布描述符则必须是S(0)，即第二输入端必须获得一个S(0)张量。很显然，此时节点A的输出张量分布描述符的P与节点B的第一输入端的输入张量的分布描述符S(0)不符，因此，要使得逻辑节点B执行正确的运算操作，就需要将节点A输出的分布描述符为P的张量变换为分布描述符为S(0)的张量。同样，如果节点E输出的张量的分布描述符为S(0)，则与节点B的第二输入端的量输入张的分布描述符S(1)不一致，因此，要使得逻辑节点B执行正确的运算操作，就需要将节点E输出的分布描述符为S(0)的张量变换为分布描述符为S(1)的张量。
在分布式计算系统中，由于各个逻辑节点的操作任务尤其是运算任务被切割分布到各个计算设备(例如计算卡CPU、GPU或TPU)上，因此为了最终获得正确的结果，需要不断对中间参数进行同步，这就会涉及到不同计算设备之间的中间参数的交换。当上一逻辑节点的SBP签名所含有的输出张量的SBP描述符与当前节点的SBP签名的对应输入张量的SBP描述符不一致时，通常在实际运行过程中进行输出转换，而这个转换过程通常需要获取位于另一个计算设备上的部分数据，以便与本地能够获得的数据一起构成当前逻辑节点输入端所需的数据，从而符合当前逻辑节点的输入端的数据张量的分布式描述符。这种从另一个设备上获取部分数据的过程将会产生比较大的数据传输开销或传输代价。因此，为当前逻辑节点选择不同的签名会产生不同的数据传输开销或代价。为此，传输数据量估算单元121会对每个未确定签名的逻辑节点估算每个候选签名将会产生的数据传输开销。例如，针对逻辑节点B，针对其三个候选SBP签名分别估算逻辑节点B在采用其中一个SBP签名的情况下会产生的数据传输代价。对于逻辑节点B而言，选择任意一个候选SBP签名都可以实现其操作任务。但是其采用不同的SBP签名情况下，其运行所产生的数据传输代价不同。因此，为了在数据处理过程中使得数据传输代价最小化，需要从各个逻辑节点的候选签名中选择数据传输量最小的签名作为其实际运行过程中的签名。
在初始逻辑节点拓扑图111中处于上下游关系的逻辑节点A和逻辑节点B之间，逻辑节点A可能是源节点，其SBP签名可以由用户配置生成，也可以基于用户的对任务的描述自然生成，或者逻辑节点A的SBP签名已经基本按照本公开的方案进行了决策选择确定，例如逻辑节点A的SBP签名的输出张量的描述符为S(0)。而作为初始逻辑节点拓扑图111中的逻辑节点B，其具有很多候选SBP签名，其可能包括(S(1),B)→S(1) , B→P , S(1))→P，以及P→B等等但是，从逻辑节点A到逻辑节点B，由于逻辑节点A的输出张量的分布描述符为S(0)，节点B可以选择的对应输入张量分布描述符可以为S(1)、B以及P。
因此，当前面一些逻辑节点的签名被确定下来以后，其下游的逻辑节点的SBP签名也基于上游逻辑节点的输出张量的逻辑分布式描述符（SBP描述符）和下游上游逻辑节点的候选逻辑分布式签名的对应输入张量的逻辑分布式描述符（SBP描述符）之间的数据传输的代价而最终被选择确定。通过这种方式，对这样当一个逻辑节点的候选SBP签名一旦被选定进行计算，意味着该逻辑节点的各个输入端和输出端的数据块的各自的SBP描述符也确定下来，从而计算或估算出当前逻辑节点的数据传输的总代价，并将总代价最小的候选逻辑分布式签名作为该当前逻辑节点的逻辑分布式签名。需要指出的是，如果当前逻辑节点的候选签名中有哪些签名的输入端的逻辑分布式描述符与其上游逻辑节点的输出张量的逻辑分布式描述符一致，则可以优先选择含有该逻辑分布式描述符的候选逻辑分布式签名，除非该候选逻辑分布式签名的其他输入端张量的逻辑分布式描述符会导致最后的总代价更大。
图2所示的是根据本公开的分布式数据处理系统的逻辑节点分布式签名决策系统100决策SBP签名的一种示意结构图。图2是对图1中节点A、B以及E之间的关系的放大示意图。如图2所示，假设逻辑节点E的已经确定的SBP签名SBP-3的输出张量的分布描述符为S(0)，逻辑节点A的已经确定的SBP签名SBP-5的输出张量的分布描述符为输入张量的分布描述符为P，逻辑节点B的候选SBP签名之一SBP-2为(S(1), S(0))→P。因此逻辑节点B的与逻辑节点E的输出张量的SBP描述符S(0)对应的输入张量的SBP描述符为S(1)，而逻辑节点B的与逻辑节点A的输出张量的SBP描述符P对应的输入张量的SBP描述符为S(0)。因此，要符合逻辑节点B的候选SBP签名的输入逻辑数据块分布要求，则需要使得其一个输入的张量分布从逻辑节点E的输出张量的SBP描述符S(0)变换为S(1)以及使得其另一个输入的张量分布从逻辑节点A的输出张量的SBP描述符P变换为S(0)。这中变换将会在实际数据处理过程中产生数据交换。
图3图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第一示意图。针对图2所示的任务节点B的候选SBP签名SBP-2，假设为(S(1), S(0))→P。为了描述方便，输入源任务节点A和E与接收的汇节点B的任务都分布在同一设备集上。如图3所示，都分布在计算卡GPU 0和GPU 1上。尽管这里只是显示了两张计算卡，实际上源任务节点和汇任务节点可以分布到更多张卡上，也可以分布到不同设备集上。图3显示了图2中的任务节点E的任务的S(0)描述符张量分布在两张计算卡上的情况下任务节点B的输入端要获得S(0)描述符的张量的情况下的数据交换过程。
任务节点B的分布在GPU 0的任务节点要获得S(1)，则除了需要直接从获得任务节点E的S(0)描述符所描述的分布在GPU 0上的张量一半外（采用实线箭头显示了这种数据部分的获取过程），还需要补充从任务节点E的S(0)描述符所描述的分布在GPU 1上的张量的另外一半（采用虚线箭头显示了这种数据部分的获取过程）。如果逻辑数据块的大小为T 1，则从GPU 1上的任务节点E的逻辑数据块上传输到任务节点B的分布在GPU 0的任务节点的数据量为T 1/2。与此同时，任务节点B的分布在GPU 1的任务节点要获得S(1)，则除了需要直接从获得任务节点E的S(0)描述符所描述的分布在GPU 1上的张量一半外（采用实线箭头显示了这种数据部分的获取过程），还需要补充从任务节点E的S(0)描述符所描述的分布在GPU 0上的张量的另外一半（采用虚线箭头显示了这种数据部分的获取过程）。如果逻辑数据块的大小为T 1，则从GPU 0的任务节点E的逻辑数据块上传输到任务节点B的分布在GPU 1的任务节点的数据量为T 1/2。因此，将任务节点E的S(0)描述符张量变换为任务节点B的输入端要获得S(0)描述符的张量，总的数据传输代价为T 1=（T 1/2+T 1/2）。T 1是源节点上所分布的逻辑数据块的大小。在图3中，逻辑数据块的大小为S(0)分布在每张卡上阴影部分的数据块的大小也就是整个张量的二分之一。在设备集的数据卡的数量为3、4或5的情况下，其传输代价也还是T 1。
图4图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第二示意图。同样，针对图2所示的任务节点B的候选SBP签名SBP-2，假设为(S(1), S(0))→P。为了描述方便，输入源任务节点A和E与接收的汇节点B的任务都分布在同一设备集上，如图4所示，都分布在计算卡GPU 0、GPU 1以及GPU 2上。尽管这里显示了三张计算卡，这仅仅是为了举例。其也可以与图3所示的那样可以是两张卡。实际上源任务节点和汇任务节点可以分布到更多张卡上，也可以分布到不同设备集上。图4显示了图2中的任务节点A的任务的P描述符张量分布在两张计算卡上的情况下任务节点B的输入端要获得S(0)描述符的张量的情况下的数据交换过程。
任务节点B的分布在GPU 0的任务节点要获得S(0)，则除了需要直接从获得任务节点A的P描述符所描述的分布在GPU 0上的张量三分之一外（采用实线箭头显示了这种数据部分的获取过程），还需要补充任务节点A的P描述符所描述的分布在GPU 1上的张量的三分之一（采用虚线箭头显示了这种数据部分的获取过程）以及任务节点A的P描述符所描述的分布在GPU 2上的张量的三分之一。为此，在三张卡上每个分布有部分值张量P，为举例而言，这里采用Ps来表示为一种部分和张量作为一种描述实例。如果A任务节点分布在每张GPU卡上的逻辑数据块的大小为T 2，则分布在GPU 0上的任务节点B要获得S(0)张量，还需要补充从GPU 1上的任务节点A的逻辑数据块上向任务节点B的分布在GPU 0的任务节点传输数据量T 2/3以及从GPU 2上的任务节点A的逻辑数据块上向任务节点B的分布在GPU 0的任务节点传输数据量T 2/3。同样，分布在GPU 1上的任务节点B要获得S(0)张量，还需要补充从GPU 0上的任务节点A的逻辑数据块向任务节点B的分布在GPU 1的任务节点传输数据量T 2/3以及从GPU 2上的任务节点A的逻辑数据块向任务节点B的分布在GPU 1的任务节点传输数据量T 2/3。类似地，分布在GPU 2上的任务节点B要获得S(0)张量，还需要补充从GPU 1上的任务节点A的逻辑数据块向任务节点B的分布在GPU 2的任务节点传输数据量T 2/3以及从GPU 0上的任务节点A的逻辑数据块向任务节点B的分布在GPU 2的任务节点传输数据量T 2/3。因此，图4所示的从P分布式张量变换成S(0)分布式张量在实际数据处理过程中的数据传输量为2T 2=（T 2/3+T 2/3+T 2/3+T 2/3+T 2/3+T 2/3）。可选择地，如果任务节点所分布的计算卡的数量为2。则数据的传输量为T 2=（T 2/2+T 2/2）。以此类推，在源节点和汇节点具有相同的设备集的情况下，如果设备集中的卡数量k，则数据的传输量为（k-1）·T 2。
很显然，如上所述，对于逻辑节点B要执行体运算操作，选择签名SBP-2签名（例如签名(S(1), S(0))→P）所需要的数据传输代价是两个输入端的传输代价的总和。综合图3和图4（如果图4中为两张计算卡的情况下），任务节点在候选签名SBP-2情况下，其需要传输的总的数据量为T 1 +T 2。为此，传输数据量估算单元121针对逻辑节点B的候选签名SBP-1所估算的传输代价需要包含针对该候选签名的两个输入端的传输代价。
根据针对源任务节点和汇任务节点的设备集之间的完全相同的情况可以归纳总结各种SBP描述符彼此之间存在的数据交换量的计算表，如下表1：
表1（源任务节点和汇任务节点的分布设备集完全相同，卡数为K）。
Figure 704992dest_path_image001
图5图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第三示意图。其中的源节点的设备集与汇节点的设备集完全不同。即源任务节点E分布在GPU 0和GPU1上，汇任务节点B分布在计算卡GPU 2和GPU 3上。如果分布在各个计算卡上的逻辑数据块大小为T 3，在需要传输的数据量为2T 3。
图6图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第四示意图。其中的源节点的设备集与汇节点的设备集完全不同。即源任务节点A分布在GPU 0、 GPU1和GPU 2上，汇任务节点B分布在计算卡GPU 3、 GPU 4和GPU 5上。举例而言，在三张卡上每个分布有部分值张量P，这里采用Ps来表示为一种部分和张量作为一种描述实例。如果分布在各个源任务节点的各个计算卡上的逻辑数据块大小为T 4，在需要传输的数据量为9个1/3 T 4，即，3T 4。如果源任务节点所分布的任务集的计算卡的数量为2， 则需要传输的数据量为2T 4。如果源任务节点A所分布的任务集的计算卡的数量为Ks，则数据的传输量则为Ks ·T 4。
根据针对源任务节点和汇任务节点的设备集之间的完全不同的情况可以归纳总结各种SBP描述符彼此之间存在的数据交换量的计算表，如下表2：
表2（源任务节点（卡数为K s)和汇任务节点（卡数为K d)各自的分布设备集完全不同）
Figure 100201dest_path_image002
图7图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第五示意图。其中的源节点的设备集与汇节点的设备集不完全相同。即源任务节点E分布在GPU 0和GPU1上，汇任务节点B分布在计算卡GPU 1和GPU 2。如果分布在各个源任务节点所分布的计算卡上的逻辑数据块大小为T 5，在需要传输的数据量为3/2 T 3=（1/2 T 3+1/2 T 3+1/2 T 3）。这种情况下，计算没有固定规律，需要根据实际设备集的具体构成以及彼此之间的交集情况进行计算。
图8图示了根据本公开的传输数据量估算单元121估算不同分布式描述符的张量之间产生的数据传输量的第六示意图。其中的源节点的设备集与汇节点的设备集不完全相同。即源任务节点A分布在GPU 0、 GPU1和GPU 2上，汇任务节点B分布在计算卡GPU 1、 GPU 2和GPU 3上。举例而言，在三张卡上每个分布有部分值张量P，这里采用Ps来表示为一种部分和张量作为一种描述实例。如果分布在各个源任务节点的各个计算卡上的逻辑数据块大小为T 6，在需要传输的数据量为在需要传输的数据量为7个1/3 T 4，即，7/3T 4。这种情况下，计算没有固定规律，需要根据实际设备集的具体构成以及彼此之间的交集情况进行计算。
如上所述，传输数据量估算单元121按照上述方式遍历逻辑节点B的所有候选签名SBP-1、SBP-2以及SBP-3，并针对每个签名获取其传输代价。随后，传输数据总量比较单元122会比较每个候选签名下的传输代价，并获取待确定逻辑节点，例如逻辑节点B，的最小传输代价。最后由SBP签名确定单元123将最小传输代价所对应的候选SBP签名确定为该逻辑节点B的最终SBP签名。
最后逻辑节点拓扑图输出组件130基于SBP签名确定单元123针对每个逻辑节点确定的SBP签名，输出最终的逻辑节点拓扑图131，构成该逻辑节点拓扑图131的每个逻辑节点都只附带有一个SBP签名，或者说每个逻辑节点都明确指定了其每个输入张量的分布方式或分布描述符，并且唯一地确定了其输入张量的分布方式或分布描述符。
尽管上面给出了如何在一些候选SBP签名确定最终SBP签名的常规情况，但是在一些特定的情况下，对于某些逻辑节点，在用户有特殊的配置的情况下或有用户指定的情况下，这些逻辑节点只有用户指定的SBP签名，因此其下游的逻辑节点将基于这种特别指定的上游逻辑节点进行SBP签名的确定。
通过根据本公开的分布式数据处理系统的逻辑节点分布式签名决策系统，能够从全局角度使得分布式数据处理系统在处理数据过程中的不同计算设备之间的数据交换量最小化，从而减少的数据交互过程产生的开销，这样有效地降低了数据交换对实际运算操作带来的不良影响，降低了运算操作的等待时间，从而加快了数据处理速度。
以上结合具体实施例描述了本公开的基本原理，但是，需要指出的是，对本领域的普通技术人员而言，能够理解本公开的方法和装置的全部或者任何步骤或者部件，可以在任何计算装置（包括处理器、存储介质等）或者计算装置的网络中，以硬件、固件、软件或者它们的组合加以实现，这是本领域普通技术人员在阅读了本公开的说明的情况下运用他们的基本编程技能就能实现的。
因此，本公开的目的还可以通过在任何计算装置上运行一个程序或者一组程序来实现。所述计算装置可以是公知的通用装置。因此，本公开的目的也可以仅仅通过提供包含实现所述方法或者装置的程序代码的程序产品来实现。也就是说，这样的程序产品也构成本公开，并且存储有这样的程序产品的存储介质也构成本公开。显然，所述存储介质可以是任何公知的存储介质或者将来所开发出来的任何存储介质。
还需要指出的是，在本公开的装置和方法中，显然，各部件或各步骤是可以分解和/或重新组合的。这些分解和/或重新组合应视为本公开的等效方案。并且，执行上述系列处理的步骤可以自然地按照说明的顺序按时间顺序执行，但是并不需要一定按照时间顺序执行。某些步骤可以并行或彼此独立地执行。
上述具体实施方式，并不构成对本公开保护范围的限制。本领域技术人员应该明白的是，取决于设计要求和其他因素，可以发生各种各样的修改、组合、子组合和替代。任何在本公开的精神和原则之内所作的修改、等同替换和改进等，均应包含在本公开保护范围之内。

Claims (10)

  1. 一种用于分布式数据处理系统的逻辑节点分布式签名决策系统，包括：
    初始逻辑节点生成组件，接收用户输入的任务配置数据，生成用于所述分布式数据处理系统的初始逻辑节点拓扑图，其中源逻辑节点具有指定的唯一逻辑分布式签名以及每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合，所述候选逻辑分布式签名集合中的每个逻辑分布式签名指定了其所属的初始逻辑节点的每个输入张量的分布式描述符以及每个输出张量的分布式描述符；以及
    逻辑分布式签名选择组件，根据每个已经唯一确定逻辑分布式签名的上游逻辑节点的输出端的分布式描述符，针对当前逻辑节点的每一个候选逻辑分布式签名，基于每个上游逻辑节点将要并行分布的设备集的数据量、当前逻辑节点将要并行分布的设备集的数据量以及每个上游逻辑节点分布在每个设备上的逻辑数据块的大小，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为当前逻辑节点的对应输入端的张量的分布式描述符所需的传输数据的代价，并选择与代价最小值所对应候选逻辑分布式签名作为所述当前逻辑节点的唯一确定逻辑分布式签名。
  2. 根据权利要求1所述的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述逻辑分布式签名选择组件对具有多个输入端的逻辑节点的每个输入端计算所需传输数据的代价并将所有输入端的所需传输数据的代价的总和作为该当前逻辑节点使用对应候选逻辑分布签名时的传输数据的总代价。
  3. 根据权利要求1或2所述的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所唯一确定的逻辑分布式签名确定了当前逻辑节点的所有输入端的输入张量的分布方式。
  4. 根据权利要求3所述的用于分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述分布式描述符包括分割张量描述符、广播张量分布描述符以及部分值张量描述符。
  5. 根据权利要求4所述的用于静态分布式数据处理系统的逻辑节点分布式签名决策系统，其中所述分割张量描述符指明了张量的分割方式。
  6. 一种用于分布式数据处理系统的逻辑节点分布式签名决策方法，包括：
    初始逻辑节点生成步骤，接收用户输入的任务配置数据，生成用于所述分布式数据处理系统的初始逻辑节点拓扑图，其中源逻辑节点具有指定的逻辑分布式签名以及每个初始逻辑节点基于所述任务配置数据附有候选逻辑分布式签名集合，所述候选逻辑分布式签名集合中的每个逻辑分布式签名指定了其所属的初始逻辑节点的每个输入张量的分布式描述符以及每个输出张量的分布式描述符；以及
    逻辑分布式签名选择步骤，根据每个已经唯一确定逻辑分布式签名的上游逻辑节点的输出端的分布式描述符，针对当前逻辑节点的每一个候选逻辑分布式签名，基于每个上游逻辑节点将要并行分布的设备集的数据量、当前逻辑节点将要并行分布的设备集的数据量以及每个上游逻辑节点分布在每个设备上的逻辑数据块的大小，计算将每个上游逻辑节点输出端的张量的分布式描述符变换为当前逻辑节点的对应输入端的张量的分布式描述符所需的传输数据的代价，并选择与代价最小值所对应候选逻辑分布式签名作为所述当前逻辑节点的唯一确定逻辑分布式签名。
  7. 根据权利要求6所述的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述逻辑分布式签名选择步骤包括对具有多个输入端的逻辑节点的每个输入端计算所需传输数据的代价并将所有输入端的所需传输数据的代价的总和作为该当前逻辑节点使用对应候选逻辑分布签名时的传输数据的总代价。
  8. 根据权利要求6或7所述的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所唯一确定的逻辑分布式签名确定了当前逻辑节点的所有输入端的输入张量的分布方式。
  9. 根据权利要求8所述的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述分布式描述符包括分割张量描述符、广播张量分布描述符以及部分值张量描述符。
  10. 根据权利要求9所述的用于分布式数据处理系统的逻辑节点分布式签名决策方法，其中所述分割张量描述符指明了张量的分割方式。
PCT/CN2021/072786 2020-02-13 2021-01-20 逻辑节点的分布式签名决策系统及其方法 WO2021159928A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/076,171 US11818231B2 (en) 2020-02-13 2022-12-06 Logical node distributed signature decision system and a method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010090335.2A CN110955734B (zh) 2020-02-13 2020-02-13 逻辑节点的分布式签名决策系统及其方法
CN202010090335.2 2020-02-13

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/076,171 Continuation US11818231B2 (en) 2020-02-13 2022-12-06 Logical node distributed signature decision system and a method thereof

Publications (1)

Publication Number Publication Date
WO2021159928A1 true WO2021159928A1 (zh) 2021-08-19

Family

ID=69985654

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/072786 WO2021159928A1 (zh) 2020-02-13 2021-01-20 逻辑节点的分布式签名决策系统及其方法

Country Status (3)

Country Link
US (1) US11818231B2 (zh)
CN (1) CN110955734B (zh)
WO (1) WO2021159928A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110955734B (zh) 2020-02-13 2020-08-21 北京一流科技有限公司 逻辑节点的分布式签名决策系统及其方法
CN111930519B (zh) * 2020-09-22 2020-12-15 北京一流科技有限公司 用于分布式数据处理的并行决策系统及其方法
CN112764940B (zh) * 2021-04-12 2021-07-30 北京一流科技有限公司 多级分布式数据处理部署系统及其方法
CN112799852B (zh) * 2021-04-12 2021-07-30 北京一流科技有限公司 逻辑节点的多维sbp分布式签名决策系统及其方法
CN114035968B (zh) * 2022-01-10 2022-03-18 北京一流科技有限公司 用于多流并行的冲突处理系统及其方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110245108A (zh) * 2019-07-15 2019-09-17 北京一流科技有限公司 执行体创建系统和执行体创建方法
CN110262995A (zh) * 2019-07-15 2019-09-20 北京一流科技有限公司 执行体创建系统和执行体创建方法
US20190332437A1 (en) * 2018-04-27 2019-10-31 Microsoft Technology Licensing, Llc Nested tenants
CN110955734A (zh) * 2020-02-13 2020-04-03 北京一流科技有限公司 逻辑节点的分布式签名决策系统及其方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104317716B (zh) * 2014-10-30 2017-10-24 华为技术有限公司 分布式节点间的数据传输方法及分布式节点设备
CN104954483A (zh) * 2015-06-30 2015-09-30 深圳清华大学研究院 云计算平台中使用竞价节点部署分布式服务的方法
CN106656798B (zh) * 2016-12-30 2020-03-27 质数链网科技成都有限公司 一种计算决策路径的方法及分布式节点
CN108199868B (zh) * 2017-12-25 2020-12-15 北京理工大学 一种基于战术云的集群系统分布式控制方法
CN108876702A (zh) * 2018-06-21 2018-11-23 北京邮电大学 一种加速分布式深度神经网络的训练方法及装置
CN109189589B (zh) * 2018-08-14 2020-08-07 北京博睿宏远数据科技股份有限公司 一种分布式大数据计算引擎及架构方法
CN110222005A (zh) * 2019-07-15 2019-09-10 北京一流科技有限公司 用于异构架构的数据处理系统及其方法
US11271816B2 (en) * 2020-02-24 2022-03-08 At&T Intellectual Property I, L.P. Network topology management using network element differential history
US20210119882A1 (en) * 2020-04-13 2021-04-22 Intel Corporation Methods and apparatus to determine topologies for networks
US11424989B2 (en) * 2020-06-15 2022-08-23 Cisco Technology, Inc. Machine-learning infused network topology generation and deployment
US11431580B2 (en) * 2020-07-13 2022-08-30 Dell Products L.P. Automated generation of a visualization of a system topology

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190332437A1 (en) * 2018-04-27 2019-10-31 Microsoft Technology Licensing, Llc Nested tenants
CN110245108A (zh) * 2019-07-15 2019-09-17 北京一流科技有限公司 执行体创建系统和执行体创建方法
CN110262995A (zh) * 2019-07-15 2019-09-20 北京一流科技有限公司 执行体创建系统和执行体创建方法
CN110955734A (zh) * 2020-02-13 2020-04-03 北京一流科技有限公司 逻辑节点的分布式签名决策系统及其方法

Also Published As

Publication number Publication date
CN110955734B (zh) 2020-08-21
US11818231B2 (en) 2023-11-14
CN110955734A (zh) 2020-04-03
US20230097489A1 (en) 2023-03-30

Similar Documents

Publication Publication Date Title
WO2021159928A1 (zh) 逻辑节点的分布式签名决策系统及其方法
US10031775B2 (en) Backfill scheduling for embarrassingly parallel jobs
WO2021159929A1 (zh) 拓扑图转换系统及其方法
Thibeault On the Comparison of IDDQ and IDDQ Testing
CN107633125B (zh) 一种基于带权有向图的仿真系统并行性识别方法
WO2022062529A1 (zh) 用于分布式数据处理的并行决策系统及其方法
CN112764940B (zh) 多级分布式数据处理部署系统及其方法
WO2015094269A1 (en) Hybrid flows containing a continuous flow
CN108108233B (zh) 任务多副本执行的集群作业调度方法及系统
Sanchez et al. Synthetic oversampling of instances using clustering
WO2019233089A1 (zh) 一种互联网测试床拓扑结构大比例规模缩减方法及装置
CN102413114B (zh) 一种面向组合的语义web服务发现方法
CN112799852B (zh) 逻辑节点的多维sbp分布式签名决策系统及其方法
Günther et al. Notes on the simplification of the Morse-Smale complex
CN111242293B (zh) 一种处理部件、数据处理的方法以及电子设备
Zhao et al. MM-Flow: Multi-modal flow network for point cloud completion
CN109088746A (zh) 一种云资源编排方法和装置
CN112799637B (zh) 一种并行环境下高吞吐量的模逆计算方法及系统
Ren et al. A provably communication-efficient asynchronous distributed inference method for convex and nonconvex problems
Zheng et al. A gradual scheduling framework for problem size reduction and cross basic block parallelism exploitation in high-level synthesis
Athanasopoulos et al. Multi-objective service similarity metrics for more effective service engineering methods (short paper)
US12086647B2 (en) Dynamic subtask creation and execution in processing platforms
Fotakis et al. Scheduling MapReduce Jobs on Unrelated Processors.
US20150169301A1 (en) Program specification estimation device, method of the same, and non-transitory computer readable medium
CN118862179A (zh) 类单机的分布式数据处理系统及其数据处理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21752991

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21752991

Country of ref document: EP

Kind code of ref document: A1