CN115118727B - Data transmission method, device, equipment and storage medium of distributed computing architecture - Google Patents

Data transmission method, device, equipment and storage medium of distributed computing architecture Download PDF

Info

Publication number
CN115118727B
CN115118727B CN202211029216.1A CN202211029216A CN115118727B CN 115118727 B CN115118727 B CN 115118727B CN 202211029216 A CN202211029216 A CN 202211029216A CN 115118727 B CN115118727 B CN 115118727B
Authority
CN
China
Prior art keywords
data
computing
computing nodes
data set
nodes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211029216.1A
Other languages
Chinese (zh)
Other versions
CN115118727A (en
Inventor
蔡超超
韦晓亚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Shudu Technology Co ltd
Original Assignee
Beijing Shudu Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Shudu Technology Co ltd filed Critical Beijing Shudu Technology Co ltd
Priority to CN202211029216.1A priority Critical patent/CN115118727B/en
Publication of CN115118727A publication Critical patent/CN115118727A/en
Application granted granted Critical
Publication of CN115118727B publication Critical patent/CN115118727B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer And Data Communications (AREA)

Abstract

The application discloses a data transmission method, a data transmission device, data transmission equipment and a data transmission storage medium of a distributed computing architecture, and belongs to the technical field of communication. The data of the first data set is stored in N computing nodes of a data sender; the data receiver distributes M computing nodes to store the data of the first data set; the method comprises the following steps: sending a data acquisition request aiming at the first data set, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes; receiving data of a first data set sent by N computing nodes through M computing nodes; wherein the data of the first data set received by the ith computing node of the M computing nodes is transmitted by at least one computing node of the N computing nodes; and the key value range of the first data set corresponding to at least one computing node is intersected with the key value range of the first data set corresponding to the ith computing node. The method and the device can realize efficient transmission and flexible transmission of the distributed computing architecture data.

Description

Data transmission method, device, equipment and storage medium of distributed computing architecture
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data transmission method, apparatus, device, and storage medium for a distributed computing architecture.
Background
With the continuous development of technologies such as cloud computing and big data, the data scale is continuously increased, and the distributed computing technology is widely applied. The core idea of distributed computing is: the huge data computing task is disassembled into a plurality of subtasks and distributed to different computing nodes in the computing cluster to be executed in parallel, so that the data volume far exceeding the single machine processing range can be processed.
In the field of multi-party private computing, a distributed computing method is also required to process data with an increasing scale. Different from the traditional big data processing technology, the privacy calculation scenario with multiple parties has the following particularity: the privacy calculation process is participated by a plurality of heterogeneous participants together, and the participants are in a cooperative relationship; in the course of privacy computing, a large amount of data needs to be transferred between the participants (across the public network) between the respective computing clusters. Therefore, when designing a multi-party private computing distributed computing architecture, it is necessary to consider how efficiently data is transferred from one computing cluster to another computing cluster in a complex network topology composed of multiple computing clusters.
Disclosure of Invention
Embodiments of the present application provide a data transmission method, apparatus, device, and storage medium in a distributed computing architecture, which can implement efficient and flexible transmission of data between multi-party computing clusters.
In a first aspect, an embodiment of the present application provides a data transmission method for a distributed computing architecture, which is applied to a data receiving party, where the data receiving party is configured to receive a first data set from a data sending party; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set, wherein N and M are positive integers;
the method comprises the following steps:
sending a data acquisition request aiming at the first data set to the data sender, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes;
receiving, by the M computing nodes, the data of the first data set sent by the N computing nodes;
the data of the first data set received by the ith computing node of the M computing nodes is sent by at least one computing node of the N computing nodes, the value range of i is from 1 to M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In a second aspect, an embodiment of the present application provides a data transmission method for a distributed computing architecture, where the data transmission method is applied to a data sender, and the data sender is configured to send a first data set to a data receiver; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set, wherein N and M are positive integers;
the method comprises the following steps:
receiving a data acquisition request aiming at the first data set and sent by the data receiver, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes;
sending, by the N computing nodes, data of the first data set to the M computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In a third aspect, an embodiment of the present application provides a data transmission apparatus of a distributed computing architecture, which is applied to a data receiving side, where the data receiving side is configured to receive a first data set from a data sending side; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set, wherein N and M are positive integers;
the device comprises:
a first sending module, configured to send a data acquisition request for the first data set to the data sender, where the data acquisition request includes key value ranges of the first data set respectively corresponding to the M computing nodes;
a first receiving module, configured to receive, by the M computing nodes, data of the first data set sent by the N computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In a fourth aspect, an embodiment of the present application provides a data transmission apparatus with a distributed computing architecture, where the data transmission apparatus is applied to a data sender, and the data sender is configured to send a first data set to a data receiver; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set, wherein N and M are positive integers;
the device comprises:
a second receiving module, configured to receive a data acquisition request for the first data set sent by the data receiving side, where the data acquisition request includes key value ranges of the first data set respectively corresponding to the M computing nodes;
a second sending module, configured to send, by the N computing nodes, data of the first data set to the M computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In a fifth aspect, an embodiment of the present application provides a data transmission device of a distributed computing architecture, where the device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a data transfer method of a distributed computing architecture as described in the first or second aspect.
In a sixth aspect, the present application provides a computer storage medium, on which computer program instructions are stored, and the computer program instructions, when executed by a processor, implement the data transmission method of the distributed computing architecture according to the first aspect or the second aspect.
In a seventh aspect, this application embodiment provides a computer program product, where instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the data transmission method of the distributed computing architecture according to the first aspect or the second party.
In the embodiment of the application, a data sending party sends a first data set to a data receiving party, wherein data of the first data set is stored in N computing nodes of the data sending party, and the data receiving party allocates M computing nodes to store the data of the first data set. Before the data sender sends the first data set to the data receiver, the data acquisition request for the first data set sent by the data receiver to the data sender may include key value ranges of the first data set corresponding to the M computing nodes, so that the data sender knows specific data of the first data set that each computing node in the M computing nodes is responsible for storing. After receiving the data acquisition request, the data sending party may directly send the data of the first data set to the M computing nodes through the N computing nodes, where the data of the first data set received by each of the M computing nodes may be sent by at least one computing node, of the N computing nodes, having an intersection with the key value range of the first data set corresponding to the computing node. Therefore, on one hand, the data sender and the data receiver can respectively and flexibly configure the number of the nodes used for data transmission in the respective computing cluster, so that the flexibility of data transmission can be improved; on the other hand, data can be directly sent from the computing node of the data sending party to the computing node of the data receiving party, and the data transmission occurs between the computing nodes with intersection in the key value range, so that the complexity of network topology can be reduced, and the data transmission efficiency can be improved. Therefore, the method and the device for transmitting the data of the distributed computing architecture can improve the flexibility and efficiency of data transmission of the distributed computing architecture, and can realize efficient transmission and flexible transmission of the data of the distributed computing architecture.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a distributed computing architecture provided by embodiments of the present application;
fig. 2 is a flowchart of a data transmission method of a distributed computing architecture according to an embodiment of the present application;
fig. 3 is a second flowchart of a data transmission method of a distributed computing architecture according to an embodiment of the present application;
FIG. 4 is a third flowchart of a data transmission method of a distributed computing architecture according to an embodiment of the present application;
FIG. 5 is a block diagram of a data transmission apparatus of a distributed computing architecture according to an embodiment of the present application;
fig. 6 is a second structural diagram of a data transmission apparatus of a distributed computing architecture according to an embodiment of the present application;
fig. 7 is a block diagram of a data transmission device of a distributed computing architecture according to an embodiment of the present application.
Detailed Description
Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element.
Different from the traditional big data processing technology, the privacy computation scene with multiple parties has the following particularity:
a traditional distributed computing architecture generally has a global scheduling mechanism, and information of all computing nodes in a cluster is mastered. And the privacy calculation process is participated by a plurality of heterogeneous participants, and the participants are in a cooperative relationship.
In the process of privacy computation, a large amount of data is often required to be transmitted between the participants (across the public network) in respective computing clusters, and the conventional distributed architecture is often required to transmit data only in the same computing cluster.
Therefore, when designing a multi-party private computing distributed computing architecture, it is necessary to consider how efficiently data is transferred from one computing cluster to another computing cluster in a complex network topology composed of multiple computing clusters.
The related technical solutions can be roughly divided into two categories: traditional (single-compute) distributed computing architectures, represented by Spark et al, and open source federal learning frameworks, such as FATE (Federated AI Technology Enabler).
In conventional distributed computing architectures, there is a need to transfer data from one computing phase to another. For example, in Spark, there are "wide dependencies" and "narrow dependencies" between different computation phases. When processing reaches a new compute stage where there is a "wide dependency" between previous compute stages, data is transferred from a set of compute nodes corresponding to the previous compute stage to a new set of compute nodes corresponding to the new compute stage. In this process, spark can ensure that data is transmitted in an efficient and robust manner. However, the design of such a computing framework does not consider the scenario of cooperation of too many participants, and is difficult to adapt to the requirement of private computing.
The FATE is used as a federal learning framework for sourcing of the micro-banking bank, and distributed computation can be executed through an eggroll engine sourced by the FATE; data transmission can be carried out between eggroll engines of different participants. While the FATE may support data transfer between multi-party computing clusters, it has the following problems:
only the full amount of data of a certain data set on one node can be transferred to another node. Data transfer supported by the FATE is to transfer all data at a certain location on one node to another node. In the process of data transfer across the computers, this means that the nodes where the two computers are involved in the data transfer need to be fully peer-to-peer. In practical applications, because the calculation amount of each participant is different, or the calculation resource amount actually owned by each participant is different, the two parties of data transmission often adopt different numbers of calculation nodes. The current framework of FATE has difficulty ensuring efficient transfer of data in this scenario.
The transmission of data involves multiple retransmissions and there is a single point of problem in the transmission process. The data transmission of the FATE needs the components of the rollite in the FATE to complete, and each computer side deploys a single rollite node. The data transmission process comprises the following steps: local eggroll node → local rolsite node → opposing eggroll node. Such a transmission process is inefficient because multiple forwarding is involved, and CPU resources are consumed for encoding and decoding data in the rollite node; meanwhile, the rollite node becomes a bottleneck in the transmission process and has the hidden trouble of single-point failure.
Based on this, the embodiment of the application provides a data transmission method of a distributed computing architecture, which is suitable for the field of private computing, and can realize efficient and flexible transmission of data among computing clusters of multiple parties. However, it should be noted that the data transmission method of the distributed computing architecture provided in the embodiment of the present application is not limited to data transmission between multi-party distributed computing architectures, and may be applied to data transmission between any distributed computing architectures.
For ease of understanding, some of the contents of the embodiments of the present application are described below:
data set: short for data set. In the present embodiment, each data set has a globally unique ID (dataset _ ID). Are typically divided into several partitions (also referred to as data slices) that are stored scattered across multiple compute nodes of a compute cluster. The specific content of the data set depends on the specific privacy computation task being performed, e.g., it may represent encrypted gradients that need to be sent from one computing party to another after one iteration of model training is completed. The first data set is a data set to be transmitted in the embodiment of the present application.
A driver: the specific algorithm logic responsible for the privacy calculation divides the whole privacy calculation process into a plurality of calculation stages, divides each calculation stage into a plurality of subtasks and distributes the subtasks to the calculation nodes for execution.
A computing resource pool: the sum of all compute nodes within a compute cluster. Typically it is controlled by a higher level resource management scheduling framework, such as kubernets.
The computing node: a component of the distributed computing architecture is responsible for performing the tasks distributed by the drives, storing the data of the data sets, and performing a computing subtask. Usually, a plurality of compute nodes are deployed in a compute cluster to achieve the purpose of parallel computing.
Data set fragmentation strategy: the strategy of deciding in what way a data set is divided into slices and stored on different compute nodes. In the embodiment of the application, the slicing strategies of multiple parties on the same data set can be different. Data in a dataset is stored in the form of key-value pairs (key-value pair), and the dataset can be sliced according to the range of keys (keys). Assuming that the data sender has N computing nodes to store the data of the first data set, the value range of the key of the first data set may be divided into N disjoint value ranges (which may also be referred to as value ranges), and each value range is allocated to one computing node, so that each computing node corresponds to the key value range of one first data set, and the computing node is responsible for storing the key value pair of the key in the allocated value range. Such as: under the condition that the value ranges of the keys of the first data set are all integers, the N value intervals may be (∞, K) 1 ], （K 1 ，K 2 ],...（K N－1 ，﹢∞）。
A data sending party: in an embodiment of the application, a participant in a privacy calculation process is used for sending a first data set. The fragmentation strategy of the data sending party for the first data set is to divide the first data set into N fragments which are respectively stored in N computing nodes, and each computing node in the N computing nodes respectively corresponds to a key value range of the first data set. Where N is a positive integer, it is understood that the number of computing nodes included in the data sender may be greater than or equal to N.
The data receiving side: in an embodiment of the application, a participant in a privacy computation process is configured to receive a first data set. The fragmentation strategy of the data receiver on the first data set is to divide the first data set into M fragments which are respectively stored in M computing nodes, and each computing node in the M computing nodes respectively corresponds to a key value range of the first data set. Where M is a positive integer, it is understood that the number of compute nodes included in the data receiver may be greater than or equal to M.
In practical application, selecting N computing nodes and corresponding relations between the N computing nodes and the N fragments; the selection of the M computing nodes and the correspondence between the M computing nodes and the M segments may be specifically determined according to the amount of computation that each node in the data sending party can bear and the amount of computation resources in practical application, which is not limited in this embodiment of the present application.
In the embodiment of the present application, N may be equal to or different from M. In addition, even if N is equal to M, the key value ranges of the first data sets corresponding to at least two computing nodes from the data sending party and the data receiving party may be different in the N computing nodes and the M computing nodes, that is, the data of the first data sets responsible for storage are different. That is to say, the embodiment of the present application does not require that the data transmission computing node at the data transmission side and the data transmission computing node at the data reception side are equal, so that the flexibility of data transmission and storage can be improved.
In addition, in the distributed computing architecture of the embodiment of the present application, the number of data sending parties may be greater than or equal to 1, and the number of data receiving parties may also be greater than or equal to 1.
To facilitate understanding of the distributed computing architecture of the embodiments of the present application, reference may be made to fig. 1. In fig. 1, the distributed computing architecture includes a participant 1 and a participant 2, where the participant 1 is a data sender and includes a driver and a computing resource pool, and the computing resource pool includes N computing nodes. Participant 1 is a data receiver and includes a drive and a computing resource pool that includes M computing nodes. Data transmission can be directly carried out between the drivers, and data transmission can be directly carried out between the computing nodes.
The data transmission method of the distributed computing architecture provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings through some embodiments and application scenarios thereof.
Referring to fig. 2, fig. 2 is a flowchart of a data transmission method of a distributed computing architecture according to an embodiment of the present application. As shown in fig. 2, the data transmission method of the distributed computing architecture may include the following steps:
step 201, a data receiving side sends a data obtaining request for the first data set to the data sending side, where the data obtaining request includes key value ranges of the first data set corresponding to the M computing nodes, respectively.
Correspondingly, the data sending party receives a data acquisition request aiming at the first data set and sent by the data receiving party, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes.
Step 202, the data receiver receives the data of the first data set sent by the N computing nodes through the M computing nodes.
Accordingly, the data sender sends the data of the first data set to the M computing nodes through the N computing nodes.
The data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; and the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In the data transmission method of the distributed computing architecture in the embodiment of the application, the data sending party sends the first data set to the data receiving party, wherein the data of the first data set is stored in the N computing nodes of the data sending party, and the data receiving party allocates the M computing nodes to store the data of the first data set. Before the data sender sends the first data set to the data receiver, the data acquisition request for the first data set sent by the data receiver to the data sender may include key value ranges of the first data set corresponding to the M computing nodes, so that the data sender knows specific data of the first data set, which is stored by each computing node in the M computing nodes. After receiving the data acquisition request, the data sending party can directly send the data of the first data set to the M computing nodes through the N computing nodes, wherein the data of the first data set received by each computing node in the M computing nodes can be sent by at least one computing node in the N computing nodes, which has an intersection with the key value range of the first data set corresponding to the computing node. Therefore, on one hand, the data sender and the data receiver can respectively and flexibly configure the number of the nodes used for data transmission in the respective computing cluster, so that the flexibility of data transmission can be improved; on the other hand, data can be directly sent from the computing node of the data sending party to the computing node of the data receiving party, and data transmission occurs between the computing nodes with intersection in the key value range, so that the complexity of network topology can be reduced, and the efficiency of data transmission can be improved. Therefore, the method and the device for transmitting the data of the distributed computing architecture can improve the flexibility and efficiency of data transmission of the distributed computing architecture, and can realize efficient transmission and flexible transmission of the data of the distributed computing architecture.
The above steps will be explained below.
With respect to step 201
In a specific implementation, the data obtaining request may include an identifier and a key value range of the first data set, so as to represent that the data obtaining request is a data obtaining request for the first data set.
The data acquisition request sent by the data receiver is used for: and informing the data sending party that the M computing nodes of the data receiving party are respectively responsible for storing the data of the first data set, so that the corresponding computing nodes of the data sending party send the data of the first data set which is responsible for storing to each computing node of the M computing nodes. Specifically, the data of the first data set, which is responsible for storage by each computing node of the M computing nodes, is data of a key value range of the key in the first data set corresponding to the computing node.
In some embodiments, the data acquisition request may be actively sent by a driver of the data receiving side (hereinafter referred to as a second driver) to a driver of the data transmitting side (hereinafter referred to as a first driver). In a specific implementation, after acquiring the first data set fragmentation policy of the data receiver, the second driver may send a data acquisition request for the first data set to the first driver. In this embodiment, the data obtaining request includes key value ranges of the first data set respectively corresponding to the M computing nodes, which may be: the data acquisition request includes a first data set fragmentation policy of the data recipient.
In this embodiment, in order for each of the N computing nodes in the data sender to determine the receiving computing node in the M computing nodes that stores the data of the first data set, further, the data obtaining request may further include address information of the M computing nodes. In this way, after acquiring the address information of the M computing nodes and the key value ranges of the first data sets corresponding to the M computing nodes, the first driver may determine the computing nodes having intersections between the N computing nodes and the M computing nodes, by combining a first data set fragmentation policy of the data receiver itself, that is, the key value ranges of the first data sets corresponding to the N computing nodes, respectively.
Thereafter, the first driver may assign data transfer tasks to the N compute nodes, respectively. The data transmission task of each of the N computing nodes is used to: and sending the data of the first data set for at least one computing node which has intersection with the key value range of the corresponding first data set in the M computing nodes. Specifically, the data transmission task of each of the N computing nodes may include: the method includes the steps that address information of at least one computing node which has intersection with a key value range of a first data set corresponding to the computing node in M computing nodes, and the key value ranges of the first data sets corresponding to the at least one computing node respectively. In this way, after each of the N computing nodes acquires its respective data transmission task, it may determine, based on the address information in the data transmission task, its corresponding data receiver computing node and the data of the first data set that it needs to send to its corresponding data receiver computing node, thereby implementing transmission of the data of the first data set stored therein.
In this embodiment, the data transmission method mainly includes the following steps:
1) A driver of a data receiver sends metadata to a driver of a data sender, wherein the metadata comprises address information of a computational node of the data receiver and a first data set fragmentation strategy of the data receiver;
2) A driver of a data sender can distribute data transmission tasks to each computing node of the data sender based on received metadata and a first dataset fragmentation strategy of the driver, wherein the data transmission tasks comprise address information of the computing nodes of a corresponding data receiver and corresponding key value ranges;
3) And the data sending party computing node directly sends the data corresponding to the first data set to the corresponding data receiving party computing node based on the distributed data transmission task.
In other embodiments, the data acquisition request may be sent by the M computing nodes to the N computing nodes. The data acquisition request sent by each of the M computing nodes includes a key value range of the corresponding first data set.
In this embodiment, in order to enable each computing node of the data receiving party to successfully request the data of the first data set that the data receiving party is responsible for storing, a data acquisition request needs to be sent to a computing node, which has an intersection with the key value range of the corresponding first data set, of the N computing nodes. Therefore, the data receiving side needs to obtain the key value ranges of the first data sets corresponding to the N computing nodes in advance. In a specific implementation, before the M computing nodes send the data obtaining request, the driver of the data sending party may send, to the driver of the data receiving party, the address information of the N computing nodes and the key value ranges of the first data sets respectively corresponding to the N computing nodes.
In this way, after acquiring the address information of the N computing nodes and the key value ranges of the first data sets corresponding to the N computing nodes, the second driver may determine the computing nodes having intersections between the N computing nodes and the M computing nodes, by combining a first data set fragmentation policy of the data receiver itself, that is, the key value ranges of the first data sets corresponding to the M computing nodes, respectively.
Thereafter, the second driver may allocate data acquisition tasks (which may also be referred to as data pull tasks) to the M compute nodes, respectively. The data acquisition task of each of the M compute nodes is used to: and sending a data acquisition request from at least one computing node which has intersection with the key value range of the corresponding first data set in the N computing nodes. Specifically, the data obtaining task of each of the M computing nodes may include: and the address information of at least one computing node which has intersection with the key value range of the corresponding first data set in the N computing nodes, and the key value range of the corresponding first data set. In this way, after each of the M computing nodes acquires its respective data acquisition task, it may send a corresponding data request task to the corresponding data sender computing node based on the address information in the data acquisition task, and pull the data of the first data set that it is responsible for storing from the corresponding data sender computing node, thereby implementing the acquisition of the data of the first data set that it is responsible for storing.
In this embodiment, the data transmission method mainly includes the following steps:
1) A driver of a data sender sends metadata to a driver of a data receiver, wherein the metadata comprises address information of a computational node of the data sender and a first data set fragmentation strategy of the data sender;
2) A driver of a data receiver can distribute data acquisition tasks to each computing node of the data receiver based on received metadata and a first dataset fragmentation strategy of the driver, wherein the data acquisition tasks comprise address information of the computing node of the data sender corresponding to the computing node and a key value range corresponding to the computing node;
3) The data receiving side computing node directly sends a data acquisition request to the corresponding data sending side computing node based on the distributed data acquisition task;
4) The data sender computing node may return data of the first data set in response to the received data acquisition request.
In an embodiment of the present application, the address information of the computing node may include an Internet Protocol (IP) address and a port of the computing node.
With respect to step 202
In step 202, the M computing nodes and the N computing nodes directly perform data transmission of the first data set without forwarding by other nodes, so that the efficiency of data transmission can be improved.
In this embodiment of the present application, it may be considered that the data sending party computing node and the data receiving party node, which have an intersection in the key value range of the corresponding first data set, have a corresponding relationship. In a specific implementation, the data receiving computing node may obtain the data of the first data set that it is responsible for storing from the data receiving node having the corresponding relationship with the data receiving node.
In the embodiment of the application, one data sending party computing node can correspond to one or more data receiving party computing nodes; one data receiver computing node may correspond to one or more data sender computing nodes. In particular implementations, one of the N computing nodes may send data of the first data set to one or more of the M computing nodes. Thus, one of the M computing nodes may receive data of the first data set transmitted by one or more of the N computing nodes.
For ease of understanding, examples are illustrated below:
assume that the first data set includes 10 key-value pairs, which are key-value pair 1 to key-value pair 10, respectively, the key corresponding to the key-value pair X is X, the value range of X is 1 to 10, and X is a positive integer.
Meanwhile, the N computing nodes of the data sender are assumed to comprise a computing node a, a computing node b and a computing node c. The key value range of the first data set corresponding to the computing node a is 1-5; calculating key value ranges of the first data set corresponding to the node b to be 6 and 7; the key value range of the first data set corresponding to the computing node c is 8-10.
The M computing nodes of the data sending party comprise a computing node d and a computing node e. The key value range of the first data set corresponding to the computing node d is 1-8; the key value range of the first data set corresponding to the computing node e is 9 and 10.
Then, since the key value ranges of the first data sets corresponding to the computing node a, the computing node b, and the computing node c all intersect with the key value range of the first data set corresponding to the computing node d, it may be determined that the computing node a, the computing node b, and the computing node c all have a corresponding relationship with the computing node d, and therefore, the computing node a may send the key value pairs 1 to 5 to the computing node d, the computing node b may send the key value pairs 6 and 7 to the computing node d, and the computing node c may send the key value pair 8 to the computing node d. Since only the key value range of the first data set corresponding to the computing node c intersects with the key value range of the first data set corresponding to the computing node e, it can be determined that the computing node c and the computing node e have a corresponding relationship, and therefore, the computing node c can send the key value pairs 9 and 10 to the computing node e.
It should be noted that, in the embodiment of the present application, information transmission between the computing nodes, such as transmission of a data acquisition request and transmission of data, may be performed in parallel, so that information transmission efficiency may be improved. In addition, the information transmission between the computing nodes can be performed through a google Remote Procedure Call (gRPC) interface, but is not limited thereto.
Referring to fig. 3, fig. 3 is a second flowchart of a data transmission method of a distributed computing architecture according to an embodiment of the present application. For the method embodiment of FIG. 3, the data sender includes a first driver and a first computing resource pool, the first computing resource pool including N computing nodes; the data receiver includes a second driver and a second computing resource pool, the second computing resource pool including M computing nodes. It will be appreciated that the N compute nodes may be some or all of the compute nodes in the first computing resource pool. The M compute nodes may be some or all of the compute nodes in the second computing resource pool.
As shown in fig. 3, the data transmission method of the distributed computing architecture may include the following steps:
step 301, the first driver sends first information to the second driver, where the first information includes address information of the N computing nodes and key value ranges of the first data set corresponding to the N computing nodes, respectively.
Accordingly, the second driver receives the first information sent by the first driver.
In this embodiment, as shown in fig. 1, data transmission between a data sender and a data receiver can be divided into two phases: the first phase is metadata transfer between drives; the second phase is the actual transmission between the compute nodes.
Step 301 is the first stage, and in step 301, the metadata is the first information. At this stage, the data sender sends, to the data receiver, address information of the computing nodes in which the data of the first data set is stored and a fragmentation policy of the first data set among the computing nodes, so that the data receiver computing node can send a data acquisition request to the corresponding data sender computing node to acquire the data of the first data set that the data receiver computing node is responsible for storing.
Step 302, the second driver determines a correspondence between each of the N computing nodes and each of the M computing nodes according to the key value ranges of the first data sets corresponding to the M computing nodes, respectively, and the key value ranges of the first data sets corresponding to the N computing nodes, respectively; and the key value range of the first data set corresponding to the computing nodes with the corresponding relation has an intersection.
In the second phase, the second driver may also obtain a sharding policy of the first data set at its own computing node. Then, in combination with the first information, the data sending party computing node and the data receiving party computing node, where the key value ranges of the corresponding first data sets have intersection, may be determined, so as to determine that the data receiving party computing node obtains the data of the first data set from those data sending party computing nodes.
And 303, the second driver sends data acquisition tasks to the M computing nodes respectively according to the corresponding relationship, where the data acquisition task of the ith computing node includes address information of at least one computing node of the N computing nodes, and a key value range of the first data set corresponding to the ith computing node.
In step 303, the second driver may allocate data acquisition tasks to each of the M computing nodes, where the data acquisition tasks allocated to each computing node include: and the address information of the corresponding data sender computational node and the key value range of the corresponding first data set.
Step 304, the M computing nodes send data acquisition requests for the first data set to the N computing nodes according to the data acquisition tasks, respectively.
Accordingly, the N computing nodes receive data acquisition requests for the first data set sent by the M computing nodes.
The destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
In step 304, after acquiring the allocated data acquisition task, each computing node may send a data acquisition request including a key value range of the first data set corresponding to the computing node based on the address information in the data acquisition task, so that the computing node of the data sender of the address information returns the data of the first data set that the computing node is responsible for storing.
Step 305, the N computing nodes send data of the first data set to the M computing nodes.
Accordingly, the M computing nodes receive the data of the first data set sent by the N computing nodes.
The data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; and the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
The implementation of step 305 can refer to the implementation of step 202 in the embodiment of the method in fig. 2, and is not described herein again. The data transfer of the first data set is the actual data transfer in the second phase.
In this embodiment, the driver of the data receiving side may determine, by receiving the first information sent by the driver of the data sending side and acquiring the fragmentation policy of the first data set in the driver of the data sending side, the computing nodes having intersections with key value ranges of the first data set corresponding to the computing nodes in the M computing nodes, respectively, so as to determine that the computing nodes in the M computing nodes need to pull data of the first data set from the computing nodes in the N computing nodes, and further directly perform transmission of the first data set through the M computing nodes and the N computing nodes, thereby improving efficiency of data transmission.
In some embodiments of the present application, after receiving, by the M computing nodes, the data of the first data set sent by the N computing nodes, the method further includes:
under the condition that the M computing nodes successfully receive the data of the first data set sent by the N computing nodes, the M computing nodes respectively store the received data of the first data set and send indication information to the second driver, wherein the indication information is used for indicating that the transmission task of the first data set is completed;
the second driver forwards the indication information to the first driver.
In this embodiment, after each of the M computing nodes successfully receives the data of the first data set that is responsible for storage, the second driver may be informed that the transmission task of the first data set of the second driver is completed, and the received data is stored.
After acquiring that all the computing nodes in the M computing nodes successfully receive the data which is in charge of being stored by the computing nodes, the second driver may notify the first driver that the transmission task of the first data set is completed, so as to end the transmission task of the first data set.
In this way, the data sender may know that the transmission of the first data set has been successfully completed, and thus may improve the reliability of data transmission.
In some embodiments of the present application, a computing node includes a task distribution service and a data transmission service; the second driver sends data acquisition tasks to the M computing nodes respectively according to the correspondence, including:
the second driver respectively sends data acquisition tasks to the task distribution services of the M computing nodes according to the corresponding relation;
the M computing nodes respectively send data acquisition requests to the N computing nodes according to the data acquisition tasks, and the data acquisition requests comprise:
the M computing nodes respectively send data acquisition requests to the data transmission services of the N computing nodes according to the data acquisition tasks;
the receiving, by the M computing nodes, the data of the first data set sent by the N computing nodes includes:
and the data transmission service of the M computing nodes receives the data of the first data set sent by the N computing nodes.
In this embodiment, the task distribution service of the computing node may be responsible for sending the data acquisition task, and the data transmission task of the computing node is responsible for transmitting the data acquisition request and the data, so that different services of the computing node may be responsible for different matters, thereby improving the efficiency of the matter processing.
It should be noted that, in the embodiments of the present application, various optional implementations that are described in the embodiments of the present application may be implemented in combination with each other or may be implemented separately without conflicting with each other, and the embodiments of the present application are not limited to this.
For ease of understanding, examples are illustrated below:
in the distributed computing architecture of the embodiment of the application, the computing node provides a gRPC interface for data transmission, and can return all key value pairs corresponding to key values in the key _ range in the dataset according to dataset _ id and the key value range (key _ range).
The data transmission across the computing clusters in the embodiment of the application is divided into two stages: the first phase is metadata transfer between drives; the second phase is the actual data transfer between the compute nodes.
In the first phase of data transmission, the driver of the data sender first sends the metadata of the data set to be transmitted to the driver of the receiver. Metadata mainly consists of two parts:
and (3) computing a node set: the sender stores a set of all the compute nodes for the data set. The set includes ip addresses and ports of the respective computing nodes, and is available for the data receiver to pull data through the gRPC interface for data transmission as described above. It should be noted that within the present computing resource pool, there may be computing nodes that do not store any data of the relevant data set, and these nodes are not within the set of computing nodes mentioned herein.
A dataset sharding policy.
In the second stage, the data receiver determines the slicing policy after receiving the data before receiving the data. After receiving the metadata of the sender data set, the transmission of the actual data is completed through the following steps:
1. the driver of the receiving party can determine the value range of the key to be stored by each computing node of the receiving party according to the data set fragmentation strategy of the receiving party. Then, according to the data set fragmentation strategy of the sender, the intersection between the value range of the key of the data stored by which computing nodes of the sender and the range of the key to be stored by each node of the sender can be obtained, so that each computing node of the sender can know which nodes to pull the data required by the computing node from which data is sent.
2. As described above, each computing node of the sender provides a gRPC interface for data transmission to the outside. The driver of the receiving party issues a task of pulling data to each computing node which needs to receive data according to the result obtained in the step 1; the task will contain the address of the data sender's compute node that each compute node needs to pull the data. After receiving the tasks, each computation receiver sends a request for acquiring data according to the value range of the key to a data transmission gPC service on a settlement node of a corresponding sender according to requirements, acquires corresponding data and stores the data locally in the computation node.
Different from the traditional distributed computing architecture, the proposal designs a distributed computing architecture facing a plurality of computing participants, and solves the problem of data transmission among computing clusters of the plurality of the computing participants. The main advantages of this proposal compared to FATE include the following two:
the data sender and the data receiver can respectively and flexibly configure the number of the nodes used for data transmission in the respective clusters. In FATE, data can only be sent from one compute node on the sender to another compute node on the recipient in full. According to the embodiment of the application, each computing node of a data receiver determines the value range of the key stored by the computing node according to the local data set fragmentation strategy, and acquires the required data from the computing node of the required data sender directly according to the range of the key. Therefore, the proposal can allow each participant to flexibly set according to different calculation amounts born by the participant or different actual calculation resource amounts owned by each participant without considering the number of the calculation nodes of the other participant. Meanwhile, because the data sending party and the data receiving party both adopt the range fragmentation strategy, the requirement that the computing node of the receiving party needs to send the data acquisition request to all the computing nodes of the sending party is not generated, and the complexity of the network topology is reduced.
Data is transmitted from a computing node on the sender side directly to a computing node on the receiver side. And the computing node of the receiver directly sends a request to the corresponding data transmission gPC service according to the address provided by the sender to acquire data without extra forwarding. At the same time, there is no single point of failure problem as in FATE.
One scenario embodiment is shown in FIG. 4:
before the computing clusters of the data sending party and the data receiving party carry out data transmission, drivers of the two parties need to establish a websocket connection. Meanwhile, the data receiving party should also determine the data fragmentation strategy adopted when the party receives data. The specific process of the whole process is as follows:
1. the sender driver sends the metadata of the data set to be sent to the receiver driver through the websocket.
The metadata includes a set of compute nodes that store the data set and a sharding policy.
2. The receiving side driver distributes the task of data pulling to the computing node of the local side according to the metadata of the data set, and the method specifically comprises the following steps:
2.1. data set metadata is received.
In specific implementation, the metadata of the data set sent by the driver of the sending party is received through the websocket connection.
2.2. It is determined that each of the local compute nodes needs to pull data from those nodes that are the sender.
In specific implementation, according to a data set fragmentation strategy of a sender, intersection between a value range of keys of data stored by which compute nodes of the sender and a range of keys to be stored by each node of the sender can be obtained, so that each compute node of the sender is informed of which compute nodes of the data sender the data which should be received by the compute node are respectively stored on.
2.3. Sending the task of pulling data.
During specific implementation, according to the result of the previous step and in combination with the address information of each computing node of a sender contained in the metadata of the data set, a data pulling task is sent to each computing node of the party in parallel through a gRPC service provided by a task distributed on the computing node, and the address of the computing node of the sender, which needs to pull data, of each computing node and the value range of the corresponding key are informed to each computing node.
3. The method for pulling the data from the sender by the computing node of the receiver according to the received data pulling task specifically comprises the following steps:
3.1. a task to pull data is received.
In specific implementation, through a gPC service distributed by a task running on the node, the value range of keys of data set fragments which are stored by the node in charge is obtained, and data is pulled from which computing nodes of a data sender
3.2. And sending a request for acquiring data according to the key value range.
In specific implementation, a data transmission gPRC service of a sending side computing node is used for sending a data pulling request to all the sending side computing nodes of data to be pulled by the nodes in parallel.
4. The method for transmitting the data of the node to the node of the receiver by the computing node of the sender specifically includes:
4.1. and receiving a request for acquiring data according to the key value range.
In specific implementation, a request for pulling data to the node is obtained through a data transmission gRPC service running on the node.
4.2. And returning the data of the data set according to the request.
In specific implementation, according to the value range of the key contained in the request for pulling the data, the data of the key stored on the node in the requested value range is returned to the computing node of the data receiver.
5. The method for receiving the data of the data set from the sender and storing the data in the local by the computing node of the receiver specifically includes:
5.1. and receiving the data returned by the computing node of the sending party.
In a specific implementation, data returned by each node is received through a data pull request sent by the data transmission gRPC service to the sending computing node in step 3.2.
5.2. And storing the data returned by each computing node of the sender.
In specific implementation, the received data of the sender is stored in the local storage unit of the computing node.
5.3. The completion of the data pull task is confirmed to the local driver.
In specific implementation, after all the requests for pulling data sent by the node in step 3.2 successfully return data and the data is already stored in the local storage unit of the node, the task distribution request sent by the receiver driver in step 2.3 is sent to confirm that the task is completed.
6. The receiving driver confirms that the data pulling is completed, and specifically includes:
6.1. and receiving confirmation of completion of the data pulling task.
In concrete implementation, it waits for all the task distribution requests sent in step 2.3 to return success.
6.2. The completion of the data transfer is confirmed to the sender driver.
In specific implementation, the Websocket is connected with a driver of a data sending party to confirm that data transmission is finished.
7. The sender driver receives a data transmission completion acknowledgement from the receiver driver.
At this point, the data transmission ends.
Based on the data transmission method of the distributed computing architecture provided in the foregoing embodiment, accordingly, the present application also provides a specific implementation manner of the data transmission device of the distributed computing architecture. Please see the examples below.
Referring to fig. 5, a data transmission apparatus of a distributed computing architecture provided in an embodiment of the present application may include:
a first sending module 501, configured to send a data acquisition request for the first data set to the data sender, where the data acquisition request includes key value ranges of the first data set corresponding to the M computing nodes, respectively;
a first receiving module 502, configured to receive, by the M computing nodes, data of the first data set sent by the N computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; and the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In some embodiments, the data sender comprises a first driver and a first computing resource pool, the first computing resource pool comprising the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
the second driver is configured to: receiving first information sent by the first driver, where the first information includes address information of the N computing nodes and key value ranges of the first data sets corresponding to the N computing nodes, respectively;
determining a correspondence between each of the N computing nodes and each of the M computing nodes according to the key value ranges of the first data sets corresponding to the M computing nodes, respectively, and the key value ranges of the first data sets corresponding to the N computing nodes, respectively; the key value ranges of the first data set corresponding to the computing nodes with the corresponding relation have intersection;
according to the corresponding relation, data acquisition tasks are respectively sent to the M computing nodes, wherein the data acquisition task of the ith computing node comprises address information of at least one computing node in the N computing nodes and a key value range of the first data set corresponding to the ith computing node;
the M compute nodes to: sending data acquisition requests aiming at the first data set to the N computing nodes respectively according to the data acquisition tasks;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
In some embodiments, the computing node includes a task distribution service and a data transmission service;
the second driver is configured to: respectively sending data acquisition tasks to the task distribution services of the M computing nodes according to the corresponding relation;
the M compute nodes to:
respectively sending data acquisition requests to the data transmission services of the N computing nodes according to the data acquisition tasks;
and receiving the data of the first data set sent by the N computing nodes through a data transmission service.
The data transmission device of the distributed computing architecture provided in the embodiment of the present application can implement each process implemented by the data receiving party in the foregoing method embodiments, and is not described here again to avoid repetition.
Referring to fig. 6, a data transmission apparatus of a distributed computing architecture provided in an embodiment of the present application may include:
a second receiving module 601, configured to receive a data acquisition request for the first data set sent by the data receiving side, where the data acquisition request includes key value ranges of the first data set corresponding to the M computing nodes, respectively;
a second sending module 602, configured to send, by the N computing nodes, data of the first data set to the M computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; and the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage.
In some embodiments, the data sender comprises a first driver and a first computing resource pool, the first computing resource pool comprising the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
the first driver is configured to:
sending first information to the second driver, where the first information includes address information of the N computing nodes and key value ranges of the first data sets corresponding to the N computing nodes, respectively;
the N compute nodes to:
receiving data acquisition requests aiming at the first data set, which are sent by the M computing nodes;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
The data transmission device of the distributed computing architecture provided in the embodiment of the present application can implement each process implemented by the data sending party in the foregoing method embodiments, and is not described here again to avoid repetition.
Fig. 7 is a hardware block diagram illustrating data transmission of a distributed computing architecture according to an embodiment of the present application.
A data transfer device in a distributed computing architecture may include a processor 701 and a memory 702 storing computer program instructions.
Specifically, the processor 701 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
Memory 702 may include a mass storage for data or instructions. By way of example, and not limitation, memory 702 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 702 may include removable or non-removable (or fixed) media, where appropriate. The memory 702 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 702 is non-volatile solid-state memory.
The Memory may include Read-Only Memory (ROM), random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash Memory devices, electrical, optical, or other physical/tangible Memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform operations described with reference to the methods according to an aspect of the present disclosure.
The processor 701 reads and executes the computer program instructions stored in the memory 702 to implement the data transmission method of any one of the distributed computing architectures in the above embodiments.
In one example, the data transport device of the distributed computing architecture may also include a communication interface 707 and a bus 710. As shown in fig. 7, the processor 701, the memory 702, and the communication interface 707 are connected to each other via a bus 710 to complete communication therebetween.
The communication interface 707 is mainly used for implementing communication between various modules, apparatuses, units and/or devices in this embodiment.
The bus 710 comprises hardware, software, or both to couple the components of the data transfer device of the distributed computing architecture to one another. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 710 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.
In addition, in combination with the data transmission method of the distributed computing architecture in the foregoing embodiments, the embodiments of the present application may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement the data transmission method of any one of the above embodiments.
It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.
The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based computer instructions which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
As described above, only the specific embodiments of the present application are provided, and it can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims (7)

1. A data transmission method in a distributed computing architecture is applied to a data receiving party, and is characterized in that the data receiving party is used for receiving a first data set from a data sending party; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set; the key value ranges of the first data sets corresponding to at least two computing nodes from the data sender and the receiver respectively exist in the N computing nodes and the M computing nodes, and N and M are positive integers;
the method comprises the following steps:
sending a data acquisition request aiming at the first data set to the data sender, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes;
receiving, by the M computing nodes, the data of the first data set sent by the N computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage;
the data sender comprises a first driver and a first computing resource pool, and the first computing resource pool comprises the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
before sending the data acquisition request for the first data set to the data sender, the method further includes:
the second driver receives first information sent by the first driver, wherein the first information comprises address information of the N computing nodes and key value ranges of the first data set corresponding to the N computing nodes respectively;
the second driver determines a corresponding relationship between each of the N computing nodes and each of the M computing nodes according to the key value range of the first data set corresponding to each of the M computing nodes and the key value range of the first data set corresponding to each of the N computing nodes; the key value ranges of the first data set corresponding to the computing nodes with the corresponding relation have intersection;
the second driver respectively sends data acquisition tasks to the M computing nodes according to the corresponding relation, wherein the data acquisition task of the ith computing node comprises address information of at least one computing node in the N computing nodes and a key value range of the first data set corresponding to the ith computing node;
the sending, to the data sender, a data acquisition request for the first data set includes:
the M computing nodes respectively send data acquisition requests aiming at the first data set to the N computing nodes according to the data acquisition tasks;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
2. The method of claim 1, the compute nodes comprising a task distribution service and a data transmission service;
the second driver sends data acquisition tasks to the M computing nodes respectively according to the correspondence, including:
the second driver respectively sends data acquisition tasks to the task distribution services of the M computing nodes according to the corresponding relation;
the M computing nodes respectively send data acquisition requests to the N computing nodes according to the data acquisition tasks, and the data acquisition requests comprise:
the M computing nodes respectively send data acquisition requests to the data transmission services of the N computing nodes according to the data acquisition tasks;
the receiving, by the M computing nodes, the data of the first data set sent by the N computing nodes includes:
and the data transmission service of the M computing nodes receives the data of the first data set sent by the N computing nodes.
3. A data transmission method in a distributed computing architecture is applied to a data sender, and is characterized in that the data sender is used for sending a first data set to a data receiver; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set; the N computing nodes and the M computing nodes have different key value ranges of the first data sets corresponding to at least two computing nodes from the data sender and the receiver respectively, and N and M are positive integers;
the method comprises the following steps:
receiving a data acquisition request aiming at the first data set and sent by the data receiver, wherein the data acquisition request comprises key value ranges of the first data set respectively corresponding to the M computing nodes;
sending, by the N computing nodes, data of the first data set to the M computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage;
the data sender comprises a first driver and a first computing resource pool, and the first computing resource pool comprises the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
before the receiving of the data acquisition request for the first data set sent by the data receiver, the method further includes:
the first driver sends first information to the second driver, wherein the first information comprises address information of the N computing nodes and key value ranges of the first data set corresponding to the N computing nodes respectively;
the receiving of the data acquisition request for the first data set sent by the data receiver includes:
the N computing nodes receive data acquisition requests aiming at the first data set and sent by the M computing nodes;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
4. A data transmission device of a distributed computing architecture is applied to a data receiving party, and is characterized in that the data receiving party is used for receiving a first data set from a data sending party; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set; the N computing nodes and the M computing nodes have different key value ranges of the first data sets corresponding to at least two computing nodes from the data sender and the receiver respectively, and N and M are positive integers;
the device comprises:
a first sending module, configured to send a data acquisition request for the first data set to the data sender, where the data acquisition request includes key value ranges of the first data set respectively corresponding to the M computing nodes;
a first receiving module, configured to receive, by the M computing nodes, data of the first data set sent by the N computing nodes;
the data of the first data set received by the ith computing node in the M computing nodes is sent by at least one computing node in the N computing nodes, the value range of i is 1-M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage;
the data sender comprises a first driver and a first computing resource pool, and the first computing resource pool comprises the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
the second driver is configured to:
receiving first information sent by the first driver, wherein the first information includes address information of the N computing nodes and key value ranges of the first data set corresponding to the N computing nodes respectively;
determining a corresponding relation between each of the N computing nodes and each of the M computing nodes according to the key value ranges of the first data sets corresponding to the M computing nodes respectively and the key value ranges of the first data sets corresponding to the N computing nodes respectively; the key value ranges of the first data set corresponding to the computing nodes with the corresponding relation have intersection;
according to the corresponding relation, data acquisition tasks are respectively sent to the M computing nodes, wherein the data acquisition task of the ith computing node comprises address information of at least one computing node in the N computing nodes and a key value range of the first data set corresponding to the ith computing node;
the M compute nodes to: sending data acquisition requests aiming at the first data set to the N computing nodes respectively according to the data acquisition tasks;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
5. A data transmission device of a distributed computing architecture is applied to a data sender, and is characterized in that the data sender is used for sending a first data set to a data receiver; wherein the data of the first data set is stored in N computing nodes of the data sender; the data receiver distributes M computing nodes to store the data of the first data set; the N computing nodes and the M computing nodes have different key value ranges of the first data sets corresponding to at least two computing nodes from the data sender and the receiver respectively, and N and M are positive integers;
the device comprises:
a second receiving module, configured to receive a data acquisition request for the first data set sent by the data receiving side, where the data acquisition request includes key value ranges of the first data set respectively corresponding to the M computing nodes;
a second sending module, configured to send, by the N computing nodes, data of the first data set to the M computing nodes;
the data of the first data set received by the ith computing node of the M computing nodes is sent by at least one computing node of the N computing nodes, the value range of i is from 1 to M, and i is a positive integer; the key value range of the first data set corresponding to at least one of the N computing nodes is intersected with the key value range of the first data set corresponding to the ith computing node; the key value range of the first data set corresponding to each computing node is used for indicating the data of the first data set which is responsible for storage;
the data sender comprises a first driver and a first computing resource pool, and the first computing resource pool comprises the N computing nodes; the data receiver comprises a second driver and a second computing resource pool, wherein the second computing resource pool comprises the M computing nodes;
the first driver is configured to:
sending first information to the second driver, where the first information includes address information of the N computing nodes and key value ranges of the first data set corresponding to the N computing nodes, respectively;
the N compute nodes to:
receiving data acquisition requests aiming at the first data set sent by the M computing nodes;
the destination address information of the data acquisition request sent by the ith computing node is address information of at least one computing node in the N computing nodes, and the data acquisition request sent by the ith computing node includes a key value range of the first data set corresponding to the ith computing node.
6. A data transmission device of a distributed computing architecture, the device comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a data transfer method of the distributed computing architecture of claim 1 or 2, or a data transfer method of the distributed computing architecture of claim 3.
7. A computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the data transfer method of the distributed computing architecture of claim 1 or 2, or the data transfer method of the distributed computing architecture of claim 3.
CN202211029216.1A 2022-08-26 2022-08-26 Data transmission method, device, equipment and storage medium of distributed computing architecture Active CN115118727B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211029216.1A CN115118727B (en) 2022-08-26 2022-08-26 Data transmission method, device, equipment and storage medium of distributed computing architecture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211029216.1A CN115118727B (en) 2022-08-26 2022-08-26 Data transmission method, device, equipment and storage medium of distributed computing architecture

Publications (2)

Publication Number Publication Date
CN115118727A CN115118727A (en) 2022-09-27
CN115118727B true CN115118727B (en) 2022-11-29

Family

ID=83335956

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211029216.1A Active CN115118727B (en) 2022-08-26 2022-08-26 Data transmission method, device, equipment and storage medium of distributed computing architecture

Country Status (1)

Country Link
CN (1) CN115118727B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5699500A (en) * 1995-06-01 1997-12-16 Ncr Corporation Reliable datagram service provider for fast messaging in a clustered environment
JPH10105522A (en) * 1996-06-27 1998-04-24 Hewlett Packard Co <Hp> Multicomputer system
CN109800236A (en) * 2019-01-28 2019-05-24 上海雷腾软件股份有限公司 Support the distributed caching method and equipment of multinode
WO2021213075A1 (en) * 2020-04-24 2021-10-28 中科寒武纪科技股份有限公司 Inter-node communication method and device based on multiple processing nodes
WO2022001375A1 (en) * 2020-06-28 2022-01-06 北京沃东天骏信息技术有限公司 Blockchain-based data storage method, system and apparatus
CN114697372A (en) * 2022-05-31 2022-07-01 深圳市泛联信息科技有限公司 Data transmission processing and storage method, system and medium in distributed system

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7395536B2 (en) * 2002-11-14 2008-07-01 Sun Microsystems, Inc. System and method for submitting and performing computational tasks in a distributed heterogeneous networked environment
US9378068B2 (en) * 2013-03-13 2016-06-28 International Business Machines Corporation Load balancing for a virtual networking system
KR102044023B1 (en) * 2013-03-14 2019-12-02 삼성전자주식회사 Data Storage System based on a key-value and Operating Method thereof
US9900386B2 (en) * 2015-04-09 2018-02-20 International Business Machines Corporation Provisioning data to distributed computing systems
US10977260B2 (en) * 2016-09-26 2021-04-13 Splunk Inc. Task distribution in an execution node of a distributed execution environment
US11588926B2 (en) * 2016-11-14 2023-02-21 Temple University—Of the Commonwealth System of Higher Education Statistic multiplexed computing system for network-scale reliable high-performance services
US10331581B2 (en) * 2017-04-10 2019-06-25 Hewlett Packard Enterprise Development Lp Virtual channel and resource assignment
CN114880133A (en) * 2017-08-31 2022-08-09 华为技术有限公司 Distributed computing system, data transmission method and device in distributed computing system
CN115668881A (en) * 2020-05-19 2023-01-31 起元技术有限责任公司 Optimization of communications in a distributed computing network
US11747996B2 (en) * 2020-08-05 2023-09-05 Dropbox, Inc. System and methods for implementing a key-value data store

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5699500A (en) * 1995-06-01 1997-12-16 Ncr Corporation Reliable datagram service provider for fast messaging in a clustered environment
JPH10105522A (en) * 1996-06-27 1998-04-24 Hewlett Packard Co <Hp> Multicomputer system
CN109800236A (en) * 2019-01-28 2019-05-24 上海雷腾软件股份有限公司 Support the distributed caching method and equipment of multinode
WO2021213075A1 (en) * 2020-04-24 2021-10-28 中科寒武纪科技股份有限公司 Inter-node communication method and device based on multiple processing nodes
WO2022001375A1 (en) * 2020-06-28 2022-01-06 北京沃东天骏信息技术有限公司 Blockchain-based data storage method, system and apparatus
CN114697372A (en) * 2022-05-31 2022-07-01 深圳市泛联信息科技有限公司 Data transmission processing and storage method, system and medium in distributed system

Also Published As

Publication number Publication date
CN115118727A (en) 2022-09-27

Similar Documents

Publication Publication Date Title
CN110855792B (en) Message pushing method, device, equipment and medium
CN110852882B (en) Packet consensus method, apparatus, device, and medium for blockchain networks
CN110915188A (en) Probabilistic relaying for efficient propagation in block-chain networks
CN112235423B (en) Cross-chain transaction processing method and device, electronic equipment and storage medium
CN115208812A (en) Service processing method and device, equipment and computer readable storage medium
CN111163130A (en) Network service system and data transmission method thereof
CN112416632B (en) Event communication method and device, electronic equipment and computer readable medium
CN112817602A (en) JSON format data sending and receiving method, device and medium
CN111431730A (en) Service processing method, system, computer equipment and readable medium
CN111343220B (en) Repeater, distributed file transmission method, distributed file transmission system, medium and electronic equipment
CN112418389A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN113448770B (en) Method, electronic device and computer program product for recovering data
CN115118727B (en) Data transmission method, device, equipment and storage medium of distributed computing architecture
CN116743785A (en) Cloud network data storage method, device, equipment and medium based on fog calculation
CN114189384B (en) Service processing method, device, equipment and storage medium
CN115776461A (en) Screen projection method, equipment and storage medium
CN114579054A (en) Data processing method and device, electronic equipment and computer readable medium
CN112019603B (en) Transaction data processing method and device
US12002043B2 (en) Method for enabling efficient evaluation of transactions in a distributed ledger network
CN109547552B (en) API request processing method and device, storage medium and electronic equipment
CN114979128A (en) Cross-region communication method and device and electronic equipment
CN107707383B (en) Put-through processing method and device, first network element and second network element
CN113765972A (en) Data request response method, device, system, server and storage medium
CN112416887A (en) Information interaction method and device and electronic equipment
CN115297046B (en) Gateway information sending method, device, electronic equipment, medium and product

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant