TWI752005B - Method, system and distributed system for intermediate data transmission - Google Patents

Abstract

The present invention relates to an intermediate data transmission method and system, and a distributed system. The intermediate data transmission method includes: determining the intermediate data to be transmitted by the upper-level subtask this time, and recording it as the data to be transmitted; The network port for data transmission is recorded as the receiving port; the data to be transmitted is directly transmitted to the receiving port through the network. In the embodiment of the present invention, the transmission of the intermediate data does not need to pass through a distributed storage system, thereby avoiding the influence of the IO rate of the disk on the transmission rate of the intermediate data, and improving the transmission rate of the intermediate data.

Description

Method, system and distributed system for intermediate data transmission

The present invention relates to the field of communications, in particular to an intermediate data transmission method and system, and a distributed system.

In distributed systems, user tasks can often be decomposed into several levels of different subtasks. There are dependencies between these subtasks, and the output results of some subtasks will be used as input data for other subtasks. This involves data transfer between subtasks, and these data transferred between subtasks are called intermediate data.

At present, in the related art, the transmission of intermediate data is completed through a distributed storage system. The process is: the upper-level subtask that generates the intermediate data writes the intermediate data in the form of files to the distributed storage system through the interface of the distributed storage system. On the disk of the storage system; the lower-level subtask that needs to take the intermediate data as input data reads the intermediate data from the disk of the distributed storage system for further processing.

Take the most common MapReduce task as an example. MapReduce tasks can be decomposed into two sub-tasks, MapTask and ReduceTask. When data sorting is done through MapReduce tasks, MapTask can Several processes are started simultaneously on different machines, each process reads a part of the input data, sorts this part of the data, and then outputs the sorted results. These output data will be used as the input of ReduceTask for further sorting to achieve global order. In this process, the data passed between the two sub-tasks of MapTask and ReduceTask are the intermediate calculation results generated in the sorting process, and do not need to be presented to the user.

FIG. 1 is a schematic diagram of an intermediate data transmission process between MapTask and ReduceTask in the related art. As shown in Figure 1, in the related art, the intermediate data transmission process between MapTask and ReduceTask is as follows: MapTask outputs intermediate data, and writes the intermediate data in the form of files into the distributed storage system through the user interface of the distributed storage system. The storage system stores the intermediate data in one or more storage nodes, that is, persists these intermediate data to the disks of one or more machines; ReduceTask, through the user interface of the distributed storage system, retrieves the intermediate data from the storage nodes that store the intermediate data. Read intermediate data.

In the related art, the following problems exist in the intermediate data transmission of the distributed system:

1. The intermediate data transmission needs to be realized by the read and write of the disk, while the average read and write rate of the traditional mechanical hard disk can only reach about 100MB/s. Therefore, this intermediate data transmission method is seriously affected by the IO (input and output) rate of the disk, and the transmission rate is low, resulting in low efficiency of user task execution.

2. A single storage node may be unavailable in a distributed storage system. In order to ensure that data will not be lost, distributed storage systems usually Multiple copies of a file will be generated and stored on different storage nodes. This process causes the same data to be transmitted multiple times in the network, occupying network bandwidth.

The purpose of the present invention is to provide an intermediate data transmission method and system, and a distributed system, so as to improve the intermediate data transmission rate of the distributed system.

In order to achieve the above object, the present invention proposes an intermediate data transmission method, which is used in a distributed system. The user tasks of the distributed system include multi-level subtasks, and the subtasks that generate intermediate data in the subtasks are called upper-level subtasks. In the subtasks, the subtasks that rely on the intermediate data for processing are called subtasks, and the method includes: determining the intermediate data to be transmitted by the superior subtask this time, and denoting it as the data to be transmitted; The network port that needs to receive the data to be transmitted is selected from the network ports used for monitoring data of the subtasks of the lower level, which is recorded as the receiving port; the data to be transmitted is directly transmitted to the receiving port through the network.

In the intermediate data transmission method in the embodiment of the present invention, the transmission of the intermediate data does not need to go through a distributed storage system, which avoids the influence of the disk IO rate on the intermediate data transmission rate, and improves the intermediate data transmission rate.

In order to achieve the above object, the present invention also proposes an intermediate data transmission system, which is used in a distributed system. The user tasks of the distributed system include multi-level subtasks, and the subtasks that generate intermediate data in the subtasks are called superiors. Subtasks, the subtasks in the subtasks that rely on the intermediate data for processing The task is called a lower-level subtask, and the intermediate data transmission system includes: a determination module, which is used to determine the intermediate data to be transmitted by the upper-level subtask this time, which is recorded as the data to be transmitted; Select the network port that needs to receive the data to be transmitted among the network ports used for monitoring data of the subtasks of the lower level, which is recorded as the receiving port; the transmission module is used to pass the data to be transmitted determined by the determining module through the network. The channel is directly transmitted to the receiving port selected by the selection module.

In the intermediate data transmission system in the embodiment of the present invention, the transmission of the intermediate data does not need to go through the distributed storage system, which avoids the influence of the IO rate of the disk on the transmission rate of the intermediate data, and improves the transmission rate of the intermediate data.

To achieve the above object, the present invention also proposes a distributed system, including the intermediate data transmission system described in any one of the foregoing.

In the distributed system in the embodiment of the present invention, the transmission of the intermediate data does not need to pass through the distributed storage system, which avoids the influence of the IO rate of the disk on the transmission rate of the intermediate data, and improves the transmission rate of the intermediate data.

800‧‧‧Intermediate Data Transmission System

810‧‧‧Determine the module

820‧‧‧Select module

830‧‧‧Transmission Module

910‧‧‧First Startup Module

920‧‧‧Notification Module

930‧‧‧Second Activation Module

940‧‧‧Determine the module

950‧‧‧Select modules

960‧‧‧Transmission Module

1000‧‧‧Transmission modules

1010‧‧‧Interprocess transfer unit

1100‧‧‧Transmission Module

1110‧‧‧Interprocess transfer unit

1120‧‧‧Response receiving unit

1130‧‧‧Licensing Unit

1140‧‧‧Retransmission Unit

1200‧‧‧Distributed Systems

FIG. 1 is a schematic diagram of an intermediate data transmission process between MapTask and ReduceTask in the related art.

FIG. 2 is a flowchart of an intermediate data transmission method in Embodiment 1 of the present invention.

FIG. 3 is a schematic diagram of the intermediate data transmission process between MapTask and ReduceTask during transmission according to the intermediate data transmission method shown in FIG. 2 one.

FIG. 4 is the second schematic diagram of the intermediate data transmission process between MapTask and ReduceTask when the transmission is performed according to the intermediate data transmission method shown in FIG. 2 .

FIG. 5 is a flowchart of an intermediate data transmission method in Embodiment 2 of the present invention.

FIG. 6 is a flowchart of an intermediate data transmission method in Embodiment 3 of the present invention.

FIG. 7 is a flowchart of an intermediate data transmission method in Embodiment 4 of the present invention.

FIG. 8 is a structural block diagram of an intermediate data transmission system in Embodiment 5 of the present invention.

FIG. 9 is a structural block diagram of an intermediate data transmission system in Embodiment 6 of the present invention.

FIG. 10 is a structural block diagram of a transmission module of the intermediate data transmission system in Embodiment 7 of the present invention.

FIG. 11 is a structural block diagram of a transmission module of the intermediate data transmission system in Embodiment 8 of the present invention.

FIG. 12 is a structural block diagram of a distributed system in Embodiment 9 of the present invention.

The principles and features of the present invention will be described below with reference to the accompanying drawings. The embodiments are only used to explain the present invention, but not to limit the scope of the present invention. For those of ordinary skill in the art, without any creative effort It is mentioned that all the embodiments obtained according to the spirit of the present invention belong to the protection scope of the present invention.

It should be noted that, the intermediate data transmission method and the intermediate data transmission system in each embodiment of the present invention are all used in a distributed system, and the user tasks of the distributed system include multi-level subtasks, wherein the intermediate tasks are generated from these subtasks. The subtasks of the data are called upper-level subtasks, and the subtasks that rely on intermediate data for processing among these subtasks are called lower-level subtasks. Superior subtasks and subordinate subtasks are relative. For example, in a user task with multiple levels of subtasks, a subtask may be a subtask for subtask A but may be a subtask for subtask B.

The user task may be a MapReduce task, a DAG (Directed Acyclic Graph, directed acyclic graph) task, or the like.

FIG. 2 is a flowchart of an intermediate data transmission method in Embodiment 1 of the present invention. As shown in FIG. 2 , in this embodiment, the method for transmitting intermediate data may include the following steps: Step S201, determining the intermediate data to be transmitted by the superior subtask this time, and denoting it as the data to be transmitted; if the superior subtask has multiple processes In this case, the intermediate data to be transmitted by the superior subtask this time (that is, the data to be transmitted) may be intermediate data generated by a process of the superior subtask, or may be intermediate data generated by multiple or all processes of the superior subtask.

Step S202 selects the network port that needs to receive the data to be transmitted from the network port used for monitoring data of the lower-level subtask, and is denoted as the receiving port; The data to be sent by the task.

Lower-level subtasks can set network ports as needed. For example, lower-level subtasks can set a network port for each process, or set one or more common network ports for all processes.

Step S203, the data to be transmitted is directly transmitted to the receiving port through the network.

That is to say, the data to be transmitted is directly transmitted from the network where the superior subtask is located to the network where the next task is located through the network, and is no longer written in the distributed storage system as in the related art mentioned in the background art. and reading process.

Take MapReduce tasks as an example. Figure 3 shows the intermediate data transfer process between MapTask and ReduceTask, the two-level subtasks of MapReduce tasks.

On this basis, users can formulate specific transmission strategies as needed.

For example, one of the transmission strategies may be: setting a first network port for each process of the upper-level subtask, setting a second network port for each process of the lower-level subtask, and transferring the data to be transmitted by The first network ports corresponding to one or more processes that transmit data are respectively transmitted to all the second network ports. That is, the intermediate data is sent directly to all processes of subtasks from the process that generates the intermediate data. Still take the MapReduce task as an example. According to this transmission strategy, the intermediate data transmission process between the two-level subtasks MapTask and ReduceTask of the MapReduce task is shown in Figure 4.

The second transmission strategy can be: set one or more first common network ports for all processes of the superior subtask, set one or more second common network ports for all processes of the inferior subtask, and transfer the The intermediate data generated by each process is directly transmitted from the first common network port to the second common network port through the network, and the second common network port then distributes the intermediate data to each process of the lower-level subtask.

The intermediate data transmission method in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate.

FIG. 5 is a flowchart of an intermediate data transmission method in Embodiment 2 of the present invention. As shown in FIG. 5 , in this embodiment, the intermediate data transmission method may include the following steps: Step S501 , start a subtask, so that the network port of the subtask used for monitoring data is in a monitoring state; because the embodiment of the present invention , the intermediate data is directly transmitted by the superior subtask to the inferior subtask through the network, so before starting the superior subtask to generate the intermediate data, the inferior subtask should be in a state that can receive the intermediate data. That is to say, the lower-level subtask should be started before the upper-level subtask, so as to ensure that the lower-level subtask can receive the intermediate data generated by the upper-level subtask.

Step S502, the information of the network port used for monitoring data of the lower-level subtask is notified to the upper-level subtask; The information of the network port may generally include an IP address and the like. The upper-level subtask can determine the destination address of the intermediate data according to the information of the network port of the lower-level subtask for monitoring data.

In a specific application, the lower-level subtask can first report the network port information to the scheduler, and the scheduler then sends the network port information of the lower-level subtask for monitoring data to the upper-level subtask.

Step S503, start the upper-level subtask after the lower-level subtask is started, and generate intermediate data; after the lower-level subtask is started, when the network port of the lower-level subtask for monitoring data is already in the monitoring state, start the upper-level subtask again . That is, start the lower-level subtask first, and then start the upper-level subtask.

Step S504, determine the intermediate data to be transmitted by the superior subtask this time, and denote it as the data to be transmitted; Step S505 selects the network port that needs to receive the data to be transmitted from the network ports of the subtask for monitoring data, denoted as A receiving port; Step S506, the data to be transmitted is directly transmitted to the receiving port through the network.

The intermediate data transmission method in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Furthermore, in the intermediate data transmission method in the embodiment of the present invention, the lower-level subtask is started before the upper-level subtask, so as to ensure that the lower-level subtask can receive the intermediate data generated by the upper-level subtask.

FIG. 6 is a flowchart of an intermediate data transmission method in Embodiment 3 of the present invention. As shown in FIG. 6 , in this embodiment, the method for transmitting intermediate data may include the following steps: Step S601 , determine the intermediate data to be transmitted by a single process of the upper-level subtask this time, and record it as the data to be transmitted, wherein the data of the upper-level subtask is determined. Each process corresponds to a first network port; in this embodiment, the intermediate data is sent in units of processes. Each process of the superior subtask determines its own intermediate data to be transmitted, and then transmits the intermediate data to be transmitted to all the processes of the inferior subtask through the network.

Step S602, each process of the lower-level subtask corresponds to a second network port, and all second network ports are selected as the receiving ports of the data to be transmitted; in this embodiment, each process of the upper and lower-level subtasks Each corresponds to a network port. The first network port corresponding to the process of the upper-level subtask is the sending port of the intermediate data, and the second network port corresponding to the process of the lower-level subtask is the receiving port of the intermediate data. Correspondingly, the process of the upper-level subtask is called the sending process, and the process of the lower-level subtask is called the receiving process. The intermediate data is directly transmitted from the sending port to the receiving port through the network. The transmission mode is, for example, the intermediate data transmission mode shown in FIG. 4 .

Step S603, the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, wherein the sending process is the process of generating the data to be transmitted.

It can be seen from the above steps that in this embodiment, the intermediate data is sent by the sending process It is directly transmitted to the receiving process through the network, and it is not necessary to collect the intermediate data generated by each sending process through the superior subtask and send it uniformly, and it is not necessary to uniformly receive the intermediate data through the subtask and then send it to each receiving process separately. Transmission rate. Not only that, the receiving process can process the intermediate data immediately after receiving it, instead of waiting for all the sending processes to generate the intermediate data and transmit it to the receiving process. Therefore, the intermediate data is directly transmitted from the sending process to the receiving process through the network. process, it can also allow the receiving process to process a part of the intermediate data that has been transmitted to the receiving process when the sending process is still generating the intermediate data, that is, the receiving process and the sending process can run in parallel, which significantly improves the The parallel execution capability between upper and lower sub-tasks shortens the overall running time of user tasks.

Since there will be a certain delay in transmitting data through the network, if the sending process only sends a small amount of data at a time, the number of times of sending data will increase, and the overhead (indirect cost) caused by network transmission will be very large. On the other hand, if the amount of data sent by the sending process each time is large, when there are many receiving processes, the memory used by the sending process will be very high. In order to solve the problem of when the sending process sends the intermediate data, in other embodiments of the present invention, before step S603, that is, the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second data through the network. Before the network port, it can also include the following steps: judging whether the data to be transmitted is greater than or equal to msgSize, msgSize represents the length of the data to be sent in each transmission of the sending process; when the data to be transmitted is greater than or equal to msgSize, start the data to be transmitted Data transmission, and determine the length of the data to be transmitted is msgSize; among them, msgSize=max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n, BuffSize represents the default upper limit of available memory for the sending process , MaxMsgSize represents the maximum data length transmitted by the preset sending process in one transmission, MinMsgSize represents the minimum data length transmitted by the preset sending process in one transmission, n represents the number of second network ports, "*" represents the multiplication With operation, "/" means division operation. The meaning of n is the number of receiving processes. In this embodiment, the number of receiving processes is equal to the number of receiving ports and the number of second network ports. Therefore, the number of receiving processes is the number of second network ports.

The above steps calculate the length of the sent data in each transmission of the sending process according to the scale of the user task (referring to the number of receiving processes) and the available memory size of each sending process, so as to realize dynamic adjustment, so as to avoid the problem of too little data sent each time. The resulting problem of high overhead costs and the problem of high memory requirements caused by too much data sent each time.

The material to be transmitted is usually sent in the form of a message. In this embodiment of the present invention, a message carrying data to be transmitted may carry a first process identifier and a message number, where the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the sending process The number of messages sent. That is to say, the first process identifier is used to explain the question of "who" sent the message, and the message number is monotonically increasing, which is used to explain the position or sequence number of the message in all the messages sent by the sending process problem.

The receiving process may judge whether the message has been received according to the first process identifier and the message number carried in the message carrying the data to be transmitted, so as to process the message according to the specific situation.

The process corresponding to the second network port can record the maximum message number that has been received from the sending process. If the message number carried in the received message is greater than the maximum message number, it means that the receiving process receives the message for the first time. The message number carried in the message is less than or equal to the maximum message number, indicating that the receiving process has received the message.

Accordingly, in this embodiment of the present invention, after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, it may further include: the second network port receives the After carrying the message of the data to be transmitted, in the case that the message number carried in the message is greater than the maximum message number currently received from the sending process recorded by the process corresponding to the second network port, the process corresponding to the second network port Save the data to be transmitted, and update the recorded maximum message number to the message number carried in the message.

Or, in this embodiment of the present invention, after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, it may further include: the second network port receives the bearer After the message of the data to be transmitted, in the case that the message number carried in the message is less than or equal to the maximum message number currently received from the sending process recorded by the process corresponding to the second network port, the corresponding The process discards the message and maintains the highest recorded message number.

The intermediate data transmission method in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Moreover, the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subtasks of the upper and lower levels, and shortens the overall running time of the user task.

In the case where the intermediate data is directly transmitted from the sending process to the receiving process in the embodiment shown in FIG. 6 , a timeout retransmission mechanism can be used to deal with possible message loss (that is, the intermediate data sent by the sending process cannot be reached). receiving process) phenomenon. The specific content of the timeout retransmission mechanism may be that after the sending process sends the intermediate data to the receiving process, if the receiving process receives the intermediate data, it will return a response to the sending process, informing the sending process that the receiving process has received its transmission. After the sending process receives the response, it can send the next intermediate data to the receiving process that returns the response. If the sending process sends the intermediate data to the receiving process, and the receiving process does not receive the intermediate data, it will not return a response to the sending process. The sending process does not receive a response within the set time limit, and it is considered that the receiving process that did not return a response did not receive the intermediate data, so it will retransmit the intermediate data of this transmission to the receiving process that did not return a response. The next intermediate data will be sent to the receiving process that does not return a response response until the sending process receives the response response returned by the receiving process. In this way, it can not only ensure that the receiving process can receive the intermediate data, but also make the same intermediate data There are no multiple transmissions in the network, thus saving network bandwidth.

Accordingly, the intermediate data transmission method of the present invention may adopt the flow shown in FIG. 7 .

FIG. 7 is a flowchart of an intermediate data transmission method in Embodiment 4 of the present invention. As shown in FIG. 7 , in this embodiment, the method for transmitting intermediate data may include the following steps: Step S701 , determining the intermediate data to be transmitted by a single process of the upper-level subtask this time, which is recorded as the data to be transmitted, wherein the data of the upper-level subtask is Each process corresponds to a first network port; in step S702, each process of the lower-level subtask corresponds to a second network port, and all second network ports are selected as the receiving ports of the data to be transmitted; step S703, the The data to be transmitted is directly transmitted by the first network port corresponding to the sending process to all the second network ports through the network, wherein the sending process is the process of generating the data to be transmitted; step S704, after the transmission, it is judged that within the set time Whether the sending process receives the response response returned by the process corresponding to the second network port, if the sending process receives the response response returned by the process corresponding to the second network port within the set time, step S705 is performed, otherwise step S706 is performed; In order to distinguish which receiving process the response comes from, the second process identifier can be carried in the response. The second process identifier is used to indicate the receiving process that issued the response, and the sending process judges the sender of the response according to the second process identifier. receive process.

Step S705, allowing the sending process to transmit the next data to the process returning the response response through the network; Step S706, the sending process again directly transmits the data to be transmitted through the network to one or more second network ports that did not return a response response , and return to step S704.

The intermediate data transmission method in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Moreover, the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subtasks of the upper and lower levels, and shortens the overall running time of the user task. Meanwhile, in the embodiment of the present invention, the same intermediate data will not be transmitted multiple times in the network, so the intermediate data transmission system in the embodiment of the present invention can also save network bandwidth.

The present invention also provides an intermediate data transmission system, which is used to implement the intermediate data transmission methods of the above embodiments. The above description of the intermediate data transmission method is applicable to the corresponding part of the intermediate data transmission system. The intermediate data transmission systems in the following embodiments of the present invention are all used in a distributed system. The user tasks of the distributed system include multi-level subtasks. The subtasks that generate intermediate data in these subtasks are called upper-level subtasks. Subtasks in tasks that rely on intermediate data for processing are called subtasks.

FIG. 8 is a structural block diagram of an intermediate data transmission system in Embodiment 5 of the present invention. As shown in FIG. 8 , in this embodiment, the intermediate data transmission system 800 may include a determination module 810 , a selection module 820 and a transmission module 830 . That Among them, the determining module 810 is used to determine the intermediate data to be transmitted by the superior subtask this time, which is recorded as the data to be transmitted. The selection module 820 is used to select the network port that needs to receive the data to be transmitted from the network ports used for monitoring data of the subtasks of the lower level, which is recorded as the receiving port. The transmission module 830 is used for directly transmitting the data to be transmitted determined by the determination module 810 to the receiving port selected by the selection module 820 through the network.

Wherein, in the case that the upper-level subtask has multiple processes, the data to be transmitted determined by the determination module 810 may be intermediate data generated by one process of the upper-level subtask, or may be generated by multiple or all processes of the upper-level subtask. intermediate data.

The receiving port selected by the selection module 820 may be a network port corresponding to a single process of the lower-level subtask, or may be one or more common network ports corresponding to all processes of the lower-level subtask.

The transmission module 830 can directly transmit the data to be transmitted to the receiving port through the network by using a specific transmission strategy formulated by the user as required. For example, the transmission strategy listed in the first embodiment of the present invention.

The intermediate data transmission system in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate.

FIG. 9 is a structural block diagram of an intermediate data transmission system in Embodiment 6 of the present invention. As shown in FIG. 9 , in this embodiment, the intermediate data transmission system may include a first activation module 910 , a notification module 920 , and a second activation module 930 Determine module 940 , select module 950 and transfer module 960 . Wherein, the first activation module 910 is used to activate the lower-level subtask, so that the network port of the lower-level subtask used for monitoring data is in a listening state. The notification module 920 is connected to the first activation module 910, and is used for notifying the upper-level subtask of the information of the network port used for monitoring data of the lower-level subtask. The second activation module 930 is respectively connected to the notification module 920 and the determination module 940, and is used to activate the upper-level subtask after the lower-level subtask is activated to generate intermediate data. The determining module 940 is used to determine the intermediate data to be transmitted by the superior subtask this time, which is recorded as the data to be transmitted. The selection module 950 is used to select the network port that needs to receive the data to be transmitted from the network ports used for monitoring data of the subtask, which is recorded as the receiving port. The transmission module 960 is used for directly transmitting the data to be transmitted determined by the determination module 940 to the receiving port selected by the selection module 950 through the network.

The intermediate data transmission system in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Furthermore, in the intermediate data transmission system in the embodiment of the present invention, the lower-level subtask is started before the upper-level subtask, so as to ensure that the lower-level subtask can receive the intermediate data generated by the upper-level subtask.

FIG. 10 is a structural block diagram of a transmission module of the intermediate data transmission system in Embodiment 7 of the present invention. As shown in FIG. 10 , in this embodiment, the transmission module 1000 of the intermediate data transmission system may include an inter-process transmission unit 1010 . The inter-process transmission unit 1010 is used to generate the data to be transmitted by a single process of the upper-level subtask, and each process of the upper-level subtask corresponds to a first. Network port. Each process of the subtask corresponds to a second network port. If all second network ports are receiving ports, the data to be transmitted is sent through the first network port corresponding to the sending process through the network. It is directly transmitted to all the second network ports, wherein the sending process refers to the process that generates the data to be transmitted.

In this embodiment, the intermediate data is directly transmitted from the sending process to the receiving process through the network, and it is not necessary to collect the intermediate data generated by each sending process through the upper-level subtask and send it uniformly, and it is also unnecessary to uniformly receive the intermediate data through the lower-level subtask and then send it to Each receiving process can thus further increase the intermediate data transfer rate. Not only that, the receiving process can process the intermediate data immediately after receiving it, instead of waiting for all the sending processes to generate the intermediate data and transmit it to the receiving process. Therefore, the intermediate data is directly transmitted from the sending process to the receiving process through the network. process, it can also allow the receiving process to process a part of the intermediate data that has been transmitted to the receiving process when the sending process is still generating the intermediate data, that is, the receiving process and the sending process can run in parallel, which significantly improves the The parallel execution capability between upper and lower sub-tasks shortens the overall running time of user tasks.

On the basis of the embodiment shown in FIG. 10 , the transmission module 1000 may further include a judgment unit and an activation unit. Wherein, the judging unit is used for judging whether the data to be transmitted is greater than or equal to msgSize, and msgSize represents the length of the data to be sent in each transmission of the sending process. The starting unit is used to start the inter-process transmission unit 1010 to transmit the data to be transmitted when the judgment result of the judging unit is that the data to be transmitted is greater than or equal to msgSize, and determine The length of the data to be transmitted is msgSize. Among them, msgSize=max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n, BuffSize represents the upper limit of available memory of the preset sending process, and MaxMsgSize represents the maximum data transmitted by the preset sending process in one sending Length, MinMsgSize represents the minimum data length transmitted by the preset sending process in one sending, n represents the number of second network ports, "*" represents multiplication, and "/" represents division.

The above judging unit and starting unit calculate the length of the sent data in each transmission of the sending process according to the scale of user tasks (referring to the number of receiving processes) and the available memory size of each sending process, so as to realize dynamic adjustment, so as to avoid The problem of high overhead costs caused by too few data transfers at one time and the problem of high memory requirements caused by too much data sent each time.

In this embodiment of the present invention, a message carrying data to be transmitted may carry a first process identifier and a message number, where the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the sending process The number of messages sent.

The receiving process may judge whether the message has been received according to the first process identifier and the message number carried in the message carrying the data to be transmitted, so as to process the message according to the specific situation.

The process corresponding to the second network port can record the maximum message number that has been received from the sending process. If the message number carried in the received message is greater than the maximum message number, it means that the receiving process receives the message for the first time. The message number carried by the message is less than or equal to the maximum The message number, indicating that the receiving process has already received the message.

Accordingly, in this embodiment of the present invention, the transmission module 1000 may include a storage unit. The saving unit is configured to, after receiving the message carrying the data to be transmitted on the second network port, the message number carried in the message is greater than that recorded by the process corresponding to the second network port that has currently been sent from the sending process (referring to the sending of the data to be transmitted). In the case of the maximum message number received by the process), make the process corresponding to the second network port save the data to be transmitted, and update the recorded maximum message number to the message number carried by the message.

In this embodiment of the present invention, the transmission module 1000 may further include a discarding unit. The discarding unit is used to, after the second network port receives the message carrying the data to be transmitted, when the message number carried in the message is less than or equal to the process record corresponding to the second network port that has been sent from the sending process (indicating that the data to be transmitted is sent out). In the case of the maximum message number received by the process), make the process corresponding to the second network port discard the message, and keep the recorded maximum message number unchanged.

The intermediate data transmission system in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Moreover, the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subtasks of the upper and lower levels, and shortens the overall running time of the user task.

In the case where the intermediate data is transmitted by the inter-process transmission unit in the embodiment shown in FIG. 10, the transmission module can use the timeout retransmission mechanism to respond For possible message loss (that is, the intermediate data sent by the sending process cannot reach the receiving process).

Accordingly, the transmission module of the intermediate data transmission system can adopt the structure shown in FIG. 11 . FIG. 11 is a structural block diagram of a transmission module of the intermediate data transmission system in Embodiment 8 of the present invention. As shown in FIG. 11 , in this embodiment, the transmission module 1100 may include an inter-process transmission unit 1110 and a response receiving unit 1120 . The function of the inter-process transmission unit 1110 is the same as that of the inter-process transmission unit 1000, which is not repeated here. The response receiving unit 1120 is configured to make the sending process receive the response response returned by the process corresponding to the second network port after the inter-process transmission unit 1110 transmits, and the response response is used to indicate that the process corresponding to the second network port has received the response. Information to be transmitted.

Referring to FIG. 11 , the transmission module 1110 may further include a license unit 1130 . The permitting unit 1130 is configured to, after the response receiving unit 1120 receives the response response, allow the sending process to transmit the next data to the process returning the response response through the network.

Referring to FIG. 11 , the transmission module 1110 may further include a retransmission unit 1140 . The retransmission unit 1140 is configured to, after the inter-process transmission unit 1110 performs transmission, in the case that the response receiving unit 1120 does not receive a response response returned by the process corresponding to the one or more second network ports within the set time, make the sending process Again, the data to be transmitted is directly transmitted through the network to one or more second network ports that have not returned a response response.

In order to distinguish which receiving process the response comes from, the response can carry a second process identifier, which is used to indicate the sending process For the receiving process of the response response, the sending process determines the receiving process that issued the response response according to the second process identifier.

The intermediate data transmission system in the embodiment of the present invention directly transmits the intermediate data in the distributed system from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through the distributed storage system in the middle, avoiding the need for disks The impact of the IO rate on the intermediate data transfer rate increases the intermediate data transfer rate. Moreover, the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subtasks of the upper and lower levels, and shortens the overall running time of the user task. Meanwhile, in the embodiment of the present invention, the same intermediate data will not be transmitted multiple times in the network, so the intermediate data transmission system in the embodiment of the present invention can also save network bandwidth.

FIG. 12 is a structural block diagram of a distributed system in Embodiment 9 of the present invention. As shown in FIG. 12 , in this embodiment, the distributed system 1200 may include an intermediate data transmission system. The intermediate data transmission system may be any of the intermediate data transmission systems in the foregoing embodiments of the present invention.

The distributed system in the embodiment of the present invention includes an intermediate data transmission system, and the intermediate data in the distributed system is directly transmitted from the upper-level subtask that generates the intermediate data to the lower-level subtask through the network, without going through distributed storage in the middle. The system avoids the impact of the disk IO rate on the intermediate data transfer rate, and improves the intermediate data transfer rate. Moreover, the distributed system of the embodiment of the present invention can also significantly improve the parallel execution capability between the subtasks of the upper and lower levels, and shorten the overall running time of the user task. At the same time, in the embodiment of the present invention, the same intermediate data will not be transmitted multiple times in the network, so the distributed system in the embodiment of the present invention can also save network bandwidth width.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection of the present invention. within the range.

Claims (19)

An intermediate data transmission method, used in a distributed system, the user tasks of the distributed system include multi-level subtasks, the subtasks that generate the intermediate data in the subtasks are called upper-level subtasks, and the subtasks in the subtasks The subtasks that rely on the intermediate data for processing are called lower-level subtasks, and the method includes: determining the intermediate data to be transmitted by the upper-level subtask this time, and denoting it as the data to be transmitted; Select the network port that needs to receive the data to be transmitted among the network ports used for monitoring data, which is recorded as the receiving port; directly transmitting the data to be transmitted to the receiving port through the network, which includes: in the The data to be transmitted is generated by a single process of the upper-level subtask, each process of the upper-level subtask corresponds to a first network port, each process of the lower-level subtask corresponds to a second network port, and all the first When the two network ports are both receiving ports, the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, and the sending process is to generate the pending data. The process of transmitting data; wherein, the message carrying the data to be transmitted carries a first process identifier and a message number, the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate The message is the first message sent by the sending process. The intermediate data transmission method according to item 1 of the scope of the application, wherein, in the determining of the current secondary transmission of the superior subtask Before the intermediate data, it also includes: starting the lower-level subtask, so that the network port of the lower-level subtask used for monitoring data is in a monitoring state; Notify the upper-level subtask; start the upper-level subtask after the lower-level subtask is started, and generate the intermediate data. The intermediate data transmission method according to item 1 of the scope of the application, wherein, before the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: judging whether the data to be transmitted is greater than or equal to msgSize, and msgSize represents the length of the data to be transmitted in each transmission of the sending process; when the data to be transmitted is greater than or equal to msgSize, start the data to be transmitted. Data transmission, and determine that the length of the data to be transmitted is msgSize; wherein, msgSize=max(min(BuffSize, MaxMsgSize*n), MinMsgSize*n)/n, and BuffSize represents the preset size of the sending process. The upper limit of available memory, MaxMsgSize represents the preset maximum data length transmitted by the sending process in one transmission, MinMsgSize represents the preset minimum data length transmitted by the sending process in one transmission, n represents the second network The number of channel ports, "*" means multiplication operation, "/" means division operation. The intermediate data transmission method according to item 1 of the scope of the application, wherein after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: after the second network port receives the message carrying the data to be transmitted, when the message number carried in the message is greater than the current record from the sending process recorded by the process corresponding to the second network port In the case of the received maximum message number, the process corresponding to the second network port saves the data to be transmitted, and updates the recorded maximum message number to the message number carried in the message. The intermediate data transmission method according to item 1 of the scope of the application, wherein after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: after the second network port receives the message carrying the data to be transmitted, when the message number carried in the message is less than or equal to the process record corresponding to the second network port that has currently been sent from the In the case of the maximum message number received by the sending process, the process corresponding to the second network port discards the message and keeps the recorded maximum message number unchanged. The intermediate data transmission method according to item 1 of the scope of the application, wherein after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: after the transmission, the sending process receives the second network terminal The response response returned by the process corresponding to the port is used to indicate that the process corresponding to the second network port has received the data to be transmitted. The intermediate data transmission method according to item 6 of the scope of the application, wherein after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: after the sending process receives the response response returned by the process corresponding to the second network port, allowing the sending process to transmit the next data to the process returning the response response through the network. The intermediate data transmission method according to item 6 of the scope of the application, wherein after the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all the second network ports through the network, It also includes: after the transmission, if the sending process does not receive the response response returned by the process corresponding to one or more second network ports within a set time, the sending process again sends the The data to be transmitted is directly transmitted to the one or more second network ports through the network. The intermediate data transmission method according to item 6 of the scope of the application, wherein the response response carries a second process identifier, and the second process identifier is used to indicate the process that sent the response response, and the sending process is based on The second process identifier determines the process that sends the response response. An intermediate data transmission system, used in a distributed system, the user tasks of the distributed system include multi-level subtasks, the subtasks that generate the intermediate data in the subtasks are called upper-level subtasks, and the subtasks in the subtasks The subtasks that rely on the intermediate data for processing are called lower-level subtasks, and the intermediate data transmission system includes: a determination module for determining the intermediate data to be transmitted this time by the upper-level subtask, which is recorded as pending transmission. data; selection module, used to select the network port that needs to receive the data to be transmitted from the network ports used for monitoring data of the lower-level subtask, which is recorded as the receiving port; transmission module, used to The data to be transmitted determined by the determination module is directly transmitted to the receiving port selected by the selection module through the network, wherein the transmission module includes: an inter-process transmission unit, used for the data to be transmitted by the A single process of the upper-level subtask is generated, each process of the upper-level subtask corresponds to a first network port, each process of the lower-level subtask corresponds to a second network port, and all second network ports are In the case of a receiving port, the data to be transmitted is directly transmitted from the first network port corresponding to the sending process to all second network ports through the network, and the sending process refers to the process that generates the data to be transmitted, wherein , the message carrying the data to be transmitted carries a first process identifier and a message number, the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the The number of messages sent by the sending process. The intermediate data transmission system according to item 10 of the scope of the application, further comprising: a first start-up module, configured to start the lower-level subtask, so as to enable the lower-level subtask The network port used for monitoring data of the subtask is in a monitoring state; the notification module is connected to the first startup module, and is used to notify the information of the network port used for monitoring data of the subtask To the upper-level subtask; the second startup module, which is respectively connected with the notification module and the determination module, is used to start the upper-level subtask after the lower-level subtask is started to generate the intermediate data . The intermediate data transmission system according to item 10 of the scope of the application, wherein the transmission module further comprises: a judging unit for judging whether the data to be transmitted is greater than or equal to msgSize, where msgSize indicates that each time the sending process is performed The length of the data to be sent in transmission; a start-up unit is used to start the inter-process transmission unit to transmit the data to be transmitted when the judgment result of the judgment unit is that the data to be transmitted is greater than or equal to msgSize, And determine that the length of the data to be transmitted is msgSize; wherein, msgSize=max(min(BuffSize, MaxMsgSize*n), MinMsgSize*n)/n, and BuffSize represents the preset upper limit of available memory for the sending process , MaxMsgSize represents the preset maximum data length transmitted by the sending process in one transmission, MinMsgSize represents the preset minimum data length transmitted by the sending process in one transmission, n represents the number of the second network port , "*" means multiply operation, "/" means divide operation. The intermediate data transmission system according to claim 10, wherein the transmission module further comprises: a storage unit for, after the second network port receives the message carrying the data to be transmitted, In the case that the message number carried in the message is greater than the largest message number currently received from the sending process recorded by the process corresponding to the second network port, make the process corresponding to the second network port save the data to be transmitted, and update the recorded maximum message number to the message number carried in the message. The intermediate data transmission system according to claim 10, wherein the transmission module further comprises: a discarding unit, configured to: after the second network port receives the message carrying the data to be transmitted, In the case that the message number carried in the message is less than or equal to the maximum message number currently received from the sending process recorded by the process corresponding to the second network port, make the number corresponding to the second network port The process discards the message and maintains the maximum message number recorded. The intermediate data transmission system according to claim 10, wherein the transmission module further comprises: a response receiving unit, configured to make the sending process receive the A response response returned by the process corresponding to the second network port, where the response response is used to indicate that the process corresponding to the second network port has received the data to be transmitted. The intermediate data transmission system according to claim 15, wherein the transmission module further comprises: The permitting unit is configured to allow the sending process to transmit the next data to the process that returns the response response through the network after the response receiving unit receives the response response. The intermediate data transmission system according to claim 15, wherein the transmission module further comprises: a retransmission unit for receiving the response within a set time after the inter-process transmission unit performs transmission In the case that the unit does not receive the response response returned by the process corresponding to one or more second network ports, the sending process is made to directly transmit the data to be transmitted to the one or more second network ports again. Second network port. The intermediate data transmission system according to item 15 of the scope of the application, wherein the response response carries a second process identifier, and the second process identifier is used to indicate the process that sent the response response, and the sending process is based on The second process identifier determines the process that sends the response response. A distributed system includes the intermediate data transmission system described in any one of items 10 to 18 of the patent application scope.

Images