WO2017136999A1 - 中间数据传输方法及系统、分布式系统 - Google Patents
中间数据传输方法及系统、分布式系统 Download PDFInfo
- Publication number
- WO2017136999A1 WO2017136999A1 PCT/CN2017/072818 CN2017072818W WO2017136999A1 WO 2017136999 A1 WO2017136999 A1 WO 2017136999A1 CN 2017072818 W CN2017072818 W CN 2017072818W WO 2017136999 A1 WO2017136999 A1 WO 2017136999A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- transmitted
- intermediate data
- network
- network port
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
Definitions
- the present invention relates to the field of communications, and in particular, to an intermediate data transmission method and system, and a distributed system.
- user tasks can often be broken down into several different subtasks. There are dependencies between these subtasks, and the output of some subtasks will be used as input data for other subtasks. This involves data transfer between subtasks, and the data passed between subtasks is called intermediate data.
- the transmission of intermediate data is performed by a distributed storage system, and the process is: the upper subtask that generates the intermediate data writes the intermediate data in the form of a file to the distributed through the interface of the distributed storage system.
- the intermediate data needs to be read as a sub-task of the input data from the disk of the distributed storage system for further processing.
- MapReduce tasks can be broken down into two subtasks: MapTask and ReduceTask.
- MapTask can start several processes simultaneously on different machines. Each process reads a part of the input data, sorts the data, and outputs the sorted results. These output data will be used as input to the ReduceTask to be further sorted to achieve global order.
- the data passed between the two sub-tasks of MapTask and ReduceTask is the intermediate calculation result generated in the sorting process, and does not need to be presented to the user.
- FIG. 1 is a schematic diagram of an intermediate data transmission process between a MapTask and a ReduceTask in the related art. As shown in FIG. 1, in the related art, the intermediate data transmission process between the MapTask and the ReduceTask is as follows:
- the MapTask outputs intermediate data, and the intermediate data is written into the distributed storage system as a file through the user interface of the distributed storage system.
- the distributed storage system stores the intermediate data in one or more storage nodes, that is, the intermediate data is persisted to On the disk of one or more machines;
- the ReduceTask reads the intermediate data from the storage node that stores the intermediate data through the user interface of the distributed storage system.
- a single storage node may be unavailable.
- a distributed storage system usually generates multiple copies of a file and stores them on different storage nodes. The process causes the same data to be transmitted multiple times in the network, taking up network bandwidth.
- the object of the present invention is to provide an intermediate data transmission method and system, and a distributed system, which improve the intermediate data transmission rate of the distributed system.
- the present invention provides an intermediate data transmission method for a distributed system
- the user task of the distributed system includes a multi-level subtask
- the subtask that generates intermediate data in the subtask is called a superior a subtask in which the subtask that depends on the intermediate data for processing is referred to as a subordinate subtask
- the method includes:
- the transmission of the intermediate data does not need to go through the distributed storage system, thereby avoiding the influence of the disk IO rate on the intermediate data transmission rate and improving the intermediate data transmission rate.
- the present invention also provides an intermediate data transmission system for a distributed system
- the user task of the distributed system includes a multi-level subtask, and a subtask for generating intermediate data in the subtask is called a sub-task in which the sub-task that depends on the intermediate data for processing is called a sub-task
- the intermediate data transmission system includes:
- a determining module configured to determine intermediate data to be transmitted by the superior subtask, and record the data to be transmitted
- a selection module configured to select, from a network port for listening to data of the subordinate subtask, a network port that needs to receive the to-be-transmitted data, and record it as a receiving port;
- a transmission module configured to directly transmit the to-be-transmitted data determined by the determining module to the receiving port selected by the selecting module.
- the transmission of the intermediate data does not need to go through the distributed storage system, thereby avoiding the influence of the disk IO rate on the intermediate data transmission rate and improving the intermediate data transmission rate.
- the present invention also provides a distributed system comprising the intermediate data transmission system of any of the preceding claims.
- the transmission of the intermediate data does not need to go through the distributed storage system, thereby avoiding the influence of the disk IO rate on the intermediate data transmission rate and improving the intermediate data transmission rate.
- FIG. 1 is a schematic diagram of an intermediate data transmission process between a MapTask and a ReduceTask in the related art
- FIG. 2 is a flowchart of an intermediate data transmission method according to Embodiment 1 of the present invention.
- FIG. 3 is a schematic diagram of an intermediate data transmission process between a MapTask and a ReduceTask according to the intermediate data transmission method shown in FIG. 2;
- FIG. 4 is a second schematic diagram of an intermediate data transmission process between a MapTask and a ReduceTask according to the intermediate data transmission method shown in FIG. 2;
- FIG. 5 is a flowchart of an intermediate data transmission method according to Embodiment 2 of the present invention.
- FIG. 6 is a flowchart of an intermediate data transmission method according to Embodiment 3 of the present invention.
- FIG. 7 is a flowchart of an intermediate data transmission method according to Embodiment 4 of the present invention.
- FIG. 8 is a structural block diagram of an intermediate data transmission system according to Embodiment 5 of the present invention.
- FIG. 9 is a structural block diagram of an intermediate data transmission system according to Embodiment 6 of the present invention.
- FIG. 10 is a structural block diagram of a transmission module of an intermediate data transmission system according to Embodiment 7 of the present invention.
- FIG. 11 is a structural block diagram of a transmission module of an intermediate data transmission system according to Embodiment 8 of the present invention.
- FIG. 12 is a structural block diagram of a distributed system according to Embodiment 9 of the present invention.
- the intermediate data transmission method and the intermediate data transmission system in the embodiments of the present invention are both used in a distributed system, and the user tasks of the distributed system include multiple levels of subtasks, wherein intermediate among the subtasks is generated.
- Subtasks of data are called superior subtasks, and subtasks that rely on intermediate data for processing in these subtasks are called Subordinate subtasks.
- the superior subtask and the subordinate subtask are relatively speaking. For example, in a user task with multiple levels of subtasks, one subtask is a subordinate subtask for subtask A, but it may be a subordinate subtask for subtask B.
- the user task may be a MapReduce task, a DAG (Directed Acyclic Graph) task, or the like.
- FIG. 2 is a flowchart of an intermediate data transmission method according to Embodiment 1 of the present invention. As shown in FIG. 2, in this embodiment, the intermediate data transmission method may include the following steps:
- Step S201 determining intermediate data to be transmitted by the superior subtask this time, and recording it as data to be transmitted;
- the intermediate data to be transmitted by the superior subtask may be the intermediate data generated by one process of the superior subtask, or may be the upper subtask. Intermediate data generated by one or all processes.
- Step S202 selecting, from the network port for monitoring data of the subordinate task of the subordinate task, a network port that needs to receive data to be transmitted, and recording it as a receiving port;
- the network port used by the subordinate subtask to listen to data is used to receive the data to be transmitted sent by the superior subtask.
- Subordinate subtasks can set network ports as needed. For example, a subordinate subtask can set a network port for each process, or set one or more common network ports for all processes.
- Step S203 The data to be transmitted is directly transmitted to the receiving port through the network.
- the data to be transmitted is directly transmitted by the network where the subordinate subtask is located to the network where the next task is located through the network, and the writing and reading of the distributed storage system are no longer carried out in the middle like the related technologies mentioned in the background art. process.
- MapReduce task Take the MapReduce task as an example.
- MapTask and ReduceTask The intermediate data transfer process between the two-level sub-task of the MapReduce task, MapTask and ReduceTask, is shown in Figure 3.
- one of the transmission policies may be: setting a first network port for each process of the upper subtask, and setting a second network port for each process of the subordinate subtask, and generating data to be transmitted by the data to be transmitted.
- the first network ports corresponding to one or more processes are respectively transmitted to all of the second network ports. That is, the intermediate data is directly sent by the process that generates the intermediate data to all processes of the subordinate subtasks.
- MapReduce task As an example.
- the intermediate data transfer process between the two-level sub-task MapTask and ReduceTask of the MapReduce task is as shown in FIG. 4 .
- the second transmission strategy may be: setting one or more first common network ports for all processes of the superior subtask, Setting one or more second common network ports for all processes of the subordinate subtasks, and transmitting intermediate data generated by each process of the subordinate subtasks by the first common network port to the second common network port through the network, the second common network The port then distributes the intermediate data to the various processes of the subordinate subtasks.
- the intermediate data in the distributed system is directly transmitted to the lower subtask through the network by the upper subtask generating the intermediate data, and the distributed storage system is not required in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- FIG. 5 is a flowchart of an intermediate data transmission method according to Embodiment 2 of the present invention. As shown in FIG. 5, in this embodiment, the intermediate data transmission method may include the following steps:
- Step S501 starting a sub-subtask, so that the network port of the sub-subtask for monitoring data is in a listening state;
- the intermediate data is directly transmitted to the sub-subtask through the network by the superior subtask. Therefore, before the intermediate subtask is generated to generate the intermediate data, the subordinate subtask should be in a state capable of receiving the intermediate data. That is to say, the subordinate subtasks should be started before the subordinate subtasks, so as to ensure that the subordinate subtasks can receive the intermediate data generated by the superior subtasks.
- Step S502 notifying the information of the network port of the subordinate task for monitoring the data to the upper subtask
- the information of the network port can generally include an IP address and the like.
- the superior subtask can determine the destination address of the intermediate data according to the information of the network port of the subordinate subtask for monitoring data.
- the sub-task can report the information of the network port to the scheduler first, and the scheduler sends the network port information of the sub-subtask for listening to the data to the sub-task.
- Step S503 after the lower subtask is started, the upper subtask is started to generate intermediate data.
- the superior subtask is started. That is, the sub-task is started first, and then the sub-task is started.
- Step S504 determining intermediate data to be transmitted by the upper subtask to be recorded as data to be transmitted;
- Step S505 selecting, from the network port for monitoring data of the subordinate task of the subordinate task, a network port that needs to receive data to be transmitted, and recording it as a receiving port;
- Step S506 the data to be transmitted is directly transmitted to the receiving port through the network.
- the intermediate data in the distributed system is directly transmitted to the lower subtask through the network by the upper subtask generating the intermediate data, and the distributed storage system is not required in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the subordinate subtasks are started before the subordinate subtasks to ensure that the subordinate subtasks can be It is enough to receive the intermediate data generated by the superior subtask.
- FIG. 6 is a flowchart of an intermediate data transmission method according to Embodiment 3 of the present invention. As shown in FIG. 6, in this embodiment, the intermediate data transmission method may include the following steps:
- Step S601 determining that the intermediate data to be transmitted by the single process of the upper subtask is recorded as data to be transmitted, wherein each process of the upper subtask corresponds to a first network port;
- the transmission of intermediate data is performed in units of processes.
- Each process of the superior subtask determines its own intermediate data to be transmitted, and then transmits the intermediate data that it wants to transmit to all processes of the subordinate subtasks through the network.
- Step S602 each process of the subordinate subtask corresponds to a second network port, and all the second network ports are selected as the receiving ports of the data to be transmitted;
- each process of the upper and lower subtasks corresponds to one network port.
- the first network port corresponding to the process of the upper subtask is the sending port of the intermediate data
- the second network port corresponding to the process of the lower subtask is the receiving port of the intermediate data.
- the process of the superior subtask is called the sending process
- the process of the subordinate subtask is called the receiving process.
- the intermediate data is transmitted directly from the sending port to the receiving port through the network.
- This transmission method is, for example, the intermediate data transmission method shown in FIG. 4 described above.
- Step S603 The data to be transmitted is directly transmitted to all the second network ports by using the first network port corresponding to the sending process, where the sending process is a process for generating data to be transmitted.
- the intermediate data is directly transmitted to the receiving process by the sending process through the network, and the intermediate data generated by each sending process is not necessarily sent through the upper sub-tasks, and the intermediate data is not uniformly received after the sub-tasks are uniformly received. It is sent to each receiving process separately, so the intermediate data transmission rate can be further improved.
- the receiving process can process the intermediate data immediately after receiving it, without having to wait until all the sending processes have generated the intermediate data and transmitted it to the receiving process before processing. Therefore, the intermediate data is directly transmitted by the sending process to the receiving process through the network. It also allows the receiving process to process a portion of the intermediate data that has been transmitted to the receiving process while the sending process is still generating intermediate data. That is, the receiving process and the sending process can run in parallel, which significantly improves the upper and lower The parallel execution capability between sub-tasks reduces the overall runtime of user tasks.
- the sending process Since there is a certain delay in transmitting data through the network, if the sending process only sends a small amount of data at a time, the number of times the data is sent will be increased, so that the overhead generated by the network transmission will be large. On the other hand, if the sending process sends a large amount of data each time, when there are more receiving processes, the memory used by the sending process is very high. In order to solve the problem of when the sending process sends intermediate data, in this issue In other embodiments, before the step S603, that is, before the data to be transmitted is directly transmitted to all the second network ports by using the first network port corresponding to the sending process, the following steps may be further included:
- msgSize indicates the length of the transmission data in each transmission of the sending process
- BuffSize indicates the upper limit of the available memory of the default sending process.
- MaxMsgSize indicates the maximum data length of the default sending process in one transmission.
- MinMsgSize indicates the minimum data length of the default sending process in one transmission, and n indicates the second.
- n is the number of receiving processes. In this embodiment, the number of receiving processes is equal to the number of receiving ports, and is also equal to the number of second network ports. Therefore, the number of receiving processes is the number of second network ports.
- the above steps calculate the length of the transmission data in each transmission of the sending process according to the size of the user task (the number of receiving processes) and the available memory size of each sending process, thereby realizing dynamic adjustment, so that the data is too small to be transmitted each time.
- the data to be transmitted is usually sent in the form of a message.
- the message carrying the data to be transmitted may carry the first process identifier and the message number, where the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the sending process.
- the first few messages sent That is to say, the first process identifier is used to indicate that the message is sent by "who", and the message number is monotonically increasing, which is used to indicate the location problem or sequence number of the message in all messages sent by the sending process. problem.
- the receiving process may determine whether the message has been received according to the first process identifier and the message number carried in the message carrying the data to be transmitted, so as to process the message according to a specific situation.
- the process corresponding to the second network port can record the maximum message number that has been received from the sending process. If the received message carries a message number greater than the maximum message number, the receiving process receives the message for the first time, if received. The message number carried in the message is less than or equal to the maximum message number, indicating that the receiving process has received the message.
- the method may further include:
- the message number carried in the message is greater than the second network. If the process corresponding to the network port records the maximum message number that has been received from the sending process, the process corresponding to the second network port saves the to-be-transmitted data, and updates the recorded maximum message number to the message carried by the message. Numbering.
- the second network port may further include: receiving, by the second network port, the data to be transmitted. After the message, if the message number carried in the message is less than or equal to the maximum message number currently received by the process corresponding to the process record corresponding to the second network port, the process corresponding to the second network port discards the message, and The maximum message number of the record is maintained.
- the intermediate data in the distributed system is directly transmitted to the lower subtask through the network by the upper subtask generating the intermediate data, and the distributed storage system is not required in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subordinate tasks of the upper and lower levels, and shortens the running time of the overall task of the user.
- the timeout retransmission mechanism may be adopted to cope with possible message loss (ie, the intermediate data sent by the sending process cannot reach the receiving. Process) phenomenon.
- the specific content of the timeout retransmission mechanism may be: after the sending process sends the intermediate data to the receiving process, if the receiving process receives the intermediate data, it returns a response response to the sending process, and informs the sending process that the receiving process has received the sending. Intermediate data; after receiving the response, the sending process can send the next intermediate data to the receiving process that returns the response. If the sending process does not receive the intermediate data after sending the intermediate data to the receiving process, it will not return a response to the sending process.
- the sending process does not receive the response response within the set time limit, and considers that the receiving process that does not return the response response does not receive the intermediate data, and therefore retransmits the intermediate data of the current transmission to the receiving process that does not return the response response.
- the next intermediate data is not sent to the receiving process that does not return a response, until the sending process receives the response returned by the receiving process. In this way, the receiving process can ensure that the intermediate data can be received, and the same intermediate data can be transmitted multiple times in the network, thereby saving network bandwidth.
- the intermediate data transmission method of the present invention can employ the flow shown in FIG.
- FIG. 7 is a flowchart of an intermediate data transmission method according to Embodiment 4 of the present invention. As shown in FIG. 7, in this embodiment, the intermediate data transmission method may include the following steps:
- Step S701 determining that the intermediate data to be transmitted by the single process of the upper subtask is recorded as data to be transmitted, wherein each process of the upper subtask corresponds to a first network port;
- Step S702 each process of the subordinate subtask corresponds to a second network port, and all the second network ports are selected as the receiving ports of the data to be transmitted;
- Step S703 The data to be transmitted is directly transmitted to all the second network ports by using the first network port corresponding to the sending process, where the sending process is a process for generating data to be transmitted.
- Step S704 after the transmission, determining whether the sending process receives the response response returned by the process corresponding to the second network port within the set time, if the sending process receives the response returned by the process corresponding to the second network port within the set time In response, step S705 is performed, otherwise step S706 is performed;
- the response response may carry a second process identifier, the second process identifier is used to indicate the receiving process that sends the response, and the sending process determines to send the response response according to the second process identifier.
- Receive process the second process identifier is used to indicate the receiving process that sends the response, and the sending process determines to send the response response according to the second process identifier.
- Step S705 the sending process is allowed to transmit the next data to the process that returns the response response through the network;
- Step S706 the sending process again transmits the data to be transmitted through the network to one or more second network ports that have not returned the response, and returns to step S704.
- the intermediate data in the distributed system is directly transmitted to the lower subtask through the network by the upper subtask generating the intermediate data, and the distributed storage system is not required in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subordinate tasks of the upper and lower levels, and shortens the running time of the overall task of the user.
- the same intermediate data is not transmitted multiple times in the network. Therefore, the intermediate data transmission system in the embodiment of the present invention can also save network bandwidth.
- the present invention also proposes an intermediate data transmission system for implementing the intermediate data transmission method of the above embodiments.
- the above description of the intermediate data transmission method is applicable to the corresponding portion of the intermediate data transmission system.
- the intermediate data transmission systems in the embodiments of the present invention are all used in a distributed system.
- the user tasks of the distributed system include multiple levels of subtasks, and the subtasks that generate intermediate data in the subtasks are called subordinate subtasks.
- a subtask that relies on intermediate data for processing in a task is called a subordinate subtask.
- FIG. 8 is a structural block diagram of an intermediate data transmission system according to Embodiment 5 of the present invention.
- the intermediate data transmission system 800 can include a determining module 810, a selecting module 820, and a transmitting module 830.
- the determining module 810 is configured to determine intermediate data to be transmitted by the upper subtask, and record it as data to be transmitted.
- the selecting module 820 is configured to select, from a network port for monitoring data of the subordinate task of the subordinate task, a network port that needs to receive data to be transmitted, and record it as a receiving port.
- the transmission module 830 is configured to directly transmit the data to be transmitted determined by the determining module 810 to the receiving port selected by the selecting module 820 through the network.
- the data to be transmitted determined by the determining module 810 may be intermediate data generated by a process of the upper subtask, or may be generated by multiple or all processes of the subordinate subtask. Intermediate data.
- the receiving port selected by the selecting module 820 may be a network port corresponding to a single process of the subordinate subtask, or may be one or more common network ports corresponding to all processes of the subordinate subtask.
- the transmission module 830 can directly transmit the data to be transmitted to the receiving port through the network by using a specific transmission policy that the user formulates according to requirements. For example, the foregoing transmission strategy listed in Embodiment 1 of the present invention.
- the intermediate data transmission system in the embodiment of the present invention directly transfers the intermediate data in the distributed system from the upper sub-task generating the intermediate data to the sub-subtask through the network, and does not need to go through the distributed storage system in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- FIG. 9 is a structural block diagram of an intermediate data transmission system according to Embodiment 6 of the present invention.
- the intermediate data transmission system may include a first startup module 910 , a notification module 920 , a second startup module 930 determination module 940 , a selection module 950 , and a transmission module 960 .
- the first startup module 910 is configured to start a subordinate subtask, so that the network port of the subordinate subtask for monitoring data is in a listening state.
- the notification module 920 is connected to the first startup module 910, and is configured to notify the upper subtask of the information of the network port of the subordinate task for monitoring the data.
- the second startup module 930 is respectively connected to the notification module 920 and the determination module 940, for starting the upper sub-task after the sub-task is started, and generating intermediate data.
- the determining module 940 is configured to determine intermediate data to be transmitted by the upper subtask, and record it as data to be transmitted.
- the selecting module 950 is configured to select, from a network port for monitoring data of the subordinate task of the subordinate task, a network port that needs to receive data to be transmitted, and record it as a receiving port.
- the transmission module 960 is configured to directly transmit the data to be transmitted determined by the determining module 940 to the receiving port selected by the selecting module 950 through the network.
- the intermediate data transmission system in the embodiment of the present invention directly transfers the intermediate data in the distributed system from the upper sub-task generating the intermediate data to the sub-subtask through the network, and does not need to go through the distributed storage system in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the subordinate subtasks are started before the subordinate subtasks to ensure that the subordinate subtasks can receive the intermediate data generated by the subordinate subtasks.
- FIG. 10 is a structural block diagram of a transmission module of an intermediate data transmission system according to Embodiment 7 of the present invention.
- the transmission module 1000 of the intermediate data transmission system may include an inter-process transmission unit 1010.
- the inter-process transmission unit 1010 is configured to generate data to be transmitted by a single process of a sub-subtask.
- Each process of the sub-subtask corresponds to a first network port, and each process of the sub-subtask corresponds to a second network port. If the two network ports are all receiving ports, the data to be transmitted is directly transmitted through the network through the first network port corresponding to the sending process. Transmitted to all second network ports, where the sending process refers to the process that generates the data to be transmitted.
- the intermediate data is directly transmitted to the receiving process by the sending process through the network, and the intermediate data generated by each sending process is not necessarily sent through the upper sub-tasks, and the intermediate data is not uniformly received by the sub-tasks, and then sent to each.
- the process is received, so the intermediate data transfer rate can be further increased.
- the receiving process can process the intermediate data immediately after receiving it, without having to wait until all the sending processes have generated the intermediate data and transmitted it to the receiving process before processing. Therefore, the intermediate data is directly transmitted by the sending process to the receiving process through the network. It also allows the receiving process to process a portion of the intermediate data that has been transmitted to the receiving process while the sending process is still generating intermediate data. That is, the receiving process and the sending process can run in parallel, which significantly improves the upper and lower The parallel execution capability between sub-tasks reduces the overall runtime of user tasks.
- the transmission module 1000 may further include a determining unit and a starting unit.
- the determining unit is configured to determine whether the data to be transmitted is greater than or equal to msgSize, and the msgSize represents the length of the sending data in each transmission of the sending process.
- the startup unit is configured to start the inter-process transmission unit 1010 to transmit the data to be transmitted, and determine that the length of the transmission data of the data to be transmitted is msgSize, if the judgment result of the determination unit is that the data to be transmitted is greater than or equal to msgSize.
- msgSize max(min(BuffSize, MaxMsgSize*n), MinMsgSize*n)/n
- BuffSize represents the upper limit of the available memory of the preset sending process
- MaxMsgSize represents the maximum data length of the preset sending process in one transmission.
- MinMsgSize indicates the minimum data length of the preset transmission process transmitted in one transmission
- n indicates the number of second network ports
- "*" indicates multiplication and operation
- "/" indicates division operation.
- the foregoing determining unit and the starting unit calculate the length of the sending data in each transmission of the sending process according to the size of the user task (refers to the number of receiving processes) and the available memory size of each sending process, thereby realizing dynamic adjustment, so that each time can be avoided.
- the message carrying the data to be transmitted may carry the first process identifier and the message number, where the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the sending process.
- the first few messages sent may carry the first process identifier and the message number, where the first process identifier is used to indicate the sending process of the message, and the message number is used to indicate that the message is the sending process. The first few messages sent.
- the receiving process may determine whether the message has been received according to the first process identifier and the message number carried in the message carrying the data to be transmitted, so as to process the message according to a specific situation.
- the process corresponding to the second network port can record the maximum message number that has been received from the sending process. If the received message carries a message number greater than the maximum message number, the receiving process receives the message for the first time, if received. The message number carried in the message is less than or equal to the maximum message number, indicating that the receiving process has received Passed the message.
- the transmission module 1000 may include a saving unit.
- the saving unit is configured to: after the second network port receives the message carrying the data to be transmitted, the message number carried in the message is greater than the current process record corresponding to the second network port, and the process is sent from the sending process.
- the process corresponding to the second network port saves the data to be transmitted, and updates the recorded maximum message number to the message number carried in the message.
- the transmission module 1000 may further include a discarding unit.
- the discarding unit is configured to: after the second network port receives the message carrying the data to be transmitted, the message number carried in the message is less than or equal to the current record of the process record corresponding to the second network port (refer to the process of sending the data to be transmitted) In the case of the largest message number received, the process corresponding to the second network port discards the message and keeps the recorded maximum message number unchanged.
- the intermediate data transmission system in the embodiment of the present invention directly transfers the intermediate data in the distributed system from the upper sub-task generating the intermediate data to the sub-subtask through the network, and does not need to go through the distributed storage system in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subordinate tasks of the upper and lower levels, and shortens the running time of the overall task of the user.
- the transmission module may adopt a timeout retransmission mechanism to cope with possible message loss (that is, the intermediate data sent by the sending process cannot reach the receiving process).
- the transmission module of the intermediate data transmission system can adopt the structure as shown in FIG. 11 is a structural block diagram of a transmission module of an intermediate data transmission system according to Embodiment 8 of the present invention.
- the transmission module 1100 may include an inter-process transmission unit 1110 and a response receiving unit 1120.
- the inter-process transmission unit 1110 has the same functions as the foregoing inter-process transmission unit 1000, and details are not described herein again.
- the response receiving unit 1120 is configured to: after the inter-process transmission unit 1110 performs the transmission, the sending process receives the response response returned by the process corresponding to the second network port, where the response response is used to indicate that the process corresponding to the second network port has received the pending transmission. data.
- the transmission module 1110 may further include a licensing unit 1130.
- the permitting unit 1130 is configured to allow the sending process to transmit the next data to the process that returns the response response through the network after the response receiving unit 1120 receives the response.
- the transmission module 1110 may further include a retransmission unit 1140.
- the retransmission unit 1140 is configured to: after the inter-process transmission unit 1110 performs transmission, in response to the receiving unit 1120 not receiving the response response returned by the process corresponding to one or more second network ports, the sending process is performed. Pass the data to be transmitted again through the network Directly transmitted to one or more second network ports that do not return an acknowledgment response.
- the response response may carry a second process identifier, the second process identifier is used to indicate the receiving process that sends the response, and the sending process determines to send the response response according to the second process identifier.
- Receive process the second process identifier is used to indicate the receiving process that sends the response, and the sending process determines to send the response response according to the second process identifier.
- the intermediate data transmission system in the embodiment of the present invention directly transfers the intermediate data in the distributed system from the upper sub-task generating the intermediate data to the sub-subtask through the network, and does not need to go through the distributed storage system in the middle, thereby avoiding the disk IO.
- the effect of the rate on the intermediate data transmission rate increases the intermediate data transmission rate.
- the intermediate data transmission system of the embodiment of the present invention also significantly improves the parallel execution capability between the subordinate tasks of the upper and lower levels, and shortens the running time of the overall task of the user.
- the same intermediate data is not transmitted multiple times in the network. Therefore, the intermediate data transmission system in the embodiment of the present invention can also save network bandwidth.
- FIG. 12 is a structural block diagram of a distributed system according to Embodiment 9 of the present invention.
- the distributed system 1200 can include an intermediate data transmission system.
- the intermediate data transmission system may be any of the intermediate data transmission systems of the foregoing embodiments of the present invention.
- the distributed system in the embodiment of the present invention includes an intermediate data transmission system, and the intermediate data in the distributed system is directly transmitted to the lower subtask through the network by the upper subtask generating the intermediate data, and does not need to pass through the distributed storage system.
- the influence of the disk IO rate on the intermediate data transmission rate is avoided, and the intermediate data transmission rate is improved.
- the distributed system of the embodiment of the present invention can significantly improve the parallel execution capability between the subordinate tasks of the upper and lower levels, and shorten the running time of the overall task of the user.
- the same intermediate data is not transmitted multiple times in the network, so the distributed system of the embodiment of the present invention can also save network bandwidth.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer And Data Communications (AREA)
Abstract
本申请涉及一种中间数据传输方法及系统、分布式系统。其中，中间数据传输方法包括：确定所述上级子任务本次要传输的中间数据，记为待传数据；从所述下级子任务的用于监听数据的网络端口中选择需要接收所述待传数据的网络端口，记为接收端口；将所述待传数据通过网络直接传输给所述接收端口。本发明实施例中，中间数据的传输不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
Description
本申请要求2016年02月14日递交的申请号为201610084419.9、发明名称为“中间数据传输方法及系统、分布式系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
本发明涉及通信领域，尤其涉及一种中间数据传输方法及系统、分布式系统。
在分布式系统中，用户任务通常可以被分解为几级不同的子任务。这些子任务之间存在依赖关系，一些子任务的输出结果会作为另一些子任务的输入数据。这就涉及到了子任务之间的数据传递，这些在子任务之间传递的数据称为中间数据。
目前，在相关技术中，中间数据的传输是通过分布式存储系统来完成的，其过程是：产生中间数据的上级子任务通过分布式存储系统的接口将中间数据以文件的形式写到分布式存储系统的磁盘上；需要将该中间数据作为输入数据的下级子任务从分布式存储系统的磁盘中读取该中间数据，以做进一步处理。
以最常见的MapReduce任务为例。MapReduce任务可以被分解为MapTask和ReduceTask这两级子任务。当通过MapReduce任务完成数据排序时，MapTask可以在不同的机器上同时启动若干个进程，每个进程分别读取一部分输入数据，并对这部分数据进行排序，然后将排序的结果输出。这些输出数据会作为ReduceTask的输入，做进一步排序，以达到全局有序。这一过程中，在MapTask和ReduceTask这两级子任务之间传递的数据是排序过程中产生的中间计算结果，是不需要被呈现给用户的。
图1为相关技术中MapTask和ReduceTask之间的中间数据传输过程示意图。如图1所示，相关技术中，MapTask和ReduceTask之间的中间数据传输过程如下：
MapTask输出中间数据，通过分布式存储系统的用户接口将中间数据以文件形式写入分布式存储系统，分布式存储系统将中间数据存储到一个或多个存储节点中，即将这些中间数据持久化到一台或多台机器的磁盘上；
ReduceTask通过分布式存储系统的用户接口，从存储中间数据的存储节点中读取中间数据。
相关技术中，分布式系统的中间数据传输存在如下问题：
1、中间数据传输需要经过磁盘的读写来实现，而传统机械硬盘的平均读写速率只能达到100MB/s左右。因此，这种中间数据传输方式严重受到磁盘IO(输入输出)速率的影响，传输速率低，导致用户任务执行效率低下。
2、分布式存储系统中可能会出现单个存储节点不可用的情况，为了保证数据不会丢失，分布式存储系统通常会为一个文件生成多份拷贝，并存放到不同的存储节点上，这一过程造成了同样的数据会在网络中进行多次传输，占用网络带宽。
发明内容
本发明的目的在于提供一种中间数据传输方法及系统、分布式系统，提高分布式系统的中间数据传输速率。
为实现上述目的，本发明提出了一种中间数据传输方法，用于分布式系统，所述分布式系统的用户任务包括多级子任务，所述子任务中产生中间数据的子任务称为上级子任务，所述子任务中依赖所述中间数据进行处理的子任务称为下级子任务，所述方法包括：
确定所述上级子任务本次要传输的中间数据，记为待传数据；
从所述下级子任务的用于监听数据的网络端口中选择需要接收所述待传数据的网络端口，记为接收端口；
将所述待传数据通过网络直接传输给所述接收端口。
本发明实施例中的中间数据传输方法，中间数据的传输不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
为实现上述目的，本发明还提出了一种中间数据传输系统，用于分布式系统，所述分布式系统的用户任务包括多级子任务，所述子任务中产生中间数据的子任务称为上级子任务，所述子任务中依赖所述中间数据进行处理的子任务称为下级子任务，所述中间数据传输系统包括：
确定模块，用于确定所述上级子任务本次要传输的中间数据，记为待传数据；
选择模块，用于从所述下级子任务的用于监听数据的网络端口中选择需要接收所述待传数据的网络端口，记为接收端口；
传输模块，用于将所述确定模块确定的待传数据通过网络直接传输给所述选择模块选择的接收端口。
本发明实施例中的中间数据传输系统，中间数据的传输不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
为实现上述目的，本发明还提出了一种分布式系统，包括前述任一项所述的中间数据传输系统。
本发明实施例中的分布式系统，中间数据的传输不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
图1为相关技术中MapTask和ReduceTask之间的中间数据传输过程示意图；
图2为本发明实施例一中中间数据传输方法的流程图；
图3为根据图2所示的中间数据传输方法进行传输时MapTask和ReduceTask之间的中间数据传输过程示意图之一；
图4为根据图2所示的中间数据传输方法进行传输时MapTask和ReduceTask之间的中间数据传输过程示意图之二；
图5为本发明实施例二中中间数据传输方法的流程图；
图6为本发明实施例三中中间数据传输方法的流程图；
图7为本发明实施例四中中间数据传输方法的流程图；
图8为本发明实施例五中中间数据传输系统的结构框图；
图9为本发明实施例六中中间数据传输系统的结构框图；
图10为本发明实施例七中中间数据传输系统的传输模块的结构框图；
图11为本发明实施例八中中间数据传输系统的传输模块的结构框图；
图12为本发明实施例九中分布式系统的结构框图。
以下结合附图对本发明的原理和特征进行描述，所举实施例只用于解释本发明，并非用于限定本发明的范围。对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，根据本发明精神所获得的所有实施例，都属于本发明的保护范围。
需要说明的是，本发明各实施例中的中间数据传输方法和中间数据传输系统均用于分布式系统，该分布式系统的用户任务包括多级子任务，其中，将这些子任务中产生中间数据的子任务称为上级子任务，将这些子任务中依赖中间数据进行处理的子任务称为
下级子任务。上级子任务和下级子任务是相对而言的。例如，在具有多级子任务的用户任务中，一个子任务对于子任务A来说是下级子任务，但对于子任务B来说是却可能是下级子任务。
其中，用户任务可以是MapReduce任务、DAG(Directed Acyclic Graph，有向无环图)任务等。
图2为本发明实施例一中中间数据传输方法的流程图。如图2所示，本实施例中，中间数据传输方法可以包括如下步骤：
步骤S201，确定上级子任务本次要传输的中间数据，记为待传数据；
在上级子任务有多个进程的情况下，上级子任务本次要传输的中间数据(也即待传数据)可以是上级子任务的一个进程产生的中间数据，也可以是上级子任务的多个或全部进程产生的中间数据。
步骤S202从下级子任务的用于监听数据的网络端口中选择需要接收待传数据的网络端口，记为接收端口；
下级子任务用于监听数据的网络端口用来接收上级子任务发送的待传数据。
下级子任务可以根据需要设置网络端口。比如，下级子任务可以为每个进程分别设置一个网络端口，也可以为所有进程设置一个或多个共同网络端口。
步骤S203，将待传数据通过网络直接传输给接收端口。
也就是说，将待传数据由上级子任务所在的网络通过网络直接传输到下任务所在的网络，中间不再像背景技术中提到的相关技术那样经过分布式存储系统的写入和读取过程。
以MapReduce任务为例。MapReduce任务的两级子任务MapTask和ReduceTask之间的中间数据传输过程如图3所示。
在此基础上，用户可以根据需要制定具体的传输策略。
例如，传输策略之一可以是：为上级子任务的每个进程分别设置一个第一网络端口，为下级子任务的每个进程分别设置一个第二网络端口，将待传数据由产生待传数据的一个或多个进程对应的第一网络端口分别传输到所有的第二网络端口。也即，将中间数据由产生中间数据的进程直接发送给下级子任务的所有进程。仍以MapReduce任务为例。按照此传输策略，MapReduce任务的两级子任务MapTask和ReduceTask之间的中间数据传递过程如图4所示。
传输策略之二可以是：为上级子任务的所有进程设置一个或多个第一共同网络端口，
为下级子任务的所有进程设置一个或多个第二共同网络端口，将上级子任务的各个进程产生的中间数据由第一共同网络端口通过网络直接传输到第二共同网络端口，第二共同网络端口再将中间数据分发给下级子任务的各个进程。
本发明实施例中的中间数据传输方法，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
图5为本发明实施例二中中间数据传输方法的流程图。如图5所示，本实施例中，中间数据传输方法可以包括如下步骤：
步骤S501，启动下级子任务，使下级子任务的用于监听数据的网络端口处于监听状态；
由于本发明实施例中，中间数据是由上级子任务通过网络直接传输给下级子任务的，因此在启动上级子任务产生中间数据前，下级子任务应处于能够接收中间数据的状态。也就是说，下级子任务应当先于上级子任务启动，这样才能保证下级子任务能够接收到上级子任务产生的中间数据。
步骤S502，将下级子任务的用于监听数据的网络端口的信息通知给上级子任务；
网络端口的信息一般可以包括IP地址等。上级子任务可以根据下级子任务的用于监听数据的网络端口的信息确定中间数据的目的地址。
在具体应用中，下级子任务可以将网络端口的信息先上报给调度器，调度器再将下级子任务的用于监听数据的网络端口信息发送给上级子任务。
步骤S503，在下级子任务启动之后启动上级子任务，产生中间数据；
在下级子任务启动后，下级子任务的用于监听数据的网络端口已经处于监听状态的情况下，再启动上级子任务。即先启动下级子任务启动，后启动上级子任务。
步骤S504，确定上级子任务本次要传输的中间数据，记为待传数据；
步骤S505从下级子任务的用于监听数据的网络端口中选择需要接收待传数据的网络端口，记为接收端口；
步骤S506，将待传数据通过网络直接传输给接收端口。
本发明实施例中的中间数据传输方法，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。再者，本发明实施例中的中间数据传输方法，下级子任务先于上级子任务启动，以保证下级子任务能
够接收到上级子任务产生的中间数据。
图6为本发明实施例三中中间数据传输方法的流程图。如图6所示，本实施例中，中间数据传输方法可以包括如下步骤：
步骤S601，确定上级子任务的单个进程本次要传输的中间数据，记为待传数据，其中，上级子任务的每个进程对应一个第一网络端口；
本实施例中，以进程为单位进行中间数据的发送。上级子任务的每个进程各自确定自己要传输的中间数据，然后将各自将自己要传输的中间数据通过网络传输给下级子任务的所有进程。
步骤S602，下级子任务的每个进程对应一个第二网络端口，将所有第二网络端口均选为待传数据的接收端口；
在本实施例中，上、下级子任务的每个进程都分别对应一个网络端口。上级子任务的进程对应的第一网络端口为中间数据的发送端口，下级子任务的进程对应的第二网络端口为中间数据的接收端口。相应地，将上级子任务的进程称为发送进程，将下级子任务的进程称为接收进程。中间数据由发送端口通过网络直接传输到接收端口。该传输方式例如前述图4所示的中间数据传输方式。
步骤S603，将待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口，其中，发送进程为产生待传数据的进程。
由上述步骤可见，本实施例中，中间数据由发送进程通过网络直接传输到接收进程，不必经由上级子任务收集各个发送进程产生的中间数据统一发送，也不必经由下级子任务统一接收中间数据后再分别发给各个接收进程，因此可以进一步提高中间数据传输速率。不仅如此，接收进程可以在接收到中间数据后立即进行处理，而不必等到所有发送进程产生完中间数据并传输到接收进程后才能进行处理，因此，中间数据由发送进程通过网络直接传输到接收进程，还可以让接收进程在发送进程仍在产生中间数据的时候就能够对已经传输到接收进程的一部分中间数据先进行处理，也就是说，接收进程和发送进程可以并行运行，这显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。
由于通过网络传输数据本身会有一定的延时，如果发送进程每次只发送少量数据，那么发送数据的次数就会变多，这样由于网络传输产生的overhead(间接成本)就会很大。另一方面，如果发送进程每次发送的数据量都较大，当需要接收进程比较多时，就会导致发送进程使用的内存很高。为了解决发送进程何时发送中间数据的问题，在本发
明其他实施例中，在步骤S603之前，也即在将待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之前，还可以包括如下步骤：
判断待传数据是否大于或等于msgSize，msgSize表示发送进程每次传输中的发送数据长度；
在待传数据大于或等于msgSize的情况下，启动待传数据的传输，并确定待传数据的发送数据长度为msgSize；
其中，msgSize＝max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n，
BuffSize表示预设的发送进程的可用内存上限，MaxMsgSize表示预设的发送进程在一次发送中传输的最大数据长度，MinMsgSize表示预设的发送进程在一次发送中传输的最小数据长度，n表示第二网络端口的数量，“*”表示乘以运算，“/”表示除以运算。n的含义是接收进程的数量，在本实施例中，接收进程的数量等于接收端口的数量，也等于第二网络端口的数量，因此，接收进程的数量就是第二网络端口的数量。
上述步骤根据用户任务的规模(指接收进程的数量)及每个发送进程可用的内存大小计算发送进程每次传输中的发送数据长度，实现动态调节，这样就可以避免由于每次发送数据过少导致的间接成本过高的问题以及每次发送数据过大导致的内存需求过高的问题。
待传数据通常是以消息的形式发送的。在本发明实施例中，承载待传数据的消息中可以携带第一进程标识和消息编号，其中，第一进程标识用于表明该消息的发送进程，消息编号用于表明该消息是该发送进程发送的第几条消息。也就是说，第一进程标识用于说明消息是由“谁”发送的问题，消息编号是单调递增的，它用于说明该消息在发送进程发送的所有消息中所处的位置问题或者说序号问题。
接收进程可以根据承载待传数据的消息中携带的第一进程标识和消息编号判断是否已经接收到该消息，以便根据具体情况处理该消息。
第二网络端口对应的进程可以记录当前已经从发送进程收到的最大消息编号，如果接收到的消息携带的消息编号大于该最大消息编号，说明接收进程第一次接收该消息，如果接收到的消息携带的消息编号小于或等于该最大消息编号，说明接收进程已经收到过该消息。
据此，在本发明实施例中，在将待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还可以包括：
第二网络端口接收到承载待传数据的消息后，在该消息携带的消息编号大于第二网
络端口对应的进程记录的当前已经从该发送进程收到的最大消息编号的情况下，第二网络端口对应的进程保存该待传数据，并将记录的最大消息编号更新为该消息携带的消息编号。
或者，在本发明实施例中，在将待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还可以包括：第二网络端口接收到承载待传数据的消息后，在该消息携带的消息编号小于或等于第二网络端口对应的进程记录的当前已经从该发送进程收到的最大消息编号的情况下，第二网络端口对应的进程丢弃该消息，并维持记录的最大消息编号不变。
本发明实施例中的中间数据传输方法，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。而且，本发明实施例的中间数据传输系统，还显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。
在中间数据采用图6所示实施例中的由发送进程直接传输到接收进程的传输方式的情况下，可以采用超时重发机制应对可能出现的消息丢失(即发送进程发送的中间数据无法到达接收进程)现象。该超时重发机制的具体内容可以是，发送进程向接收进程发送中间数据后，如果接收进程接收到该中间数据，就向发送进程返回应答响应，告知发送进程该接收进程已经接收到其发送的中间数据；发送进程接收到应答响应后，就可以向返回应答响应的接收进程发送下一次的中间数据了。如果发送进程向接收进程发送中间数据后，接收进程没有接收到该中间数据，就不会向发送进程返回应答响应。发送进程在设定的时限内没有接收到应答响应，就认为没有返回应答响应的接收进程没有接收到该中间数据，因此会向没有返回应答响应的接收进程重发本次传输的中间数据，而不会向没有返回应答响应的接收进程发送下一次的中间数据，直到发送进程接收到该接收进程返回的应答响应为止。这样，既可以保证接收进程能够接收到中间数据，又可以使同样的中间数据不会在网络中进行多次传输，因此能够节约网络带宽。
据此，本发明的中间数据传输方法可以采用图7所示的流程。
图7为本发明实施例四中中间数据传输方法的流程图。如图7所示，本实施例中，中间数据传输方法可以包括如下步骤：
步骤S701，确定上级子任务的单个进程本次要传输的中间数据，记为待传数据，其中，上级子任务的每个进程对应一个第一网络端口；
步骤S702，下级子任务的每个进程对应一个第二网络端口，将所有第二网络端口均选为待传数据的接收端口；
步骤S703，将待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口，其中，发送进程为产生待传数据的进程；
步骤S704，在传输之后，判断在设定时间内发送进程是否接收到第二网络端口对应的进程返回的应答响应，如果在设定时间内发送进程接收到第二网络端口对应的进程返回的应答响应，则执行步骤S705，否则执行步骤S706；
为了区分应答响应来自于哪一个接收进程，应答响应中可以携带第二进程标识，该第二进程标识用于表明发出该应答响应的接收进程，发送进程根据第二进程标识判断发出该应答响应的接收进程。
步骤S705，允许发送进程通过网络向返回应答响应的进程传输下一次的数据；
步骤S706，发送进程再次将待传数据通过网络直接传输给未返回应答响应的一个或多个第二网络端口，返回步骤S704。
本发明实施例中的中间数据传输方法，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。而且，本发明实施例的中间数据传输系统，还显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。同时，在本发明实施例中，同样的中间数据不会在网络中进行多次传输，因此本发明实施例的中间数据传输系统，还能够节约网络带宽。
本发明还提出了中间数据传输系统，用以实施上述各实施例的中间数据传输方法。上述对中间数据传输方法的说明均适用于中间数据传输系统的相应部分。下面本发明各实施例中的中间数据传输系统都用于分布式系统，该分布式系统的用户任务包括多级子任务，这些子任务中产生中间数据的子任务称为上级子任务，这些子任务中依赖中间数据进行处理的子任务称为下级子任务。
图8为本发明实施例五中中间数据传输系统的结构框图。如图8所示，本实施例中，中间数据传输系统800可以包括确定模块810、选择模块820和传输模块830。其中，确定模块810用于确定上级子任务本次要传输的中间数据，记为待传数据。选择模块820用于从下级子任务的用于监听数据的网络端口中选择需要接收待传数据的网络端口，记为接收端口。传输模块830用于将确定模块810确定的待传数据通过网络直接传输给选择模块820选择的接收端口。
其中，在上级子任务有多个进程的情况下，确定模块810所确定的待传数据可以是上级子任务的一个进程产生的中间数据，也可以是上级子任务的多个或全部进程产生的中间数据。
其中，选择模块820选择的接收端口可以是下级子任务的单个进程对应的网络端口，也可以为下级子任务的所有进程对应的一个或多个共同网络端口。
其中，传输模块830可以使用用户根据需要制定的具体传输策略将待传数据通过网络直接传输给接收端口。例如前述本发明实施例一中列举的传输策略。
本发明实施例中的中间数据传输系统，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。
图9为本发明实施例六中中间数据传输系统的结构框图。如图9所示，本实施例中，中间数据传输系统可以包括第一启动模块910、通知模块920、第二启动模块930确定模块940、选择模块950和传输模块960。其中，第一启动模块910用于启动下级子任务，使下级子任务的用于监听数据的网络端口处于监听状态。通知模块920与第一启动模块910相连，用于将下级子任务的用于监听数据的网络端口的信息通知给上级子任务。第二启动模块930分别与通知模块920和确定模块940相连，用于在下级子任务启动之后启动上级子任务，产生中间数据。确定模块940用于确定上级子任务本次要传输的中间数据，记为待传数据。选择模块950用于从下级子任务的用于监听数据的网络端口中选择需要接收待传数据的网络端口，记为接收端口。传输模块960用于将确定模块940确定的待传数据通过网络直接传输给选择模块950选择的接收端口。
本发明实施例中的中间数据传输系统，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。再者，本发明实施例中的中间数据传输系统，下级子任务先于上级子任务启动，以保证下级子任务能够接收到上级子任务产生的中间数据。
图10为本发明实施例七中中间数据传输系统的传输模块的结构框图。如图10所示，本实施例中，中间数据传输系统的传输模块1000可以包括进程间传输单元1010。进程间传输单元1010用于在待传数据由上级子任务的单个进程产生，上级子任务的每个进程对应一个第一网络端口，下级子任务的每个进程对应一个第二网络端口，所有第二网络端口均为接收端口的情况下，将待传数据由发送进程对应的第一网络端口通过网络直接
传输给所有第二网络端口，其中，发送进程指产生待传数据的进程。
本实施例中，中间数据由发送进程通过网络直接传输到接收进程，不必经由上级子任务收集各个发送进程产生的中间数据统一发送，也不必经由下级子任务统一接收中间数据后再分别发给各个接收进程，因此可以进一步提高中间数据传输速率。不仅如此，接收进程可以在接收到中间数据后立即进行处理，而不必等到所有发送进程产生完中间数据并传输到接收进程后才能进行处理，因此，中间数据由发送进程通过网络直接传输到接收进程，还可以让接收进程在发送进程仍在产生中间数据的时候就能够对已经传输到接收进程的一部分中间数据先进行处理，也就是说，接收进程和发送进程可以并行运行，这显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。
在图10所示实施例的基础上，传输模块1000还可以进一步包括判断单元和启动单元。其中，判断单元用于判断待传数据是否大于或等于msgSize，msgSize表示发送进程每次传输中的发送数据长度。启动单元用于在判断单元的判断结果为待传数据大于或等于msgSize的情况下，启动进程间传输单元1010进行待传数据的传输，并确定待传数据的发送数据长度为msgSize。其中，msgSize＝max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n，BuffSize表示预设的发送进程的可用内存上限，MaxMsgSize表示预设的发送进程在一次发送中传输的最大数据长度，MinMsgSize表示预设的发送进程在一次发送中传输的最小数据长度，n表示第二网络端口的数量，“*”表示乘以运算，“/”表示除以运算。
上述判断单元和启动单元根据用户任务的规模(指接收进程的数量)及每个发送进程可用的内存大小计算发送进程每次传输中的发送数据长度，实现动态调节，这样就可以避免由于每次发送数据过少导致的间接成本过高的问题以及每次发送数据过大导致的内存需求过高的问题。
在本发明实施例中，承载待传数据的消息中可以携带第一进程标识和消息编号，其中，第一进程标识用于表明该消息的发送进程，消息编号用于表明该消息是该发送进程发送的第几条消息。
接收进程可以根据承载待传数据的消息中携带的第一进程标识和消息编号判断是否已经接收到该消息，以便根据具体情况处理该消息。
第二网络端口对应的进程可以记录当前已经从发送进程收到的最大消息编号，如果接收到的消息携带的消息编号大于该最大消息编号，说明接收进程第一次接收该消息，如果接收到的消息携带的消息编号小于或等于该最大消息编号，说明接收进程已经收到
过该消息。
据此，在本发明实施例中，传输模块1000可以包括保存单元。保存单元用于在第二网络端口接收到承载待传数据的消息后，在该消息携带的消息编号大于第二网络端口对应的进程记录的当前已经从该发送进程(指发出待传数据的进程)收到的最大消息编号的情况下，令第二网络端口对应的进程保存待传数据，并将记录的最大消息编号更新为该消息携带的消息编号。
在本发明实施例中，传输模块1000还可以包括丢弃单元。丢弃单元用于在第二网络端口接收到承载待传数据的消息后，在消息携带的消息编号小于或等于第二网络端口对应的进程记录的当前已经从发送进程(指发出待传数据的进程)收到的最大消息编号的情况下，令第二网络端口对应的进程丢弃该消息，并维持记录的最大消息编号不变。
本发明实施例中的中间数据传输系统，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。而且，本发明实施例的中间数据传输系统，还显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。
在中间数据采用图10所示实施例中的进程间传输单元进行传输的情况下，传输模块可以采用超时重发机制应对可能出现的消息丢失(即发送进程发送的中间数据无法到达接收进程)现象。
据此，中间数据传输系统的传输模块可以采用如图11所示的结构。图11为本发明实施例八中中间数据传输系统的传输模块的结构框图。如图11所示，本实施例中，传输模块1100可以包括进程间传输单元1110、响应接收单元1120。其中，进程间传输单元1110同前述的进程间传输单元1000的功能相同，此处不再赘述。响应接收单元1120用于在进程间传输单元1110进行传输之后，令发送进程接收第二网络端口对应的进程返回的应答响应，该应答响应用于表示第二网络端口对应的进程已经接收到待传数据。
参见图11，传输模块1110还可以包括许可单元1130。许可单元1130用于在响应接收单元1120接收到应答响应后，允许发送进程通过网络向返回应答响应的进程传输下一次的数据。
参见图11，传输模块1110还可以包括重传单元1140。重传单元1140用于在进程间传输单元1110进行传输之后，在设定时间内响应接收单元1120未接收到一个或多个第二网络端口对应的进程返回的应答响应的情况下，令发送进程再次将待传数据通过网络
直接传输给未返回应答响应的一个或多个第二网络端口。
为了区分应答响应来自于哪一个接收进程，应答响应中可以携带第二进程标识，该第二进程标识用于表明发出该应答响应的接收进程，发送进程根据第二进程标识判断发出该应答响应的接收进程。
本发明实施例中的中间数据传输系统，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。而且，本发明实施例的中间数据传输系统，还显著提高了上下级子任务之间的并行执行能力，缩短了用户任务整体的运行时间。同时，在本发明实施例中，同样的中间数据不会在网络中进行多次传输，因此本发明实施例的中间数据传输系统还能够节约网络带宽。
图12为本发明实施例九中分布式系统的结构框图。如图12所示，本实施例中，分布式系统1200可以包括中间数据传输系统。该中间数据传输系统可以是本发明前述实施例中的任一种中间数据传输系统。
本发明实施例中的分布式系统中包括中间数据传输系统，将分布式系统中的中间数据由产生该中间数据的上级子任务通过网络直接传输给下级子任务，中间不需要经过分布式存储系统，避免了磁盘IO速率对中间数据传输速率的影响，提高了中间数据传输速率。而且，本发明实施例的分布式系统，还能够显著提高上下级子任务之间的并行执行能力，缩短用户任务整体的运行时间。同时，在本发明实施例中，同样的中间数据不会在网络中进行多次传输，因此本发明实施例的分布式系统还能够节约网络带宽。
以上所述仅为本发明的较佳实施例，并不用以限制本发明，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。
Claims (23)
- 一种中间数据传输方法，用于分布式系统，所述分布式系统的用户任务包括多级子任务，所述子任务中产生中间数据的子任务称为上级子任务，所述子任务中依赖所述中间数据进行处理的子任务称为下级子任务，其特征在于，所述方法包括：确定所述上级子任务本次要传输的中间数据，记为待传数据；从所述下级子任务的用于监听数据的网络端口中选择需要接收所述待传数据的网络端口，记为接收端口；将所述待传数据通过网络直接传输给所述接收端口。
- 根据权利要求1所述的中间数据传输方法，其特征在于，在所述确定所述上级子任务本次要传输的中间数据之前，还包括：启动所述下级子任务，使所述下级子任务的用于监听数据的网络端口处于监听状态；将所述下级子任务的用于监听数据的网络端口的信息通知给所述上级子任务；在所述下级子任务启动之后启动所述上级子任务，产生中间数据。
- 根据权利要求1所述的中间数据传输方法，其特征在于，所述将所述待传数据通过网络直接传输给所述接收端口包括：在所述待传数据由所述上级子任务的单个进程产生，所述上级子任务的每个进程对应一个第一网络端口，所述下级子任务的每个进程对应一个第二网络端口，所有第二网络端口均为接收端口的情况下，将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口，所述发送进程为产生所述待传数据的进程。
- 根据权利要求3所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之前，还包括：判断所述待传数据是否大于或等于msgSize，msgSize表示产生所述发送进程每次传输中的发送数据长度；在所述待传数据大于或等于msgSize的情况下，启动所述待传数据的传输，并确定所述待传数据的发送数据长度为msgSize；其中，msgSize＝max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n，BuffSize表示预设的所述发送进程的可用内存上限，MaxMsgSize表示预设的所述发送进程在一次发送中传输的最大数据长度，MinMsgSize表示预设的所述发送进程在一次发送中传输的最小数据长度，n表示所述第二网络端口的数量，“*”表示乘以运算， “/”表示除以运算。
- 根据权利要求3所述的中间数据传输方法，其特征在于，承载所述待传数据的消息中携带第一进程标识和消息编号，所述第一进程标识用于表明所述消息的发送进程，所述消息编号用于表明所述消息是所述发送进程发送的第几条消息。
- 根据权利要求5所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还包括：所述第二网络端口接收到承载所述待传数据的消息后，在所述消息携带的消息编号大于所述第二网络端口对应的进程记录的当前已经从所述发送进程收到的最大消息编号的情况下，所述第二网络端口对应的进程保存所述待传数据，并将记录的所述最大消息编号更新为所述消息携带的消息编号。
- 根据权利要求5所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还包括：所述第二网络端口接收到承载所述待传数据的消息后，在所述消息携带的消息编号小于或等于所述第二网络端口对应的进程记录的当前已经从所述发送进程收到的最大消息编号的情况下，所述第二网络端口对应的进程丢弃所述消息，并维持记录的所述最大消息编号不变。
- 根据权利要求3所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还包括：在所述传输之后，所述发送进程接收所述第二网络端口对应的进程返回的应答响应，所述应答响应用于表示所述第二网络端口对应的进程已经接收到所述待传数据。
- 根据权利要求8所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还包括：在所述发送进程接收所述第二网络端口对应的进程返回的应答响应后，允许所述发送进程通过网络向返回所述应答响应的进程传输下一次的数据。
- 根据权利要求8所述的中间数据传输方法，其特征在于，在将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口之后，还包括：在所述传输之后，在设定时间内所述发送进程未接收到一个或多个第二网络端口对应的进程返回的应答响应的情况下，所述发送进程再次将所述待传数据通过网络直接传输给所述一个或多个第二网络端口。
- 根据权利要求8所述的中间数据传输方法，其特征在于，所述应答响应中携带 第二进程标识，所述第二进程标识用于表明发出所述应答响应的进程，所述发送进程根据所述第二进程标识判断发出所述应答响应的进程。
- 一种中间数据传输系统，用于分布式系统，所述分布式系统的用户任务包括多级子任务，所述子任务中产生中间数据的子任务称为上级子任务，所述子任务中依赖所述中间数据进行处理的子任务称为下级子任务，其特征在于，所述中间数据传输系统包括：确定模块，用于确定所述上级子任务本次要传输的中间数据，记为待传数据；选择模块，用于从所述下级子任务的用于监听数据的网络端口中选择需要接收所述待传数据的网络端口，记为接收端口；传输模块，用于将所述确定模块确定的待传数据通过网络直接传输给所述选择模块选择的接收端口。
- 根据权利要求12所述的中间数据传输系统，其特征在于，还包括：第一启动模块，用于启动所述下级子任务，使所述下级子任务的用于监听数据的网络端口处于监听状态；通知模块，与所述第一启动模块相连，用于将所述下级子任务的用于监听数据的网络端口的信息通知给所述上级子任务；第二启动模块，分别与所述通知模块和所述确定模块相连，用于在所述下级子任务启动之后启动所述上级子任务，产生中间数据。
- 根据权利要求12所述的中间数据传输系统，其特征在于，所述传输模块包括：进程间传输单元，用于在所述待传数据由所述上级子任务的单个进程产生，所述上级子任务的每个进程对应一个第一网络端口，所述下级子任务的每个进程对应一个第二网络端口，所有第二网络端口均为接收端口的情况下，将所述待传数据由发送进程对应的第一网络端口通过网络直接传输给所有第二网络端口，所述发送进程指产生所述待传数据的进程。
- 根据权利要求14所述的中间数据传输系统，其特征在于，所述传输模块还包括：判断单元，用于判断所述待传数据是否大于或等于msgSize，msgSize表示所述发送进程每次传输中的发送数据长度；启动单元，用于在所述判断单元的判断结果为所述待传数据大于或等于msgSize的情况下，启动所述进程间传输单元进行所述待传数据的传输，并确定所述待传数据的发 送数据长度为msgSize；其中，msgSize＝max(min(BuffSize,MaxMsgSize*n),MinMsgSize*n)/n，BuffSize表示预设的所述发送进程的可用内存上限，MaxMsgSize表示预设的所述发送进程在一次发送中传输的最大数据长度，MinMsgSize表示预设的所述发送进程在一次发送中传输的最小数据长度，n表示所述第二网络端口的数量，“*”表示乘以运算，“/”表示除以运算。
- 根据权利要求14所述的中间数据传输系统，其特征在于，承载所述待传数据的消息中携带第一进程标识和消息编号，所述第一进程标识用于表明所述消息的发送进程，所述消息编号用于表明所述消息是所述发送进程发送的第几条消息。
- 根据权利要求16所述的中间数据传输系统，其特征在于，所述传输模块还包括：保存单元，用于在所述第二网络端口接收到承载所述待传数据的消息后，在所述消息携带的消息编号大于所述第二网络端口对应的进程记录的当前已经从所述发送进程收到的最大消息编号的情况下，令所述第二网络端口对应的进程保存所述待传数据，并将记录的所述最大消息编号更新为所述消息携带的消息编号。
- 根据权利要求16所述的中间数据传输系统，其特征在于，所述传输模块还包括：丢弃单元，用于在所述第二网络端口接收到承载所述待传数据的消息后，在所述消息携带的消息编号小于或等于所述第二网络端口对应的进程记录的当前已经从所述发送进程收到的最大消息编号的情况下，令所述第二网络端口对应的进程丢弃所述消息，并维持记录的所述最大消息编号不变。
- 根据权利要求14所述的中间数据传输系统，其特征在于，所述传输模块还包括：响应接收单元，用于在所述进程间传输单元进行传输之后，令所述发送进程接收所述第二网络端口对应的进程返回的应答响应，所述应答响应用于表示所述第二网络端口对应的进程已经接收到所述待传数据。
- 根据权利要求19所述的中间数据传输系统，其特征在于，所述传输模块还包括：许可单元，用于在所述响应接收单元接收到所述应答响应后，允许所述发送进程通过网络向返回所述应答响应的进程传输下一次的数据。
- 根据权利要求19所述的中间数据传输系统，其特征在于，所述传输模块还包括：重传单元，用于在所述进程间传输单元进行传输之后，在设定时间内所述响应接收单元未接收到一个或多个第二网络端口对应的进程返回的应答响应的情况下，令所述发送进程再次将所述待传数据通过网络直接传输给所述一个或多个第二网络端口。
- 根据权利要求19所述的中间数据传输系统，其特征在于，所述应答响应中携带第二进程标识，所述第二进程标识用于表明发出所述应答响应的进程，所述发送进程根据所述第二进程标识判断发出所述应答响应的进程。
- 一种分布式系统，其特征在于，包括权利要求12至22任一项所述的中间数据传输系统。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610084419.9 | 2016-02-14 | ||
| CN201610084419.9A CN107087010B (zh) | 2016-02-14 | 2016-02-14 | 中间数据传输方法及系统、分布式系统 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2017136999A1 true WO2017136999A1 (zh) | 2017-08-17 |
Family
ID=59562909
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2017/072818 Ceased WO2017136999A1 (zh) | 2016-02-14 | 2017-02-03 | 中间数据传输方法及系统、分布式系统 |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN107087010B (zh) |
| TW (1) | TWI752005B (zh) |
| WO (1) | WO2017136999A1 (zh) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115987812A (zh) * | 2022-12-27 | 2023-04-18 | 上海天旦网络科技发展有限公司 | 海量会话数据聚合分析方法、系统、设备及介质 |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103279390A (zh) * | 2012-08-21 | 2013-09-04 | 中国科学院信息工程研究所 | 一种面向小作业优化的并行处理系统 |
| CN104063486A (zh) * | 2014-07-03 | 2014-09-24 | 四川中亚联邦科技有限公司 | 一种大数据分布式存储方法和系统 |
| US20150095308A1 (en) * | 2013-10-01 | 2015-04-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
| CN105138679A (zh) * | 2015-09-14 | 2015-12-09 | 桂林电子科技大学 | 一种基于分布式缓存的数据处理系统及处理方法 |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8077602B2 (en) * | 2008-02-01 | 2011-12-13 | International Business Machines Corporation | Performing dynamic request routing based on broadcast queue depths |
| CN101621429B (zh) * | 2009-07-20 | 2012-05-23 | 中兴通讯股份有限公司 | 一种报文监控的方法和系统 |
| US8346845B2 (en) * | 2010-04-14 | 2013-01-01 | International Business Machines Corporation | Distributed solutions for large-scale resource assignment tasks |
| CN102655480B (zh) * | 2011-03-03 | 2015-12-02 | 腾讯科技（深圳）有限公司 | 相似邮件处理系统和方法 |
| CN102508704A (zh) * | 2011-11-10 | 2012-06-20 | 上海市共进通信技术有限公司 | 计算机软件系统中实现任务分解和并行处理控制的方法 |
| US9405572B2 (en) * | 2014-04-07 | 2016-08-02 | International Business Machines Corporation | Optimized resource allocation and management in a virtualized computing environment |
| CN104035817A (zh) * | 2014-07-08 | 2014-09-10 | 领佰思自动化科技(上海)有限公司 | 大规模集成电路物理实现的分布式并行计算方法及其系统 |
-
2016
- 2016-02-14 CN CN201610084419.9A patent/CN107087010B/zh active Active
-
2017
- 2017-02-03 WO PCT/CN2017/072818 patent/WO2017136999A1/zh not_active Ceased
- 2017-02-07 TW TW106103972A patent/TWI752005B/zh not_active IP Right Cessation
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103279390A (zh) * | 2012-08-21 | 2013-09-04 | 中国科学院信息工程研究所 | 一种面向小作业优化的并行处理系统 |
| US20150095308A1 (en) * | 2013-10-01 | 2015-04-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
| CN104063486A (zh) * | 2014-07-03 | 2014-09-24 | 四川中亚联邦科技有限公司 | 一种大数据分布式存储方法和系统 |
| CN105138679A (zh) * | 2015-09-14 | 2015-12-09 | 桂林电子科技大学 | 一种基于分布式缓存的数据处理系统及处理方法 |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115987812A (zh) * | 2022-12-27 | 2023-04-18 | 上海天旦网络科技发展有限公司 | 海量会话数据聚合分析方法、系统、设备及介质 |
Also Published As
| Publication number | Publication date |
|---|---|
| TWI752005B (zh) | 2022-01-11 |
| CN107087010A (zh) | 2017-08-22 |
| CN107087010B (zh) | 2020-10-27 |
| TW201732632A (zh) | 2017-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7290066B2 (en) | Methods and structure for improved transfer rate performance in a SAS wide port environment | |
| JP6290462B2 (ja) | ネットワーク・アクセス可能なブロック・ストレージのための協調アドミッション制御 | |
| US8478926B1 (en) | Co-processing acceleration method, apparatus, and system | |
| WO2017133623A1 (zh) | 一种数据流处理方法、装置和系统 | |
| JP6580307B2 (ja) | マルチコア装置及びマルチコア装置のジョブスケジューリング方法 | |
| US7240135B2 (en) | Method of balancing work load with prioritized tasks across a multitude of communication ports | |
| US9178839B2 (en) | Sharing buffer space in link aggregation configurations | |
| JP2015537307A (ja) | コンポーネント指向ハイブリッドクラウドオペレーティングシステムのアーキテクチャ及びその通信方法 | |
| WO2011142087A1 (ja) | 中継器およびチップ回路 | |
| WO2016030973A1 (ja) | マルチテナントリソース調停方法 | |
| JP5362975B2 (ja) | ストレージ・デバイス間のデータ複製を制御する制御装置、方法、プログラム及びストレージ・システム | |
| CN108616398B (zh) | 一种基于dns负载均衡技术的容器动态缩容方法 | |
| WO2023116438A1 (zh) | 一种数据访问方法、装置以及设备 | |
| US20070220217A1 (en) | Communication Between Virtual Machines | |
| CN106412039A (zh) | 一种分布式存储的网络集成的方法及系统 | |
| WO2021203680A1 (zh) | 一种业务流的传输方法、装置、设备及存储介质 | |
| US20160261484A9 (en) | Chip multi processor and router for chip multi processor | |
| CN104834565B (zh) | 一种系统服务动态部署方法及装置 | |
| WO2017136999A1 (zh) | 中间数据传输方法及系统、分布式系统 | |
| CN106372013B (zh) | 远程内存访问方法、装置和系统 | |
| WO2015062390A1 (zh) | 虚拟机迁移方法、装置及系统 | |
| CN118175111B (zh) | 一种数据传输方法、dma控制器、设备及存储介质 | |
| US10182019B2 (en) | Interconnected hardware infrastructure resource control | |
| JP4317348B2 (ja) | 情報処理装置及び入出力方法並びにプログラム | |
| CN115484318A (zh) | 访问请求控制方法、装置及电子设备 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17749890 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 17749890 Country of ref document: EP Kind code of ref document: A1 |