WO2023280207A1 - 数据处理方法、执行工作站、分布式计算系统和存储介质 - Google Patents

数据处理方法、执行工作站、分布式计算系统和存储介质 Download PDF

Info

Publication number
WO2023280207A1
WO2023280207A1 PCT/CN2022/104127 CN2022104127W WO2023280207A1 WO 2023280207 A1 WO2023280207 A1 WO 2023280207A1 CN 2022104127 W CN2022104127 W CN 2022104127W WO 2023280207 A1 WO2023280207 A1 WO 2023280207A1
Authority
WO
WIPO (PCT)
Prior art keywords
sub
execution
segments
workstation
segment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/104127
Other languages
English (en)
French (fr)
Inventor
俞博文
陈文光
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to US18/576,856 priority Critical patent/US12477032B2/en
Publication of WO2023280207A1 publication Critical patent/WO2023280207A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9027—Trees
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying

Definitions

  • Embodiments of the present disclosure relate to a data processing method, an execution workstation, a distributed computing system, and a computer-readable storage medium.
  • Spark is one of the most popular big data processing frameworks today. Spark connects various data analysis domains via its elastic distributed dataset programming model. Spark's machine learning framework MLlib is widely used because MLlib can be easily integrated into Spark's data analysis workflow. Tree aggregation is an important research hotspot in MLlib. Tree aggregation in Spark can be roughly divided into two consecutive steps: calculation and reduction. In a compute step, the value for the current partition is computed locally. In the reduction step, the computed values (called aggregators) are further globally reduced to a single aggregator.
  • At least one embodiment of the present disclosure provides a data processing method for distributed computing performed by an execution workstation, including: splitting an aggregation sub-segment with a predetermined data format into a plurality of aggregation sub-segments; The first part of the aggregation sub-segment of is sent to other execution workstations for reduction on other execution workstations; the aggregation sub-segments corresponding to the second part of the aggregation sub-segments in the plurality of aggregation sub-segments generated by other execution workstations are received from other execution workstations fragment and reduce it with the second part of the aggregate sub-fragment to update the second part of the aggregate sub-fragment, the first part being different from the second part; Partially re-determine the first part of the aggregated sub-segment and the second part of the aggregated sub-segment and perform the sending, receiving, and reducing described above until each of the plurality of aggregated sub-segments
  • the number of aggregated sub-segments to be split by the aggregated sub-segment is equal to the number of execution workstations participating in the aggregation; The number is one each.
  • the number of aggregated sub-segments of the aggregated sub-segments is equal to the number of execution workstations participating in the aggregation multiplied by the number of parallel communication lines of each execution workstation; and the first part The numbers of the aggregation sub-segment and the second part of the aggregation sub-segment are respectively the number of parallel communication lines of each execution workstation.
  • the first part of aggregated sub-segments re-determined in the remaining part after removing the sent first part of aggregated sub-segments from the plurality of aggregated sub-segments is performed in the previous round
  • the second part of the aggregated subfragment that is updated in the receive and reduce is performed in the previous round
  • At least one embodiment of the present disclosure provides an execution workstation for distributed computing, including: a processor and a memory, the memory stores one or more computer program instructions, one or more computer program instructions are stored in the memory, and are executed by The processor implements the steps of the data processing method provided by at least one embodiment of the present disclosure when executed.
  • At least one embodiment of the present disclosure provides a data processing method executed by a distributed computing system.
  • the distributed computing system includes a plurality of execution workstations and a management workstation.
  • the data processing method includes: each of the plurality of execution workstations performs the following steps: Splitting an aggregation sub-segment with a predetermined data format into a plurality of aggregation sub-segments; sending the first part of the aggregation sub-segments among the plurality of aggregation sub-segments to other execution workstations for reduction on other execution workstations; from other execution workstations receiving the aggregation sub-segment corresponding to the second part of the aggregation sub-segment generated by other execution workstations and reducing it with the second part of the aggregation sub-segment to update the second part of the aggregation sub-segment, which is different from the first part In the second part; re-determining the first part of the
  • the number of aggregated sub-segments to be split by the aggregated sub-segment is equal to the number of a plurality of execution workstations; respectively for one.
  • the number of multiple execution workstations is N, and they are sequentially numbered as 0,1,...i,...N-1;
  • the quantity is N, and is sequentially numbered as 0, 1, ... i, ...
  • the number of aggregated sub-segments of the aggregated sub-segment is equal to the number of multiple execution workstations multiplied by the number of parallel communication lines for each execution workstation; and the first part aggregates The number of sub-segments and second-part aggregated sub-segments is the number of parallel communication lines per execution workstation, respectively.
  • the distributed computing system includes a plurality of execution workstations and a management workstation. aggregate sub-segments; send the first part of aggregated sub-segments in multiple aggregated sub-segments to other execution workstations for reduction on other execution workstations; receive from other execution workstations and multiple aggregated sub-segments generated by other execution workstations The aggregate sub-fragment corresponding to the second part of the aggregated sub-fragment in and reduce it with the second part of the aggregated sub-fragment to update the second part of the aggregated sub-fragment, the first part is different from the second part; Re-determine the first part of the aggregated subsegment and the second part of the aggregated subsegment and perform the above sending, receiving, and reduction in the remaining part after removing the sent first part of the aggregated subsegment, until each of the plurality of aggregated subsegments is sent or complete all reductions; and if there is an aggregate
  • At least one embodiment of the present disclosure provides a computer-readable storage medium that stores computer-readable instructions in a non-transitory manner, and implements the data processing method provided by at least one embodiment of the present disclosure when the computer-readable instructions are executed by a processor.
  • the data processing method, execution workstation, distributed computing system, and computer-readable storage medium according to the embodiments of the present disclosure can split and reduce aggregates, improve the parallelism of data aggregation, and thereby improve processing performance.
  • Fig. 1 shows a schematic diagram of comparison between split reduction and non-split reduction provided by at least one embodiment of the present disclosure
  • Fig. 2A shows a flow chart of a data processing method for distributed computing executed by an execution workstation provided by at least one embodiment of the present disclosure
  • Fig. 2B shows a flowchart of a data processing method executed by a distributed computing system provided by at least one embodiment of the present disclosure
  • Fig. 3 shows a schematic topology diagram of a communication infrastructure of an execution workstation provided by at least one embodiment of the present disclosure
  • Fig. 4 shows a flow chart of sending, receiving and reducing processes performed by multiple execution workstations provided by at least one embodiment of the present disclosure
  • Fig. 5 shows the schematic diagram of the algorithm provided by Fig. 4 when the number of execution workstations is 4;
  • Fig. 6 shows a schematic block diagram of a distributed computing system provided by at least one embodiment of the present disclosure
  • Fig. 7 shows a schematic block diagram of an execution workstation for distributed computing provided by at least one embodiment of the present disclosure.
  • Fig. 8 shows a schematic diagram of a computer-readable storage medium provided by at least one embodiment of the present disclosure.
  • MLlib is poorly scalable and its bottleneck is the reduction step in the tree aggregation.
  • Scalable reduction algorithms e.g., Rabenseifner reduction algorithm
  • the non-splittable object interface in Spark does not support scalable reduction due to the lack of object splitting functionality algorithm. Therefore, the inventor has designed a new technical solution, which can split objects to obtain more parallelism, so that it can be applied to a scalable reduction algorithm.
  • At least one embodiment of the present disclosure provides a data processing method, an execution workstation, a distributed computing system, and a computer-readable storage medium.
  • the data processing method is used for distributed computing and executed by an execution workstation, which may include: splitting an aggregation sub-segment with a predetermined data format into a plurality of aggregation sub-segments; The segment is sent to other execution workstations for reduction on other execution workstations; the aggregation sub-segment corresponding to the second part of the aggregation sub-segments in the plurality of aggregation sub-segments generated by other execution workstations is received from other execution workstations and combined with The second part of the aggregated sub-segment is reduced to update the second part of the aggregated sub-segment, the first part being different from the second part; the first part is re-determined in the remaining part after removing the sent first part of the aggregated sub-segment from the plurality of aggregated sub-
  • the data processing method of this embodiment can split the object to be reduced into small pieces (fragments) and reduce them separately to obtain higher parallelism, thereby improving the processing performance, and therefore applicable to scalable reduction algorithms.
  • big data processing framework includes but not limited to Spark, and the data processing method provided by at least one embodiment of the present disclosure may also be applicable to other big data processing frameworks.
  • the distributed computing system may include a management workstation, an execution workstation, and an internal memory.
  • An execution workstation is a computing device for task processing in a distributed computing system, for example, it can run an executor in the Spark architecture;
  • a management workstation is a computing device responsible for managing each execution workstation in a distributed computing system, for example, its The driver in the Spark architecture can be run, and the management workstation can coordinate, schedule, and monitor the tasks of each execution workstation.
  • the task results of the execution workstations can finally be summarized in the management workstation.
  • the management workstation interacts with a plurality of executive workstations through the communication network, and the executive workstation interacts with other executive workstations through the communication network to receive or send messages.
  • the communication network is a medium used to provide communication links between the management workstation and multiple execution workstations and between multiple execution workstations.
  • the communication network may comprise various connection types, such as wired or wireless communication links, such as in particular WIFI, 3G, 4G, 5G, and fiber optic cables, among others.
  • reduction means combining two or more data, and the total data volume after merging is smaller than the total data volume before merging, for example, two or more data may be summed and other operations.
  • Aggregators represent data units with a predetermined data format, which can be used for aggregation (merging) or other calculations.
  • the predetermined data structure can be predetermined according to the specific application, for example, it can be an array, or it can also be other data structures defined by the user.
  • Fig. 1 shows a schematic diagram of comparison between split reduction and non-split reduction provided by at least one embodiment of the present disclosure.
  • the reduction processes shown on the left and right sides of the diagram all show the reduction of multiple aggregators into one aggregator.
  • a non-split reduction is shown on the left side of the diagram, while a split reduction is shown on the right side of the diagram.
  • the first fragment V 1,1 , V 2,1 , V 3,1 , V 4,1 of the polymeron V i is a group, similarly, the second fragment V 1,2 , V 2,2 , V 3,2 , V 4,2 are a group, and the third segment V 1,3 , V 2,3 , V 3,3 , V 4,3 is a group.
  • Each set of fragments can be reduced independently to form the results V *,1 , V *,2 , V *,3 respectively. Compared with non-split reduction, split reduction achieves 3 times parallelism, thus improving processing performance.
  • Fig. 2A shows a flowchart of a data processing method for distributed computing executed by an execution workstation provided by at least one embodiment of the present disclosure.
  • the data processing method includes steps S201 to S205.
  • Steps S201-S205 are executed by the execution workstation.
  • Step S201 Split the aggregation sub-segment with a predetermined data format into a plurality of aggregation sub-segments.
  • Step S202 Send the first part of the aggregated sub-segments among the multiple aggregated sub-segments to other execution workstations for reduction on other execution workstations.
  • Step S203 receiving aggregated sub-segments corresponding to the second part of the aggregated sub-segments generated by other executing workstations from other executing workstations and reducing them with the second part of the aggregated sub-segments to update the second part Aggregated subfragments, the first part differs from the second part.
  • Step S204 Re-determine the first part of the aggregated sub-segment and the second part of the aggregated sub-segment in the remainder of the multiple aggregated sub-segments after removing the sent first part of the aggregated sub-segment and perform the above-mentioned sending, receiving and reduction (i.e. steps S202 and S203), until each of the multiple aggregation sub-segments is sent or all reductions are completed.
  • Step S205 If there is an aggregation sub-segment that has completed all reductions, send the aggregation sub-segments that have completed all reductions to the management workstation, so that the management workstation can splice it with other aggregation sub-segments that have completed all reductions to form an aggregation sub-segment.
  • the aggregator is usually used as a data unit for data aggregation or reduction, and its data format can be predetermined according to specific applications, for example, it can be an array or other data structures defined by users.
  • the aggregation sub-segment is split into multiple aggregation sub-segments, and each aggregation sub-segment can be independently reduced, thereby improving parallelism.
  • the division of aggregated sub-segments can be determined according to the needs of the application.
  • the number of aggregated sub-segments can be determined according to the desired number of parallel processing, and then the aggregated sub-segments can be divided according to the number of aggregated sub-segments, such as evenly divided or unevenly divided.
  • the number of aggregated sub-segments may be equal to the number of execution workstations participating in the aggregation. In this case, each execution workstation may perform reduction of an aggregated sub-segment at the same time, so that the parallelism of multiple execution workstations may be fully utilized.
  • the first part of aggregated sub-segments refers to the aggregated sub-segments that are sent to other execution workstations and reduced on other execution workstations.
  • Other execution workstations refer to one or more execution workstations in the distributed computing system that are different from this execution workstation and participate in aggregation.
  • the first part of aggregated sub-segments may be one or more segments, and these segments are sent to other execution workstations for reduction with corresponding aggregated sub-segments on other workstations.
  • Corresponding aggregated sub-segment indicates that the position of the aggregated sub-segment in its aggregated sub-segment is the same as that of the sent aggregated sub-segment in its aggregated sub-segment.
  • two corresponding aggregated sub-fragments are combined to compute a single fragment.
  • the first part can be sent to the same other execution workstation, or to different other execution workstations respectively.
  • the second part of the aggregated sub-segment refers to the aggregated sub-segment that is reduced with the aggregated sub-segment received from other execution workstations, that is, the second part of the aggregated sub-segment is the aggregated sub-segment that is reduced on this execution workstation .
  • the other execution workstations in step S203 may be the same as or different from the other execution workstations in step S202.
  • the number of aggregated sub-segments to be split by the aggregated sub-segment is equal to the number of execution workstations participating in the aggregation
  • the number of the first part of the aggregated sub-segment and the number of the second part of the aggregated sub-segment can be one respectively, so that in the distributed system All execution stations of can reduce the aggregation sub-segment individually one at a time, in order to take full advantage of the parallelism of multiple execution stations.
  • step S204 re-determine the first part of the aggregated sub-segment and the second part of the aggregated sub-segment after removing the sent aggregated sub-segment, and iteratively execute the above steps S202 and S203 until each of the multiple aggregated sub-segments is sent or All reductions are done.
  • the "all reduction" mentioned here means that the aggregation sub-segment has already aggregated the corresponding segments of all aggregation sub-segments that need to be reduced in the distributed system.
  • the distributed system has M execution workstations in total, and each execution workstation has an aggregation sub-segment If reduction is required, if a certain aggregation sub-segment has already aggregated corresponding segments in M aggregation sub-segments of M execution workstations, it means that the aggregation sub-segment has completed all reductions.
  • the re-determining of the first part and the second part can be set according to the needs of the application, for example, the re-determined first part of the aggregated sub-segment can be It is the second part of aggregated sub-segments updated in the previous round of receiving and reducing, that is, the aggregated sub-segments that have been reduced are sent preferentially.
  • the execution workstation sends the reduced aggregation sub-segments in step S204 to the management workstation, so that the management workstation splices all the reduced aggregation sub-segments into a result aggregate.
  • Fig. 2B shows a flowchart of a data processing method executed by a distributed computing system provided by at least one embodiment of the present disclosure.
  • Distributed computing system includes a management workstation and multiple execution workstations
  • the data processing method includes steps S301 to S306.
  • Steps S301-S305 are executed by each execution workstation, and step S306 is executed by the management workstation.
  • Step S301 Split the aggregation sub-segment with a predetermined data format into a plurality of aggregation sub-segments.
  • Step S302 Send the first part of the aggregated sub-segments among the multiple aggregated sub-segments to other execution workstations for reduction on other execution workstations.
  • Step S303 receiving aggregated sub-segments corresponding to the second part of aggregated sub-segments generated by other executing workstations from other executing workstations and reducing them with the second part of aggregated sub-segments to update the second part Aggregated subfragments, the first part differs from the second part.
  • Step S304 Re-determine the first part of the aggregated sub-segment and the second part of the aggregated sub-segment in the remainder of the multiple aggregated sub-segments after removing the sent first part of the aggregated sub-segment, and perform the above sending, receiving and reduction until the multiple aggregated sub-segments Each of the aggregated sub-segments is sent or all reductions are done.
  • Step S305 If there is an aggregation sub-segment that has completed all the reductions, send the aggregation sub-segments that have completed all the reductions to the management workstation.
  • Step S306 The management workstation receives aggregated sub-segments sent by multiple execution workstations that have completed all the reductions, and splices the received aggregated sub-segments into an aggregated sub-segment.
  • Steps S301-S305 are the same as steps S201-S205 shown in FIG. 2A, which have been described in detail above, and will not be repeated here.
  • the management workstation receives all the reduced aggregation sub-segments, and then stitches these aggregation sub-segments into a result aggregation sub-segment.
  • Fig. 3 shows a schematic topology diagram of a communication infrastructure of an execution workstation provided by at least one embodiment of the present disclosure. It should be noted that the connection relationship between each execution workstation in Figure 3 only represents the logical communication relationship between them, and does not represent the actual network connection relationship. The actual network connection can be in any suitable way, as long as it can realize the The logical communication relationship shown in .
  • execution workstations there are four execution workstations in the distributed computing system, which are execution workstation 0 , execution workstation 1 , execution workstation 2 , and execution workstation 3 .
  • the four execution stations communicate in parallel directional rings.
  • Each execution workstation can send messages to its next execution workstation and receive messages from its previous workstation, for example, execution workstation 3 sends a message to execution workstation 0 and receives a message from execution workstation 2.
  • the arrows in Fig. 3 represent message channels, and there may be one or more parallel message channels between two adjacent execution workstations.
  • Fig. 3 shows 2 parallel message channels (parallel communication lines), which are marked as "0" respectively and 1".
  • the number of aggregation sub-segments to be split by the aggregation sub-segment may be equal to the number of execution workstations participating in the aggregation multiplied by the number of parallel communication lines of each execution workstation, and the first part of aggregation sub-segments and the second The number of partially aggregated sub-segments is the number of parallel communication lines per execution workstation respectively. In this case, communication resources can be fully utilized to improve parallelism.
  • the number of aggregation sub-segments to be split by the aggregation sub is equal to the number of execution workstations participating in the aggregation 4 multiplied by the number of parallel communication lines for each execution workstation 2, that is, it is split into 8 Aggregated sub-fragments.
  • the numbers of the first partial aggregation sub-segment and the second partial aggregation sub-segment are two, respectively. Therefore, during each round of processing, an execution workstation can simultaneously send 2 aggregation sub-segments and receive 2 aggregation sub-segments simultaneously through 2 communication lines.
  • Fig. 4 shows a flow chart of sending, receiving and reducing processes performed by multiple execution workstations provided by at least one embodiment of the present disclosure.
  • the number of multiple execution workstations is N, and they are sequentially numbered as 0,1,...i,...N-1; the number of multiple aggregation sub-segments is also N, and they are sequentially numbered as 0,1,...N-1 .i,...N-1.
  • the method may include step S401 to step S402.
  • step S402 is repeated.
  • N 4, that is, the number of execution workstations is 4, and the number of multiple aggregation sub-segments is 4.
  • execution workstation 0 sends its aggregation sub-segment 0 to execution workstation 1, and receives aggregation sub-segment 3 from execution workstation 3 and performs reduction.
  • execution workstation 0 sends its aggregation sub-segment 3 to execution workstation 1, and from The execution workstation 3 receives the aggregated sub-segment 2 and performs reduction.
  • execution workstation 1 sends its aggregation sub-segment 0 to execution workstation 2, and receives aggregation sub-segment 3 from execution workstation 0 and performs reduction.
  • execution workstation 2 sends its aggregation sub-segment 1 to execution workstation 3, and receives aggregation sub-segment 0 from execution workstation 1 and performs reduction.
  • execution workstation 1 For execution workstation 1, execution workstation 1 sends its aggregation sub-segment 3 to execution workstation 2, and receives aggregation sub-segment 2 from execution workstation 0 and performs reduction.
  • the execution workstation 0 sends the aggregation sub-segment 1 to the management workstation, the execution workstation 1 sends the aggregation sub-segment 2 to the management workstation, and the execution workstation 2 sends the aggregation sub-segment 3 to the management workstation.
  • FIG. 5 shows a schematic diagram of the algorithm provided in FIG. 4 when the number of execution workstations is four.
  • the four rounded rectangles represent four execution workstations, and the numbers 0, 1, 2, and 3 on the left side of the rounded rectangles represent the numbers of the execution workstations.
  • Each execution workstation divides the aggregate into four segments, using Fragment 0, Fragment 1, Fragment 2, and Fragment 3 represent.
  • a rectangle marked with a circle symbol represents the current value to be sent
  • a rectangle marked with a square symbol represents receiving and merging the next value of an aggregated subsegment from another execution station
  • a rectangle marked with a triangle symbol represents receiving and merging the next value from another execution station. Executes the final value of the workstation's aggregated subsegment.
  • execution station 0 sends its aggregated subsegment 0 to execution station 1, and receives aggregated subsegment 3 from execution station 3, which sends its aggregated subsegment 1 to execution station 2, and receiving aggregated subsegment 0 from execution station 0, execution station 2 sent its aggregated subsegment 2 to execution station 3, and receiving aggregated subsegment 1 from execution station 1, execution station 3 sent its aggregated subsegment 3 to Workstation 0 is executed, and aggregated subsegment 2 is received from execution workstation 2.
  • execution workstation 0 sends its aggregated subsegment 3 to execution workstation 1, and receives aggregated subsegment 2 from execution workstation 3, and execution workstation 1 sends its aggregated subsegment 0 to execution workstation 2, and receiving aggregated subsegment 3 from execution station 0, execution station 2 sent its aggregated subsegment 1 to execution station 3, and receiving aggregated subsegment 0 from execution station 1, execution station 3 sent its aggregated subsegment 2 to Workstation 0 is executed, and aggregated subsegment 1 is received from execution station 2.
  • execution station 0 sends its aggregated subsegment 2 to execution station 1 and merges it to form the final value, and receives aggregated subsegment 1 from execution station 3 and merges it with the current value to form the final value value
  • execution station 1 sends its aggregated subsegment 3 to execution station 2 and merges it to form the final value, and receives aggregated subsegment 2 from execution station 0 and merges it with the current value to form the final value
  • execution station 2 aggregates its subsegment 0 is sent to execution workstation 3 and merged to form the final value
  • aggregated sub-segment 3 is received from execution workstation 1 and merged with the current value to form the final value
  • execution workstation 3 sends its aggregated sub-segment 1 to execution workstation 0 and merged to form The final value, and aggregated subsegment 0 is received from execution station 2 and merged with the current value to form the final value.
  • Fig. 6 shows a schematic block diagram of a distributed computing system provided by at least one embodiment of the present disclosure.
  • the distributed computing system 600 includes a plurality of execution workstations 610 and a management workstation 620 .
  • Each of the plurality of execution workstations 610 is configured to: split the aggregation sub-segment having a predetermined data format into a plurality of aggregation sub-segments; send the first part of the aggregation sub-segments to other execution workstations for processing performing reduction on other execution workstations; receiving from other execution workstations the aggregation sub-segments corresponding to the second part of the aggregation sub-segments generated by other execution workstations and reducing them with the second part of the aggregation sub-segments to update the second part of the aggregation sub-segment, the first part being different from the second part; re-determining the first part of the aggregation sub-segment and the second part of the aggregation sub-segment in the remaining part after removing the transmitted first part of the aggregation sub-segment subfragments and perform the send, receive, and reduce described above until each of the multiple aggregate
  • the execution workstation 610 can, for example, execute steps S301-S305 described in FIG. 2B.
  • the management workstation 620 is configured to receive aggregated sub-segments that have completed all reductions sent by multiple execution workstations, and splice the received aggregated sub-segments into an aggregated sub-segment.
  • the management workstation 620 may, for example, execute step S306 described in FIG. 2B .
  • executive workstation 610 and management workstation 620 may be implemented as hardware, software, firmware, and any feasible combination thereof.
  • the execution workstation 610 and the management workstation 620 may be dedicated or general-purpose circuits, chips or devices, or a combination of processors and memories.
  • the embodiment of the present disclosure does not limit it.
  • each module used in the distributed computing system 600 corresponds to each step of the aforementioned data processing method, and for the specific functions used in the distributed computing system 600, please refer to About the data processing method The related descriptions will not be repeated here.
  • Components and structures of the distributed computing system 600 shown in FIG. 6 are exemplary rather than limiting, and the distributed computing system 600 may further include other components and structures as required.
  • At least one embodiment of the present disclosure also provides an execution workstation for distributed computing
  • the execution workstation for distributed computing includes a processor and a memory
  • the memory includes one or more computer program modules.
  • One or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the above data processing method.
  • the execution workstation for distributed computing can split the object to be reduced into small pieces to obtain higher parallelism, thereby improving processing performance, and thus can be applicable to a scalable reduction algorithm.
  • Fig. 7 shows a schematic block diagram of an execution workstation 700 for distributed computing provided by at least one embodiment of the present disclosure.
  • the execution workstation 700 for distributed computing includes a processor 710 and a memory 720 .
  • Memory 720 is used to store non-transitory computer readable instructions (eg, one or more computer program modules).
  • the processor 710 is configured to execute non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by the processor 710, one or more steps in the data processing method described above may be performed.
  • the memory 720 and the processor 710 may be interconnected by a bus system and/or other forms of connection mechanisms (not shown).
  • the processor 710 may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capabilities and/or program execution capabilities.
  • the central processing unit (CPU) may be of X86 or ARM architecture or the like.
  • the processor 710 may be a general-purpose processor or a special-purpose processor, and may control other components in the execution workstation 700 for distributed computing to perform desired functions.
  • memory 720 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may include random access memory (RAM) and/or cache memory (cache), etc., for example.
  • Non-volatile memory may include, for example, read only memory (ROM), hard disks, erasable programmable read only memory (EPROM), compact disc read only memory (CD-ROM), USB memory, flash memory, and the like.
  • One or more computer program modules can be stored on the computer-readable storage medium, and the processor 710 can run one or more computer program modules to realize various functions of the execution workstation 700 for distributed computing.
  • Various application programs, various data, and various data used and/or generated by the application programs can also be stored in the computer-readable storage medium.
  • At least one embodiment of the present disclosure also provides a computer-readable storage medium for storing non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a computer, the above-mentioned data processing method.
  • the object to be reduced can be split into small pieces to obtain more parallelism, thereby improving the scalability of the reduction.
  • Fig. 8 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure.
  • storage medium 800 is used to store non-transitory computer readable instructions 810 .
  • non-transitory computer readable instructions 810 are executed by a computer, one or more steps in the data processing method described above may be performed.
  • the storage medium 800 may be applied to the above-mentioned execution workstation 700 for distributed computing.
  • the storage medium 800 may be the memory 720 in the execution workstation 700 for distributed computing shown in FIG. 7 .
  • related descriptions about the storage medium 800 may refer to the corresponding description of the memory 720 in the execution workstation 700 for distributed computing shown in FIG. 7 , which will not be repeated here.
  • the drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to common designs. In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Multi Processors (AREA)

Abstract

一种数据处理方法、执行工作站、分布式计算系统和计算机可读存储介质。该数据处理方法包括：将具有预定数据格式的聚合子拆分为多个聚合子片段(S201)；将多个聚合子片段中的第一部分聚合子片段发送至其它执行工作站用于在其它执行工作站上进行归约(S202)；从其它执行工作站接收其它执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段(S203)；重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约(S204)；以及将完成所有归约的聚合子片段发送至管理工作站(S205)。

Description

数据处理方法、执行工作站、分布式计算系统和存储介质
本申请要求于2021年7月7日递交的中国专利申请第202110766780.0号的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分。
技术领域
本公开的实施例涉及一种数据处理方法、执行工作站、分布式计算系统和计算机可读存储介质。
背景技术
现如今是数据时代，数据信息化与我们的生活和工作息息相关，由于数据量的急剧增加，单机难以处理，因此设计出了分布式计算用于大数据处理框架。Spark是当今最流行的大数据处理框架之一。Spark经由其弹性分布式数据集编程模型连接各种数据分析域，Spark的机器学习框架MLlib被广泛地使用，这是由于MLlib可以被很容易地集成到Spark的数据分析工作流中。树聚合是MLlib的一个重要的研究热点，Spark中的树聚合可以被大致分为两个连续的步骤：计算与归约。在计算步骤中，当前分区的值在本地计算。在归约步骤中，计算出的值(称为聚合子)被进一步全局归约为单个聚合子。
发明内容
本公开至少一实施例提供一种由执行工作站执行的用于分布式计算的数据处理方法，包括：将具有预定数据格式的聚合子拆分为多个聚合子片段；将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约；从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分；在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约；以及如果存在完成所 有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站，用于管理工作站将其与其他完成所有归约的聚合子片段拼接为聚合子。
例如，在本公开至少一实施例提供的数据处理方法中，聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目；并且第一部分聚合子片段和第二部分聚合子片段的数目分别为一个。
例如，在本公开至少一实施例提供的数据处理方法中，聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目乘以每个执行工作站的并行通信线路数；并且第一部分聚合子片段和第二部分聚合子片段的数目分别为每个执行工作站的并行通信线路数。
例如，在本公开至少一实施例提供的数据处理方法中，在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定的第一部分聚合子片段是在前一轮的接收和归约中被更新的第二部分聚合子片段。
本公开至少一实施例提供一种用于分布式计算的执行工作站，包括：处理器和存储器，存储器存储一个或多个计算机程序指令，一个或多个计算机程序指令被存储在存储器中，并由处理器执行时实现本公开至少一实施例提供的数据处理方法的步骤。
本公开至少一实施例提供一种由分布式计算系统执行的数据处理方法，分布式计算系统包括多个执行工作站和管理工作站，数据处理方法包括：多个执行工作站中的每个执行以下步骤：将具有预定数据格式的聚合子拆分为多个聚合子片段；将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约；从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分；在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约；并且如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站；以及管理工作站接收多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
例如，在本公开至少一实施例提供的数据处理方法中，聚合子被拆分的聚合子片段的数目等于多个执行工作站的数目；并且第一部分聚合子片段和第二 部分聚合子片段的数目分别为一个。
例如，在本公开至少一实施例提供的数据处理方法中，多个执行工作站的数量为N，并被顺序编号为0,1,...i,…N-1；多个聚合子片段的数量为N，并被顺序编号为0,1,...i,…N-1；多个执行工作站如下执行上述发送、接收和归约的步骤：第一步：设j＝N-1，第二步：对于i＝0至N-1，第i个执行工作站将其第i+j+1个聚合子片段发送至第i+1个执行工作站，以及从第i-1个执行工作站接收第i+j个聚合子片段并进行归约，其中，若i-1<0，则用i+N-1代替i-1，若i+j>N-1，则用i+j-N代替i+j，若i+j+1>N-1，则用i+j+1-N代替i+j+1；若j>1，则将j值减1后重复第二步；第三步：对于i＝0至N-1，第i个执行工作站将第i+1个聚合子片段发送至管理工作站，并且若i+1＝N，则用0替代i+1。
例如，在本公开至少一实施例提供的数据处理方法中，聚合子被拆分的聚合子片段的数目等于多个执行工作站的数目乘以每个执行工作站的并行通信线路数；并且第一部分聚合子片段和第二部分聚合子片段的数目分别为每个执行工作站的并行通信线路数。
本公开至少一实施例提供一种分布式计算系统，分布式计算系统包括多个执行工作站和管理工作站，多个执行工作站中的每个用于：将具有预定数据格式的聚合子拆分为多个聚合子片段；将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约；从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分；在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约；并且如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站；以及管理工作站用于接收多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
本公开至少一实施例提供一种计算机可读存储介质，非暂时性存储有计算机可读指令，当计算机可读指令由处理器执行时实现本公开至少一实施例提供的数据处理方法。
根据本公开实施例的数据处理方法、执行工作站、分布式计算系统和计算机可读存储介质能够将聚合子进行拆分后归约，提高数据聚合的并行性，从而 提高处理性能。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例的附图作简单地介绍，显而易见地，下面描述中的附图仅仅涉及本公开的一些实施例，而非对本公开的限制。
图1示出了本公开至少一实施例提供的拆分归约和非拆分归约之间的对比示意图；
图2A示出了本公开至少一实施例提供的一种由执行工作站执行的用于分布式计算的数据处理方法的流程图；
图2B示出了本公开至少一实施例提供的由分布式计算系统执行的数据处理方法的流程图；
图3示出了本公开至少一实施例提供的执行工作站的通信基础结构的拓扑示意图；
图4示出了本公开至少一实施例提供的多个执行工作站执行发送、接收和归约过程的流程图；
图5示出了图4所提供的算法在执行工作站的数量为4的情况下的示意图；
图6示出了本公开至少一实施例提供的一种分布式计算系统的示意框图；
图7示出了本公开至少一实施例提供的一种用于分布式计算的执行工作站的示意框图；以及
图8示出了本公开至少一实施例提供的一种计算机可读存储介质的示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例的附图，对本公开实施例的技术方案进行清楚、完整地描述。显然，所描述的实施例是本公开的一部分实施例，而不是全部的实施例。基于所描述的本公开的实施例，本领域普通技术人员在无需创造性劳动的前提下所获得的所有其它实施例，都属于本公开保护的范围。
除非另外定义，本公开使用的技术术语或者科学术语应当为本公开所属领域内具有一般技能的人士所理解的通常意义。本公开中使用的“第一”、“第二” 以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的组成部分。同样，“一个”、“一”或者“该”等类似词语也不表示数量限制，而是表示存在至少一个。“包括”或者“包含”等类似的词语意指出现该词前面的元件或者物件涵盖出现在该词后面列举的元件或者物件及其等同，而不排除其它元件或者物件。“连接”或者“相连”等类似的词语并非限定于物理的或者机械的连接，而是可以包括电性的连接，不管是直接的还是间接的。“上”、“下”、“左”、“右”等仅用于表示相对位置关系，当被描述对象的绝对位置改变后，则该相对位置关系也可能相应地改变。
MLlib的可扩展性较差，其瓶颈在于树聚合中的归约步骤。可扩展的归约算法(例如，Rabenseifner归约算法)的性能比树聚合的性能好，然而，Spark中的不可拆分对象接口由于缺少对象拆分功能，从而使得MLlib不支持可扩展的归约算法。因此，发明人设计了一种新的技术方案，其能够将对象进行拆分以获得更多的并行性，从而能够适用于可扩展的归约算法。
本公开至少一个实施例提供一种数据处理方法、执行工作站、分布式计算系统和计算机可读存储介质。例如，该数据处理方法用于分布式计算并由执行工作站执行，其可以包括：将具有预定数据格式的聚合子拆分为多个聚合子片段；将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约；从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分；在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约；以及如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站，用于管理工作站将其与其他完成所有归约的聚合子片段拼接为聚合子。
该实施例的数据处理方法可以将要被归约的对象拆分成小块(片段)分别归约以获得更高的并行性，从而提高处理性能，并因此能够适用于可扩展的归约算法。
需要说明的是，大数据处理框架包括但不限于Spark，本公开至少一实施例提供的数据处理方法还可以适用于其它大数据处理框架。
本公开至少一实施例提供的分布式计算系统可以包括管理工作站和执行 工作站和内部存储器。执行工作站是分布式计算系统中进行任务处理的计算设备，例如，其可以运行Spark架构中的执行器(executor)；管理工作站是分布式计算系统中负责管理各个执行工作站的计算设备，例如，其可以运行Spark架构中的驱动器(driver)，管理工作站可以协调、调度、并监控每个执行工作站的任务。执行工作站的任务结果最终可以汇总到管理工作站中。管理工作站通过通信网络与多个执行工作站交互，并且执行工作站通过通信网络与其它执行工作站交互，以接收或发送消息。通信网络用以在管理工作站和多个执行工作站之间以及多个执行工作站之间提供通信链路的介质。通信网络可以包括各种连接类型，例如有线或无线通信链路，具体地例如WIFI、3G、4G、5G和光纤电缆等。
在本公开的实施例中，归约表示将两个或多个数据进行合并，合并后的总数据量小于合并前的总数据量，例如可以将两个或多个数据进行求和等运算。聚合子表示具有预定数据格式的数据单元，其可以用于聚合(合并)或者其它计算。预定数据结构可以根据具体的应用而预先确定，例如，可以是数组，也可以是用户定义的其它数据结构。
图1示出了本公开至少一实施例提供的拆分归约和非拆分归约之间的对比示意图。
如图1所示，在示意图左右两侧示出的归约过程都展示了将多个聚合子归约为一个聚合子。该示意图左侧示出了非拆分归约，而示意图右侧示出了拆分归约。
图中四个聚合子V 1、V 2、V 3、V 4被归约为一个聚合子。非拆分归约将聚合子视为不可拆分的对象，直接将V 1、V 2、V 3、V 4归约为V *。而对于拆分归约，每个聚合子V i(i＝1、2、3、4)都是可拆分的并且每个聚合子V i(i＝1、2、3、4)可以被拆分为例如三个片段V i,1、V i,2、V i,3。聚合子V i(i＝1、2、3、4)的第一个片段V 1,1、V 2,1、V 3,1、V 4,1为一组，类似地，第二个片段V 1,2、V 2,2、V 3,2、V 4,2为一组，第三个片段V 1,3、V 2,3、V 3,3、V 4,3为一组。每组片段可以被独立地归约以分别形成结果V *,1、V *,2、V *,3。与非拆分归约相比，拆分归约获得了3倍的并行性，因此提高了处理性能。
图2A示出了本公开至少一实施例提供的一种由执行工作站执行的用于分布式计算的数据处理方法的流程图。
如图2A所示，该数据处理方法包括步骤S201～步骤S205。
步骤S201-S205由执行工作站执行。
步骤S201：将具有预定数据格式的聚合子拆分为多个聚合子片段。
步骤S202：将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约。
步骤S203：从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分。
步骤S204：在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约(即步骤S202和S203)，直至多个聚合子片段的每一个都被发送或完成所有归约。
步骤S205：如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站，用于管理工作站将其与其他完成所有归约的聚合子片段拼接为聚合子。
对于步骤S201，聚合子通常是作为数据聚合或归约的数据单位，其数据格式可以根据具体的应用而预先确定，例如，可以是数组，也可以是用户定义的其它数据结构。在本公开的实施例中，聚合子被拆分为多个聚合子片段，每个聚合子片段可以独立地进行归约，从而提高了并行性。聚合子片段的划分可以根据应用的需要而确定，例如，可以根据希望并行处理的数量确定聚合子片段的数目，再根据聚合子片段的数目对聚合子进行划分，例如均匀划分或不均匀划分。例如，聚合子片段的数目可以等于参与聚合的执行工作站的数目，在此情况下，每个执行工作站可以同时各自进行一个聚合子片段的归约，从而可以充分利用多个执行工作站的并行性。
对于步骤S202，第一部分聚合子片段是指发送至其它执行工作站并在其它执行工作站上进行归约的聚合子片段。其它执行工作站是指分布式计算系统中不同于本执行工作站且参与聚合的一个或多个执行工作站。第一部分聚合子片段可以是一个或多个片段，这些片段被发送至其它执行工作站上与其它工作站上的对应聚合子片段进行归约。“对应聚合子片段”表示该聚合子片段在其聚合子中的位置与被发送的聚合子片段在其聚合子中的位置相同。在归约时，两个对应的聚合子片段被合并计算为一个片段。在第一部分具有多个片段时，该第一部分可以发送给同一个其它执行工作站，也可以分别发送给不同的其它执 行工作站。
对于步骤S203，第二部分聚合子片段是指与接收自其它执行工作站的聚合子片段进行归约的聚合子片段，即，第二部分聚合子片段是在本执行工作站进行归约的聚合子片段。需要说明的是，步骤S203中的其它执行工作站与步骤202中的其它执行工作站可以相同也可以不同。此外，在聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目的情况下，第一部分聚合子片段和第二部分聚合子片段的数目可以分别为一个，从而分布式系统中的所有执行工作站可以每次分别归约一个聚合子片段，以便充分利用多个执行工作站的并行性。
对于步骤S204，在去除已经发送的聚合子片段后重新确定第一部分聚合子片段和第二部分聚合子片段，并迭代执行上述步骤S202和S203，直至多个聚合子片段的每一个都被发送或完成所有归约。这里所述的“所有归约”表示该聚合子片段已经聚合了分布式系统需要归约的所有聚合子的对应片段，例如，分布式系统共有M个执行工作站，每个执行工作站有一个聚合子需要归约，则如果某个聚合子片段已经聚合了M个执行工作站的M个聚合子中的对应片段，则表示该聚合子片段已经完成了所有归约。对于第一部分和第二部分的重新确定可以根据应用的需要设定，例如，在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定的第一部分聚合子片段可以是在前一轮的接收和归约中被更新的第二部分聚合子片段，即优先将已经归约过的聚合子片段进行发送。对于步骤S205，执行工作站将步骤S204中完成归约的聚合子片段发送给管理工作站，以便管理工作站将完成所有归约的聚合子片段拼接为结果聚合子。
图2B示出了本公开至少一实施例提供的由分布式计算系统执行的数据处理方法的流程图。分布式计算系统包括管理工作站和多个执行工作站
如图2B所示，该数据处理方法包括步骤S301～步骤S306。
步骤S301-S305由每个执行工作站执行，步骤S306由管理工作站执行。
步骤S301：将具有预定数据格式的聚合子拆分为多个聚合子片段。
步骤S302：将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约。
步骤S303：从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段 进行归约以更新第二部分聚合子片段，第一部分不同于第二部分。
步骤S304：在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约。
步骤S305：如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站。
步骤S306：管理工作站接收多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
步骤S301-S305和图2A所示的步骤S201-S205相同，已经在上面进行了详细说明，在此不再赘述。
对于步骤S306，管理工作站接收完成所有归约的聚合子片段，然后将这些聚合子片段拼接成一个结果聚合子。
图3示出了本公开至少一实施例提供的执行工作站的通信基础结构的拓扑示意图。需要说明的是，图3中每个执行工作站之间的连接关系仅表示它们之间的逻辑通信关系，并不表示实际网络连接关系，实际网络连接可以采用任意合适的方式，只要能够实现图3中所示的逻辑通信关系即可。
如图3所示，例如，分布式计算系统存在四个执行工作站，分别为执行工作站0、执行工作站1、执行工作站2、执行工作站3。四个执行工作站按并行定向环的形式通信。每个执行工作站可以向其下一个执行工作站发送消息，并且从其前一个工作站接受消息，例如，执行工作站3向执行工作站0发送消息并从执行工作站2接受消息。图3中的箭头表示消息信道，相邻两个执行工作站之间可以有一个或多个并行消息信道，图3中示出了2个并行消息信道(并行通信线路)，分别标示为“0”和“1”。
在本公开的一些实施例中，聚合子被拆分的聚合子片段的数目可以等于参与聚合的执行工作站的数目乘以每个执行工作站的并行通信线路数，并且第一部分聚合子片段和第二部分聚合子片段的数目分别为每个执行工作站的并行通信线路数。在此情况下，可以充分利用通信资源提高并行性。
例如，在图3所示的情况下，聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目4乘以每个执行工作站的并行通信线路数2，即被拆分成8个聚合子片段。第一部分聚合子片段和第二部分聚合子片段的数目分别为2。从而，在每一轮处理时，一个执行工作站可以通过2个通信线路同时发 送2个聚合子片段并同时接收2个聚合子片段。
图4示出了本公开至少一个实施例提供的多个执行工作站执行发送、接收和归约过程的流程图。多个执行工作站的数量为N，并被顺序编号为0,1,...i,…N-1；多个聚合子片段的数量也为N，并被顺序编号为0,1,...i,…N-1。
如图4所示，该方法可以包括步骤S401～步骤S402。
步骤S401：设j＝N-1。
步骤S402：对于i＝0至N-1，执行工作站i将其聚合子片段i+j+1发送至执行工作站i+1，以及从执行工作站i-1接收聚合子片段i+j并进行归约，其中，若i-1<0，则用i+N-1代替i-1，若i+j>N-1，则用i+j-N代替i+j，若i+j+1>N-1，则用i+j+1-N代替i+j+1；
若j>1，则将j值减1后重复步骤S402。
步骤S403：对于i＝0至N-1，执行工作站i将聚合子片段i+1发送至管理工作站，并且若i+1＝N，则用0替代i+1。
例如，N＝4，即执行工作站的数量为4，多个聚合子片段的数量为4。设j＝N-1＝3。
在第一轮迭代中，对于执行工作站0，由于i-1<0，则设i-1＝i+N-1＝3，并且由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝0，执行工作站0将其聚合子片段0发送至执行工作站1，以及从执行工作站3接收聚合子片段3并进行归约。对于执行工作站1，由于i+j＝1+3>3，则设i+j＝i+j-N＝0，并且由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝1，执行工作站1将其聚合子片段1发送至执行工作站2，以及从执行工作站0接收聚合子片段0并进行归约。对于执行工作站2，由于i+j＝2+3>3，则设i+j＝i+j-N＝1，并且由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝2，执行工作站2将其聚合子片段2发送给执行工作站3，以及从执行工作站1接受聚合子片段1并进行归约。对于执行工作站3，由于i+j＝3+3>3，则设i+j＝i+j-N＝2，并且由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝3，执行工作站3将其聚合子片段3发送至执行工作站0，以及从执行工作站2接收聚合子片段2并进行归约。由于j＝3>1，则将j值减1，此时j＝2，并进行第二轮迭代。
在第二轮迭代中，对于执行工作站0，由于i-1<0，则设i-1＝i+N-1＝3，执行工作站0将其聚合子片段3发送给执行工作站1，以及从执行工作站3接收聚合子片段2并进行归约。对于执行工作站1，执行工作站1将其聚合子片段0发送给执行工作站2，以及从执行工作站0接收聚合子片段3并进行归约。 对于执行工作站2，执行工作站2将其聚合子片段1发送给执行工作站3，以及从执行工作站1接收聚合子片段0并进行归约。对于执行工作站3，执行工作站3将其聚合子片段2发送给执行工作站0，以及从执行工作站2接收聚合子片段1并进行归约。由于j＝2>1，则将j值减1，此时j＝1，并进行第三轮迭代。
在第三轮迭代中，对于执行工作站0，由于i-1<0，则设i-1＝i+N-1＝3，执行工作站0将其聚合子片段2发送给执行工作站1，以及从执行工作站3接收聚合子片段1并进行归约。对于执行工作站1，执行工作站1将其聚合子片段3发送给执行工作站2，以及从执行工作站0接收聚合子片段2并进行归约。对于执行工作站2，由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝0，执行工作站2将其聚合子片段0发送给执行工作站3，以及从执行工作站1接收聚合子片段3并进行归约。对于执行工作站3，由于i+j＝3+1>3，则设i+j＝i+j-N＝0，并且由于i+j+1>N-1，则设i+j+1＝i+j+1-N＝1，执行工作站3将其聚合子片段1发送给执行工作站0，以及从执行工作站2接收聚合子片段0并进行归约。由于j＝1，循环结束，并进入到步骤S403。
对于步骤S403，执行工作站0将聚合子片段1发送至管理工作站，执行工作站1将聚合子片段2发送至管理工作站，执行工作站2将聚合子片段3发送至管理工作站。对于执行工作站3，由于i+1＝N，则i+1＝0，执行工作站3将聚合子片段0发送至管理工作站。
图5示出了图4所提供的算法在执行工作站的数量为4的情况下的示意图。
如图5所示，4个圆角矩形表示4个执行工作站，圆角矩形左侧的数字0、1、2、3表示执行工作站的编号，每个执行工作站将聚合子分成4个片段，用片段0、片段1、片段2、片段3表示。标有圆形符号的矩形表示要被发送的当前值，标有正方形符号的矩形表示接收并合并来自其它执行工作站的聚合子片段的下一个值，标有三角形符号的矩形表示接收并合并来自其它执行工作站的聚合子片段的最终值。在第一轮迭代(迭代0)中，执行工作站0将其聚合子片段0发送给执行工作站1，以及从执行工作站3接收聚合子片段3，执行工作站1将其聚合子片段1发送给执行工作站2，以及从执行工作站0接收聚合子片段0，执行工作站2将其聚合子片段2发送给执行工作站3，以及从执行工作站1接收聚合子片段1，执行工作站3将其聚合子片段3发送给执行工作站0，以及从执行工作站2接收聚合子片段2。在第二轮迭代(迭代1)中，执 行工作站0将其聚合子片段3发送给执行工作站1，以及从执行工作站3接收聚合子片段2，执行工作站1将其聚合子片段0发送给执行工作站2，以及从执行工作站0接收聚合子片段3，执行工作站2将其聚合子片段1发送给执行工作站3，以及从执行工作站1接收聚合子片段0，执行工作站3将其聚合子片段2发送给执行工作站0，以及从执行工作站2接收聚合子片段1。在第三轮迭代(迭代2)中，执行工作站0将其聚合子片段2发送给执行工作站1并进行合并形成最终值，以及从执行工作站3接收聚合子片段1并与当前值进行合并形成最终值，执行工作站1将其聚合子片段3发送给执行工作站2并进行合并形成最终值，以及从执行工作站0接收聚合子片段2并与当前值合并形成最终值，执行工作站2将其聚合子片段0发送给执行工作站3并进行合并形成最终值，以及从执行工作站1接收聚合子片段3并与当前值合并形成最终值，执行工作站3将其聚合子片段1发送给执行工作站0并进行合并形成最终值，以及从执行工作站2接收聚合子片段0并与当前值合并形成最终值。
图6示出了本公开至少一实施例提供的一种分布式计算系统的示意框图。
该分布式计算系统600包括多个执行工作站610和管理工作站620。
多个执行工作站610中的每个配置为：将具有预定数据格式的聚合子拆分为多个聚合子片段；将多个聚合子片段中的第一部分聚合子片段发送至其他执行工作站用于在其他执行工作站上进行归约；从其他执行工作站接收其他执行工作站产生的与多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与第二部分聚合子片段进行归约以更新第二部分聚合子片段，第一部分不同于第二部分；在从多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至多个聚合子片段的每一个都被发送或完成所有归约；并且如果存在完成所有归约的聚合子片段，则将完成所有归约的聚合子片段发送至管理工作站。
执行工作站610例如可以执行图2B描述的步骤S301-S305。
管理工作站620配置为接收多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
管理工作站620例如可以执行图2B描述的步骤S306。
例如，执行工作站610和管理工作站620可以实现为硬件、软件、固件以及它们的任意可行的组合。例如，执行工作站610和管理工作站620可以为专 用或通用的电路、芯片或装置等，也可以为处理器和存储器的结合。关于上述各个模块的具体实现形式，本公开的实施例对此不作限制。
需要说明的是，本公开的实施例中，用于分布式计算系统600的各个模块与前述的数据处理方法的各个步骤对应，关于用于分布式计算系统600的具体功能可以参考关于数据处理方法的相关描述，此处不再赘述。图6所示的用于分布式计算系统600的组件和结构只是示例性的，而非限制性的，根据需要，该用于分布式计算系统600还可以包括其它组件和结构。
本公开的至少一个实施例还提供了一种用于分布式计算的执行工作站，该用于分布式计算的执行工作站包括处理器和存储器，存储器包括一个或多个计算机程序模块。一个或多个计算机程序模块被存储在存储器中并被配置为由处理器执行，一个或多个计算机程序模块包括用于实现上述的数据处理方法的指令。该用于分布式计算的执行工作站可以将要被归约的对象拆分成小块以获得更高的并行性，从而提高处理性能，并因此能够适用于可扩展的归约算法。
图7示出了本公开至少一实施例提供的一种用于分布式计算的执行工作站700的示意框图。如图7所示，该用于分布式计算的执行工作站700包括处理器710和存储器720。存储器720用于存储非暂时性计算机可读指令(例如一个或多个计算机程序模块)。处理器710用于运行非暂时性计算机可读指令，非暂时性计算机可读指令被处理器710运行时可以执行上文所述的数据处理方法中的一个或多个步骤。存储器720和处理器710可以通过总线系统和/或其它形式的连接机构(未示出)互连。
例如，处理器710可以是中央处理单元(CPU)、图形处理单元(GPU)或者具有数据处理能力和/或程序执行能力的其它形式的处理单元。例如，中央处理单元(CPU)可以为X86或ARM架构等。处理器710可以为通用处理器或专用处理器，可以控制用于分布式计算的执行工作站700中的其它组件以执行期望的功能。
例如，存储器720可以包括一个或多个计算机程序产品的任意组合，计算机程序产品可以包括各种形式的计算机可读存储介质，例如易失性存储器和/或非易失性存储器。易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。非易失性存储器例如可以包括只读存储器(ROM)、硬盘、可擦除可编程只读存储器(EPROM)、便携式紧致盘只读存储器(CD-ROM)、USB存储器、闪存等。在计算机可读存储介质上可以存 储一个或多个计算机程序模块，处理器710可以运行一个或多个计算机程序模块，以实现用于分布式计算的执行工作站700的各种功能。在计算机可读存储介质中还可以存储各种应用程序和各种数据以及应用程序使用和/或产生的各种数据等。
需要说明的是，本公开的实施例中，用于分布式计算的执行工作站700的具体功能和技术效果可以参考上文中关于数据处理方法的描述，此处不再赘述。
本公开的至少一个实施例还提供了一种计算机可读存储介质，该计算机可读存储介质用于存储非暂时性计算机可读指令，当非暂时性计算机可读指令由计算机执行时可以实现上述的数据处理方法。利用该计算机可读存储介质，可以将要被归约的对象拆分成小块以获得更多的并行性，从而提高了归约的可扩展性。
图8为本公开一些实施例提供的一种存储介质的示意图。如图8所示，存储介质800用于存储非暂时性计算机可读指令810。例如，当非暂时性计算机可读指令810由计算机执行时可以执行根据上文所述的数据处理方法中的一个或多个步骤。
例如，该存储介质800可以应用于上述用于分布式计算的执行工作站700中。例如，存储介质800可以为图7所示的用于分布式计算的执行工作站700中的存储器720。例如，关于存储介质800的相关说明可以参考图7所示的用于分布式计算的执行工作站700中的存储器720的相应描述，此处不再赘述。需要说明的是，本公开实施例附图只涉及到本公开实施例涉及到的结构，其它结构可参考通常设计。在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合以得到新的实施例。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，本公开的保护范围应以所述权利要求的保护范围为准。

Claims (11)

  1. 一种由执行工作站执行的用于分布式计算的数据处理方法，包括：
    将具有预定数据格式的聚合子拆分为多个聚合子片段；
    将所述多个聚合子片段中的第一部分聚合子片段发送至其它执行工作站用于在其它执行工作站上进行归约；
    从其它执行工作站接收其它执行工作站产生的与所述多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与所述第二部分聚合子片段进行归约以更新所述第二部分聚合子片段，所述第一部分不同于所述第二部分；
    在从所述多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至所述多个聚合子片段的每一个都被发送或完成所有归约；以及
    如果存在完成所有归约的聚合子片段，则将所述完成所有归约的聚合子片段发送至管理工作站，用于所述管理工作站将其与其它完成所有归约的聚合子片段拼接为聚合子。
  2. 根据权利要求1所述的数据处理方法，其中
    所述聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目；并且
    所述第一部分聚合子片段和所述第二部分聚合子片段的数目分别为一个。
  3. 根据权利要求1所述的数据处理方法，其中
    所述聚合子被拆分的聚合子片段的数目等于参与聚合的执行工作站的数目乘以每个执行工作站的并行通信线路数；并且
    所述第一部分聚合子片段和所述第二部分聚合子片段的数目分别为每个执行工作站的并行通信线路数。
  4. 根据权利要求1-3中的任一项所述的数据处理方法，其中
    在从所述多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定的第一部分聚合子片段是在前一轮的接收和归约中被更新的第二部分聚合子片段。
  5. 一种用于分布式计算的执行工作站，包括处理器和存储器，所述存储器存储一个或多个计算机程序指令，其中，所述一个或多个计算机程序指令被 存储在所述存储器中，并由所述处理器执行时实现权利要求1-4任一项所述的数据处理方法的步骤。
  6. 一种由分布式计算系统执行的数据处理方法，所述分布式计算系统包括多个执行工作站和管理工作站，其中所述数据处理方法包括：
    所述多个执行工作站中的每个执行以下步骤：
    将具有预定数据格式的聚合子拆分为多个聚合子片段；
    将所述多个聚合子片段中的第一部分聚合子片段发送至其它执行工作站用于在其它执行工作站上进行归约；
    从其它执行工作站接收其它执行工作站产生的与所述多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与所述第二部分聚合子片段进行归约以更新所述第二部分聚合子片段，所述第一部分不同于所述第二部分；
    在从所述多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至所述多个聚合子片段的每一个都被发送或完成所有归约；并且
    如果存在完成所有归约的聚合子片段，则将所述完成所有归约的聚合子片段发送至管理工作站；以及
    所述管理工作站接收所述多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
  7. 根据权利要求6所述的数据处理方法，其中
    所述聚合子被拆分的聚合子片段的数目等于所述多个执行工作站的数目；并且
    所述第一部分聚合子片段和所述第二部分聚合子片段的数目分别为一个。
  8. 根据权利要求6或7所述的数据处理方法，其中
    所述多个执行工作站的数量为N，并被顺序编号为0,1,...i,…N-1；
    所述多个聚合子片段的数量为N，并被顺序编号为0,1,...i,…N-1；
    所述多个执行工作站如下执行上述发送、接收和归约的步骤：
    第一步：设j＝N-1，
    第二步：对于i＝0至N-1，第i个执行工作站将其第i+j+1个聚合子片段发送至第i+1个执行工作站，以及从第i-1个执行工作站接收第i+j个聚 合子片段并进行归约，其中，若i-1<0，则用i+N-1代替i-1，若i+j>N-1，则用i+j-N代替i+j，若i+j+1>N-1，则用i+j+1-N代替i+j+1；
    若j>1，则将j值减1后重复第二步；
    第三步：对于i＝0至N-1，第i个执行工作站将第i+1个聚合子片段发送至所述管理工作站，其中若i+1＝N，则用0替代i+1。
  9. 根据权利要求6-8中的任一项所述的数据处理方法，其中
    所述聚合子被拆分的聚合子片段的数目等于所述多个执行工作站的数目乘以每个执行工作站的并行通信线路数；并且
    所述第一部分聚合子片段和所述第二部分聚合子片段的数目分别为每个执行工作站的并行通信线路数。
  10. 一种分布式计算系统，所述分布式计算系统包括多个执行工作站和管理工作站，其中
    所述多个执行工作站中的每个用于：
    将具有预定数据格式的聚合子拆分为多个聚合子片段；
    将所述多个聚合子片段中的第一部分聚合子片段发送至其它执行工作站用于在其它执行工作站上进行归约；
    从其它执行工作站接收其它执行工作站产生的与所述多个聚合子片段中的第二部分聚合子片段对应的聚合子片段并将其与所述第二部分聚合子片段进行归约以更新所述第二部分聚合子片段，所述第一部分不同于所述第二部分；
    在从所述多个聚合子片段中去除已发送的第一部分聚合子片段的剩余部分中重新确定第一部分聚合子片段和第二部分聚合子片段并执行上述发送、接收和归约，直至所述多个聚合子片段的每一个都被发送或完成所有归约；并且
    如果存在完成所有归约的聚合子片段，则将所述完成所有归约的聚合子片段发送至管理工作站；以及
    所述管理工作站用于接收所述多个执行工作站发送的完成所有归约的聚合子片段，并将所接收的聚合子片段拼接为聚合子。
  11. 一种计算机可读存储介质，非暂时性存储有计算机可读指令，其中，当所述计算机可读指令由处理器执行时实现权利要求1-4任一项所述的数据处理方法。
PCT/CN2022/104127 2021-07-07 2022-07-06 数据处理方法、执行工作站、分布式计算系统和存储介质 Ceased WO2023280207A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/576,856 US12477032B2 (en) 2021-07-07 2022-07-06 Data processing method, execution workstation, distributed computing system, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110766780.0A CN115599956B (zh) 2021-07-07 2021-07-07 数据处理方法、执行工作站、分布式计算系统和存储介质
CN202110766780.0 2021-07-07

Publications (1)

Publication Number Publication Date
WO2023280207A1 true WO2023280207A1 (zh) 2023-01-12

Family

ID=84801289

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/104127 Ceased WO2023280207A1 (zh) 2021-07-07 2022-07-06 数据处理方法、执行工作站、分布式计算系统和存储介质

Country Status (3)

Country Link
US (1) US12477032B2 (zh)
CN (1) CN115599956B (zh)
WO (1) WO2023280207A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170262517A1 (en) * 2012-07-26 2017-09-14 Eliot Horowitz Aggregation framework system architecture and method
CN111694876A (zh) * 2019-03-15 2020-09-22 北京奇虎科技有限公司 一种基于Spark框架实现ID mapping的方法和装置
CN111949681A (zh) * 2020-06-22 2020-11-17 中科驭数（北京）科技有限公司 数据的聚合处理装置、方法和存储介质
CN112506950A (zh) * 2020-12-10 2021-03-16 深圳前海微众银行股份有限公司 数据聚合处理方法、计算节点、计算集群及存储介质

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8065503B2 (en) * 2006-12-15 2011-11-22 International Business Machines Corporation Iteratively processing data segments by concurrently transmitting to, processing by, and receiving from partnered process
US10231077B2 (en) * 2007-07-03 2019-03-12 Eingot Llc Records access and management
EP2765510A4 (en) * 2011-10-06 2016-07-06 Fujitsu Ltd DATA PROCESSING, DISTRIBUTED PROCESSING SYSTEM AND PROGRAM
US8977723B2 (en) * 2012-10-02 2015-03-10 Nextbit Systems Inc. Cloud based application fragmentation
US9753807B1 (en) * 2014-06-17 2017-09-05 Amazon Technologies, Inc. Generation and verification of erasure encoded fragments
US10735823B2 (en) * 2015-03-13 2020-08-04 Telefonaktiebolaget Lm Ericsson (Publ) System and method for optimized delivery of live ABR media
EP3376372A4 (en) * 2015-12-31 2018-12-26 Huawei Technologies Co., Ltd. Data processing method, device, and system
US10356158B2 (en) * 2016-05-16 2019-07-16 Carbonite, Inc. Systems and methods for aggregation of cloud storage
CN107885595B (zh) * 2016-09-30 2021-12-14 华为技术有限公司 一种资源分配方法、相关设备及系统
US11270201B2 (en) * 2017-12-29 2022-03-08 Intel Corporation Communication optimizations for distributed machine learning
US10936239B2 (en) * 2019-01-29 2021-03-02 EMC IP Holding Company LLC Cluster contraction of a mapped redundant array of independent nodes
CN111309702A (zh) * 2020-02-28 2020-06-19 苏州浪潮智能科技有限公司 一种文件内、文件间聚合方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170262517A1 (en) * 2012-07-26 2017-09-14 Eliot Horowitz Aggregation framework system architecture and method
CN111694876A (zh) * 2019-03-15 2020-09-22 北京奇虎科技有限公司 一种基于Spark框架实现ID mapping的方法和装置
CN111949681A (zh) * 2020-06-22 2020-11-17 中科驭数（北京）科技有限公司 数据的聚合处理装置、方法和存储介质
CN112506950A (zh) * 2020-12-10 2021-03-16 深圳前海微众银行股份有限公司 数据聚合处理方法、计算节点、计算集群及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YU BOWEN, CAO HUANQI, SHAN TIANYI, WANG HAOJIE, TANG XIONGCHAO, CHEN WENGUANG: "Sparker: Efficient Reduction for More Scalable Machine Learning with Spark", ACM TURING AWARD CELEBRATION CONFERENCE - CHINA ( ACM TURC 2021), ACMPUB27, NEW YORK, NY, USA, 9 August 2021 (2021-08-09) - 25 June 2021 (2021-06-25), New York, NY, USA, pages 1 - 11, XP058657978, ISBN: 978-1-4503-8988-4, DOI: 10.1145/3472456.3472499 *

Also Published As

Publication number Publication date
US12477032B2 (en) 2025-11-18
CN115599956A (zh) 2023-01-13
US20240291886A1 (en) 2024-08-29
CN115599956B (zh) 2024-09-24

Similar Documents

Publication Publication Date Title
CN110262901B (zh) 一种数据处理方法及数据处理系统
US11144323B2 (en) Independent mapping of threads
US8949577B2 (en) Performing a deterministic reduction operation in a parallel computer
US9612856B2 (en) Administering virtual machines in a distributed computing environment
US9223505B2 (en) Administering inter-core communication via shared memory
US9250948B2 (en) Establishing a group of endpoints in a parallel computer
US20110296139A1 (en) Performing A Deterministic Reduction Operation In A Parallel Computer
US20230038051A1 (en) Data transmission method and apparatus
Nukada et al. Scalable multi-gpu 3-d fft for tsubame 2.0 supercomputer
US20070180161A1 (en) DMA transfer apparatus
US9503515B2 (en) Administering virtual machines in a distributed computing environment
CN111078394B (zh) 一种gpu线程负载均衡方法与装置
US9612857B2 (en) Administering virtual machines in a distributed computing environment
WO2017113277A1 (zh) 数据处理方法、装置和系统
US9507637B1 (en) Computer platform where tasks can optionally share per task resources
Zeng et al. Enabling efficient spatio-temporal GPU sharing for network function virtualization
US9342564B2 (en) Distributed processing apparatus and method for processing large data through hardware acceleration
CN110399394A (zh) 计算节点的数据遍历方法、装置及计算节点
CN115599956A (zh) 数据处理方法、执行工作站、分布式计算系统和存储介质
CN119003208A (zh) 数据处理方法、装置、电子设备及存储介质
WO2023280208A1 (zh) 数据处理方法、执行工作站、电子设备和存储介质
WO2023045478A1 (zh) 图任务调度方法、执行端设备、存储介质及程序产品
WO2022141344A1 (zh) 执行程序的编译方法、芯片、电子设备及计算机可读存储介质
US20260017226A1 (en) Hybrid network-on-chip (noc) for thread synchronization in many-core neural network accelerators
US12625725B2 (en) Dynamic provisioning of portions of a data processing array for spatial and temporal sharing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22836953

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18576856

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22836953

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 18576856

Country of ref document: US