WO2025256136A1 - 数据洗牌shuffle方法、装置、设备及集群 - Google Patents

数据洗牌shuffle方法、装置、设备及集群

Info

Publication number
WO2025256136A1
WO2025256136A1 PCT/CN2025/072572 CN2025072572W WO2025256136A1 WO 2025256136 A1 WO2025256136 A1 WO 2025256136A1 CN 2025072572 W CN2025072572 W CN 2025072572W WO 2025256136 A1 WO2025256136 A1 WO 2025256136A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
computing
node
computing node
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/072572
Other languages
English (en)
French (fr)
Inventor
王世杰
贾佳峰
周坤
熊钦
王工艺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2025256136A1 publication Critical patent/WO2025256136A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers

Definitions

  • This application relates to the field of data analysis technology, and in particular to a data shuffle method, apparatus, device and cluster.
  • Data shuffling is a fundamental core operation in data analysis. Data shuffling can repartition data across different computing nodes in a data analysis system, allowing data with the same key on different computing nodes to be assigned to the same data partition, thereby integrating and classifying the data across different computing nodes.
  • This application provides a data shuffle method, apparatus, device, and cluster that can effectively improve data shuffle efficiency.
  • a data shuffle method is provided, applied to a computing cluster, which includes multiple computing nodes and a management node.
  • the method includes: the management node acquiring the data volume of each computing node among the multiple computing nodes, and partitioning multiple data points of the multiple computing nodes according to a shuffle strategy and the data volume of each computing node, obtaining partitioned data for each computing node; the management node sending partition information to a first computing node, the partition information indicating the partitioned data of the first computing node; the first computing node being any one of the multiple computing nodes; the first computing node determining, within its first memory space, a first group of partitioned data belonging to the first computing node and a second group of partitioned data not belonging to the first computing node; the first computing node writing the first group of partitioned data from its first memory space into its second memory space; the second memory space storing the partitioned data of the first computing node; and the first computing node writing the second group of partitioned data from
  • the management node when shuffling multiple data points across multiple compute nodes is required, the management node obtains the data volume of each compute node and partitions the data across multiple compute nodes according to the shuffle strategy and the data of each compute node, resulting in partitioned data for each compute node. Then, the management node sends partition information to each compute node (e.g., the first compute node) to indicate the partitioned data for each compute node. Upon receiving the partition information, each compute node determines, based on the partition information, the first group of partitioned data belonging to its own compute node and the second group of partitioned data not belonging to its own compute node within the first memory space.
  • each compute node determines, based on the partition information, the first group of partitioned data belonging to its own compute node and the second group of partitioned data not belonging to its own compute node within the first memory space.
  • the first group of partitioned data belonging to its own compute node is directly written to the second memory space of its own compute node, while the second group of partitioned data belonging to other compute nodes is written to the first disk for transfer from the first disk to other compute nodes.
  • each compute node directly writes its own partition data into its own memory space (e.g., a second memory space)
  • a portion of each compute node's data is no longer written to its local disk. This reduces the amount of data written to the local disk by each compute node, improving disk write efficiency and consequently data shuffle efficiency.
  • since data belonging to each compute node is not written to its local disk there is no need to write data from the local disk to the local memory of each compute node. This avoids redundant data transfer between the local disk and local memory of each compute node, reducing the amount of data that needs to be distributed to different data partitions on the local disk of each compute node. This improves disk read efficiency and consequently data shuffle efficiency, preventing data shuffle operations from impacting data analysis efficiency.
  • the first and second memory spaces reside within the memory of the Trusted Execution Environment (TEE) of the first compute node.
  • TEE Trusted Execution Environment
  • the first and second memory spaces reside within the memory of the rich execution environment (REE) of the first compute node. This allows the above scheme to be implemented within the REE, enabling the shuffling of plaintext data.
  • REE rich execution environment
  • the data type of multiple data is key-value data.
  • the method further includes: the first computing node grouping the keys of each data in the first computing node to obtain each group of the first computing node; the management node obtaining the data volume of each computing node in the multiple computing nodes, and partitioning the multiple data of the multiple computing nodes according to the shuffle strategy and the data volume of each computing node, including: the management node obtaining the data volume of each group of each computing node to obtain the data volume of multiple groups of the multiple computing nodes; the management node partitioning the multiple groups of the multiple computing nodes according to the shuffle strategy and the data volume of the multiple groups of the multiple computing nodes.
  • the management node when the management node needs to partition multiple data points across multiple compute nodes, it obtains the data volume of each group on each compute node, thus generating the data volume of multiple groups across multiple compute nodes. Then, based on the shuffle strategy and the number of groups, the multiple groups are partitioned. Since identical keys on different compute nodes belong to the same group, and some different keys also belong to the same group, the number of groups is less than the number of keys. Based on this, since the number of groups on each compute node is less than the number of keys, the number of elements requiring partitioning on each compute node is reduced, thereby reducing the number of elements requiring partitioning across multiple compute nodes. This improves the efficiency of partition determination, and consequently, the efficiency of partitioning multiple data points across multiple compute nodes.
  • partitioning is determined at the group level, the same group on different computing nodes belongs to the same partition. Combined with the fact that each computing node assigns the same key to the same group, partitioning at the group level ensures that data with the same key on multiple computing nodes all belong to the same partition, thus helping to ensure the reliability and accuracy of subsequent data analysis processes.
  • the first computing node groups the keys of each piece of data within the first computing node, including: the first computing node determining the hash value of the key for each piece of data; the first computing node taking the remainder of the hash value of the key for each piece of data with respect to a target value, obtaining the remainder corresponding to the key of each piece of data; wherein, the target value is the number of groups; the first computing node groups the remainders corresponding to the keys of each piece of data; keys with the same remainder belong to the same group. This helps ensure the accuracy of the determined groupings.
  • the first computing node groups the keys of each piece of data on the first computing node, including: the first computing node groups the keys of each piece of data on the first computing node according to multiple key group information; wherein, a key group information is used to indicate at least one key included in a group. This helps to improve the diversity of ways to determine the grouping.
  • the target group among multiple groups is the partition data of the first compute node; the data volume of the target group on the first compute node is greater than or equal to the data volume of the target group on non-first compute nodes. This helps to reduce the amount of cross-node transmission, thereby improving shuffle efficiency.
  • the management node partitions multiple groups across multiple compute nodes based on the shuffle strategy and the data volume of multiple groups on multiple compute nodes. This includes: the management node determining the group associated with each compute node from the multiple groups; wherein the data volume of the target group associated with the first compute node on the first compute node is greater than or equal to the data volume of the target group on non-first compute nodes; the management node polls the allocated partition data for each compute node from the groups associated with each compute node; wherein, in each round of polling, if the allocated data volume of the first compute node is less than or equal to the allocated data volume of non-first compute nodes, partition data is allocated to the first compute node from the groups associated with the first compute node. This helps to balance the data volume across different compute nodes.
  • the method further includes: if the management node has already assigned a compute node to the group associated with the first compute node, it allocates partitioned data to the first compute node from the groups not associated with the first compute node. This helps to balance the amount of data on different compute nodes.
  • the management node partitions multiple groups across multiple compute nodes based on the shuffle strategy and the data volume of multiple groups across multiple compute nodes. This includes: the management node allocating partitioned data to each compute node from multiple groups; and ensuring that the data volume of the target group allocated to the first compute node on the first compute node is greater than or equal to the data volume of the target group on non-first compute nodes. This helps to provide diversity in determining the shuffle strategy.
  • the data type of the multiple data is not key-value data.
  • the method further includes: the first computing node grouping each data of the first computing node to obtain each group of the first computing node; the management node obtaining the data volume of each computing node in the multiple computing nodes, and partitioning the multiple data of the multiple computing nodes according to the shuffle strategy and the data volume of each computing node, including: the management node obtaining the data volume of each group of each computing node to obtain the data volume of multiple groups of the multiple computing nodes; the management node partitioning the multiple groups of the multiple computing nodes according to the shuffle strategy and the data volume of the multiple groups of the multiple computing nodes.
  • the first compute node groups each piece of data of the first compute node, including: the first compute node takes the data of each partition of the first compute node as each group of the first compute node.
  • the first computing node groups each piece of data of the first computing node, including: the first computing node determining the key of each piece of data of the first computing node; the first computing node grouping the keys of each piece of data of the first computing node; wherein some different keys belong to the same group.
  • the first computing node groups each piece of data, including: the first computing node determines the hash value of each piece of data; the first computing node takes the remainder of the hash value of each piece of data modulo a target value to obtain the remainder corresponding to each piece of data; wherein, the target value is the number of groups; the first computing node groups the remainders corresponding to each piece of data; data with the same remainder belong to the same group, and data with different remainders belong to the same group.
  • a data shuffle apparatus comprising: functional units for performing any of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented by hardware or by hardware executing corresponding software.
  • the data shuffle device may include: a partitioning module, a writing module, and a transmission module; the partitioning module is used to obtain data volume information of multiple computing nodes, and partition multiple data of multiple computing nodes according to the shuffle strategy and the data volume information of multiple computing nodes to obtain partition data of each computing node; the partitioning module is also used to send partition information to a first computing node, the partition information being used to indicate the partition data of the first computing node; the first computing node is any one of the multiple computing nodes; the writing module is used to determine the first group of partition data belonging to the first computing node and the second group of partition data not belonging to the first computing node in the first memory space of the first computing node; the writing module is used to write the first group of partition data in the first memory space of the
  • a computing cluster comprising multiple computing nodes and a management node.
  • the management node is used to obtain the data volume of each computing node among the multiple computing nodes, and partition the data of the multiple computing nodes according to the shuffle strategy and the data volume of each computing node, obtaining partitioned data for each computing node;
  • the management node is also used to send partition information to a first computing node, the partition information indicating the partitioned data of the first computing node;
  • the first computing node is any one of the multiple computing nodes;
  • the first computing node is also used to determine, within the data of the first computing node's first memory space, a first group of partitioned data belonging to the first computing node and a second group of partitioned data not belonging to the first computing node;
  • the first computing node is also used to write the first group of partitioned data from the first computing node's first memory space into the second memory space of the first computing node;
  • the second memory space is used to store the partition
  • the management node can also be used to execute the steps performed by the management node in the first aspect above, and the first computing node can also be used to execute the steps performed by the first computing node in the first aspect above, which will not be elaborated here.
  • a processor is provided that can be used to perform the steps of any of the methods provided in the first aspect above.
  • a chip comprising: a processor and a power supply circuit; the power supply circuit can be used to supply power to the chip; the processor can be used to execute the operation steps of any of the methods provided in the first aspect above.
  • a sixth aspect provides a computing device, comprising: a processor, a memory, and a computer program/instructions stored in the memory; the processor executes the computer program to cause the computing device to perform the steps of any of the methods provided in the first aspect.
  • a seventh aspect provides a computing device cluster, comprising: at least one computing device, each computing device including a processor and a memory; each computing device including a processor, a memory, and computer programs/instructions stored in the memory; the processor executing the computer programs/instructions to cause the computing device cluster to implement the steps of any of the methods provided in the first aspect, or to implement the steps of any of the methods provided in the third aspect.
  • a computer program product including instructions, comprising a computer program/instructions that, when executed by a processor, implement the steps of any of the methods provided in the first aspect above.
  • a computer-readable storage medium on which a computer program/instructions are stored, which, when executed by a processor, implement the steps of any of the methods provided in the first aspect above.
  • FIG. 1 is a schematic diagram of data shuffling in the related technology provided in this application.
  • Figure 2 is an architecture diagram of a data analysis system provided in this application.
  • FIG. 3 is a schematic diagram of the modules of a target software program provided in this application.
  • FIG. 4 is a flowchart of a data shuffling method provided in this application.
  • FIG. 5 is a schematic diagram of a data shuffling method provided in this application.
  • FIG. 6 is a schematic diagram of a data shuffling device provided in this application.
  • Figure 7 is a schematic diagram of a computing device provided in this application.
  • Figure 8 is a schematic diagram of the computing device cluster provided in this application.
  • Figure 9 is a schematic diagram of the connection of a computing device cluster provided in this application.
  • REE Rich execution environment
  • OS operating system
  • the data analysis system running in the REE can be used to analyze plaintext data.
  • a Trusted Execution Environment is a hardware-supported secure isolation environment that can coexist with an REE. Data within the TEE is isolated from the outside world; code outside the TEE cannot read or tamper with the data within the TEE. Based on this, plaintext computation can be performed on encrypted data within the TEE, thus processing the encrypted data.
  • the data analysis system running in the TEE can be used to analyze encrypted data.
  • Data partitioning also known simply as partitioning, refers to a group of data that shares the same attributes. For example, daily temperature data for January could form one data partition, and daily rainfall data for February could form another.
  • the data partition before shuffle is referred to as the old partition or old data partition
  • the data partition obtained after shuffle is referred to as the new partition or new data partition.
  • Data shuffle operations are a fundamental core operation in data analysis.
  • Data shuffle operations can repartition data on different computing nodes of a data analysis system, so that data with the same key on different computing nodes can be assigned to the same data partition.
  • the data analysis system includes computing node a and computing node b.
  • Computing node a includes memory a and hard disk a
  • computing node b includes memory b and hard disk b.
  • Memory space a1 of memory a stores data a1 and data a2
  • memory space b1 of memory b stores data b1 and data b2.
  • the data analysis system When the data analysis system performs a shuffle operation on data a1, a2, b1, and b2, it first determines the partitioning strategy based on the key of each data item in data a1, a2, b1, and b2. Since the keys of data a1 and b1 are the same, and the keys of data a2 and b2 are the same, the partitioning strategy indicates that data a1 and b1 belong to new partition a and compute node a, while data a2 and b2 belong to new partition b and compute node b. Specifically, memory space a2 in memory a is used to store data in new partition a, and memory space b2 in memory b is used to store data in new partition b.
  • compute node a writes data a1 and data a2 from memory space a1 to disk a, writes data a1 from disk a to memory space a2, and sends data a2 from disk a to compute node b, which then writes data a2 to memory space b2.
  • Compute node b writes data b1 and data b2 from memory space b1 to disk b, writes data b2 from disk b to memory space b2, and sends data b1 from disk b to compute node 1, which then writes it to memory space a2.
  • the computing nodes include multiple computing nodes.
  • the management node obtains the data volume of each computing node and partitions the multiple data on multiple computing nodes according to the shuffle strategy and the data of each computing node, obtaining the partition data of each computing node. Then, the management node sends partition information to each computing node to indicate the partition data of each computing node. After receiving the partition information, each computing node determines the first group of partition data belonging to its own computing node and the second group of partition data not belonging to its own computing node in the first memory space according to the partition information. Then, the first group of partition data belonging to its own computing node is directly written into the second memory space of its own computing node, and the second group of partition data belonging to other computing nodes is written into the first disk for transfer from the first disk to other computing nodes.
  • each compute node directly writes its own partition data into its own memory space (e.g., a second memory space)
  • a portion of each compute node's data is no longer written to its local disk. This reduces the amount of data written to the local disk by each compute node, improving disk write efficiency and consequently data shuffle efficiency.
  • since data belonging to each compute node is not written to its local disk there is no need to write data from the local disk to the local memory of each compute node. This avoids redundant data transfer between the local disk and local memory of each compute node, reducing the amount of data that needs to be distributed to different data partitions on the local disk of each compute node. This improves disk read efficiency and consequently data shuffle efficiency, preventing data shuffle operations from impacting data analysis efficiency.
  • this application provides a data analysis system that can be used to execute the above-mentioned data shuffling method, thereby improving data shuffle efficiency without increasing hardware costs, avoiding the impact of data shuffle operations on data analysis efficiency, and thus improving the overall performance of the data analysis system.
  • a data analysis system can also be called a computing cluster.
  • the data analysis system may include multiple computing nodes. These computing nodes can communicate with each other, and each computing node performs computing tasks.
  • the computing tasks may include data analysis tasks such as data encryption, data decryption, data sorting, data aggregation, and data shuffling.
  • the data analysis system may also include a management node, which is used to perform resource scheduling tasks and manage computing tasks.
  • resource scheduling tasks may include determining the number of data partitions, etc.
  • Managing computing tasks may include monitoring the execution status of computing tasks on each computing node, etc.
  • the management node can be any one of the multiple compute nodes. In another example, the management node can be a compute device other than the multiple compute nodes. When the management node is a compute device other than the multiple compute nodes, the management node can communicate with each of the multiple compute nodes.
  • the following example illustrates this application by taking a computing device other than multiple computing nodes as the management node.
  • the management node can be used to determine the partitioning strategy for shuffle operations, such as determining the partition data for each compute node.
  • the compute nodes can be used to directly write the partition data belonging to their own compute node into their own memory space, determine groupings, etc.
  • the operation of determining the partitioning strategy performed by the management node can also be performed by the compute node, and this application does not impose any restrictions on this.
  • the following example illustrates this application by showing that the operation of determining the partitioning strategy is performed by the management node.
  • management node can be considered as a cluster relationship or a distributed computing cluster, etc.
  • the data analysis system can provide a REE (Resource Equivalent to a Data Shuffling Element). Based on this, the data analysis system can execute the data shuffling method provided in this application within the REE, so as to perform data shuffling operations on plaintext data.
  • REE Resource Equivalent to a Data Shuffling Element
  • the management node and each compute node can provide a REE (Resource Equivalent to a Data Buffer).
  • a data shuffle method can be executed within the REEs provided by the management node and each compute node.
  • the processor of the management node and the processor of each compute node can include a first processing unit, which can be used to provide the REE.
  • each compute node can include a first memory, which can be used to store data in the REE provided by each compute node, such as plaintext data to be shuffled.
  • the management node can include a third memory, which can be used to store data in the REE provided by the management node, such as partitioning strategies for the plaintext data to be shuffled.
  • the first memory of each computing node and the third memory of the management node can be referred to as REE memory, which will not be elaborated further.
  • the data analysis system can provide a TEE (Transaction Execution Environment). Based on this, the data analysis system can execute the data shuffling method provided in this application within the TEE, so as to perform data shuffling operations on encrypted data.
  • TEE Transaction Execution Environment
  • the management node and each compute node can provide a TEE (Transport Equipment Environment).
  • the data shuffle method can be executed within the TEE provided by the management node and each compute node.
  • the processor of the management node and the processor of each compute node can also include a second processing unit, which can be used to provide the REE.
  • the second processing unit can include ARM TrustZone, CCA, etc.
  • each compute node may include a second memory, which can be used to store data from the TEE provided by each compute node, such as ciphertext data for the shuffling operation.
  • the management node may include a fourth memory, which can be used to store data from the TEE provided by the management node, such as the partitioning strategy for the ciphertext data for the shuffling operation.
  • TEE memory The second memory of each compute node and the fourth memory of the management node can be referred to as TEE memory, which will not be elaborated further.
  • first memory second memory
  • third memory third memory
  • fourth memory can be the same or different, and this application does not impose any restrictions on this.
  • each compute node may also include a disk.
  • the disk of each compute node may store the shuffle file for that compute node, and each compute node's shuffle file may include data to be transferred from that compute node.
  • the data to be transferred may be partition data from other compute nodes.
  • the disk can be a solid state disk/drive (SSD), a hard disk drive (HDD), a hybrid hard disk (HHD), etc.
  • SSD solid state disk/drive
  • HDD hard disk drive
  • HD hybrid hard disk
  • the computing node can be a computing device, which can be used to perform computing tasks.
  • the computing device can be a terminal device or a network device.
  • the terminal devices may include ultra-mobile personal computers (UMPCs), laptops, netbooks, desktop computers, all-in-one computers, etc.
  • UMPCs ultra-mobile personal computers
  • laptops netbooks
  • desktop computers all-in-one computers, etc.
  • Network devices can include servers.
  • a server can be a single physical server, or two or more physical servers that share different responsibilities and work together to achieve the various functions of the server.
  • a server can be a blade server, a high-density server, a rack server, or a tower server.
  • the management node can also be a computing device, which can be used to perform resource scheduling tasks and manage computing tasks, etc.
  • the device types of the management node and the computing node can be the same or different, and this application does not impose any restrictions on this.
  • Figure 2 is a schematic diagram of the architecture of a data analysis system provided in this application.
  • the data analysis system may include a management node, computing node 1 and computing node 2.
  • the management node can communicate with computing node 1 and computing node 2, and computing node 1 and computing node 2 can also communicate with each other.
  • the management node can communicate with compute nodes 1 and 2 via control signals.
  • the management node can transmit task requests to compute nodes 1 and 2 via control signals, thereby instructing compute nodes 1 and 2 to execute the data shuffle method.
  • management node, compute node 1, and compute node 2 can all provide TEE and REE.
  • compute node 1 includes one disk core, one REE memory (not shown in Figure 2), and one TEE memory (not shown in Figure 2).
  • the REE memory 1 is used to store data in the REE provided by compute node 1
  • the TEE memory 1 is used to store data in the TEE provided by compute node 1
  • the disk core is used to store data to be transferred on compute node 1.
  • the data to be transferred could be partition data of compute node 2.
  • the second computing node 2 includes a disk 2, REE memory 2 (not shown in Figure 2), and TEE memory 2 (not shown in Figure 2).
  • REE memory 2 stores data in the REE provided by computing node 2
  • TEE memory 2 stores data in the TEE provided by computing node 2
  • disk 2 stores data to be transferred on computing node 2.
  • the data to be transferred can be partition data from computing node 1.
  • the management node includes disk 3, REE memory 3 (not shown in Figure 2), and TEE memory 3 (not shown in Figure 2).
  • REE memory 3 is used to store data in the REE provided by the management node
  • TEE memory 3 is used to store memory in the TEE provided by the management node
  • disk 3 can be used to store data received by the management node, etc.
  • the data analysis system shown in Figure 2 can be a system provided by a cloud service platform.
  • the cloud computing provided by the cloud service platform can include the data shuffling method provided in this application.
  • a user can communicate with a data analysis system provided by a cloud service platform through a terminal device to instruct the data analysis system provided by the cloud service platform to perform the data shuffling method provided in this application.
  • the terminal device may include a personal computer (PC), a personal digital assistant (PDA), an ultra-mobile personal computer (UMPC), a laptop, a netbook, a desktop computer, or an all-in-one computer, etc.
  • PC personal computer
  • PDA personal digital assistant
  • UMPC ultra-mobile personal computer
  • laptop a netbook
  • desktop computer a desktop computer
  • all-in-one computer etc.
  • the cloud service platform shown in Figure 2 can be a public cloud, a private cloud, a hybrid cloud, etc.
  • this application provides a computing engine, such as Apache Spark, which can be used to perform data analysis tasks.
  • the computing engine may include a driver (e.g., a Spark driver) and an executor (e.g., a Spark executor).
  • a driver e.g., a Spark driver
  • an executor e.g., a Spark executor
  • the driver can be deployed on a management node, which can execute resource scheduling tasks and manage computing tasks by running the driver.
  • the executor can be deployed on each computing node, and each computing node can execute computing tasks by running the executor.
  • the computing engine can be deployed in the TEE provided by the data analysis system, such as the driver being deployed in the TEE provided by the management node and the executor being deployed in the TEE provided by each computing node, thereby enabling data analysis of encrypted data.
  • the computing engine can be deployed in the REE provided by the data analysis system, such as: the driver is deployed in the REE provided by the management node, and the actuator is deployed in the REE provided by each computing node, thereby enabling data analysis of plaintext data.
  • this application also provides a target software program that can be used to implement the data shuffling method provided in this application.
  • This target software program can be deployed on a data analysis system, which can then execute the data shuffling method provided in this application by running the target software program.
  • the target software program can be embedded in the computing engine.
  • the target software program can be a library file of the computing engine.
  • the computing engine can call the target software program, causing the data analysis system to run the target software program and thus execute the data shuffling method provided in this application.
  • the management node can determine partitioning strategies by running the target software program, such as determining the partition data for each computing node.
  • Each computing node can transfer data to be transmitted from its own computing node to other computing nodes by running the target software, and the partition data of its own computing node can be directly written from the memory space of the old partition to the memory space of the new partition.
  • the target software program can be called data shuffle software, data shuffle application, data shuffle device, etc.
  • the target software program can be software, application program, or component, etc.
  • the target software program can be deployed in a TEE provided by the data analysis system, such as in the TEE provided by the management node and each compute node, for shuffling encrypted data.
  • the target software program can be deployed in an REE provided by the data analysis system, such as in the REE provided by the management node and each compute node, for shuffling plaintext data.
  • FIG. 3 is a schematic diagram of the modules of a target software program provided in this application.
  • the target software program provided in this application may include a partitioning module, a shuffleWrite module, and a shuffleRead module.
  • the partitioning module can be used to determine the partitioning strategy for multiple data sets.
  • the shuffleWrite module can be used to prune the dataset according to the partitioning strategy, obtaining a first set of partitioned data and a second set of partitioned data.
  • the first set of partitioned data belongs to the data partition of the current computing node, and the second set of partitioned data belongs to the data partition of other computing nodes. Then, the first set of partitioned data is directly written from the memory space of the old partition to the memory space of the new partition.
  • the shuffleRead module can be used to perform data transfer according to the partitioning strategy, such as writing the second set of partitioned data to the local disk of each computing node and transferring the second set of partitioned data from the local disk of each computing node to other computing nodes.
  • the shuffle writing module can be simply referred to as the writing module, and the shuffle reading module can be called the transmission module, which will not be elaborated further.
  • Figure 4 is a flowchart of a data shuffling method provided in this application. Exemplarily, the method may include steps 401-407.
  • step in this application can be abbreviated as "S”, and will not be elaborated further hereafter.
  • the data shuffling method shown in Figure 4 can be used to process ciphertext data or plaintext data.
  • a target user obtains data to be analyzed provided by one or more users and sends a storage request to the data analysis system via a terminal device.
  • the storage request requests the storage of the data to be analyzed.
  • the data analysis system can store the data to be analyzed on multiple computing nodes of the data analysis system, where different computing nodes store different data to be analyzed.
  • this application does not restrict the relationship between the target user and the data to be analyzed.
  • the target user may or may not be the owner of the data to be analyzed.
  • the amount of data stored on different computing nodes i.e., the quantity of data to be analyzed
  • this application does not impose any restrictions on this.
  • Example 1 shows that the target user can send a storage request to the management node through the terminal device. After receiving the storage request, the management node can store a portion of the data to be analyzed in computing node 1 and store another portion of the data to be analyzed in computing node 2.
  • Example 2 shows that a target user can send a first storage request to computing node 1 via a terminal device to request the storage of a portion of the data to be analyzed. Computing node 1 can then store this portion of the data based on the first storage request. The target user can also send a second storage request to computing node 2 via the terminal device to request the storage of another portion of the data to be analyzed. Computing node 2 can then store this other portion of the data based on the second storage request.
  • the following provides an example of the types of data to be analyzed.
  • Example a The data to be analyzed is plaintext data, which includes plaintext data 1 and plaintext data 2.
  • the target user can send a storage request 1 to computing node 1 via a terminal device, requesting the storage of plaintext data 1, and a storage request 2 to computing node 2, requesting the storage of plaintext data 2.
  • Computing node 1 based on the received storage request 1, stores the received plaintext data 1 on disk 1, and then writes the plaintext data 1 from disk 1 into REE memory 1 for data analysis in the first REE.
  • Computing node 2, based on the received storage request 2 stores the received plaintext data 2 on disk 2, and then writes the plaintext data 2 from disk 2 into REE memory 2 for data analysis in the second REE.
  • Example b The data to be analyzed is encrypted data, which includes encrypted data 1 and encrypted data 2.
  • the target user can send a storage request 3 to computing node 1 via a terminal device, requesting the storage of encrypted data 1, and a storage request 4 to computing node 2, requesting the storage of encrypted data 2.
  • computing node 1 stores the received encrypted data 1 on disk 1, and then writes the encrypted data 1 from disk 1 into TEE memory 1 for data analysis in the first TEE.
  • computing node 2 stores the received encrypted data 2 on disk 2, and then writes the encrypted data 2 from disk 2 into TEE memory 2 for data analysis in the second TEE.
  • computing node 1 after computing node 1 writes the encrypted data 1 into TEE memory 1, it can partition the encrypted data 1 to divide it into multiple partitions 1. Since these multiple partitions 1 are partitions from the data storage stage, that is, partitions before the shuffle operation, they are called old partitions. To facilitate the distinction between partitions from different stages, these multiple partitions 1 will be referred to as multiple old partitions 1 below. Similarly, computing node 2 can partition the sealed data 2 to obtain multiple old partitions 2.
  • computing node 1 after computing node 1 identifies multiple old partitions 1, it can allocate memory space from TEE memory 1 for each old partition 1 to store the data of each old partition 1.
  • computing node 2 after computing node 2 identifies multiple old partitions 2, it can allocate memory space from TEE memory 2 for each old partition 2 to store the data of each old partition 2.
  • each computing node when storing data to be analyzed, such as encrypted data 1 and encrypted data 2, each computing node can obtain multiple old partitions by partitioning the data to be analyzed. In this way, when each computing node performs shuffle, it can perform shuffle on multiple partitions in parallel, which helps to improve shuffle efficiency.
  • computing node 1 can divide the ciphertext data 1 into multiple old partitions 1 based on the amount of ciphertext data 1 and the parallel computing capability.
  • a target user when a target user needs to perform data analysis on the data to be analyzed, they can send a data analysis request to the management node through their terminal device.
  • This data analysis request requests the analysis of the data.
  • the management node Upon receiving the data analysis request, the management node can parse it to obtain at least one data analysis task indicated by the request, such as a target data analysis task. Then, the management node sends a target task request to each computing node to request the execution of the target data analysis task.
  • the management node and each computing node can perform shuffle operations on the data to be analyzed on multiple computing nodes through the data shuffling method provided in this application.
  • the data types of multiple data items can be key-value data (hereinafter referred to as key-value data) or non-key-value data.
  • key-value data hereinafter referred to as key-value data
  • non-key-value data The following description uses key-value data as an example to illustrate this application.
  • a dataset can contain multiple keys, where each key can correspond to at least one piece of data.
  • the multiple keys include city a and city b, wherein at least one piece of data corresponding to city a includes data with the key of city a among multiple pieces of data, and the data corresponding to city b includes data with the key of city b among multiple pieces of data.
  • the multiple computing nodes may include a first computing node, which can be any one of the multiple computing nodes.
  • non-first computing nodes performing the data shuffling method can be referred to as the process of the first computing node performing the data shuffling method described below, and will not be repeated hereafter.
  • non-first computing nodes refer to the computing nodes other than the first computing node among multiple computing nodes.
  • non-first computing nodes will be referred to as second computing nodes, and will not be elaborated further.
  • the first computing node includes a first TEE memory, which stores data to be analyzed, namely the data for which a shuffle operation needs to be performed.
  • the data to be analyzed is divided into multiple old partitions. These multiple old partitions include a first old partition, which can be any one of the multiple old partitions.
  • the following example uses the data from the first old partition to illustrate the process of performing a shuffle operation on the first compute node.
  • the memory space corresponding to the first old partition is called the first memory space; that is, the first memory space is used to store the data from the first old partition.
  • the first computing node can perform shuffle operations in parallel on multiple old partitions, and group the data of each old partition and transmit the data of each old partition in parallel, thereby improving the efficiency of the shuffle operation.
  • Step 401 Each of the multiple compute nodes sends the amount of data for each compute node to the management node.
  • the amount of data for each computing node can include multiple cases.
  • the following uses the first computing node as an example to illustrate the multiple cases of the amount of data for each computing node through case 1 and case 2.
  • the amount of data in the first compute node can be the amount of data corresponding to each key of the first compute node.
  • the amount of data corresponding to each key refers to the quantity of at least one piece of data corresponding to each key.
  • the amount of data corresponding to each key can also be referred to as the amount of data for each first key.
  • the data on the first compute node will be referred to as the first data
  • the keys of the data on the first compute node will be referred to as the first keys.
  • a first compute node includes multiple sets of first data, which are key-value data.
  • the multiple sets of first data may include multiple first keys, each of which is different.
  • the first compute node can send the amount of data corresponding to each first key to the management node, so that the management node can partition at least one set of data corresponding to each first key based on the amount of data for each first key.
  • each computing node sends the amount of data corresponding to the key to the management node. This helps to reduce the operations that each computing node needs to perform, thereby improving the efficiency of each computing node sending the amount of data to the management node.
  • the data volume of the first computing node can be the data volume corresponding to each group of the first computing node.
  • the data volume corresponding to each group refers to the number of at least one piece of data corresponding to each group. At least one piece of data corresponding to each group refers to at least one piece of data corresponding to each first key included in each first group.
  • the data volume corresponding to each first group can also be referred to as the data volume of each first group.
  • the data volume of the SKa group of the first compute node is 5, and the data volume of the SKb group is [missing data].
  • the first compute node sends first data volume information to the management node, indicating that the data volume of the SKa group of the first compute node is 5, and the data volume of the SKb group is 3.
  • the group on the first computing node will be referred to as the first group below.
  • the first compute node includes multiple first groups, wherein the number of the multiple first groups is less than the number of multiple first keys.
  • the first compute node can send the data volume of each first group to the management node, so that the management node can partition at least one piece of data corresponding to each first group based on the data volume of each first group.
  • each computing node sends the data volume corresponding to each group to the management node, and the management node partitions the data at the group level, which reduces the number of elements that need to be partitioned, thereby improving the efficiency of the shuffle process.
  • multiple groups of multiple computing nodes include multiple keys of the multiple computing nodes, and a group includes at least one key from the multiple keys.
  • partitioning at the group level it helps to avoid omitting some keys from the multiple keys, thereby helping to ensure that each key from the multiple keys can be assigned to a new partition.
  • the same key belongs to the same group. This ensures that when partitioning at the group level, identical groups on each computing node can be assigned to the same data partition, thus helping to guarantee the reliability and accuracy of subsequent data analysis.
  • the number of groups across multiple computing nodes is less than the number of keys across multiple computing nodes. Therefore, when partitioning at the group level, compared to partitioning at a single key, the number of elements requiring partitioning on each computing node can be reduced, thereby reducing the total number of elements requiring partitioning across the entire data analysis system. This improves the efficiency of partitioning, that is, the efficiency of partitioning data across multiple computing nodes, and consequently, the efficiency of data shuffle.
  • each group of the first computing node may include key-value data corresponding to at least one key; that is, each group may include both the key and the value in the key-value data.
  • the multiple first groups of the first computing node may include multiple first data, that is, the multiple first groups may simultaneously include multiple keys of first data and multiple values of first data.
  • the first computing node divides the data in the first old partition into SKa group (hereinafter referred to as group 1) and SKb group (hereinafter referred to as group 2).
  • group 1 includes data with pentagonal keys and data with circular keys, and contains 5 data items.
  • Group 2 includes data with hexagonal keys and data with triangular keys, and contains 3 data items.
  • the second computing node divides the data in the second old partition into SKa group (group 1) and SKb group (group 2).
  • Group 1 contains 3 data items, and group 2 contains 5 data items.
  • the SKa group includes pentagonal and circular keys, and the SKb group includes hexagonal and triangular keys.
  • some groups in the first compute node may include key-value data corresponding to multiple keys. This ensures that the total number of groups in the first compute node is less than the total number of keys, thereby reducing the number of elements that need to be partitioned when partitioning at the group level, and thus improving shuffle efficiency.
  • each group of the first computing node may include at least one key; that is, each group may include only the key of the key-value data.
  • multiple first groups of the first computing node may include only different first keys of multiple first data.
  • first group 1 may include pentagons and circles
  • first group 2 may include hexagons and triangles.
  • each group by setting each group to include only the key, the number of elements in the grouping can be reduced when there are many identical keys in multiple data sets, thereby helping to improve grouping efficiency and thus improving shuffle efficiency.
  • some groups in the first compute node may include multiple keys. This ensures that the total number of groups in the first compute node is less than the total number of keys, thereby reducing the number of elements that need to be partitioned when partitioning at the group level, and thus improving shuffle efficiency.
  • the data shuffling method may further include: the first computing node grouping each data item of the first computing node by its key, resulting in each group of the first computing node. Where identical keys belong to the same group, and partially different keys also belong to the same group.
  • the first computing node can group multiple first keys to obtain multiple first groups for the first computing node. After obtaining multiple first groups, the first computing node sends the data volume of each first group to the management node. For example, as shown in Figure 4, the first computing node groups each data key to obtain two groups, namely the SKa group and the SKb group, where the data volume of the SKa group is 5 and the data volume of the SKb group is 3.
  • different computing nodes in multiple computing nodes can group the keys on their respective computing nodes according to the target rules. In this way, the same group on different computing nodes corresponds to the same key.
  • the same group on different computing nodes can be assigned to the same data partition, thereby ensuring that data with the same key on different computing nodes are assigned to the same data partition.
  • each computing node when each computing node groups its keys, it assigns identical keys to the same group, and also assigns some different keys to the same group. This ensures that when partitioning at the group level later, identical keys are assigned to the same new partition, thus helping to guarantee the accuracy and reliability of subsequent data analysis tasks. Furthermore, because some different keys are also assigned to the same group, it ensures that the number of groups on each computing node is less than the number of keys on each computing node, and consequently, that the number of groups across multiple computing nodes is less than the number of keys across multiple computing nodes.
  • method 1 may include the following S1-S3.
  • the first computing node determines the hash value of the key for each piece of data in the first computing node.
  • the first computing node can perform a hash algorithm on each of the multiple first keys to obtain the hash value of each first key, and thus obtain the hash values of multiple first keys.
  • the data on the second compute node will be referred to as the second data
  • the keys of the data on the second compute node will be referred to as the second keys
  • the groups on the second compute node will be referred to as the second groups.
  • computing node 1 as the first computing node
  • computing node 2 as the second computing node
  • the multiple first data items of computing node 1 are data 1, ..., data 6, which are respectively (key1, value1), (key3, value2), (key1, value3), (key5, value4), (key1, value5), and (key1, value).
  • the multiple first data items include multiple first keys, key1, key3, and key5.
  • the first computing node can perform a hash algorithm on key1, key3, and key5 to obtain the hash values of key1, key3, and key5.
  • the hash values of key1, key3, and key5 are 13, 14, and 15 respectively.
  • Computation node 2 has multiple second data items: data 7, ..., data 12. These data items are (key2, value17), (key4, value8), (key6, value9), (key2, value10), (key7, value11), and (key8, value12), respectively.
  • the multiple second data items include multiple second keys: key2, key4, key6, key7, and key8.
  • computation node 2 can perform a hash algorithm on key2, key4, key6, key7, and key8 to obtain the hash values of key2, key4, key6, key7, and key8. For example, the hash values of key2, key4, key6, key7, and key8 are 17, 18, 19, 20, and 24, respectively.
  • the first computing node takes the remainder of the hash value of the key of each data in the first computing node modulo the target value to obtain the remainder corresponding to the key of each data in the first computing node; where the target value is the number of multiple groups.
  • the remainder corresponding to the key refers to the value obtained by taking the remainder of the hash value of the key with respect to the target value.
  • the first computing node takes the remainder of the hash value of each of the multiple first keys with respect to the target value, thereby obtaining the remainder corresponding to each first key, and thus obtaining the remainder corresponding to the multiple first keys.
  • the target value is the number of groups.
  • the remainders corresponding to multiple keys can be divided into M different values. Based on this, keys with the same remainder are grouped into the same group, and keys with different remainders are grouped into different groups. This allows multiple keys on multiple computing nodes to be divided into M groups, thereby enabling accurate control of the number of multiple groups and thus accurate control of the shuffle efficiency.
  • the target value can be provided by the target user. This helps to improve the reasonableness of the target value.
  • a target user can send a target value to the management node via a terminal device, whereby the target value indicates the number of multiple groups.
  • the management node After receiving the target value, the management node forwards it to the first computing node, so that the first computing node can determine multiple first groups based on the target value.
  • a target user when a target user sends a data analysis request to the management node, the target user may also send the target value at the same time, or the target value may be sent to the management node separately. This application does not impose any restrictions on this.
  • the target value can be determined by the management node. This helps to increase the diversity of ways to determine the target value.
  • the management node can determine the target value based on the data analysis task indicated by the data analysis request.
  • the management node stores the correspondence between data analysis tasks and numerical values. Based on this, the management node can determine the target value according to the numerical value corresponding to the data analysis task indicated by the currently received data analysis request. Then, the management node can send the target value to the first computing node.
  • this application does not limit the method of determining the target value; the above is merely an illustrative example. Furthermore, the target values corresponding to different data analysis requests may be the same or different; this application does not impose any restrictions on this.
  • the target value is 4.
  • Calculation node 1 takes the hash values of key1, key3, and key5 respectively and modulo 4 to obtain the remainders corresponding to key1, key3, and key5.
  • the remainders corresponding to key1, key3, and key5 are 1, 2, and 3 respectively.
  • Computing node 2 takes the hash values of key2, key4, key6, key7, and key8 respectively, modulo 4 to obtain the remainders for key2, key4, key6, key7, and key8.
  • the remainders for key2, key4, key6, key7, and key8 are 1, 2, 3, 0, and 0 respectively.
  • the number of multiple groups (i.e., the target value) can be greater than the number of multiple data partitions on multiple computing nodes.
  • These multiple data partitions refer to the new data partitions after the shuffle operation, where each new data partition includes the data partition corresponding to each of the multiple groups. This allows for the balancing of data volume across different data partitions through the redundant groups, thus helping to ensure the balance of data volume across different data partitions and ultimately facilitating the full utilization of the computing resources of different computing nodes.
  • the following is an exemplary description of the process for determining the number of data partitions across multiple computing nodes.
  • the target user sends a first value to the management node via a terminal device.
  • This first value indicates the number of new partitions created during the shuffle operation.
  • the first value can be sent simultaneously with the data analysis request, or it can be sent separately; this application does not impose any restrictions on this.
  • the management node determines the number of new partitions for the shuffle operation based on the data analysis task indicated in the data analysis request.
  • the management node stores the correspondence between data analysis tasks and the number of new partitions. Based on this, the management node can determine the number of new partitions for the data shuffling operation according to the data analysis task indicated by the data analysis request and the correspondence between the data analysis task and the number of new partitions.
  • the number of groups is equal to the number of data partitions on the multiple compute nodes. This not only helps improve the efficiency of determining the data partition to which a group belongs, thereby improving the efficiency of shuffle determination, but also helps increase the diversity of partition determination.
  • the first computing node groups the remainders corresponding to the keys of each data item in the first computing node; keys with the same remainder belong to the same group.
  • the first computing node when the first group only includes the first key, after the first computing node obtains the remainder of each first key, it directly identifies the first keys with the same remainder as the same group, and the first keys with different remainders as different groups, thus obtaining multiple first groups. This helps to improve the efficiency of identifying multiple groups.
  • compute node 1 can determine that key1 is a group 1, key3 is a group 2, and key5 is a group 3.
  • Computation node 2 can determine that key2 is a group 4, key4 is a group 5, key6 is a group 6, and key7 and key8 are a group 7.
  • group 1 and group 4 are the same group, group 2 and group 5 are the same group, and group 3 and group 4 are the same group.
  • the first computing node when the first group includes both a first key and a value, after the first computing node obtains the remainder for each first key, it identifies the data corresponding to keys with the same remainder as the same group, and the data corresponding to keys with different remainders as different groups, thus obtaining multiple groups. This helps improve the efficiency of subsequent data transmission.
  • compute node 1 can group data 1, data 3, data 5, and data 6 into one group, data 2 into another group, and data 4 into a third group.
  • Compute node 2 can group data 7 and data 10 into one group, data 8 into another group, data 9 into another group, and data 11 and data 12 into a third group.
  • the identifier of a group can be determined based on the fact that a group includes at least one key.
  • the identifier of a group can be a superkey constructed from the at least one key included in a group.
  • determining the identifier of a group can also be considered as establishing a mapping relationship between the group and the identifier (i.e., the hyperkey).
  • key1 and key2 form a group, which can be identified as key1key2 (abbreviated as SK1).
  • the remainders of key3 and key4 are the same, so key3 and key4 can form a group, which can be identified as key3key4 (abbreviated as SK2).
  • the remainders of key5 and key6 are the same, so key5 and key6 can form a group, which can be identified as key5key6 (abbreviated as SK3).
  • the remainders of key7 and key8 are the same, so key7 and key8 can form a group, which can be identified as key7key8 (abbreviated as SK4).
  • the identifier of a group is determined by at least one key in the group. This not only establishes the association between the key and the identifier in the group, which helps to understand the key in each group through the group identifier, but also helps to determine whether the same key is assigned to the same group, thereby improving the accuracy of grouping.
  • identifiers for different groups can also be generated randomly, or identifiers for different groups can be generated according to preset rules, or identifiers for different groups can be specified by the user.
  • multiple groups are determined by grouping keys with the same remainder into the same group and keys with different remainders into different groups. Since keys with the same hash value have the same remainder, the remainders for keys with the same hash value are identical, ensuring that identical keys are grouped into the same group. Furthermore, keys with different hash values may have the same remainder, such as the remainders of key1 and key2, thus allowing different keys to be grouped into the same group, resulting in a number of groups that is less than the number of keys.
  • method 2 may include the following S4.
  • the first computing node groups the keys of each piece of data in the first computing node according to multiple key group information; wherein, a key group information is used to indicate at least one key included in a group.
  • the first computing node determines a first group based on one key group information from the multiple key group information, thereby obtaining multiple first groups.
  • the multiple key group information can be provided by the target user.
  • the target user sends the multiple key group information to the management node, and the management node forwards the multiple key group information to each compute node.
  • the multiple key group information can be sent to the management node together with the data analysis request, or it can be sent separately; this application does not impose any restrictions on this.
  • the management node can send multiple key group information simultaneously when sending the target task request to each compute node, or it can send it separately; this application also does not impose any restrictions on this.
  • multiple key group information includes first key group information, second key group information, third key group information, and fourth key group information.
  • the first key group information includes key1 and key2
  • the second key group information includes key3 and key4
  • the third key group information includes key5 and key6,
  • the fourth key group information includes key7 and key8.
  • compute node 1 according to the first key group information, filters data 1, data 3, data 5, and data 6 from data 1, ..., data 6, and determines data 1, data 3, data 5, and data 6 as the SK1 group of compute node 1.
  • the keys for data 1, data 3, data 5, and data 6 are all key1.
  • Compute node 2 according to the first key group information, filters data 7 and data 10 from data 7, ..., data 12, and determines data 7 and data 10 as the SK1 group of compute node 2.
  • the keys for data 7 and data 10 are both key2.
  • each computing node determines each group of its own computing node based on multiple key group information. In this way, each computing node can directly filter out the key belonging to a group from multiple data according to the key indicated by a key group information, thereby helping to improve the efficiency of group determination and thus helping to improve shuffle efficiency.
  • the first compute node can treat the data from each old partition as one group for the first compute node. For instance, if the first compute node includes old partition 1 and old partition 2, then the first compute node will treat old partition 1 as one group and old partition 2 as another group. This helps improve the efficiency of determining the grouping.
  • the first compute node can assign a key to each piece of data, thus transforming non-key-value data into key-value data. Then, the data is grouped according to the grouping method used for key-group data. In this way, each compute node can group two different types of data using a single scheme, which helps to improve the applicability of the grouping scheme.
  • the target feature of non-key value data can be used as the key of non-key group data, such as: the target feature can be the city name, the province name, the value of a certain attribute, etc.
  • the first computing node determines the hash value of each data point. Then, it takes the remainder of each data point modulo the target value. Next, the remainders for each data point are grouped, with data points having the same remainder belonging to the same group.
  • this application does not limit the method for determining the hash value of each piece of data.
  • a hash algorithm can be performed on the content of each piece of data to obtain the hash value of each piece of data.
  • a hash algorithm can be performed on the values of some attributes of each piece of data to obtain the hash value of each piece of data.
  • Step 402 The management node obtains the data volume of each compute node, and partitions the multiple data of multiple compute nodes according to the shuffle strategy and the data volume of each compute node to obtain the partition data of each compute node.
  • the partition data of the first compute node refers to the data in the new partition of the first compute node.
  • the shuffle strategy includes assigning data with the same key to the same new partition.
  • the management node includes a third memory. After receiving the data volume from each computing node, the management node stores the data volume of each computing node in the third memory.
  • the third memory is the TEE memory of the management node. In another example, when the data shuffling method of this application is used to shuffle plaintext data, the third memory is the REE memory of the management node.
  • the management node After the management node obtains the data volume corresponding to each key on each compute node, it partitions multiple keys across multiple compute nodes according to the shuffle strategy and the data volume of each key on each compute node, thereby partitioning multiple data across multiple compute nodes. For example, identical keys on different compute nodes belong to the same new partition, and some different keys on different compute nodes can also belong to the same new partition. This helps to balance the data volume across different compute nodes.
  • the management node After the management node obtains the data volume corresponding to each group of each compute node, it partitions multiple groups on multiple compute nodes according to the shuffle strategy and the data volume of each group of each compute node, thereby partitioning multiple data on multiple compute nodes to obtain the group of each compute node, and then obtaining the partition data of each compute node.
  • the first computing node determines multiple first groups, such as the SKa group and the SKb group, it sends the group information of the first computing node to the management node.
  • This group information includes the data volume of each group of the first computing node.
  • the SKa group includes 5 data items
  • the SKb group includes 3 data items.
  • grouping information can be used to indicate the group identifier, the amount of data corresponding to the group, the key corresponding to the group, etc.
  • the grouping information of the first computing node indicates that the group identifiers are SKa group and SKb group, with the SKa group corresponding to 5 data items and the SKb group corresponding to 3 data items.
  • the keys corresponding to the SKa group are pentagons and circles, and the keys corresponding to the SKb group are hexagons and triangles.
  • the target group among the multiple groups is assigned to the first compute node.
  • the data of the target group among the multiple groups is the partition data of the first compute node.
  • the data volume of the target group on the first compute node is greater than or equal to the data volume of the target group on non-first compute nodes. This helps to allocate the larger data groups on each compute node to that node, thereby reducing the amount of cross-node transmission and improving data shuffling efficiency.
  • the data volume of the SKa group on the first compute node is 5, and the number of SKa groups on the second compute node is 3. Therefore, the SKa group belongs to the first compute node.
  • the data volume of the SKb group on the first compute node is 3, and the number of SKb groups on the second compute node is 5. Therefore, the SKb group belongs to the second compute node.
  • multiple groups on multiple computing nodes are partitioned according to the shuffle strategy and the amount of data in each group on each computing node.
  • This can include various implementation methods, and the following describes them by way A and way B.
  • method A may include the following S5-S6.
  • S5 The management node determines the group associated with each compute node from multiple groups of multiple compute nodes.
  • the amount of data of the target group associated with the first computing node on the first computing node is greater than or equal to the amount of data of the target group on non-first computing nodes.
  • the management node can determine multiple groups to be assigned, including the grouping information of each computing node. For example, as shown in Figure 5, after receiving grouping information 1 from computing node 1 and grouping information 2 from computing node 2, the management node can determine four groups to be assigned, including SK1, SK2, SK3, and SK4. Computing node 1 includes SK1, SK2, and SK3, and computing node 2 includes SK1, SK2, SK3, and SK4.
  • the management node can determine the group associated with each computing node from multiple groups to be assigned, according to the amount of data of each group on each computing node. For example, as shown in Figure 5, SK1 has four data items on computing node 1 (i.e., data 1, data 3, data 5, and data 6), and SK1 has two data items on computing node 2 (i.e., data 7 and data 10). Since the amount of data of SK1 on computing node 1 is greater than the amount of data of SK1 on computing node 2, SK1 is determined to be the group associated with computing node 2. Based on the same principle, SK4 is the group associated with computing node 2.
  • SK2 and SK3 can be grouped together as an association of either compute node 1 or compute node 2. Alternatively, SK2 and SK3 may not be grouped together as an association of compute node 1 and compute node 2.
  • the following description uses SK2 and SK3 as an example of a group associated with compute node 2 to illustrate this application.
  • this target group can be the first group selected by compute node 1, or in other words, the target group can be the preferred group selected by compute node 1.
  • SK1 can be the first group selected by compute node 1
  • SK4 can be the first group selected by compute node 2.
  • compute node 1 can be called the preferred compute node for the target group.
  • S6 The management node polls the groups associated with each compute node and assigns a group to each compute node in a round-robin fashion.
  • the management node can assign groups to the first computing node from the groups associated with the first computing node, thereby determining the computing node to which each group of each computing node belongs, and thus determining the group data of each computing node.
  • the groups to be allocated include SK1, SK2, SK3, and SK4.
  • SK1 can be allocated to a new partition on compute node 1 in the first round of polling.
  • SK1 on compute node 1 and SK1 on compute node 2 both belong to compute node 1, and compute node 1 has already allocated 6 data entries.
  • the data corresponding to SK1 on compute node 1 and the data corresponding to SK1 on compute node 2 constitute the partition data of compute node 1.
  • a group can be assigned to the computing node with the largest amount of data in that group. For example, SK1 is assigned to computing node 1 and SK4 is assigned to computing node 2. This helps to reduce the amount of data transmitted across nodes, thereby helping to improve data transmission efficiency and thus helping to improve shuffle efficiency.
  • groups when assigning groups to each compute node, groups can be randomly assigned to each compute node from the groups associated with each compute node. This helps improve allocation efficiency.
  • any one of the groups of SK2, SK3 and SK4 can be assigned to the new partition of computing node 2, such as assigning SK2 to computing node 2.
  • the group with the largest data volume among the groups associated with each compute node can be assigned to that compute node. This assigns a group to the compute node with the largest data volume in that group, helping to reduce the amount of data transferred across nodes, thereby improving data transfer efficiency and, consequently, shuffle efficiency.
  • the priority computing node for SK2, SK3 and SK4 is computing node 2.
  • SK2 and SK3 each have 1 data point, while SK4 has 2 data points. Based on this, SK4 can be assigned to computing node 2 during the first round of polling.
  • a group is allocated to the first computing node from the group associated with the first computing node.
  • the first compute node will be skipped in this round of polling, that is, no data partitions will be allocated to the data on the first compute node.
  • the management node can allocate a group to compute node 1 from the unallocated groups, that is, allocate a group to compute node 1 from SK3 and SK4, such as allocating SK3 to a new partition of compute node 1.
  • the data shuffling method may further include: when the groups associated with the first computing node have already been assigned to computing nodes, the management node allocates data to the first computing node from groups not associated with the first computing node. This increases the range of groups that each computing node can select, thereby helping to better balance the amount of data across different computing nodes.
  • a group in the second round of polling, if a group is selected for compute node 1, since compute node 1 is associated with SK1 and has been assigned a compute node, a group can be selected for compute node 1 from the groups associated with compute node 2, such as SK3 and SK4.
  • the data shuffling method may further include: the management node assigning a group to the first computing node from the groups of computing nodes that have not yet been associated. This increases the range of groups that each computing node can select, thereby helping to better balance the amount of data on different computing nodes.
  • the management node can assign a group to compute node 1 from SK2 and SK3.
  • the fact that the allocated data amount of the first computing node is less than or equal to the allocated data amount of the non-first computing node may include: the difference between the allocated data amount of the first computing node and the allocated data amount of the non-first computing node is less than or equal to a difference threshold.
  • the difference threshold is determined based on the amount of data allocated to different computing nodes.
  • the difference threshold can satisfy the following formula: (K-P)*u/P.
  • K is the maximum allocated data amount among multiple computing nodes
  • P is the minimum allocated data amount among multiple computing nodes.
  • u can be 5%-20%.
  • u can be 5%. In another example, u can be 20. In yet another example, u can be 10%. This helps to increase the diversity of u choices.
  • computing nodes are assigned groups only when the difference in data volume is less than a difference threshold, which helps to better balance the data volume on different computing nodes.
  • method B may include the following S7.
  • S7 The management node assigns a group to each compute node from multiple groups of multiple compute nodes.
  • the amount of data in the target group allocated to the first computing node on the first computing node is greater than or equal to the amount of data in the target group on non-first computing nodes.
  • the management node can determine the computing node to which each unassigned group belongs based on the amount of data of each unassigned group on different computing nodes.
  • the amount of data SK1 has on compute node 1 is greater than the amount of data SK1 has on compute node 2. Therefore, SK1 is allocated to a new partition on compute node 1.
  • the amount of SK4 on compute node 2 is greater than the amount of data SK4 has on compute node 1. Therefore, SK4 is allocated to a data partition on compute node 2.
  • SK2/SK3 can be allocated to either compute node 1 or compute node 2. For example, SK2/SK3 can be allocated to the compute node with less data, i.e., compute node 2. This helps to balance the amount of data across different compute nodes.
  • groups are directly allocated to each computing node based on the amount of data in each group on different computing nodes. This helps improve allocation efficiency, which in turn helps improve the efficiency of determining the partitioning strategy.
  • the following describes the process by which the management node determines the new partition for each compute node.
  • the management node can determine a new partition for each compute node based on the amount of data corresponding to different keys/groups.
  • the management node After the management node determines at least one target group belonging to the first computing node from multiple groups of multiple computing nodes, it can determine at least one new partition of the first computing node based on the data volume of each target group in the at least one target group.
  • the management node determines that SK2, SK3, and SK4 belong to compute node 2, it determines the number of partitions for compute node 2 based on the data volume of SK2 being 1, SK3 being 1, and SK4 being 2. For instance, SK2 and SK3 can be designated as one new partition, and SK4 as another new partition.
  • the management node can first determine the total number of new partitions across multiple compute nodes. Then, based on the amount of partition data allocated to each compute node, the data for the new partitions on each compute node is determined. Next, based on the amount of data in each group belonging to each compute node and the number of new partitions on each compute node, the grouping of each new partition belonging to each compute node is determined.
  • the management node can predetermine N new partitions for each compute node. Based on this, during each round of polling, it can select groups for each new partition of each compute node, or select groups for one new partition of each compute node.
  • compute node 1 includes new partition a and new partition b
  • compute node 2 includes new partition c and new partition d.
  • the management node can select groups for the new partitions and new partition b sequentially.
  • the management node selects groups only for new partition a
  • the management node selects groups for new partition b.
  • the grouping of different new partitions can be adjusted according to the amount of data in each new partition to balance the number of different new partitions.
  • this application does not limit the method by which the management node determines the new partition of each computing node; the above is merely an illustrative example.
  • step 402 can also be performed by the first computing node or a non-first computing node; the above is only an illustrative example.
  • Step 403 The management node sends partition information to the first compute node.
  • the partition information is used to indicate the partition data of the first compute node.
  • the management node after the management node determines the partition data of each computing node, it can send partition information to each computing node so that each computing node can distribute the data on its computing node according to the partition information.
  • partition information is used to indicate that data with a pentagonal key and data with a circle key both belong to the first new partition of the first computing node. That is, data with a pentagonal key and data with a circle key are partitioned data of the first computing node.
  • Partition information can also be used to indicate that data with a hexagonal key and data with a triangle key both belong to the new partition of the second computing node. That is, data with a hexagonal key and data with a triangle key are partitioned data of the second computing node.
  • the management node sends partition information to the first compute node, including the compute node to which each group belongs. This helps reduce the amount of information the management node sends to each compute node, thereby improving information transmission efficiency.
  • the partition information sent by the management node to compute node 1 only includes the compute nodes to which SK1, SK2 and SK3 belong, but does not include the compute node to which SK4 belongs.
  • the partition information sent by the management node to the first compute node includes the compute nodes to which each of the multiple groups belongs. This eliminates the need for the management node to filter partition information for each compute node, thus improving information transmission efficiency.
  • the partition information sent by the management node to compute node 1 only includes the compute nodes to which SK1, SK2, SK3, and SK4 belong.
  • Step 404 Based on the partition information, the first computing node determines the first group of partition data belonging to the first computing node and the second group of partition data not belonging to the first computing node in the first memory space of the first computing node.
  • the first group of partitioned data includes data grouped into SKa groups, that is, data with pentagonal keys and data with circular keys.
  • the second group of partitioned data includes data grouped into SKb groups, that is, data with triangular keys and data with hexagonal keys.
  • the reason for dividing the data in the first memory space into a first group of partitioned data and a second group of partitioned data is to distinguish the data in the first memory space belonging to different computing nodes. Specifically, for each computing node, the first group of partitioned data consists of partitioned data belonging to that computing node, while the second group of partitioned data consists of partitioned data not belonging to that computing node.
  • the first computing node when transmitting data from the first memory space, the first computing node can determine the first group of partitioned data belonging to the first computing node and the second group of partitioned data not belonging to the first computing node. This helps to improve data transmission efficiency.
  • compute node 1 after receiving the partition information, compute node 1 indicates that the data of SK1 is the partition data of the first compute node, and the data of SK2 and SK3 are the partition data of compute node 2. Based on this, when compute node 1 reads the data of SK1 from the first memory space of TEE memory 1, it can determine that the data of SK1 belongs to the first group of partition data according to the partition information. Therefore, compute node 1 processes the data of SK1 through step 405. Afterwards, when compute node 1 reads the data of SK2 from the first memory space of TEE memory 1, it can determine that the data of SK2 belongs to the second group of partition data according to the partition information. Therefore, compute node 1 can process the data of SK2 through steps 406 and 407.
  • the first computing node after receiving the partitioning information, can divide the data in the first memory space into multiple groups of partitioned data, such as a first group of partitioned data and a second group of data. This helps to improve the accuracy of data transmission.
  • compute node 1 after receiving the partition information, identifies the data of SK1 as the first group of partitioned data and the data of SK2 and SK3 as the second group of partitioned data. Then, compute node 1 processes the first group of partitioned data through step 405, and processes the second group of partitioned data through steps 406 and 407.
  • Step 405 The first computing node writes the first set of partition data in the first memory space into the second memory space of the first computing node; the second memory space is used to store the partition data of the first computing node.
  • the partition information can also be used to indicate the number of new partitions on the first computing node and the storage space occupied by the data in each new partition. After obtaining the partition information, the first computing node allocates memory space from the first memory for each new partition of the first computing node.
  • the number of new partitions on the first computing node is 1.
  • this 1 new partition is the first new partition.
  • the first computing node allocates the second memory space on the first memory to the first new partition.
  • the second memory space can be used to store the data of the first new partition of the first computing node.
  • the first computing node writes the first set of partitioned data directly from the first memory space to the second memory space.
  • the first set of partitioned data is read from the first memory space, it is not written to the first disk of the first computing node, but directly to the second memory space.
  • the first compute node reads the data of the SKa group from the first memory space and writes it directly into the second memory space.
  • compute node 1 reads the data of SK1 from the first memory space and writes it directly into the second memory space.
  • the second compute node obtains the number of new partitions on its own compute node and the storage space occupied by the data in each new partition, and allocates memory space from the second memory for each new partition.
  • the second compute node has one new partition, which is the second new partition. Based on this, the second compute node allocates the third memory space on the second memory to the second new partition, and the third memory space can be used to store the data of the second new partition. Then, the second compute node writes the SKb-grouped data directly from the memory space of the second old partition to the third memory space, without writing it to the second disk of the second compute node.
  • Step 406 The first computing node writes the second set of partitioned data from the first memory space to the first disk of the first computing node.
  • the first memory space is the memory space on the first memory and the first memory is TEE memory
  • the first computing node reads the second group partition data from the first memory space, it encrypts the second group partition data and writes the encrypted second group partition data to the first disk of the first computing node, so as to facilitate the transmission of the second group partition data to the second computing node.
  • the first disk is the disk of the REE.
  • the first computing node after the first computing node reads the data of the SKb group from the first memory space, it encrypts the data of the SKb group to obtain the encrypted SKb group data, and then writes the encrypted SKb group data to the first disk.
  • Step 407 The first computing node sends the second set of partition data from the first disk to non-first computing nodes.
  • the first computing node can transmit the encrypted second set of partitioned data from the first disk to the second computing node.
  • compute node 1 can encrypt the data of SK2 and SK3 to obtain encrypted data of SK2 and SK3. Then, compute node 1 writes the encrypted data of SK2 and SK3, along with partition information, to a shuffling file 1 on disk 1. The partition information indicates that the data of SK2 and SK3 are the score data for compute node 2. Afterward, compute node 1 transmits the encrypted data of SK2 and SK3 to compute node 2.
  • the second computing node can determine the new partition to which the second set of partitioned data belongs based on the identifier of each group of the second set of partitioned data. For instance, as shown in Figure 4, the second computing node can write the data of the SKb group into the third memory space of the second memory based on the identifier of each group of the second set of partitioned data including the SKb group and the fact that the SKb group belongs to the second new partition. It should be noted that for the relevant explanations regarding the new partition and memory space on the second computing node, please refer to the explanation of the new partition and memory space of the first computing node in step 405, which will not be repeated here.
  • the second computing node when the second computing node executes the data shuffling method provided in this application, it can send the SKa packet data from the second disk to the first computing node. After receiving the SKa packet data sent by a non-second computing node, the first computing node writes the SKa packet data into the second memory space.
  • each computing node directly writes the partition data belonging to its own computing node into the memory space (e.g., the second memory space) used to store partition data on its own computing node, on the one hand, a portion of the data of each computing node will no longer be written to the local disk. Therefore, the amount of data written to the local disk by each computing node is reduced, thereby improving the disk write efficiency of each computing node and thus improving the data shuffle efficiency.
  • the memory space e.g., the second memory space
  • the data shuffling device includes corresponding hardware structures and/or software modules for executing each function.
  • the data shuffling device includes corresponding hardware structures and/or software modules for executing each function.
  • this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
  • the data shuffling device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module.
  • the integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
  • Figure 6 shows a possible structural schematic diagram of the data shuffling device (denoted as data shuffling device 600) involved in the above embodiments.
  • the actions performed by the data shuffling device 600 are implemented by a computing device or by executing corresponding software on the computing device.
  • the data shuffling device 600 may include a partitioning module 601, a writing module 602, and a transmission module 603.
  • the partitioning module 601 is used to obtain data volume information of multiple computing nodes, and partition multiple data of multiple computing nodes according to the shuffle strategy and the data volume information of multiple computing nodes to obtain partition data of each computing node. For example, as shown in S403 of Figure 4.
  • the partitioning module 601 is also used to send partition information to a first computing node, the partition information being used to indicate the partition data of the first computing node; the first computing node is any one of the multiple computing nodes.
  • the writing module 602 is used to determine the first group of partition data belonging to the first computing node and the second group of partition data not belonging to the first computing node in the data of the first memory space of the first computing node.
  • the write module 602 is used to write the first set of partition data from the first memory space of the first computing node to the second memory space of the first computing node; the second memory space is used to store the partition data of the first computing node.
  • the write module 602 is also used to write the second set of partition data from the first memory space of the first computing node to the first disk of the first computing node; the transmission module 603 is used to transmit the second set of partition data from the first disk to non-first computing nodes.
  • the write module 602 is also used to write the second set of partition data from the first memory space of the first computing node to the first disk of the first computing node; the transmission module 603 is used to transmit the second set of partition data from the first disk to non-first computing nodes.
  • the first memory space and the second memory space are located in the memory of the Trusted Execution Environment (TEE) of the first compute node.
  • TEE Trusted Execution Environment
  • the first memory space and the second memory space are located in the memory of the rich execution environment REE of the first computing node.
  • the data type of the multiple data is key-value data
  • the data shuffling device also includes a grouping module 604.
  • the grouping module 604 is used to: group the keys of each data item of the first computing node to obtain each group of the first computing node;
  • the partitioning module 601 is specifically used to: have the management node obtain the data volume of each group of each computing node to obtain the data volume of multiple groups of multiple computing nodes;
  • the multiple data items include multiple keys, with identical keys belonging to the same group, and some different keys belonging to the same group;
  • the management node partitions the multiple groups of multiple computing nodes according to the shuffle strategy and the data volume of the multiple groups of multiple computing nodes.
  • the grouping module 604 is specifically used to: determine the hash value of the key of each data in the first computing node; take the remainder of the hash value of the key of each data in the first computing node modulo a target value to obtain the remainder corresponding to the key of each data in the first computing node; wherein, the target value is the number of multiple groups; group the remainders corresponding to the keys of each data in the first computing node; keys with the same remainder belong to the same group.
  • the grouping module 604 is specifically used to: group the keys of each data of the first computing node according to multiple key group information; wherein, a key group information is used to indicate at least one key included in a group.
  • the target group among the multiple groups is the partition data of the first computing node; the data volume of the target group on the first computing node is greater than or equal to the data volume of the target group on non-first computing nodes.
  • the partitioning module 601 is specifically used to: determine the group associated with each computing node from multiple groups; wherein the amount of data of the target group associated with the first computing node on the first computing node is greater than or equal to the amount of data of the target group on non-first computing nodes; poll the allocated partition data for each computing node from the groups associated with each computing node; wherein, in each round of polling, if the allocated data of the first computing node is less than or equal to the allocated data of non-first computing nodes, partition data is allocated for the first computing node from the groups associated with the first computing node.
  • the partitioning module 601 is further configured to: allocate partition data to the first computing node from groups not associated with the first computing node, provided that computing nodes have been allocated to the groups associated with the first computing node.
  • the partitioning module 601 is specifically used to: allocate partition data for each computing node from multiple groups; and allocate the target group to the first computing node such that the amount of data on the first computing node is greater than or equal to the amount of data on non-first computing nodes.
  • the grouping module 604 is further configured to: group each data of the first computing node to obtain each group of the first computing node; the partitioning module 601 is specifically configured to: obtain the data volume of each group of each computing node to obtain the data volume of multiple groups of multiple computing nodes; and partition the multiple groups of multiple computing nodes according to the shuffle strategy and the data volume of multiple groups of multiple computing nodes.
  • the grouping module 604 is specifically used to: treat the data of each partition of the first computing node as each group of the first computing node.
  • the grouping module 604 is specifically used to: determine the key of each data of the first computing node; the first computing node groups the keys of each data of the first computing node; wherein some different keys belong to the same group.
  • the grouping module 604 is specifically used to: determine the hash value of each data of the first computing node; the first computing node takes the remainder of the hash value of each data of the first computing node modulo a target value to obtain the remainder corresponding to each data of the first computing node; wherein, the target value is the number of multiple groups; the first computing node groups the remainders corresponding to each data of the first computing node; data with the same remainder belong to the same group, and data with different remainders belong to the same group.
  • partitioning module 601, writing module 602, transmission module 603, and packetizing module 604 can all be implemented in software or in hardware.
  • the implementation of partitioning module 601 will be described below.
  • the implementation of writing module 602, transmission module 603, and packetizing module 604 can refer to the implementation of partitioning module 601.
  • the partition module 601 may include code running on a computing instance.
  • the computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance may be one or more.
  • partitioning module 601 may include code running on multiple hosts/virtual machines/containers. It should be noted that the multiple hosts/virtual machines/containers used to run this code can be distributed within the same region or in different regions. Furthermore, the multiple hosts/virtual machines/containers used to run this code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.
  • VPC Virtual Private Cloud
  • multiple hosts/virtual machines/containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs.
  • VPC Virtual Private Cloud
  • a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
  • partition module 601 may include at least one computing device, such as a server.
  • partition module 601 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL generic array logic
  • the partitioning module 601 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the partitioning module 601 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the partitioning module 601 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
  • the partitioning module 601 can be used to execute any step in the data shuffling method
  • the writing module 602 can be used to execute any step in the data shuffling method
  • the transmission module 603 can be used to execute any step in the data shuffling method
  • the grouping module 604 can be used to execute any step in the data shuffling method.
  • the steps implemented by the partitioning module 601, the writing module 602, the transmission module 603, and the grouping module 604 can be specified as needed. By implementing different steps in the data shuffling method through the partitioning module 601, the writing module 602, the transmission module 603, and the grouping module 604, all functions of the data shuffling device can be realized.
  • the computing device 700 includes: a bus 702, a processor 704, a memory 706, and a communication interface 708.
  • the processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702.
  • the computing device 700 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.
  • Bus 702 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 7, but this does not imply that there is only one bus or one type of bus. Bus 702 can include pathways for transmitting information between various components of computing device 700 (e.g., memory 706, processor 704, communication interface 708).
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • Processor 704 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • the memory 706 may include volatile memory, such as random access memory (RAM).
  • RAM random access memory
  • the processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
  • ROM read-only memory
  • HDD hard disk drive
  • SSD solid state drive
  • the memory 706 stores executable program code, which the processor 704 executes to implement the functions of the aforementioned partitioning module 601, writing module 602, transmission module 603, and grouping module 604, thereby realizing the data shuffling method.
  • the memory 706 stores instructions for executing the data shuffling method.
  • the communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.
  • transceiver modules such as, but not limited to, network interface cards and transceivers
  • the computing device cluster includes at least one computing device.
  • the computing device can be a server, such as a central server, an edge server, or a local server in a local data center.
  • the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
  • the computing device cluster 800 includes at least one computing device 700.
  • the memory 706 of one or more computing devices 700 in the computing device cluster 800 may store the same instructions for performing the data shuffling method.
  • the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data shuffling method.
  • a combination of one or more computing devices 700 can jointly execute the instructions for executing the data shuffling method.
  • the memory 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data shuffling device. That is, the instructions stored in the memory 706 of different computing devices 700 can implement the functions of one or more modules among the partitioning module 601, writing module 602, transmission module 603, and grouping module 604.
  • one or more computing devices in a computing device cluster can be connected via a network.
  • This network can be a wide area network (WAN) or a local area network (LAN), etc.
  • Figure 9 illustrates one possible implementation.
  • two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.
  • the memory 706 in computing device 700A stores instructions for executing the functions of partitioning module 601.
  • the memory 706 in computing device 700B stores instructions for executing the functions of writing module 602, transmission module 603, and packet module 604.
  • connection method between the computing device clusters shown in Figure 9 can be considered as follows: taking into account that the data shuffling method provided in this application requires a large amount of computation, the functions implemented by the writing module 602, the transmission module 603 and the grouping module 604 are to be performed by the computing device 700B.
  • computing device 700A shown in Figure 9 can also be performed by multiple computing devices 700.
  • computing device 700B can also be performed by multiple computing devices 700.
  • This application also provides another computing device cluster.
  • the connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 8 and 9.
  • the difference is that the memory 706 of one or more computing devices 700 in this computing device cluster can store the same instructions for executing the data shuffling method.
  • the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data shuffling method.
  • a combination of one or more computing devices 700 can jointly execute the instructions for executing the data shuffling method.
  • This application also provides a computer program product containing instructions.
  • the computer program product may be a software or program product containing instructions that can run on a computing device or be stored on any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute the aforementioned data shuffling method.
  • the computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media.
  • the available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).
  • the computer-readable storage medium includes instructions that instruct the computing device to perform a data shuffling method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

公开了一种数据洗牌shuffle方法、装置、设备及集群，涉及数据分析技术领域。该方法中，在需要对多个计算节点上的多个数据进行shuffle时，每个计算节点从老分区对应的存储空间读取本计算节点的分区数据后，直接将读取到的分区数据写入新分区对应的存储空间。而从老分区对应的存储空间读取到其他计算节点的分区数据后，将读取的分区数据写入本计算节点的磁盘中，之后，向其他计算节点传输磁盘中数据。这样，不仅可以减少向本地磁盘写入的数据量，从而提高磁盘的数据写入效率，还可以减少本地磁盘中需要分发到不同数据分区的数据，从而提高磁盘的数据读取效率，进而能够提高shuffle效率。

Description

数据洗牌shuffle方法、装置、设备及集群
本申请要求于2024年06月13日提交国家知识产权局、申请号为202410764426.8、申请名称为“数据洗牌shuffle方法、装置、设备及集群”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据分析技术领域，尤其涉及一种数据洗牌shuffle方法、装置、设备及集群。
背景技术
当前，数据洗牌shuffle操作是数据分析中的基础核心操作。数据洗牌shuffle操作可以对数据分析系统的不同计算节点上的数据进行数据重分区，以便于将不同计算节点上键相同的数据分配到相同的数据分区，从而对不同计算节点上的数据进行整合和分类。
然而，由于当前数据分析系统的硬件性能有限，导致数据洗牌shuffle效率较低，严重影响了数据分析的效率。因此，如何提升数据分析系统的数据洗牌shuffle效率，成为亟需解决的技术问题。
发明内容
本申请提供了一种数据洗牌shuffle方法、装置、设备及集群，能够有效提升数据洗牌shuffle效率。
第一方面，提供了一种数据洗牌shuffle方法，应用于计算集群，计算集群包括多个计算节点及管理节点，该方法包括：管理节点获取多个计算节点中每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据量，对多个计算节点的多个数据进行分区，得到每个计算节点的分区数据；管理节点向第一计算节点发送分区信息，分区信息用于指示第一计算节点的分区数据；第一计算节点为多个计算节点中的任意一个计算节点；第一计算节点确定第一计算节点的第一内存空间的数据中，属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据；第一计算节点将第一计算节点的第一内存空间中的第一组分区数据，写入第一计算节点的第二内存空间；第二内存空间用于存储第一计算节点的分区数据；第一计算节点将第一计算节点的第一内存空间中的第二组分区数据，写入第一计算节点的第一磁盘，并向非第一计算节点传输第一磁盘中的第二组分区数据。
上述shuffle方案，在需要对多个计算节点上的多个数据进行shuffle时，管理节点获取每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据，对多个计算节点的多个数据进行分区，得到每个计算节点的分区数据，之后，管理节点向每个计算节点(如：第一计算节点)发送分区信息，用于指示每个计算节点的分区数据。每个计算节点接收到分区信息后，根据分区信息，确定第一内存空间中属于本计算节点的第一组分区数据，以及不属于本计算节点的第二组分区数据，然后，将属于本计算节点的第一组分区数据直接写入本计算节点的第二内存空间，将属于其他计算节点的第二组分区数据写入第一磁盘，以从第一磁盘传输给其他计算节点。
由于每个计算节点将属于本计算节点分区数据直接写入本计算节点上用于存储分区数据的内存空间(如：第二内存空间)，一方面，每个计算节点的一部分数据将不再写入本地磁盘，因此，减少了每个计算节点向本地磁盘写入数据的数量，从而提高了每个计算节的磁盘写入效率，进而能够提高数据shuffle效率。另一方面，每个计算节点的属于本计算节点的数据未写入本地磁盘，也就不再需要从每个计算节的本地磁盘写入每个计算节点的本地内存，因此，避免了每个计算节点的本地磁盘与本地内存之间的冗余数据传输，减少了每个计算节点的本地磁盘中需要分发到不同数据分区的数据，从而提高了每个计算节点的磁盘读取效率，进而能够提高数据shuffle效率，避免数据shuffle操作影响数据分析效率。
在一种可能的设计中，第一内存空间和第二内存空间位于第一计算节点的受信任执行环境TEE的内存。这样，上述方案可以在TEE中实现，从而可以对密文数据进行shuffle。
在另一种可能的设计中，第一内存空间和第二内存空间位于第一计算节点的富执行环境REE的内存。这样，上述方案可以在REE中实现，从而可以对明文数据进行shuffle。
在另一种可能的设计中，多个数据的数据类型为键-值key-value数据，该方法还包括：第一计算节点对第一计算节点的每个数据的键进行分组，得到第一计算节点的每个分组；管理节点获取多个计算节点中每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据量，对多个计算节点的多个数据进行分区，包括：管理节点获取每个计算节点的每个分组的数据量，得到多个计算节点的多个分组的数据量；管理节点根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区。
该设计中，管理节点需要对多个计算节点的多个数据进行分区时，获取每个计算节点的每个分组的数据量，从而得到多个计算节点的多个分组的数据量。之后，根据shuffle策略以及多个分组的数量，对多个分组进行分区。由于各个计算节点上相同的键属于相同的分组，部分不同的键也属于相同的分组，因此，分组的数量小于多个键的数量。在此基础上，由于每个计算节点上的分组的数量小于键的数量，实现了减少每个计算节点上需要进行分区的元素的数量，从而减少多个计算节点上需要进行分区的元素的数量，这样，便能够提高确定分区的效率，进而提升为多个计算节点的多个数据进行分区的效率。
此外，由于以分组为粒度确定分区时，不同计算节点上的相同分组属于相同的分区，结合每个计算节点将相同的键划分在相同的分组，因此，在以分组为粒度确定分区后，能够保证多个计算节点上键相同的数据全部属于相同的分区，从而有助于保证后续数据分析过程的可靠性和准确性。
在另一种可能的设计中，第一计算节点对第一计算节点的每个数据的键进行分组，包括：第一计算节点确定第一计算节点的每个数据的键的哈希值；第一计算节点将第一计算节点的每个数据的键的哈希值对目标数值取余，得到第一计算节点的每个数据的键对应的余数；其中，目标数值为多个分组的数量值；第一计算节点对第一计算节点的每个数据的键对应的余数进行分组；对应余数相同的键属于相同的分组。这样，有助于保证所确定的分组的准确性。
在另一种可能的设计中，第一计算节点对第一计算节点的每个数据的键进行分组，包括：第一计算节点根据多个键组信息，对第一计算节点的每个数据的键进行分组；其中，一个键组信息用于指示一个分组包括的至少一个键。这样，有助于提高确定分组的方式多样性。
在另一种可能的设计中，多个分组中的目标分组为第一计算节点的分区数据；第一计算节点上的目标分组的数据量大于或等于非第一计算节点上的目标分组的数据量。这样，有助于减少跨节点传输的数量，从而提高shuffle效率。
在另一种可能的设计中，管理节点根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区，包括：管理节点从多个分组中，确定每个计算节点关联的分组；其中，第一计算节点关联的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量；管理节点从每个计算节点关联的分组中，轮询为每个计算节点的分配分区数据；其中，在每轮轮询过程中，若第一计算节点的已分配数据量小于或等于非第一计算节点的已分配数据量，从第一计算节点关联的分组中，为第一计算节点分配分区数据。这样，有助于均衡不同计算节点上的数据量。
在另一种可能的设计中，在每轮轮询过程中，该方法还包括：管理节点在第一计算节点关联的分组已分配计算节点的情况下，从非第一计算节点关联的分组中，为第一计算节点分配分区数据。这样，有助于均衡不同计算节点上的数据量。
在另一种可能的设计中，管理节点根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区，包括：管理节点从多个分组中，为每个计算节点的分配分区数据；为第一计算节点分配的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量。这样，有助于提供确定shuffle策略的多样性。
在另一种可能的设计中，多个数据的数据类型不是键-值key-value数据，该方法还包括：第一计算节点对第一计算节点的每个数据进行分组，得到第一计算节点的每个分组；管理节点获取多个计算节点中每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据量，对多个计算节点的多个数据进行分区，包括：管理节点获取每个计算节点的每个分组的数据量，得到多个计算节点的多个分组的数据量；管理节点根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区。
在另一种可能的设计中，第一计算节点对第一计算节点的每个数据进行分组，包括：第一计算节点将第一计算节点的每个分区的数据作为第一计算节点的每个分组。
在另一种可能的设计中，第一计算节点对第一计算节点的每个数据进行分组，包括：第一计算节点确定第一计算节点的每个数据的键；第一计算节点对第一计算节点的每个数据的键进行分组；其中，部分不同的键属于相同的分组。
在另一种可能的设计中，第一计算节点对第一计算节点的每个数据进行分组，包括：第一计算节点确定第一计算节点的每个数据的哈希值；第一计算节点将第一计算节点的每个数据的哈希值对目标数值取余，得到第一计算节点的每个数据对应的余数；其中，目标数值为多个分组的数量值；第一计算节点对第一计算节点的每个数据对应的余数进行分组；对应余数相同的数据属于相同的分组，对应余数不同的部分数据属于相同的分组。
第二方面，提供了一种数据洗牌shuffle装置，该装置包括：用于执行第一方面提供的任意一种方法的功能单元，各个功能单元所执行的动作通过硬件实现或通过硬件执行相应的软件实现。例如，该数据洗牌shuffle装置可以包括：分区模块、写入模块、传输模块；分区模块，用于获取多个计算节点的数据量信息，并根据shuffle策略及多个计算节点的数据量信息，对多个计算节点的多个数据进行分区，得到多个计算节点中每个计算节点的分区数据；分区模块，还用于向第一计算节点发送分区信息，分区信息用于指示第一计算节点的分区数据；第一计算节点为多个计算节点中的任意一个计算节点；写入模块，用于确定第一计算节点的第一内存空间的数据中，属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据；写入模块，用于将第一计算节点的第一内存空间中的第一组分区数据，写入第一计算节点的第二内存空间；第二内存空间用于存储第一计算节点的分区数据；写入模块，用于将第一计算节点的第一内存空间中的第二组分区数据，写入第一计算节点的第一磁盘；传输模块，用于向非第一计算节点传输第一磁盘中的第二组分区数据。
第三方面，提供了一种计算集群，计算集群包括多个计算节点和管理节点。管理节点，用于获取多个计算节点中每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据量，对多个计算节点的多个数据进行分区，得到每个计算节点的分区数据；管理节点，还用于向第一计算节点发送分区信息，分区信息用于指示第一计算节点的分区数据；第一计算节点为多个计算节点中的任意一个计算节点；第一计算节点，还用于确定第一计算节点的第一内存空间的数据中，属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据；第一计算节点，还用于将第一计算节点的第一内存空间中的第一组分区数据，写入第一计算节点的第二内存空间；第二内存空间用于存储第一计算节点的分区数据；第一计算节点，还用于将第一计算节点的第一内存空间中的第二组分区数据，写入第一计算节点的第一磁盘，并向非第一计算节点传输第一磁盘中的第二组分区数据。
在第三方面，管理节点还可以用于执行上述第一方面中的管理节点所执行步骤，第一计算节点还可以用于执行上述第一方面中的第一计算节点所执行的步骤，此处不再赘述。
第四方面，提供了一种处理器，处理器可以用于执行上述第一方面提供的任意一种方法的步骤。
第五方面，提供了一种芯片，包括：处理器和供电电路；供电电路可以用于为芯片供电；处理器可以用于执行上述第一方面提供的任意一种方法的操作步骤。
第六方面，提供了一种计算设备，包括：处理器、存储器及存储在存储器上的计算机程序/指令；处理器执行计算机程序，以使计算设备执行上述第一方面提供的任意一种方法的步骤。
第七方面，提供了一种计算设备集群，包括：至少一个计算设备，至少一个计算设备中的每个计算设备包括处理器和存储器；每个计算设备包括处理器、存储器及存储在存储器上的计算机程序/指令；处理器执行计算机程序/指令，以使计算设备集群实现上述第一方面提供的任意一种方法的步骤，或者，实现上述第三方面提供的任意一种方法的步骤。
第八方面，提供了一种包括指令的计算机程序产品，包括计算机程序/指令，计算机程序/指令被处理器执行时实现上述第一方面提供的任意一种方法的步骤。
第九方面，提供了一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序/指令，计算机程序/指令被处理器执行时实现上述第一方面提供的任意一种方法的步骤。
其中，第二方面至第九方面中任一种实现方式所带来的技术效果，可以参见上述第一方面中不同设计所带来的技术效果，此处不再赘述。
附图说明
图1为本申请提供的相关技术中数据洗牌的示意图；
图2为本申请提供的一种数据分析系统的架构图；
图3为本申请提供的一种目标软件程序的模块示意图；
图4为本申请提供的一种数据洗牌方法的流程图；
图5为本申请提供的一种数据洗牌方法的示意图；
图6为本申请提供的一种数据洗牌装置的示意图；
图7为本申请提供的一种计算设备的示意图；
图8为本申请提供的计算设备集群的示意图；
图9为本申请提供的一种计算设备集群的连接示意图。
具体实施方式
为了便于理解，首先对本申请涉及的相关术语进行简要介绍。
富执行环境(rich execution environment，REE)：也可以称为普通执行环境，是指操作系统(operating system，OS)运行时的环境。REE是一个容易受到攻击的开放环境。
在本申请中，在REE中运行的数据分析系统，可以用于分析明文数据。
受信任执行环境(trusted execution environment，TEE)：是一种硬件支持的安全隔离环境，可以与REE并存运行。TEE内的数据是与外部隔离的，TEE外的代码无法读取或篡改TEE内的数据，基于此，TEE内可以对密文数据进行明文计算，从而对密文数据进行处理。
在本申请中，在TEE中运行的数据分析系统，可以用于分析密文数据。
数据分区：也可以称为简称为分区，是指具有相同属性的数据构成的组。如：1月份每天的温度数据可以构成一个数据分区，2月份每天的降雨量可以一个数据分区。
在本申请中，为了便于描述，将执行shuffle前的数据分区称为老分区或老数据分区，执行shuffle后得到数据分区称为新分区或新数据分区。
下面将结合附图对本申请所提供的技术方案进行详细介绍。
当前，数据shuffle操作是数据分析中的基础核心操作。数据shuffle操作可以对数据分析系统的不同计算节点上的数据进行数据重分区，以便于将不同计算节点上键相同的数据分配到相同的数据分区。
示例性的，如图1所示，数据分析系统包括计算节点a和计算节点b，计算节点a包括内存a和硬盘a，计算节点b包括内存b和硬盘b，内存a的内存空间a1存储有数据a1和数据a2，内存b的内存空间b1存储有数据b1和数据b2。
数据分析系统对数据a1、数据a2、数据b1和数据b2执行shuffle操作时，先根据数据a1、数据a2、数据b1和数据b2中每个数据的键确定分区策略。基于数据a1的键和数据b1的键相同，数据a2和数据b2的键相同，分区策略指示数据a1和数据b1属于新分区a和计算节点a，数据a2和数据b2属于新分区b和计算节点b。其中，内存a的内存空间a2用于存储新分区a的数据，内存b的内存空间b2用于存储新分区b的数据。
之后，基于分区策略，计算节点a将数据a1和数据a2从内存空间a1写入磁盘a，并将磁盘a中的数据a1写入内存空间a2，将磁盘a中的数据a2发送给计算节点b，由计算节点b将数据a2写入内存空间b2。计算节点b将内存空间b1中的数据b1和数据b2写入磁盘b，将磁盘b中的数据b2写入内存空间b2，并将磁盘b中的数据b1发送给计算节点1，以由计算节点1写入内存空间a2。
然而，由于当前数据分析系统的硬件性能有限，导致数据shuffle效率较低，严重影响了数据分析的效率。因此，如何提升数据洗牌效率，成为亟需解决的技术问题。
有鉴于此，本申请提供了一种数据洗牌方法，应用于计算集群，计算节点包括多个计算节点，在需要对多个计算节点上的多个数据进行shuffle时，管理节点获取每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据，对多个计算节点的多个数据进行分区，得到每个计算节点的分区数据，之后，管理节点向每个计算节点发送分区信息，用于指示每个计算节点的分区数据。每个计算节点接收到分区信息后，根据分区信息，确定第一内存空间中属于本计算节点的第一组分区数据，以及不属于本计算节点的第二组分区数据，然后，将属于本计算节点的第一组分区数据直接写入本计算节点的第二内存空间，将属于其他计算节点的第二组分区数据写入第一磁盘，以从第一磁盘传输给其他计算节点。
由于每个计算节点将属于本计算节点分区数据直接写入本计算节点上用于存储分区数据的内存空间(如：第二内存空间)，一方面，每个计算节点的一部分数据将不再写入本地磁盘，因此，减少了每个计算节点向本地磁盘写入数据的数量，从而提高了每个计算节的磁盘写入效率，进而能够提高数据shuffle效率。另一方面，每个计算节点的属于本计算节点的数据未写入本地磁盘，也就不再需要从每个计算节的本地磁盘写入每个计算节点的本地内存，因此，避免了每个计算节点的本地磁盘与本地内存之间的冗余数据传输，减少了每个计算节点的本地磁盘中需要分发到不同数据分区的数据，从而提高了每个计算节点的磁盘读取效率，进而能够提高数据shuffle效率，避免数据shuffle操作影响数据分析效率。
接下来，结合附图进一步节点本申请提供的技术方案所涉及的系统架构。
在硬件方面，本申请提供了一种数据分析系统，该数据分析系统可以用于执行上述的数据洗牌方法，从而在不增加硬件成本的情况下，提高数据shuffle效率，避免数据shuffle操作影响数据分析效率，进而提高数据分析系统的整体性能。
需要说明的，本申请对数据分析系统的名称不做限制，如：数据分析系统也可以叫做计算集群，
本申请中，数据分析系统可以包括多个计算节点。其中，多个计算节点中的不同计算节点之间可以通信，多个计算节点中的每个计算节点用于执行计算任务。示例性的，计算任务可以包括数据加密、数据解密、数据排序、数据聚合、数据洗牌等数据分析任务。
需要说明的，在本申请中，多个是指两个或者两个以上，后续不再赘述。
在此基础上，数据分析系统还可以包括管理节点，管理节点用于执行资源调度任务和管理计算任务。
示例性的，资源调度任务可以包括确定数据分区的数量等。管理计算任务可以包括监测每个计算节点上的计算任务的执行状态等。
在一个示例中，管理节点可以多个计算节点中的任意一个计算节点。在另一个示例中，管理节点可以多个计算节点之外的计算设备。在管理节点为多个计算节点之外的计算设备的情况下，管理节点与多个计算节点中的每个计算节点之间可以通信。
以下，以管理节点为多个计算节点之外的计算设备为例，对本申请进行示例性说明。
在本申请中，管理节点可以用于确定shuffle操作的分区策略，如：确定每个计算节点的分区数据等。计算节点可以用于将属于本计算节点的分区数据直接写入本计算节点的内存空间、确定分组等。
需要说明的，管理节点所执行的确定分区策略的操作，也可以由计算节点执行，本申请对此不做限制。以下，以确定分区策略的操作由管理节点执行为例，对本申请进行示例性介绍。
需要说明的，管理节点和多个计算节点之间可以认为是集群关系或分布式计算集群等。
本申请中，数据分析系统可以提供REE。在此基础上，数据分析系统可以在REE中执行本申请提供的数据洗牌方法，以便于对明文数据执行数据洗牌操作。
示例性的，管理节点和每个计算节点可以提供REE。数据shuffle方法可以在管理节点和每个计算节点提供的REE中执行。基于此，管理节点的处理器和每个计算节点的处理器可以包括第一处理单元，第一处理单元可以用于提供REE。在此基础上，每个计算节点可以包括第一内存，第一内存可以用于存储每个计算节点提供的REE中的数据，如：待执行shuffle操作的明文数据等。管理节点可以包括第三内存，第三内存可以用于存储管理节点提供的REE中的数据，如：待执行shuffle操作的明文数据的分区策略等。
其中，每个计算节点的第一内存及管理节点的第三内存可以称为REE内存，后续不再赘述。
本申请中，数据分析系统可以提供TEE。在此基础上，数据分析系统可以在TEE中执行本申请提供的数据洗牌方法，以便于对密文数据执行数据洗牌操作。
示例性的，管理节点和每个计算节点可以提供TEE。数据shuffle方法可以在管理节点和每个计算节点提供的TEE中执行。基于此，管理节点的处理器和每个计算节点的处理器可以还包括第二处理单元，第二处理单元可以用于提供REE。例如，第二处理单元可以包括ARM TrustZone、CCA等。
需要说明的，本申请对第二处理单元的类型不做限制，以上仅为示例性说明。
在此基础上，每个计算节点可以包括第二内存，第二内存可以用于存储每个计算节点提供的TEE中的数据，如：待执行洗牌操作的密文数据等。管理节点可以包括第四内存，第四内存可以用于存储管理节点提供的TEE中的数据，如：待执行洗牌操作的密文数据的分区策略等。
其中，每个计算节点的第二内存及管理节点的第四内存可以称为TEE内存，后续不再赘述。
需要说明的，第一内存、第二内存、第三内存以及第四内存的类型、型号、存储容量等参数，可以相同，或者，也可以不同，本申请对此不做限制。
需要说明的，本申请对管理节点和每个计算节点提供REE和TEE的方式不做限制。
可选地，每个计算节点还可以包括磁盘。其中，每个计算节点的磁盘可以存储每个计算节点的shuffle文件，每个计算节点shuffle文件可以包括本计算节点上的待传输数据。例如，待传输数据可以是其他计算节点的分区数据。
示例性的，磁盘可以是固态硬盘(solid state disk/drive，SSD)、机械硬盘(hard disk drive，HDD)、混合硬盘(hybrid hard disk，HHD)等。
需要说明的，本申请对磁盘的类型不做限制，以上仅为示例性说明。
本申请中，计算节点可以是计算设备，该计算设备可以用于执行计算任务。
可选地，计算设备可以为终端设备或网络设备。
其中，终端设备可以包括超级移动个人计算机(ultra-mobile personal computer，UMPC)、笔记本电脑、上网本、台式计算机、一体机等。
需要说明的，本申请对终端设备的设备形态不做限制，以上仅为示例性说明。
其中，网络设备可以包括服务器等。服务器可以是一个物理服务器，或者，也可以是两个或两个以上分担不同职责的物理服务器、相互协同来实现服务器的各项功能。示例性的，服务器可以是刀片服务器、高密服务器、机架服务器或塔式服务器等。
需要说明的，本申请对网络设备的设备形态不做限制，以上仅为示例性说明。
本申请中，管理节点也可以是计算设备，该计算设备可以用于执行资源调度任务和管理计算任务等。
其中，管理节点的设备类型和计算节点的设备类型可以相同，或，也可以不同，本申请对此不做限制。
图2为本申请提供的一种数据分析系统的架构示意图。
如图2所示，该数据分析系统可以包括管理节点、计算节点1和计算节点2，管理节点与计算节点1、计算节点2之间可以通信，计算节点1和计算节点2也可以通信。
示例性的，管理节点与计算节点1、计算节点2可以通过控制信号通信。管理节点可以通过控制信号向计算节点1、计算节点2传输任务请求，从而指示计算节点1和计算节点2执行数据shuffle方法。
本申请中，管理节点、计算节点1和计算节点2均可以提供TEE和REE。
在此基础上，计算节点1包括磁盘1核、REE内存1(图2中未示出)、TEE内存1(图2中未示出)。其中，REE内存1用于存储计算节点1提供的REE中的数据，TEE内存1用于存储计算节点1提供的TEE中的数据，磁盘1用于存储计算节点1上的待传输数据。例如，待传输数据可以为计算节点2的分区数据。
第二计算节点2包括磁盘2、REE内存2(图2中未示出)、TEE内存2(图2中未示出)。其中，REE内存2用于存储计算节点2提供的REE中的数据，TEE内存2用于存储计算节点2提供的TEE中的数据，磁盘2用于存储计算节点2上的待传输数据。例如，待传输数据可以为计算节点1的分区数据。
管理节点包括磁盘3、REE内存3(图2中未示出)、TEE内存3(图2中未示出)。其中，REE内存3用于存储管理节点提供的REE中的数据，TEE内存3用于存储管理节点提供的TEE中的内存，磁盘3可以用于存储管理节点接收到的数据等。
需要说明的，本申请对数据分析系统包括的计算节点的数量不做限制，图2所示的数据分析系统包括两个计算节点(即计算节点1和计算节点2)仅为示例性说明。
可选地，图2所示的数据分析系统可以是云服务平台提供的系统。基于此，云服务平台提供的云计算可以包括本申请提供的数据洗牌方法。
示例性的，用户可以通过终端设备与云服务平台提供的数据分析系统通信，以指示云服务平台提供的数据分析系统执行本申请提供的数据洗牌方法。
可选地，终端设备可以包括个人计算机(personal computer，PC)、个人数字助理(personal digital assistant，PDA)、超级移动个人计算机(ultra-mobile personal computer，UMPC)、笔记本电脑、上网本、台式计算机或一体机等。
需要说明的，本申请对终端设备的设备形态不做限制，以上仅为示例性说明。
示例性的，图2所示的云服务平台可以是公有云(Public Cloud)、私有云(Private Cloud)、混合云(Hybrid Cloud)等。
需要说明的，图2所示的系统架构，不构成对执行本申请提供的数据洗牌方法的系统架构的限制。
在软件方面，本申请提供了一种计算引擎，如：Apache Spark，计算引擎可以用于执行数据分析任务。
本申请中，计算引擎可以包括驱动器(如：spark driver)和执行器(如：spark executor)。其中，驱动器可以部署在管理节点上，管理节点可以通过运行驱动器实现执行资源调度任务、管理计算任务等。执行器可以部署在每个计算节点上，每个计算节点可以通过运行执行器实现执行计算任务等。
在一个示例中，计算引擎可以部署在数据分析系统提供的TEE中，如：驱动器部署在管理节点提供的TEE，执行器部署在每个计算节点提供的TEE，从而实现对密文数据进行数据分析。
在另一个示例中，计算引擎可以部署在数据分析系统提供的REE中，如：驱动器部署在管理节点提供的REEE，执行器部署在每个计算节点提供的REE，从而实现对明文数据进行数据分析。
在软件方面，本申请还提供了一种目标软件程序，该目标软件程序可以用于实现本申请提供的数据洗牌方法。该目标软件程序可以部署在数据分析系统上，数据分析系统可以通过运行该目标软件程序执行本申请提供的数据洗牌方法。
示例性的，该目标软件程序可以嵌入在计算引擎中。例如，目标软件程序可以为计算引擎的库文件，基于此，数据分析系统运行计算引擎的过程中，计算引擎可以调用目标软件程序，使得数据分析系统运行目标软件程序，从而执行本申请提供的数据洗牌方法。例如，管理节点可以通过运行目标软件程序实现确定分区策略等，如：确定每个计算节点的分区数据等，每个计算节点可以通过运行目标软件实现将本计算节点上的待传输数据传输给其他计算节点，本计算节点的分区数据直接从老分区的内存空间写入新分区的内存空间等。
需要说明的，本申请对目标软件程序的名称不做限制，如：目标软件程序可以叫做数据shuffle软件、数据shuffle应用程序、数据shuffle装置等。此外，目标软件程序可以是软件、应用程序或组件等。
在一个示例中，目标软件程序可以部署在数据分析系统提供的TEE中，如：部署在管理节点和每个计算节点提供的TEE中，从而用于对密文数据进行数据洗牌。在另一个示例中，目标软件程序可以部署在数据分析系统提供的REE中，如：部署在管理节点和每个计算节点提供的REE中，从而用于对明文数据进行数据洗牌。
图3为本申请提供的一种目标软件程序的模块示意图。
如图3所示，本申请提供的目标软件程序可以包括分区模块、洗牌写入(ShuffleWrite)模块和洗牌读取(ShuffleRead)模块。分区模块可以用于确定多个数据的分区策略等。洗牌写入模块可以用于根据分区策略对数据集进行裁剪，得到第一组分区数据和第二组分区数据，第一组分区数据为属于的本计算节点的数据分区，第二组分区数据为属于其他计算节点的数据分区，之后，将第一组分区数据直接从老分区的内存空间写入新分区的内存空间。洗牌读取模块可以用于根据分区策略进行数据传输，如：将第二组分区数据写入每个计算节点的本地磁盘，并将每个计算节点的本地磁盘中的第二组分区数据传输给其他计算节点。
需要说明的，洗牌写入模块可以简称为写入模块，洗牌读取模块可以称为传输模块，后续不再赘述。
需要说明的，本申请描述的系统架构以及应用场景是为了更加清楚的说明本申请的技术方案，并不构成对于本申请提供的技术方案的限定，本领域普通技术人员可知，随着系统架构的演变和新应用场景的出现，本申请提供的技术方案对于类似的技术问题，同样适用。
为了便于理解，以下结合上述系统架构以及附图，对本申请提供的数据洗牌方法进行示例性介绍。
图4为本申请提供的一种数据洗牌方法的流程图。示例性的，该方法可以包括以下步骤401-步骤407。
需要说明的，本申请中的“步骤”可以简写为“S”，后续不再赘述。
本申请中，图4所示的数据洗牌方法可以用于处理密文数据或者明文数据。
示例性的，目标用户获取一个用户或多个用户提供的待分析数据，并通过终端设备向数据分析系统发送存储请求，存储请求用于请求存储待分析数据。数据分析系统接收到存储请求后，可以将待分析数据存储在数据分析系统的多个计算节点，其中，不同计算节点存储的待分析数据不同。
需要说明的，本申请对目标用户与待分析数据的关系不做限制，如：目标用户可以是待分析数据的所有者，或者，也可以不是待分析数据的所有者。此外，不同计算节点上存储的数据量(即待分析数据的数量)可以相同，或者，也可以不同，本申请对此也不做限制。
示例1，结合图2，目标用户可以通过终端设备向管理节点发送存储请求，管理节点接收到存储请求后，可以将待分析数据中的一部分数据存储在计算节点1，并将待分析数据中的另一部分数据存储在的计算节点2。
示例2，结合图2，目标用户可以通过终端设备可以向计算节点1发送第一存储请求，用于请求存储待分析数据中的一部分数据，计算节点1可以基于第一存储请求，存储待分析数据中的一部分数据。目标用户还可以通过终端设备向计算节点2发送第二存储请求，用于请求存储待分析数据的另一部分数据，计算节点2可以基于第二存储请求，存储待分析数据中的另一部分数据。
需要说明的，以下以示例2为例，对本申请进行示例性介绍。
以下，对待分析数据的类型进行示例性介绍。
示例a，待分析数据为明文数据，待分析数据包括明文数据1和明文数据2。
结合图2，目标用户可以通过终端设备向计算节点1发送存储请求1，存储请求1用于请求存储明文数据1，并向计算节点2发送存储请求2，存储请求2用于请求存储明文数据2。计算节点1基于接收到的存储请求1，将接收到的明文数据1存储在磁盘1中，之后，将磁盘1中的明文数据1写入REE内存1，以便于在第一REE中对明文数据1进行数据分析。计算节点2基于接收到的存储请求2，将接收到的明文数据2存储在磁盘2中，之后，将磁盘2中的明文数据2写入REE内存2中，以便于在第二REE中对明文数据2进行数据分析。
需要说明的，之所以将待分析数据分为明文数据1和明文数据2，是为了区分不同计算节点上的待分析数据。此外，本申请对明文数据1和明文数据2包括的数据量不做限制。
示例b，待分析数据为密文数据，待分析数据包括密文数据1和密文数据2。
结合图2，目标用户可以通过终端设备向计算节点1发送存储请求3，存储请求3用于请求存储密文数据1，并向计算节点2发送存储请求4，存储请求4用于请求存储密文数据2。计算节点1基于接收到的存储请求3，将接收到的密文数据1存储在磁盘1中，之后，将磁盘1中的密文数据1写入TEE内存1，以便于在第一TEE中对密文数据1进行数据分析。计算节点2基于接收到的存储请求4，将接收到密文数据2存储在磁盘2中，之后，将磁盘2中的密文数据2写入TEE内存2，以便于在第二TEE中对密文数据2进行数据分析。
需要说明的，之所以将待分析数据分为密文数据1和密文数据2，是为了区分不同计算节点上的待分析数据。此外，本申请对密文数据1和密文数据2包括的数据量不做限制。
以下，以上述示例b为例，对本申请提供的数据洗牌方法对密文数据进行shuffle的过程，进行介绍。
本申请中，计算节点1将密文数据1写入TEE内存1后，可以密文数据1进行分区，以将密文数据1分成多个分区1。由于多个分区1为存储数据阶段的分区，也即，执行shuffle操作前的分区，因此，多个分区1为老分区。为了便于区分不同阶段的分区，以下，将多个分区1称为多个老分区1。同样地，计算节点2可以对密封数据2进行分区，从而得到多个老分区2。
示例性的，计算节点1确定多个老分区1后，可以从TEE内存1上为每个老分区1分配内存空间，以用于存储每个老分区1的数据。同样地，计算节点2确定多个老分区2后，可以从TEE内存2上为每个老分区2分配内存空间，以用于存储每个老分区2的数据。
该实施例中，每个计算节点在存储待分析数据时，如：密文数据1、密文数据2，通过待分析数据分区，从而每个计算节点可以得到多个老分区，这样，每个计算节点在执行shuffle时，可以并对对多个分区执行shuffle，从而有助于提高shuffle效率。
需要说明的，本申请对确定老分区的方式不做限制，如：计算节点1可以根据密文数据1的数据量以及并行计算能力，将密文数据1划分成多个老分区1等。
在此基础上，目标用户需要对待分析数据进行数据分析时，可以通过终端设备向管理节点发送数据分析请求，数据分析请求用于请求对待分析数据进行数据分析。管理节点接收到数据分析请求后，可以解析数据分析请求，以得到数据分析请求指示的至少一个数据分析任务，如：至少一个数据分析任务可以包括目标数据分析任务。之后，管理节点向每个计算节点发送目标任务请求，用于请求执行目标数据分析任务。
需要说明的，本申请对数据分析请求所指示的数据分析任务的数量、不同数据分析任务的排序等均不做限制。
本申请中，在需要对多个计算节点上的多个数据(即待分析数据)进行shuffle时，管理节点和每个计算节点可以通过本申请提供的数据洗牌方法，对多个计算节点上的待分析数据执行shuffle操作。
本申请中，多个数据的数据类型可以为键-值key-value数据(以下简写为键值数据)，或者，也可以为非键值数据。以下，以多个数据的数据类型为键值数据为例，对本申请进行介绍。
在多个数据中，一部分数据的键相同，另一部分数据的键不同。基于此，多个数据可以包括多个键，其中，多个键中的每个键可以对应至少一个数据。
示例性的，多个键包括城市a和城市b，其中，城市a对应的至少一个数据包括多个数据中键为城市a的数据，城市b对应的数据包括多个数据中键为城市b的数据。
示例性的，多个计算节点可以包括第一计算节点，第一计算节点可以是多个计算节点中的任意一个计算节点。以下，以第一计算节点为例，对每个计算节点执行数据洗牌方法的过程进行示例性介绍。
需要说明的，非第一计算节点执行数据洗牌方法的过程，可以参考下述的第一计算节点执行数据洗牌方法的过程，后续不再赘述。其中，非第一计算节点是指多个计算节点中除第一计算节点的其他计算节点。
为了便于描述，以将非第一计算节点称为第二计算节点，后续不再赘述。
本申请中，第一计算节点包括第一TEE内存，第一TEE内存中存储有待分析数据，该待分析数据即需要执行shuffle操作的数据。
示例性的，该待分析数据被分成多个老分区。其中，多个老分区包括第一老分区，第一老分区可以是多个老分区中的任意一个老分区。以下，第一老分区的数据为例，对第一计算节点执行shuffle操作的过程进行示例性介绍。例如，第一老分区对应的内存空间为第一内存空间，也就是说，第一内存空间用于存储第一老分区的数据。
该实施例中，第一计算节点可以多个老分区并行执行shuffle操作，并对每个老分区的数据进行分组、并行传输每个老分区的数据等，从而提供shuffle操作的效率。
步骤401：多个计算节点中的每个计算节点向管理节点发送每个计算节点的数据量。
可选地，对于每个计算节点的数据量可以包括多种情况，以下，以第一计算节点为例，通过情况1和情况2，对每个计算节点的数据量的多种情况进行示例性说明。
情况1：第一计算节点的数据量可以为第一计算节点的每个键对应的数据量。
其中，每个键对应的数据量是指每个键对应的至少一个数据的数量。为了便于描述，每个键对应的数据量也可以称为每个第一键的数据量。
为了区分不同于计算节点上的数据和键，以下，将第一计算节点上的数据称为第一数据，第一计算节点上的数据的键称为第一键。
示例性的，第一计算节点包括多个第一数据，多个第一数据为键值数据。多个第一数据可以包括多个第一键，该多个第一键中的每个第一键不同。在需要进行shuffle操作时，第一计算节点可以将管理节点发送每个第一键对应的数据量，以便于管理节点可以根据每个第一键的数据量，对每个第一键对应的至少一个数据进行分区。
该实施例中，每个计算节点通过向管理节点发送键对应的数据量，这样，有助于减少每个计算需要执行的操作，从而提高每个计算节点向管理节点发送数据量的效率。
情况2：第一计算节点数据量可以为第一计算节点的每个分组对应的数据量。
其中，每个分组对应的数据量是指每个分组对应的至少一个数据的数量。每个分组对应的至少一个数据是指每个第一分组包括的每个第一键对应的至少一个数据。为了便于描述，每个第一分组对应的数据量也可以称为每个第一分组的数据量。
示例性的，如图4所示，第一计算节点的SKa分组的数据量为5，SKb分组的数据量为。第一计算节点向管理节点第一数据量信息，第一数据量信息指示第一计算节点的SKa分组的数据量为5，SKb分组的数据量为3。
为了区别不同计算节点上的分组，以下，将第一计算节点上的分组称为第一分组。
示例性的，第一计算节点包括多个第一分组，其中，多个第一分组的数量小于多个第一键的数量。在需要进行shuffle操作时，第一计算节点可以将管理节点发送每个第一分组的数据量，以便于管理节点可以根据每个第一分组的数据量，对每个第一分组对应的至少一个数据进行分区。
该实施例中，由于第一计算节点的多个第一分组的数量小于多个第一键的数量，因此，多个计算节点的多个分组的数量将小于多个计算节点的多个键的数量。在此基础上，每个计算节点通过向管理节点发送每个分组对应的数据量，管理节点以分组为粒度进行分区，能够减少需要分区的元素的数量，从而能够提高shuffle的效率。
本申请中，多个计算节点的多个分组包括多个计算节点的多个键，一个分组包括多个键中的至少一个键。这样，后续以分组为粒度进行分区时，有助于避免遗漏多个键中的部分键，从而有助于保证多个键中每个键都可以分配到新分区。
本申请中，多个计算节点的多个键中，相同的键属于相同的分组。这样，后续以分组为粒度进行分区时，便可以保证每个计算节点上相同的分组可以被分配到同一个数据分区，从而有助于保证后续数据分析过程的可靠性和准确性。
本申请中，多个计算节点的多个分组的数量小于多个计算节点的多个键的数量。这样，后续以分组为粒度进行分区时，相对于以单个键为单位进行分区，能够减少每个计算节点上需要进行分区的元素的数量，从而能够减少整个数据分析系统上需进行分区的元素的数量，这样，便能够提高确定进行分区效率，也即，提升为多个计算节点上的数据进行分区的效率，进而能够提升数据shuffle效率。
可选地，第一计算节点的每个分组可以包括至少一个键对应的键-值数据，也就是说，每个分组可以同时包括键-值数据中键和值。
示例性的，第一计算节点的多个第一分组可以包括多个第一数据，也就是说，多个第一分组可以同时包括多个第一数据的键和多个第一数据的值。
如图4所示，第一计算节点将第一老分区的数据划分为SKa分组(以下称为第一分组1)和SKb分组(以下称为第一分组2)，其中，第一分组1包括键为五角形的数据和键为圆的数据，第一分组1包括5个数据，第一分组2包括键为六边形的数据和键为三角形的数据，第一分组2包括3个数据。第二计算节点将第二老分区的数据划分为SKa分组(第二分组1)和SKb分组(第二分组2)，第二分组1包括3个数据，第二分组2包括5个数据。其中，SKa分组包括五角形和圆，SKb分组包括六边形和三角形。
该实施例中，通过将多个数据划分为多个分组，这样，后续以分组为粒度确定新分区后，可以直接以分组为粒度读写数据，有助于提高数据读写效率，进而有助于提高shuffle效率。
可选地，第一计算节点的部分分组可以包括多个键对应的键-值数据。这样，能够保证第一计算节点的分组的总数量小于键的总数量，从而在以分组为粒度进行分区时，可以减少需要分区的元素的数量，进而提高shuffle效率。
可选地，第一计算节点的每个分组可以包括至少一个键，也就是说，每个分组可以仅包括键值数据的键。示例性的，第一计算节点的多个第一分组可以仅包括多个第一数据的不同第一键。如图4所示，第一分组1可以包括五角形和圆，第一分组2可以包括六边形和三角形。
该实施例中，通过设置每个分组仅包括键，这样，在多个数据中相同键的数量较多时，能够减少分组时的元素的数量，从而有助于提高分组效率，进而有助于提高shuffle效率。
可选地，第一计算节点的部分分组可以包括多个键。这样，能够保证第一计算节点的分组的总数量小于键的总数量，从而在以分组为粒度进行分区时，可以减少需要分区的元素的数量，进而提高shuffle效率。
需要说明的，本申请对多个分组的形式不做限制，以上仅为示例性说明。以下，对第一计算节点确定分组的过程进行示例性介绍。
可选地，数据洗牌方法还可以包括：第一计算节点对第一计算节点的每个数据的键进行分组，得到第一计算节点的每个分组。其中，相同的键属于相同的分组，部分不同的键属于相同的分组。
示例性的，第一计算节点接收到管理节点发送的目标任务请求后，可以对多个第一键进行分组，从而得到第一计算节点的多个第一分组。第一计算节点得到多个第一分组后，向所述管理节点发送第一计算节点的每个第一分组的数据量。例如，如图4所示，第一计算节点对每个数据的键进行分组，得到两个分组，分别为SKa分组和SKb分组，其中，SKa分组的数据量为5，SKb分组的数据量为3。
本申请中，多个计算节点中的不同计算节点可以均根据目标规则，对本计算节点上的键进行分组，这样，不同计算节点上的相同分组对应相同的键，后续以分组为粒度进行分区时，不同计算节点上的相同分组便可以被分配到同一个数据分区，从而能够保证不同计算节点上键相同的数据被分配到相同的数据分区。
该实施例中，每个计算节点在对本计算节点的键进行分组时，将相同的键分配在相同的分组，部分不同的键也分配到相同的分组，这样，后续以分组为粒度进行分区时，由于相同的键在相同的分组，因此，可以保证相同的键被分配到相同的新分区，从而有助于保证后续执行数据分析任务时的准确性和可靠性。另外，由于部分不同的键也分盘到了相同的分组，因此，能够保证每个计算节点的分组的数量小于每个计算节点的键的数量，进而保证多个计算节点的分组的数量小于多个计算节点的键的数量。
本申请中，对键进行分组包括多种实现方式，以下，通过方式1和方式2进行示例性介绍。
可选地，方式1可以包括以下的S1-S3。
S1：第一计算节点确定第一计算节点的每个数据的键的哈希值。
示例性的，第一计算节点接收到管理节点发送的目标任务请求后，可以对多个第一键中的每个第一键执行哈希算法，从而得到每个第一键的哈希值，进而得到多个第一键的哈希值。
需要说明的，本申请对哈希算法的类型不做限制，只要保证相同键的哈希值相同，不同键的哈希值不同即可。
为了区分不同于计算节点上的数据、键及分组，以下将第二计算节点上的数据称为第二数据，第二计算节点上的数据的键称为第二键，第二计算节点上的分组称为第二分组。
示例性的，以计算节点1为第一计算节点，计算节点2为第二计算节点为例，对本申请进行介绍。
如图5所示，计算节点1的多个第一数据为数据1、……数据6，数据1、……数据6依次为(key1，value1)、(key3，value2)、(key1，value3)、(key5，value4)、(key1，value5)、(key1，value)，多个第一数据包括的多个第一键为key1、key3和key5。基于此，第一计算节点可以对key1、key3和key5执行哈希算法，得到key1的哈希值、key3的哈希值和key5的哈希值，如：key1的哈希值、key3的哈希值和key5的哈希值依次为13、14、15。
计算节点2的多个第二数据为数据7、……、数据12，数据7、……、数据12依次为(key2，value17)、(key4，value8)、(key6，value9)、(key2，value10)、(key7，value11)、(key8，value12)，多个第二数据包括的多个第二键为key2、key4、key6、key7和key8。基于此，计算节点2可以对key2、key4、key6、key7和key8执行哈希算法，得到key2的哈希值、key4的哈希值、key6的哈希值、key7的哈希值和key8的哈希值，如：key2的哈希值、key4的哈希值、key6的哈希值、key7的哈希值和key8的哈希值依次为17、18、19、20、24。
S2：第一计算节点将第一计算节点的每个数据的键的哈希值对目标数值取余，得到第一计算节点的每个数据的键对应的余数；其中，目标数值为多个分组的数量值。
其中，键对应的余数是指键的哈希值对目标数值取余所得到的数值。
示例性的，第一计算节点将多个第一键中每个第一键的哈希值对目标数值取余，从而得到每个第一键对应的余数，进而得到多个第一键对应的余数。
可选地，目标数值为多个分组的数量。
该方案中，通过对目标数值(如：目标数值为M)取余数，而非对其他数值取余，这样，可以将多个键对应的余数划分为M个不同值，在此基础上，将对应余数相同的键划分到相同分组，对应余数不同的键划分到不同分组，能够将多个计算节点上的多个键划分为M个分组，从而实现准确控制多个分组的数量，进而能够准确控制shuffle的效率。
可选地，目标数值可以由目标用户提供。这样，有助于提高目标数值的合理性。
示例性的，目标用户可以通过终端设备向管理节点发送目标数值，其中，目标数值用于指示多个分组的数量。管理节点接收到目标数值后，将目标数值转发给第一计算节点，以便于第一计算节点可以根据目标数值，确定多个第一分组。
需要说明的，目标用户可以向管理节点发送数据分析请求时，同时发送目标数值，或者，也可以单独向管理节点发送目标数值，本申请对此不做限制。
可选地，目标数值可以由管理节点确定。这样，有助于提高确定目标数值的方式多样性。
本申请中，管理节点可以根据数据分析请求所指示的数据分析任务，确定目标数值。
示例性的，管理节点存储有数据分析任务与数值的对应关系。在此基础上，管理节点可以根据将当前接收到的数据分析请求指示的数据分析任务对应的数值，确定为目标数值。之后，管理节点可以向第一计算节点发送目标数值。
需要说明的，本申请对确定目标数值的方式不做限制，以上仅为示例性说明。此外，不同数据分析请求对应的目标数值可以相同，或者，也可以不同，本申请对此不做限制。
示例性的，如图5所示，目标数值为4，计算节点1将key1的哈希值、key3的哈希值和key5的哈希值依次对4取余，得到key1对应的余数、key3对应的余数和key5对应的余数，如：key1对应的余数、key3对应的余数和key5对应的余数依次为1、2、3。
计算节点2将key2的哈希值、key4的哈希值、key6的哈希值、key7的哈希值和key8的哈希值依次对4取余，得到key2对应的余数、key4对应的余数、key6对应的余数、key7对应的余数和key8对应的余数，如：key2对应的余数、key4对应的余数、key6对应的余数、key7对应的余数和key8对应的余数依次为1、2、3、0、0。
可选地，多个分组的数量(即目标数值)可以大于多个计算节点的多个数据分区的数量，该多个数据分区是指shuffle操作后的新数据分区，其中，多个新数据分区包括多个分组中每个分组对应的数据分区。这样，可以通过多余的分组平衡不同数据分区的数据量，从而有助于保证不同数据分区的数据量的均衡性，进而有助于充分利用不同计算节点的计算资源。
以下，对多个计算节点的多个数据分区的数量的确定过程，进行示例性介绍。
在一个示例中，目标用户通过终端设备向管理节点发送第一数值，第一数值用于指示shuffle操作的新分区的数量。其中，第一数值可以与数据分析请求同时发送，或，也可以单独发送，本申请对此不做限制。
在另一个示例中，管理节点接收到用户通过终端设备发送的数据分析请求后，根据数据分析请求指示的数据分析任务，确定shuffle操作的新分区的数量。
示例性的，管理节点中存储有数据分析任务与新分区数量的对应关系。在此基础上，管理节点可以根据数据分析请求指示的数据分析任务，以及数据分析任务与新分区数量的对应关系，确定数据洗牌操作的新分区的数量。
需要说明的，本申请对新分区数量的确定方式不做限制，以上仅为示例性说明。
可选地，多个分组的数量等于多个计算节点的多个数据分区的数量。这样，不仅有助于提高确定分组所属数据分区的效率，从而提高确定shuffle的效率，还有助于提高确定分区的多样性。
S3：第一计算节点对第一计算节点的每个数据的键对应的余数进行分组；其中，对应余数相同的键属于相同的分组。
需要说明的，为了便于理解，一个键对应的余数可以称为一个键的余数，后续步骤赘述。
示例性的，在第一分组仅包括第一键的情况下，第一计算节点得到每个第一键的余数后，直接将余数相同的第一键确定为同一个分组，余数不同的第一键确定为不同的分组，从而得到多个第一分组。这样，有助于提高确定多个分组的效率。
如图5所示，key1的余数和key2的余数相同，key3的余数和key4的余数相同，key5的余数和key6的余数相同，key7的余数和key8的余数相同。基于此，计算节点1可以确定key1为分组1，key3为一个分组2，key5为一个分组3。计算节点2可以确定key2为一个分组4，key4为一个分组5，key6为一个分组6，key7和key8为一个分组7。其中，分组1和分组4为相同的分组，分组2和分组5为相同的分组，分组3和分组4为相同分组。
示例性的，在第一分组同时包括第一键和值的情况下，第一计算节点得到每个第一键的余数后，将余数相同的键所对应的数据确定为同一个分组，余数不同的键所对应的数据确定为不同的分组，从而得到多个分组。这样，有助于提高后续传输数据的效率。
如图5所示，计算节点1可以确定数据1、数据3、数据5和数据6为一个分组，数据2为一个分组，数据4为一个分组。计算节点2可以确定数据7和数据10为一个分组，数据8为一个分组，数据9为一个分组，数据11和数据12为一个分组。
本申请中，多个计算节点的多个分组中的不同分组具有不同的标识，多个计算节点上的相同分组具有相同的标识。
可选地，一个分组的标识可以根据一个分组包括至少一个键确定。示例性的，一个分组的标识可以为一个分组包括的至少一个键所构建的超键。
需要说明的，确定分组的标识也可以认为建立分组和标识(即超键)之间的映射关系。
如图5所示，在计算节点1和计算节点2上，key1的余数和key2的余数相同，因此，key1和key2构成一个分组，该分组的标识可以为key1key2(简写为SK1)。key3的余数和key4的余数相同，key3和key4可以构成一个分组，该分组的标识可以为key3key4(简写为SK2)。key5的余数和key6的余数相同，key5和key6可以构成一个分组，该分组的标识可以为key5key6(简写为SK3)。在计算节点2上，key7的余数和key8的余数相同，key7和key8可以构成一个分组，该分组的标识可以为key7key8(简写为SK4)。
该方案中，通过分组中的至少一个键确定分组的标识，不仅能够建立分组中的键与标识之间的关联关系，有助于通过分组的标识了解各个分组中的键，还有助于通过分组的标识判断是否将相同的键划分到相同的分组，进而有助于提高分组的准确性。
需要说明的，本申请对确定标识的方式不做限制，以上仅为示例性说明。如：还可以随机生成不同分组的标识，或，根据预设规则生成不同分组的标识，或，由用户指定不同分组的标识。
上述方式中，确定每个键的哈希值对目标数值的余数后，通过将对应余数相同的键划分到相同分组，对应余数不同的键划分到不同分组，实现确定多个分组。由于相同的键所对应的哈希值相同，哈希值相同的键所对应的余数相同，因此，相同的键所对应的余数相同，从而能够保证相同的键被划分在相同分组。此外，由于哈希值不同的键，所对应的余数也可能相同，如：key1的余数和key2的余数，从而能够将不同的键划分在相同的分组，进而能够使得多个分组的数量小于多个键的数量。
可选地，方式2可以包括以下的S4。
S4：第一计算节点根据多个键组信息，对第一计算节点的每个数据的键进行分组；其中，一个键组信息用于指示一个分组包括的至少一个键。
示例性的，第一计算节点获取到多个键组信息后，根据多个键组信息的一个键组信息，确定一个第一分组，从而得到多个第一分组。
可选地，多个键组信息可以目标用户提供的。目标用户将多个键组信息发送给管理节点，管理节点将多个键组信息转发给每个计算节点。多个键组信息可以与数据分析请求一起发送给管理节点，或，也可以单独发送，本申请对此不做限制。此外，管理节点可以在向每个计算节点发送目标任务请求时，同时发送多个键组信息，或者，也可以单独发送，本申请对此也不做限制。
如图5所示，多个键组信息包括第一键组信息、第二键组信息、第三键组信息和第四键组信息，其中，第一键组信息包括key1和key2，第二键组信息包括key3和key4，第三键组信息包括key5和key6，第四键组信息包括key7和key8。基于此，计算节点1根据第一键组信息，从数据1、……、数据6中筛选出数据1、数据3、数据5和数据6，将数据1、数据3、数据5和数据6确定为计算节点1的SK1分组，其中，数据1的键、数据3的键、数据5的键和6的键均为key1。计算节点2根据第一键组信息，从数据7、……、数据12中筛选出数据7和数据10，将数据7和数据10确定为计算节点2的SK1分组，其中，数据7的键和数据10的键均为key2。
需要说明的，S4的其他相关说明，可以参考上述的S3，此处不再赘述。
上述实施例中，每个计算节点根据多个键组信息，确定本计算节点的每个分组，这样，每个计算节点可以根据一个键组信息指示的键，直接从多个数据中筛选出属于一个分组的键，从而有助于提高确定分组的效率，进而有助于提高shuffle效率。
上述实施例详细介绍了对键组数据进行分组的方案。以下，对非键值数据进行分组的方案进行介绍。需要说明的，以下，仅对两个方案的不同之处进行介绍，对于相同之处，不再赘述。
在一个示例中，第一计算节点可以将每个老分区的数据作为第一计算节点的每个分组。例如，第一计算节点包括老分区1和老分区2，则第一计算将老分区1作为一个分组，将老分区2作为另一个分组。这样，有助于提高，确定分组的效率。
在另一个示例中，第一计算节点可以为第一计算节点的每个数据确定键，从而将非键值数据转换成键值数据。之后，按照键组数据的分组方式，进行分组。这样，每个计算节点可以通过一套方案对两种不同类型的数据进行分组，有助于提高分组的方案的适用范围。
需要说明的，本申请对非键值数据确定键的方式不做限制。例如，可以将非键值数据的目标特征作为非键组数据的键，如：目标特征可以为城市名称、省名称、某个属性的数值等。
在又一个示例中，第一计算节点可以确定每个数据的哈希值，之后，将每个数据的哈希值对目标数值取余，得到每个数据对应的余数。然后，对每个数据对应的余数进行分组，其中，余数相同的数据属于相同的分组。
需要说明的，本申请对确定每个数据的哈希值的方式不做限制。例如，可以是对每个数据的内容执行哈希算法，得到每个数据的哈希值。或者，也可以是每个数据的部分属性的值执行哈希算法，得到每个数据的哈希值。
步骤402：管理节点获取每个计算节点的数据量，并根据shuffle策略及每个计算节点的数据量，对多个计算节点的多个数据进行分区，得到每个计算节点的分区数据。
其中，第一计算节点的分区数据是指第一计算节点的新分区的数据。shuffle策略包括将键相同的数据分配到相同的新分区。
示例性的，如图4所示，管理节点包括第三内存，管理节点接收到每个计算节点的数据量后，将每个计算节点的数据量存储在第三内存中。在一个示例中，本申请的数据洗牌方法用于对密文数据进行shuffle时，第三内存为管理节点的TEE内存。在另一个示例中，本申请的数据洗牌方法用于对明文数据进行shuffle时，第三内存为管理节点的REE内存。
结合上述情况1，管理节点获取每个计算节点的每个键对应的数据量后，根据shuffle策略以及每个计算节点的每个键的数据量，对多个计算节点上的多个键进行分区，从而实现对多个计算节点上的多个数据进行分区。示例性的，各个计算节点的相同的键属于相同的新分区，各个计算节点上的部分不同的键也可以属于相同的新分区，这样，有助于均衡各个计算节点上的数据量。
结合上述情况2，管理节点获取每个计算节点的每个分组对应的数据量后，根据shuffle策略以及每个计算节点的每个分组的数据量，对多个计算节点上的多个分组进行分区，从而实现对多个计算节点上的多个数据进行分区，以得到每个计算节点的分组，进而得到每个计算节点的分区数据。
示例性的，结合图4，第一计算节点确定多个第一分组后，如：SKa分组和SKb分组，向管理节点发送第一计算节点的分组信息，该分组信息包括第一计算节点的每个分组的数据量。例如，SKa分组包括5个数据，SKb分组包括3个数据。
可选地，分组信息可以用于指示分组的标识、分组对应的数据量、分组对应的键等。
示例性的，第一计算节点的分组信息指示分组的标识为SKa分组和SKb分组，SKa分组对应的数据量为5个数据，SKb分组对应的数据量为3个数据。SKa分组对应键为五角形和圆，SKb分组对应的键为六边形和三角形。
该实施例中，通过设置分组信息可以指示标识、数据量、键等，这样，有助于管理节点掌握每个计算节点的分组的详情，从而有助于提高确定每个计算节点的分区数据的准确性和合理性。
本申请中，根据shuffle策略对多个计算节点的多个分组进行分区时，多个计算节点中不同计算节点上的相同分组属于相同的数据分区。如图4所示，第一计算节点上的SKa分组和第二计算节点上的SKa分组均属于第一计算节点的第一新分区。由于不同计算节点上相同分组所属的分组相同，这样，不同计算节点上的键相同的数据便可以被分配到相同的数据分区，从而有助于保证后续数据分析的可靠性和准确性。
可选地，根据shuffle策略对多个计算节点的多个分组进行分区时，多个分组中目标分组分配给第一计算节点，也就是说，多个分组中的目标分组的数据为第一计算节点的分区数据。其中，第一计算节点上的目标分组的数据量大于或等于非第一计算节点上的目标分组的数据量。这样，有助于将每个计算节点上数据量较多的分组分配给本计算节点，从而有助于减少跨节点传输的数量，进而有助于提高数据洗牌效率。
示例性的，如图4所示，第一计算节点上的SKa分组的数据量为5，第二计算节点上的SKa分组的数量为3，因此，SKa分组属于第一计算节点。第一计算节点上的SKb分组的数据量为3，第二计算节点上的SKb分组的数量为5，因此，SKb分组属于第二计算节点。
本申请中，根据shuffle策略以及每个计算节点的每个分组的数据量，对多个计算节点上的多个分组进行分区，可以包括多种实现方式，以下，通过方式A和方式B进行示例性介绍。
可选地，方式A可以包括以下的S5-S6。
S5：管理节点从多个计算节点的多个分组中，确定每个计算节点关联的分组。
其中，第一计算节点关联的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量。
本申请中，管理节点接收到多个计算节点的分组信息后，可以确定多个待分配分组，多个待分配分组包括每个计算节点的分组。示例性的，如图5所示，管理节点接收到计算节点1的分组信息1和计算节点2的分组信息2后，可以确定4个待分配分组，4个待分配分组包括SK1、SK2、SK3和SK4。其中，计算节点1包括SK1、SK2和SK3，计算节点2包括SK1、SK2、SK3和SK4。
在此基础上，管理节点可以根据每个待分配分组在每个计算节点上的数据量，从多个待分分键组中，确定每个计算节点关联的分组。示例性的，如图5所示，SK1在计算节点1上的数据量为4个数据(即数据1、数据3、数据5和数据6)，SK1在计算节点2的数据量为2个数据(即数据7和数据10)，由于SK1在计算节点1上的数据量大于SK1在计算节点2上的数据量，因此，确定SK1为计算节点关联的分组。基于相同的原理，SK4为计算节点2关联的分组。
由于SK2和SK3在计算节点1上的数据量，等于在计算节点2上的数据量，因此，SK2和SK3可以作为计算节点1或计算节点2中任意一个计算节点的关联的分组。或者，SK2和SK3也可以不作为计算节点1和计算节点2关联的分组。以下，以SK2和SK3为计算节点2关联的分组为例，对本申请进行介绍。
需要说明的，在计算节点1关联的目标分组的情况，为计算节点1分配分组时，该目标分组可以作为计算节点1最先选择的分组，或者说，目标分组可以作为计算节点1优先选择的分组，如：SK1可以作为计算节点1最先选择的分组，SK4可以作为计算节点2最先选择的分组。基于此，计算节点1可以称为目标分组的优先计算节点。
S6：管理节点从每个计算节点关联的分组中，轮询为每个计算节点分配分组。
示例性的，在每轮轮询过程中，管理节点可以从第一计算节点关联的分组中，为第一计算节点分配分组，从而确定每个计算节点的每个分组属于的计算节点，进而确定每个计算节点的分组数据。
示例性的，如图5所示，待分配分组包括SK1、SK2、SK3和SK4。在为计算节点1分配分组时，由于SK1的优先计算节点为计算节点1，因此，在第一轮轮询时，可以将SK1分配给计算节点1的新分区，基于此，计算节点1上的SK1和计算节点2上的SK1均属于计算节点1，且计算节点1已分配数据量为6个。其中，计算节点1上的SK1对应的数据和计算节点2上的SK1对应的数据为计算节点1的分区数据。
该实施例中，通过从每个计算节点关联的分组中，为每个计算节点分配分组，能够将一个分组分配给拥有该一个分组的数据量最多的计算节点，如：将SK1分配给计算节点1，SK4分配给计算节点2，这样，有助于减少跨节点传输的数据量，从而有助于提高数据传输效率，进而有助于提高shuffle效率。
在一个示例中，在为每个计算节点分配分组时，可以从每个计算节点关联的分组中，随机为每个计算节点分配分组。这样，有助于提高分配效率。
示例性的，如图5所示，在第一轮轮询时，由于SK2、SK3和SK4的优先计算节点均为计算节点2，因此，在为计算节点2分配分组时，可以将SK2、SK3和SK4中的任意一个分组分配给计算节点2的新分区，如：将SK2分配给计算节点2。
在另一个示例中，在为每个计算节点的新分区分配分组时，可以将每个计算节点关联的分组中，数据量最大的分组分配给每个计算节点。这样，可以将一个分组分配给拥有该一个分组的数据量最多的计算节点，有助于减少跨节点传输的数据量，从而有助于提高数据传输效率，进而有助于提高shuffle效率。
示例性的，如图5所示，SK2、SK3和SK4的优先计算节点均为计算节点2，SK2和SK3的数据量均为1个，SK4的数据量为2个，基于此，在第一轮轮询时，可以将SK4分配给计算节点2。
本申请中，在轮询过程中，若第一计算节点的已分配数据量小于或等于非第一计算节点的已分配数据量，则从第一计算节点关联的分组中，为第一计算节点分配分组。
反之，第一计算节点的已分配数据量大于非第一计算节点的已分配数据量，则本次轮询跳过第一计算节点，也即，不为第一计算节点上的数据分区分配分组。
需要说明的，由于第一轮轮询时，每个计算节点的已分配数据量均为0，因此，不跳过任何计算节点。
示例性的，如图5所示，第二轮轮询时，由于计算节点1的已分配数据量为6个数据，而计算节点2的已分配数据量为1(即SK2的数据量)，也就是说，计算节点1的已分配数据量大于计算节点2的已分配数据量，因此，在第二轮轮询时，可以跳过计算节点1，也即，不为计算节点1分配分组。在跳过计算节点1后，由于计算节点2的已分配数据量小于计算节点1的已分配数据量，因此，管理节点可以从未分配分组中为计算节点1分配分组，也即，从SK3和SK4中为计算节点1分配分组，如：将SK3分配给计算节点1的新分区。
示例性的，如图5所示，第三轮轮询时，由于计算节点的已分配数据量为6，计算节点2的已分配数据量为2，因此，在第三轮轮询时，依然可以跳过计算节点1，将剩余的SK4分配给计算节点2。
上述实施例中，通过在轮询过程中，跳过已分配数据量较大的计算节点，这样，有助于均衡不同计算节点上的数据量，从而有助于充分利用不同计算节点上的计算资源。
本申请中，数据洗牌方法还可以包括：在第一计算节点关联的分组已分配计算节点的情况下，管理节点从非第一计算节点关联的分组中，为第一计算节点分配数据。这样，提高了每个计算节点选取分组的范围，从而有助于更好的均衡不同计算节点上的数据量。
示例性的，如图5所示，在第二轮轮询过程中，若为计算节点1选取分组，由于计算节点1关联的SK1已分配计算节点，因此，可以从计算节点2关联的分组中，如：SK3和SK4中，为计算节点1选取分组。
本申请中，数据洗牌方法还可以包括：管理节点从未确定关联的计算节点的分组中，为第一计算节点分配分组。这样，提高了每个计算节点选取分组的范围，从而有助于更好的均衡不同计算节点上的数据量。
示例性的，如图5所示，若SK2和SK3未与计算节点1和计算节点2建立关联关系，也就是说，SK2和SK3不是计算节点1关联的分组，也不是计算节点2关联的分组。基于此，在计算节点1关联的分组已分配计算节点的情况下，管理节点可以SK2和SK3中，为计算节点1分配分组。
可选地，第一计算节点的已分配数据量小于或等于非第一计算节点的已分配数据量可以包括：第一计算节点的已分配数据量与非第一计算节点的已分配数据量的差值小于或等于差值阈值。
本申请中，差值阈值根据不同计算节点的已分配数据量确定。
可选地，差值阈值可以满足以下公式：(K-P)*u/P。其中，K为多个计算节点中的最大已分配数据量，P为多个计算节点中的最小已分配数据量。u可以为5％-20％。
在一个示例中，u可以为5％。在另一个示例中，u可以以为20。在又一个示例中，u可以为10％。这样，有助于提高u的选择多样性。该实施例中，在数据量的差值小于差值阈值时才为计算节点分配分组，这样，有助于更好的平衡不同计算节点上的数据量。
可选地，方式B可以包括以下的S7。
S7：管理节点从多个计算节点的多个分组中，为每个计算节点分配分组。
其中，为第一计算节点分配的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量。
本申请中，管理节点可以根据多个待分配分组中每个待分配分组在不同计算节点上的数据量，确定每个待分配分组所属的计算节点。
示例性的，如图2所示，SK1在计算节点1上的数据量，大于SK1在计算节点2上的数据量，基于此，SK1分配给计算节点1上的新分区。SK4在计算节点2上的数量，大于SK4在计算节点1上的数据量，基于此，SK4分配给计算节点2上的数据分区。此外，由于SK2/SK3在计算节点1上的数据量，等于SK2/SK3在计算节点2上的数据量，基于此，SK2/SK3可以分配给计算节点1或计算节点2中的任意一个计算节点。如：SK2/SK3可以分配给数据量少的计算节点，也即，分配给计算节点2，这样，有助于均衡不同计算节点上的数据量。
需要说明的，S7的其他相关说明，可以参考上述的S5的说明，此次不再赘述。
该实现方式中，根据每个分组在不同计算节点上的数据量，直接为每个计算节点分配分组，这样，有助于提高分配效率，从而有助于提高分区策略的确定效率。
以下，对管理节点确定每个计算节点的新分区的过程进行介绍。
可选地，管理节点可以在确定属于每个计算节点的键/分组之后，根据不同键/不同分组对应的数据量，确定每个计算节点的新分区。
结合上述情况2，管理节点从多个计算节点的多个分组中，确定属于第一计算节点的至少一个目标分组后，可以根据至少一个目标分组中每个目标分组的数据量，确定第一计算节点的至少一个新分区。
示例性的，如图5所示，管理节点确定SK2、SK3和SK4属于计算节点2后，基于SK2的数据量为1、SK3的数据量为1、SK4的数据量为2，确定计算节点2的分区数量。例如，可以将SK2和SK3确定为一个新分区，并将SK4确定另一个新分区。
示例性的，管理节点可以先确定多个计算节点上的新分区的总数量。之后，根据每个计算节点的分配的分区数据的数据量，确定每个计算节点的新分区的数据。之后，根据属于每个计算节点的每个分组的数据量以及每个计算节点的新分区的数量，确定属于每个计算节点的每个新分区的分组。
可选地，管理节点可以预先为每个计算节点确定N个新分区。在此基础上，在每轮轮询过程中，可以为每个计算节点的每个新分区选取分组，或者，为每个计算节点的一个新分区选取分组。
示例性的，计算节点1包括新分区a和新分区b，计算节点2包括新分区c和新分区d。在第一轮轮询过程中，管理节点可以依次为新分区和新分区b选取分组。或者，在第一轮轮询过程中，管理节点仅为新分区a选取分组，在第二轮轮询过程中，管理节点再为新分区b选取分组。
示例性的，在每个计算节点的每个新分区选取分组后，可以根据每个新分区的数据量，调整不同新分区的分组，以均衡不同新分区的数量。
需要说明的，本申请对管理节点确定每个计算节点的新分区的方式不做限制，以上仅为示例性说明。
需要说明的，步骤402也可以由第一计算节点或非第一计算节点，以上仅为示例性说明。
步骤403：管理节点向第一计算节点发送分区信息，分区信息用于指示第一计算节点的分区数据。
本申请中，管理节点确定每个计算节点的分区数据后，可以向每个计算节点发送分区信息，以便于每个计算节点可以根据分区信息分发本计算节点上的数据。
示例性的，如图4所示，分区信息用于指示键为五角形的数据以及键为圆的数据，均属于第一计算节点的第一新分区，也就是说，键为五角形的数据以及键为圆的数据是第一计算节点的分区数据。分区信息还可以用于指示键为六边形的数据以及键为三角形的数据，均属于第二计算节点的新分区，也就是说，键为六边形的数据以及键为三角形的数据是第二计算节点的分区数据。
在一个示例中，管理节点向第一计算节点发送分区信息包括第一计算节点的每个分组所属的计算节点。这样，有助于减少管理节点向每个计算节点发送的信息量，从而有助于提高信息发送效率。
示例性的，结合图6，管理节点向计算节点1发送的分区信息仅包括SK1、SK2和SK3所属的计算节点，不包括SK4所属的计算节点。
在另一个示例中，管理节点向第一计算节点发送的分区信息包括多个分组中每个分组所属的计算节点。这样，管理节点便不用为每个计算节点筛选分区信息，从而有助于提高信息传输效率。示例性的，结合图6，管理节点向计算节点1发送的分区信息仅包括SK1、SK2、SK3和SK4所属的计算节点。
步骤404：第一计算节点根据分区信息，确定第一计算节点的第一内存空间的数据中，属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据。
示例性的，如图4所示，第一组分区数据包括SKa分组的数据，也即，键为五角形的数据和键为圆形的数据。第二组分区数据包括SKb分组的数据，也即，键为三角形的数据和键为六边形的数据。
需要说明的，之所以将第一内存空间的数据划分为第一组分区数据和第二组分区数据，是为了区分将第一内存空间的数据中属于不同计算节点的数据。其中，对于每个计算节点来说，第一组分区数据为属于本计算节点的分区数据，第二组分区数据为不属于本计算节点的分区数据。
本申请中，第一计算节点可以在传输第一内存空间的数据时，确定属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据。这样，有助于提高数据传输效率。
示例性的，如图4所示，计算节点1接收到分区信息后，分区信息指示SK1的数据为第一计算节点的分区数据，SK2的数据和SK3的数据为计算节点2的分区数据。基于此，计算节点1从TEE内存1的第一内存空间中读取SK1的数据时，可以根据分区信息确定SK1的数据属于第一组分区数据，因此，计算节点1通过步骤405处理SK1的数据。之后，计算节点1从TEE内存1的第一内存空间中读取SK2的数据时，可以根据分区信息确定SK2的数据属于第二组分区数据，因此，计算节点可以通过步骤406和步骤407处理SK2的数据。
或者，本申请中，第一计算节点可以在接收到分区信息后，将第一内存空间的数据裁剪为多组分区数据，如：第一组分区数据和第二分组数据。这样，有助于提高数据传输的准确性。
示例性的，如图4所示，计算节点1接收到分区信息后，将SK1的数据确定为第一组分区数据，并将SK2的数据和SK3的数据确定为第二组分区数据。之后，计算节点1通过步骤405处理第一组分区数据，并通过步骤406和步骤407处理第二组分区数据。
步骤405：第一计算节点将第一内存空间中的第一组分区数据，写入第一计算节点的第二内存空间；第二内存空间用于存储第一计算节点的分区数据。
本申请中，分区信息还可用于指示第一计算节点的新分区数量以及每个新分区的数据占用的存储空间。第一计算节点获取到分区信息后，从第一内存上为第一计算节点的每个新分区分配内存空间。
示例性的，如图4所示，第一计算节点上的新分区的数量为1，如：该1个新分区为第一新分区，基于此，第一计算节点将第一内存上的第二内存空间分配给第一新分区，第二内存空间可以用于存储第一计算节点的第一新分区的数据。
在此基础上，第一计算节点将第一组分区数据，直接从第一内存空间写入第二内存空间。也就是说，第一组分区数据从第一内存空间被读出后，不写入第一计算节点的第一磁盘，而是直接写入第二内存空间。
在一个示例中，如图4所示，第一计算节点将SKa分组的数据从第一内存空间读出后，直接写入第二内存空间。在另一个示例中，如图5所示，计算节点1将SK1的数据从从第一内存空间读取后，直接写入第二内存空间。
示例性的，第二计算节点获取到本计算节点的新分区数量以及每个新分区的数据占用的存储空间，从第二内存上为每个新分区分配内存空间。如图4所示，第二计算节点上的新分区的数量为1，该1个新分区为第二新分区，基于此，第二计算节点将第二内存上的第三内存空间分配给第二新分区，第三内存空间可以用于存储第二新分区的数据。之后，第二计算节点将SKb分组的数据从第二老分区的内存空间直接写入第三内存空间，而不写入第二计算节点的第二磁盘。
步骤406：第一计算节点将第一内存空间中的第二组分区数据，写入第一计算节点的第一磁盘。
本申请中，在第一内存空间为第一内存上的内存空间，第一内存为TEE内存时，第一计算节点从第一内存空间读取第二组分区数据后，对第二组分区数据进行加密，并加密后的第二组分区数据写入第一计算节点的第一磁盘，以便于向第二计算节点传输第二组分区数据。其中，第一磁盘为REE的磁盘。
示例性的，如图4所示，第一计算节点从第一内存空间中读取到SKb分组的数据后，对SKb分组的数据进行加密，得到加密后的SKb分组的数据，之后，将加密后的SKb分组的数据写入第一磁盘。
步骤407：第一计算节点向非第一计算节点发送第一磁盘中的第二组分区数据。
本申请中，加密后的第二组分区数据写入第一磁盘后，第一计算节点可以向第二计算节点传输第一磁盘中的加密后的第二组分区数据。
示例性的，如图5所示，计算节点1可以对SK2的数据和SK3的数据进行加密，得到SK2的加密数据和SK3的加密数据。之后，计算节点1将SK2的加密数据和SK3的加密数据以及分区信息写入磁盘1的洗牌文件1中，分区信息用于指示SK2的数据和SK3的数据为计算节点2的分数数据。之后，计算节点将SK2的加密数据和SK3的加密数据传输给计算节点2。
示例性的，第二计算节点接收到加密后的第二组分区数据后，可以根据第二组分区数据的每个分组的标识，确定第二组分区所属的新分区。例如，如图4所示，第二计算节点可以基于第二组分区数据的每个分组的标识包括SKb分组，以及SKb分组属于第二新分区，将SKb分组的数据写入第二内存的第三内存空间。需要说明的，第二计算节点上，关于新分区和内存空间的相关说明，可以参考步骤405中第一计算节点的新分区以及内存空间的说明，此处步骤赘述。
示例性的，结合图4，第二计算节点在执行本申请提供的数据洗牌方法时，可以向第一计算节点发送第二磁盘中的SKa分组的数据。第一计算节点接收到非第二计算节点发送SKa分组的数据后，将SKa分组的数据写入第二内存空间。
需要说明的，本申请对步骤405和步骤406-步骤407的执行顺序不做限制，以上仅为示例性说明。
上述实施例中，以第一计算节点为例，介绍了对多个计算节点上的多个数据进行shuffle时，每个计算节点执行本申请提供的数据洗牌方法的过程。由于每个计算节点将属于本计算节点分区数据直接写入本计算节点上用于存储分区数据的内存空间(如：第二内存空间)，一方面，每个计算节点的一部分数据将不再写入本地磁盘，因此，减少了每个计算节点向本地磁盘写入数据的数量，从而提高了每个计算节的磁盘写入效率，进而能够提高数据shuffle效率。另一方面，每个计算节点的属于本计算节点的数据未写入本地磁盘，也就不再需要从每个计算节的本地磁盘写入每个计算节点的本地内存，因此，避免了每个计算节点的本地磁盘与本地内存之间的冗余数据传输，减少了每个计算节点的本地磁盘中需要分发到不同数据分区的数据，从而提高了每个计算节点的磁盘读取效率，进而能够提高数据shuffle效率，避免数据shuffle操作影响数据分析效率。
上述主要从方法的角度对本申请提供的方案进行了介绍。为了实现上述功能，数据洗牌装置包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请可以根据上述方法，示例性的对数据洗牌装置进行功能模块的划分，例如，数据洗牌装置可以包括对应各个功能划分的各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本申请中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
示例性的，图6示出了上述实施例中所涉及的数据洗牌装置(记为数据洗牌装置600)的一种可能的结构示意图，该数据洗牌装置600所执行的动作通过计算设备实现或通过计算设备执行相应的软件实现。该数据洗牌装置600可以包括分区模块601、写入模块602、传输模块603。分区模块601，用于获取多个计算节点的数据量信息，并根据shuffle策略及多个计算节点的数据量信息，对多个计算节点的多个数据进行分区，得到多个计算节点中每个计算节点的分区数据。例如，如图4所示的S403。分区模块601，还用于向第一计算节点发送分区信息，分区信息用于指示第一计算节点的分区数据；第一计算节点为多个计算节点中的任意一个计算节点。例如，如图4所示的S404。写入模块602，用于确定第一计算节点的第一内存空间的数据中，属于第一计算节点的第一组分区数据以及不属于第一计算节点的第二组分区数据。例如，如图4所示的S405。写入模块602，用于将第一计算节点的第一内存空间中的第一组分区数据，写入第一计算节点的第二内存空间；第二内存空间用于存储第一计算节点的分区数据。例如，如图4所示的S406。写入模块602，用于将第一计算节点的第一内存空间中的第二组分区数据，写入第一计算节点的第一磁盘；传输模块603，用于向非第一计算节点传输第一磁盘中的第二组分区数据。例如，如图4所示的S407。
可选地，第一内存空间和第二内存空间位于第一计算节点的受信任执行环境TEE的内存。
可选地，第一内存空间和第二内存空间位于第一计算节点的富执行环境REE的内存。
可选地，多个数据的数据类型为键-值key-value数据，数据洗牌装置还包括分组模块604。分组模块604用于：对第一计算节点的每个数据的键进行分组，得到第一计算节点的每个分组；分区模块601具体用于：管理节点获取每个计算节点的每个分组的数据量，得到多个计算节点的多个分组的数据量；多个数据包括多个键，相同的键属于相同的分组，部分不同的键属于相同的分组；管理节点根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区。
可选地，分组模块604具体用于：确定第一计算节点的每个数据的键的哈希值；将第一计算节点的每个数据的键的哈希值对目标数值取余，得到第一计算节点的每个数据的键对应的余数；其中，目标数值为多个分组的数量值；对第一计算节点的每个数据的键对应的余数进行分组；对应余数相同的键属于相同的分组。
可选地，分组模块604具体用于：根据多个键组信息，对第一计算节点的每个数据的键进行分组；其中，一个键组信息用于指示一个分组包括的至少一个键。
可选地，多个分组中的目标分组为第一计算节点的分区数据；第一计算节点上的目标分组的数据量大于或等于非第一计算节点上的目标分组的数据量。
可选地，分区模块601具体用于：从多个分组中，确定每个计算节点关联的分组；其中，第一计算节点关联的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量；从每个计算节点关联的分组中，轮询为每个计算节点的分配分区数据；其中，在每轮轮询过程中，若第一计算节点的已分配数据量小于或等于非第一计算节点的已分配数据量，从第一计算节点关联的分组中，为第一计算节点分配分区数据。
可选地，在每轮轮询过程中，分区模块601还用于：在第一计算节点关联的分组已分配计算节点的情况下，从非第一计算节点关联的分组中，为第一计算节点分配分区数据。
可选地，分区模块601具体用于：从多个分组中，为每个计算节点的分配分区数据；为第一计算节点分配的目标分组在第一计算节点上的数据量，大于或等于目标分组在非第一计算节点上的数据量。
可选地，多个数据的数据类型不是键-值key-value数据，分组模块604还用于：对第一计算节点的每个数据进行分组，得到第一计算节点的每个分组；分区模块601具体用于：获取每个计算节点的每个分组的数据量，得到多个计算节点的多个分组的数据量；根据shuffle策略及多个计算节点的多个分组的数据量，对多个计算节点的多个分组进行分区。
可选地，分组模块604具体用于：将第一计算节点的每个分区的数据作为第一计算节点的每个分组。
可选地，分组模块604具体用于：确定第一计算节点的每个数据的键；第一计算节点对第一计算节点的每个数据的键进行分组；其中，部分不同的键属于相同的分组。
可选地，分组模块604具体用于：确定第一计算节点的每个数据的哈希值；第一计算节点将第一计算节点的每个数据的哈希值对目标数值取余，得到第一计算节点的每个数据对应的余数；其中，目标数值为多个分组的数量值；第一计算节点对第一计算节点的每个数据对应的余数进行分组；对应余数相同的数据属于相同的分组，对应余数不同的部分数据属于相同的分组。
关于上述可选方式的具体描述可以参见前述的方法实施例，此处不再赘述。此外，上述提供的任一种数据洗牌装置600的解释以及有益效果的描述均可参考上述对应的方法实施例，不再赘述。
本申请中，分区模块601、写入模块602、传输模块603和分组模块604均可以通过软件实现，或者可以通过硬件实现。示例性的，接下来以分区模块601为例，介绍分区模块601的实现方式。类似的，写入模块602、传输模块603和分组模块604的实现方式可以参考分区模块601的实现方式。
模块作为软件功能单元的一种举例，分区模块601可以包括运行在计算实例上的代码。其中，计算实例可以包括物理主机(计算设备)、虚拟机、容器中的至少一种。进一步地，上述计算实例可以是一台或者多台。
例如，分区模块601可以包括运行在多个主机/虚拟机/容器上的代码。需要说明的是，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的区域(region)中，也可以分布在不同的region中。进一步地，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的可用区(availability zone，AZ)中，也可以分布在不同的AZ中，每个AZ包括一个数据中心或多个地理位置相近的数据中心。其中，通常一个region可以包括多个AZ。
同样，用于运行该代码的多个主机/虚拟机/容器可以分布在同一个虚拟私有云(virtual private cloud，VPC)中，也可以分布在多个VPC中。其中，通常一个VPC设置在一个region内，同一region内两个VPC之间，以及不同region的VPC之间跨区通信需在每个VPC内设置通信网关，经通信网关实现VPC之间的互连。
模块作为硬件功能单元的一种举例，分区模块601可以包括至少一个计算设备，如服务器等。或者，分区模块601也可以是利用专用集成电路(application-specific integrated circuit，ASIC)实现、或可编程逻辑器件(programmable logic device，PLD)实现的设备等。其中，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可编程门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合实现。
分区模块601包括的多个计算设备可以分布在相同的region中，也可以分布在不同的region中。分区模块601包括的多个计算设备可以分布在相同的AZ中，也可以分布在不同的AZ中。同样，分区模块601包括的多个计算设备可以分布在同一个VPC中，也可以分布在多个VPC中。其中，所述多个计算设备可以是服务器、ASIC、PLD、CPLD、FPGA和GAL等计算设备的任意组合。
需要说明的是，在其他实施例中，分区模块601可以用于执行数据洗牌方法中的任意步骤，写入模块602可以用于执行数据洗牌方法中的任意步骤，传输模块603可以用于执行数据洗牌方法中的任意步骤，分组模块604可以用于执行数据洗牌方法中的任意步骤，分区模块601、写入模块602、传输模块603和分组模块604负责实现的步骤可根据需要指定，通过分区模块601、写入模块602、传输模块603和分组模块604分别实现数据洗牌方法中不同的步骤来实现数据洗牌装置的全部功能。
本申请还提供一种计算设备700。如图7所示，计算设备700包括：总线702、处理器704、存储器706和通信接口708。处理器704、存储器706和通信接口708之间通过总线702通信。计算设备700可以是服务器或终端设备。应理解，本申请不限定计算设备700中的处理器、存储器的个数。
总线702可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图7中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线702可包括在计算设备700各个部件(例如，存储器706、处理器704、通信接口708)之间传送信息的通路。
处理器704可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器706可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。处理器704还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。
存储器706中存储有可执行的程序代码，处理器704执行该可执行的程序代码以分别实现前述分区模块601、写入模块602、传输模块603和分组模块604的功能，从而实现数据洗牌方法。也即，存储器706上存有用于执行数据洗牌方法的指令。
通信接口708使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算设备700与其他设备或通信网络之间的通信。
本申请实施例还提供了一种计算设备集群。该计算设备集群包括至少一台计算设备。该计算设备可以是服务器，例如是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。
如图8所示，所述计算设备集群800包括至少一个计算设备700。计算设备集群800中的一个或多个计算设备700中的存储器706中可以存有相同的用于执行数据洗牌方法的指令。
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备700的存储器706中也可以分别存有用于执行数据洗牌方法的部分指令。换言之，一个或多个计算设备700的组合可以共同执行用于执行数据洗牌方法的指令。
需要说明的是，计算设备集群中的不同的计算设备700中的存储器706可以存储不同的指令，分别用于执行数据洗牌装置的部分功能。也即，不同的计算设备700中的存储器706存储的指令可以实现分区模块601、写入模块602、传输模块603和分组模块604中的一个或多个模块的功能。
在一些可能的实现方式中，计算设备集群中的一个或多个计算设备可以通过网络连接。其中，所述网络可以是广域网或局域网等等。图9示出了一种可能的实现方式。
如图9所示，两个计算设备700A和700B之间通过网络进行连接。具体地，通过各个计算设备中的通信接口与所述网络进行连接。在这一类可能的实现方式中，计算设备700A中的存储器706中存有执行分区模块601的功能的指令。同时，计算设备700B中的存储器706中存有执行写入模块602、传输模块603和分组模块604的功能的指令。
图9所示的计算设备集群之间的连接方式可以是考虑到本申请提供的数据洗牌方法需要执行大量的计算，因此考虑将写入模块602、传输模块603和分组模块604实现的功能交由计算设备700B执行。
应理解，图9中示出的计算设备700A的功能也可以由多个计算设备700完成。同样，计算设备700B的功能也可以由多个计算设备700完成。
本申请实施例还提供了另一种计算设备集群。该计算设备集群中各计算设备之间的连接关系可以类似的参考图8和图9所述计算设备集群的连接方式。不同的是，该计算设备集群中的一个或多个计算设备700中的存储器706中可以存有相同的用于执行数据洗牌方法的指令。
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备700的存储器706中也可以分别存有用于执行数据洗牌方法的部分指令。换言之，一个或多个计算设备700的组合可以共同执行用于执行数据洗牌方法的指令。本申请实施例还提供了一种包含指令的计算机程序产品。计算机程序产品可以是包含指令的，能够运行在计算设备上或被储存在任何可用介质中的软件或程序产品。当计算机程序产品在至少一个计算设备上运行时，使得至少一个计算设备执行上述的数据洗牌方法。
本申请实施例还提供了一种计算机可读存储介质。计算机可读存储介质可以是计算设备能够存储的任何可用介质或者是包含一个或多个可用介质的数据中心等数据存储设备。可用介质可以是磁性介质，(如：软盘、硬盘、磁带)、光介质(如：DVD)、或者半导体介质(例如固态硬盘)等。该计算机可读存储介质包括指令，指令指示计算设备执行数据洗牌方法。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的保护范围。

Claims (15)

  1. 一种数据洗牌shuffle方法，其特征在于，应用于计算集群，所述计算集群包括多个计算节点及管理节点，所述方法包括：
    所述管理节点获取所述多个计算节点中每个计算节点的数据量，并根据shuffle策略及所述每个计算节点的数据量，对所述多个计算节点的多个数据进行分区，得到所述每个计算节点的分区数据；
    所述管理节点向第一计算节点发送分区信息，所述分区信息用于指示所述第一计算节点的分区数据；所述第一计算节点为所述多个计算节点中的任意一个计算节点；
    所述第一计算节点根据所述分区信息，确定所述第一计算节点的第一内存空间的数据中，属于所述第一计算节点的第一组分区数据以及不属于所述第一计算节点的第二组分区数据；
    所述第一计算节点将所述第一计算节点的第一内存空间中的所述第一组分区数据，写入所述第一计算节点的第二内存空间；所述第二内存空间用于存储所述第一计算节点的分区数据；
    所述第一计算节点将所述第一计算节点的第一内存空间中的所述第二组分区数据，写入所述第一计算节点的第一磁盘，并向非第一计算节点传输所述第一磁盘中的所述第二组分区数据。
  2. 根据权利要求1所述的方法，其特征在于，
    所述第一内存空间和所述第二内存空间位于所述第一计算节点的受信任执行环境TEE的内存。
  3. 根据权利要求1或2所述的方法，其特征在于，所述多个数据的数据类型为键-值key-value数据，
    所述方法还包括：所述第一计算节点对所述第一计算节点的每个数据的按照键进行分组，得到所述第一计算节点的每个分组；每个分组包括至少一个键对应的键-值数据，部分分组包括多个键对应的键-值数据；
    所述管理节点获取所述多个计算节点中每个计算节点的数据量，并根据shuffle策略及所述每个计算节点的数据量，对所述多个计算节点的多个数据进行分区，包括：所述管理节点获取所述每个计算节点的每个分组的数据量，得到所述多个计算节点的多个分组的数据量；所述管理节点根据shuffle策略及所述多个计算节点的多个分组的数据量，对所述多个计算节点的多个分组进行分区。
  4. 根据权利要求3所述的方法，其特征在于，
    所述多个分组中的目标分组的数据为所述第一计算节点的分区数据；所述第一计算节点上的所述目标分组的数据量大于或等于非第一计算节点上的所述目标分组的数据量。
  5. 根据权利要求3或4所述的方法，其特征在于，所述管理节点根据shuffle策略及所述多个计算节点的多个分组的数据量，对所述多个计算节点的多个分组进行分区，包括：
    所述管理节点从所述多个分组中，确定所述每个计算节点关联的分组；其中，所述第一计算节点关联的目标分组在所述第一计算节点上的数据量，大于或等于所述目标分组在非第一计算节点上的数据量；
    所述管理节点从所述每个计算节点关联的分组中，轮询为所述每个计算节点的分配分组；
    其中，在每轮轮询过程中，若所述第一计算节点的已分配数据量小于或等于所述非第一计算节点的已分配数据量，从所述第一计算节点关联的分组中，为所述第一计算节点分配分组。
  6. 根据权利要求3或4所述的方法，其特征在于，所述管理节点根据shuffle策略及所述多个计算节点的多个分组的数据量，对所述多个计算节点的多个分组进行分区，包括：
    所述管理节点从所述多个计算节点的多个分组中，为所述每个计算节点的分配分组；
    其中，为所述第一计算节点分配的目标分组在所述第一计算节点上的数据量，大于或等于所述目标分组在非第一计算节点上的数据量。
  7. 一种数据洗牌shuffle装置，其特征在于，应用于计算集群，计算集群包括多个计算节点；所述装置包括：
    分区模块，用于获取所述多个计算节点的数据量信息，并根据shuffle策略及所述多个计算节点的数据量信息，对所述多个计算节点的多个数据进行分区，得到所述多个计算节点中每个计算节点的分区数据；
    所述分区模块，还用于向第一计算节点发送分区信息，所述分区信息用于指示所述第一计算节点的分区数据；所述第一计算节点为所述多个计算节点中的任意一个计算节点；
    写入模块，用于确定所述第一计算节点的第一内存空间的数据中，属于所述第一计算节点的第一组分区数据以及不属于所述第一计算节点的第二组分区数据；
    所述写入模块，用于将所述第一计算节点的第一内存空间中的所述第一组分区数据，写入所述第一计算节点的第二内存空间；所述第二内存空间用于存储所述第一计算节点的分区数据；
    所述写入模块，用于将所述第一计算节点的第一内存空间中的所述第二组分区数据，写入所述第一计算节点的第一磁盘；
    传输模块，用于向非第一计算节点传输所述第一磁盘中的所述第二组分区数据。
  8. 根据权利要求7所述的装置，其特征在于，
    所述第一内存空间和所述第二内存空间位于所述第一计算节点的受信任执行环境TEE的内存。
  9. 根据权利要求7或8所述的装置，其特征在于，所述装置还包括：分组模块；
    所述分组模块，用于对所述第一计算节点的每个数据的按照键进行分组，得到所述第一计算节点的每个分组；每个分组包括至少一个键对应的键-值数据；
    所述分区模块具体用于：获取所述每个计算节点的每个分组的数据量，得到所述多个计算节点的多个分组的数据量；根据shuffle策略及所述多个计算节点的多个分组的数据量，对所述多个计算节点的多个分组进行分区。
  10. 根据权利要求9所述的装置，其特征在于，
    所述多个分组中的目标分组的数据为所述第一计算节点的分区数据；所述第一计算节点上的所述目标分组的数据量大于或等于非第一计算节点上的所述目标分组的数据量。
  11. 根据权利要求9或10所述的装置，其特征在于，所述分区模块具体用于：
    从所述多个分组中，确定所述每个计算节点关联的分组；其中，所述第一计算节点关联的目标分组在所述第一计算节点上的数据量，大于或等于所述目标分组在非第一计算节点上的数据量；
    从所述每个计算节点关联的分组中，轮询为所述每个计算节点的分配分组；
    其中，在每轮轮询过程中，若所述第一计算节点的已分配数据量小于或等于所述非第一计算节点的已分配数据量，从所述第一计算节点关联的分组中，为所述第一计算节点分配分组。
  12. 根据权利要求9或10所述的装置，其特征在于，所述分区模块具体用于：
    从所述多个计算节点的多个分组中，为所述每个计算节点的分配分组；
    其中，为所述第一计算节点分配的目标分组在所述第一计算节点上的数据量，大于或等于所述目标分组在非第一计算节点上的数据量。
  13. 一种计算设备集群，其特征在于，所述计算设备集群包括至少一个计算设备；
    所述至少一个计算设备中的每个计算设备包括处理器、存储器及存储在所述存储器上的计算机程序/指令；所述每个计算设备的处理器执行所述每个计算设备的存储器中存储的计算机程序，以使所述每个计算设备实现如权利要求1-6中任一项所述方法的步骤。
  14. 一种计算机程序产品，其特征在于，
    所述计算机程序产品包括计算机程序/指令，所述计算机程序/指令被处理器执行时实现如权利要求1-6中任一项所述方法的步骤。
  15. 一种计算机可读存储介质，其特征在于，
    所述计算机可读存储介质上存储有计算机程序/指令，所述计算机程序/指令被处理器执行时实现如权利要求1-6中任一项所述方法的步骤。
PCT/CN2025/072572 2024-06-13 2025-01-15 数据洗牌shuffle方法、装置、设备及集群 Pending WO2025256136A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410764426.8 2024-06-13
CN202410764426.8A CN121144409A (zh) 2024-06-13 2024-06-13 数据洗牌shuffle方法、装置、设备及集群

Publications (1)

Publication Number Publication Date
WO2025256136A1 true WO2025256136A1 (zh) 2025-12-18

Family

ID=97981053

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/072572 Pending WO2025256136A1 (zh) 2024-06-13 2025-01-15 数据洗牌shuffle方法、装置、设备及集群

Country Status (2)

Country Link
CN (1) CN121144409A (zh)
WO (1) WO2025256136A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109189746A (zh) * 2018-07-12 2019-01-11 北京百度网讯科技有限公司 通用流式Shuffle引擎的实现方法、装置、设备及存储介质
US10642520B1 (en) * 2017-04-18 2020-05-05 EMC IP Holding Company LLC Memory optimized data shuffle
CN114237510A (zh) * 2021-12-17 2022-03-25 北京达佳互联信息技术有限公司 数据处理方法、装置、电子设备及存储介质
US20220164122A1 (en) * 2020-11-20 2022-05-26 Samsung Electronics Co., Ltd. Systems, methods, and devices for shuffle acceleration
CN117891391A (zh) * 2023-12-14 2024-04-16 中科驭数（北京）科技有限公司 一种Spark应用的数据洗牌传输过程的数据读写方法、系统和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10642520B1 (en) * 2017-04-18 2020-05-05 EMC IP Holding Company LLC Memory optimized data shuffle
CN109189746A (zh) * 2018-07-12 2019-01-11 北京百度网讯科技有限公司 通用流式Shuffle引擎的实现方法、装置、设备及存储介质
US20220164122A1 (en) * 2020-11-20 2022-05-26 Samsung Electronics Co., Ltd. Systems, methods, and devices for shuffle acceleration
CN114237510A (zh) * 2021-12-17 2022-03-25 北京达佳互联信息技术有限公司 数据处理方法、装置、电子设备及存储介质
CN117891391A (zh) * 2023-12-14 2024-04-16 中科驭数（北京）科技有限公司 一种Spark应用的数据洗牌传输过程的数据读写方法、系统和存储介质

Also Published As

Publication number Publication date
CN121144409A (zh) 2025-12-16

Similar Documents

Publication Publication Date Title
US11789614B2 (en) Performance allocation among users for accessing non-volatile memory devices
EP4152140B1 (en) Network card and method for network card to process data
EP3574613B1 (en) Unified resource management in a data center cloud architecture
US11710206B2 (en) Session coordination for auto-scaled virtualized graphics processing
CN109983449A (zh) 数据处理的方法和存储系统
CN112181632A (zh) 在适配器设备中部署服务容器
US9940020B2 (en) Memory management method, apparatus, and system
US20080162735A1 (en) Methods and systems for prioritizing input/outputs to storage devices
EP4318251A1 (en) Data access system and method, and device and network card
CN116560785A (zh) 一种访问存储节点的方法、装置及计算机设备
WO2022063273A1 (zh) 一种基于numa属性的资源分配方法及装置
US11824752B2 (en) Port-to-port network routing using a storage device
CN108228099B (zh) 一种数据存储的方法及装置
WO2025139140A1 (zh) 一种多主机与多计算快速链接内存设备系统及其应用设备
JP5186555B2 (ja) 共有イーサネット・ポート用の論理ポート・レベルにおいて帯域幅キャッピングを実装するための方法、装置及びコンピュータ・プログラム
WO2025256136A1 (zh) 数据洗牌shuffle方法、装置、设备及集群
CN119071310A (zh) 云存储处理方法、设备、存储介质和系统
CN115544489A (zh) 一种认证方法、装置和存储系统
CN121523872A (zh) 数据聚合方法、装置及集群
CN119336519B (zh) 一种存内计算系统、方法、服务器及存内计算网络
CN120602160A (zh) 一种用于计算机局域网络的数据共享方法及系统
WO2025140222A1 (zh) 计算设备内组件间功能调用的方法、装置和设备
CN121935912A (zh) 可信根架构的构建方法、可信体系的应用方法和装置
WO2025001473A1 (zh) 一种基于块存储的文件处理方法、系统及相关装置
WO2025201241A1 (zh) 一种内存共享方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25820820

Country of ref document: EP

Kind code of ref document: A1