CN116932579A

CN116932579A - Data parallel aggregation method and device, electronic equipment and storage medium

Info

Publication number: CN116932579A
Application number: CN202210320481.9A
Authority: CN
Inventors: 王思远
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2022-03-29
Filing date: 2022-03-29
Publication date: 2023-10-24

Abstract

The disclosure relates to a data parallel aggregation method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: distributing data of a data source to a plurality of computing threads through the data threads according to a first preset rule; distributing the calculation results obtained by the parallel calculation of the plurality of calculation threads to a plurality of aggregation threads through a coordination thread according to a second preset rule; and acquiring an aggregation result obtained by parallel aggregation of the plurality of aggregation threads according to a third preset rule by the result thread. By using the technical scheme provided by the disclosure, the same parallel hash aggregation query instruction can be executed on the data of the same data source whenever the summary aggregation results obtained each time are consistent.

Description

Data parallel aggregation method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of database query, and in particular relates to a data parallel aggregation method, a data parallel aggregation device, electronic equipment and a storage medium.

Background

In database systems, parallel hash aggregate queries are the primary way to perform data analysis. However, in practice, the same parallel hash aggregate query instruction is executed at different times for the same data set, and the resulting query results may not be consistent. For example, suppose table T has columns a and b, with 3 pieces of data (1, 2), (1, 3), and (2, 4), and a parallel hash aggregate query instruction select a, sum (b) from T group by a limit 1 is executed. The meaning of the instruction is to group the data of table T by the value of a, sum the b values of each group, and output the first of the aggregated results. In practice, (1, 5) is sometimes outputted, and (2, 4) is sometimes outputted. Obviously, such a result is not reasonable. Therefore, how to ensure that the same parallel hash aggregation query instruction is executed at different times for the same data set, and the obtained query results are consistent is a problem to be solved at present.

Disclosure of Invention

In order to solve the technical problems, the present disclosure provides a data parallel aggregation method, a device, an electronic apparatus, and a storage medium.

In a first aspect, the present disclosure provides a data parallel aggregation method, which is characterized in that the method includes:

distributing data of a data source to a plurality of computing threads through the data threads according to a first preset rule;

distributing the calculation results obtained by the parallel calculation of the plurality of calculation threads to a plurality of aggregation threads through a coordination thread according to a second preset rule;

and acquiring an aggregation result obtained by parallel aggregation of the plurality of aggregation threads according to a third preset rule by the result thread.

In a second aspect, the present disclosure further provides a data parallel aggregation apparatus, where the apparatus includes:

the first distribution unit is used for distributing the data of the data source to a plurality of computing threads through the data threads according to a first preset rule;

the second distributing unit is used for distributing the calculation results obtained by the parallel calculation of the plurality of calculation threads to the plurality of aggregation threads through the coordination threads according to a second preset rule;

the acquisition unit is used for acquiring an aggregation result obtained by parallel aggregation of the plurality of aggregation threads according to a third preset rule by the result threads.

In a third aspect, the present disclosure also provides an electronic device, including: a processor and a memory;

the processor is operable to perform the steps of any of the methods described above by invoking a program or instruction stored in the memory.

In a fourth aspect, the present disclosure also provides a computer-readable storage medium storing a program or instructions that cause a computer to perform the steps of any of the methods described above.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:

according to the technical scheme provided by the embodiment of the disclosure, the data of the data source are distributed to a plurality of computing threads through the data threads according to the first preset rule, so that the same parallel hash aggregation query instruction is executed on the data of the same data source whenever the input data (namely, original data) and output data (namely, a computing result or an intermediate result) of each computing thread are identical; distributing the calculation results obtained by parallel calculation of the plurality of calculation threads to the plurality of aggregation threads through the coordination threads according to a second preset rule, so that the input data (namely the calculation results or the intermediate results) and the output data (namely the aggregation results or the final results) of each aggregation thread are identical whenever the same parallel hash aggregation query instruction is executed on the data of the same data source; and acquiring an aggregation result obtained by parallel aggregation of a plurality of aggregation threads through the result thread according to a third preset rule, so that whenever the same parallel hash aggregation query instruction is executed on the data of the same data source, the sequence of receiving each piece of input data (namely the aggregation result or the final result) by the result thread is consistent. Since in practice the result thread typically outputs the input data in the order in which it receives it. If the result thread receives the sequence determination of the aggregation results, the sequence of the aggregation results output by the result thread is also certain. Therefore, the technical scheme provided by the disclosure can ensure that the same parallel hash aggregation query instruction is executed on the data of the same data source whenever the summarized aggregation result obtained each time is consistent. Here, the aggregate result agreement includes agreement of the output order of the respective aggregate results.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

Fig. 1 is a flowchart of a data parallel aggregation method provided in an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a parallel hash aggregate query execution stage provided by an embodiment of the present disclosure;

FIG. 3 is a flow chart of a method for data parallel aggregation provided by an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a data parallel aggregation device according to an embodiment of the present disclosure;

fig. 5 is a schematic hardware structure of an electronic device according to an embodiment of the disclosure.

Detailed Description

In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.

Fig. 1 is a flowchart of a data parallel aggregation method provided in an embodiment of the present disclosure. Fig. 2 is a schematic diagram of a parallel hash aggregate query execution stage according to an embodiment of the present disclosure. Referring to fig. 1 and 2, the method includes:

s110, distributing the data of the data source to a plurality of computing threads through the data threads according to a first preset rule.

The data thread, namely DataWorker, functions to acquire data from a data source and distribute the acquired data to a plurality of computing threads.

The computing thread, namely PartialWorker, is used for pre-aggregating the original data received through the data thread to form an intermediate result.

The first preset rule is a distribution rule that distributes data (i.e., raw data) of the data source to the computing thread. The first preset rule may specify in what order to distribute the raw data to the computing threads. Illustratively, referring to FIG. 2, assuming that there are three computing threads, a first preset rule may specify that one piece of raw data be sent to each computing thread in the order of computing thread 1, computing thread 2, and computing thread 3 in sequence. By setting the first preset rule, the same piece of data can be distributed to the same computing thread when the data of the same data source is distributed in response to the same parallel hash aggregation query instruction at any time. That is, whenever the same parallel hash aggregate query instruction is executed on data of the same data source, the input data for each compute thread is the same. Those skilled in the art will appreciate that for any computational thread, if the input data is the same, the output data will be the same. That is, by setting the first preset rule, the output data of each computing thread can be the same whenever the same parallel hash aggregate query instruction is executed on the data of the same data source.

Optionally, the first preset rule is polling, that is, sending one piece of original data to each computing thread sequentially according to the sequence numbers of the computing threads. Illustratively, with continued reference to FIG. 2, the first preset rule is to send one piece of raw data to each computing thread in the order of computing thread 1, computing thread 2, and computing thread 3. This arrangement ensures that all computing threads participate in the data aggregation operation, and increases the rate of data aggregation.

There are various ways to implement this step, and the present application is not limited thereto. Illustratively, the method for implementing the step includes:

first, a plurality of pieces of data are acquired from a data source by a data thread.

And secondly, distributing at least one piece of data to each computing thread through the data threads according to a first preset rule, so that the plurality of computing threads calculate the received data in parallel.

And after the parallel computation of the plurality of computing threads is completed, distributing at least one piece of data to each computing thread again through the data threads according to a first preset rule until the distribution of the plurality of pieces of data is completed.

And S120, distributing the calculation results obtained by parallel calculation of the plurality of calculation threads to the plurality of aggregation threads through the coordination threads according to a second preset rule.

The coordination thread, i.e., the Coordinator worker, functions to distribute the computation results (i.e., intermediate results) of multiple computation threads to the aggregate thread.

And the aggregation thread is the FinalWorker, and the received intermediate result is calculated to form a final result.

The second preset rule is a distribution rule for distributing the calculation result (i.e. the intermediate result) obtained after the calculation of the calculation thread to the aggregation thread. The second preset rule is set to ensure that at any time, in response to the same parallel hash aggregate query instruction, the same intermediate result is distributed to the same aggregate thread. That is, whenever the same parallel hash aggregate query instruction is executed on data of the same data source, the input data for each aggregate thread is the same. Those skilled in the art will appreciate that for any aggregate thread, if the input data is the same, the output data will be the same. That is, by setting the second preset rule, the same parallel hash aggregate query instruction is executed on the data of the same data source whenever the aggregate results of the aggregate threads are the same.

S130, acquiring an aggregation result obtained by parallel aggregation of a plurality of aggregation threads according to a third preset rule by the result threads.

The Result thread, namely the Aggregate Result, is used for receiving the final Result (i.e. the aggregation Result) obtained by the aggregation thread and outputting each final Result. In other words, the results thread is used to aggregate all aggregate results and output.

The third preset rule is a sending rule for sending the aggregation result (i.e. the final result) obtained after the calculation of the aggregation thread to the result thread. Setting the third preset rule has two purposes, namely, ensuring that the sequence of receiving each final result (i.e., the aggregation result) by the result thread is consistent when the final result (i.e., the aggregation result) is sent in response to the same parallel hash aggregation query instruction at any time.

Optionally, there are various specific implementation methods of this step, and an exemplary specific implementation method of this step may be that, by using a result thread, corresponding aggregation results are sequentially obtained from multiple aggregation threads. That is, the result thread is in an active acquisition state, and the result thread requests the aggregation result from each aggregation thread according to a preset sequence (for example, the sequence of each aggregation thread sequence number is from small to large, or the sequence of each aggregation thread sequence number is from large to small). For example, if the result thread is asking for the aggregate result from the mth aggregate thread, the mth aggregate thread does not complete the aggregate calculation, then wait. When the mth aggregation thread completes aggregation calculation, the result thread obtains an aggregation result from the mth aggregation thread, and then requests the aggregation result from the (m+1) th aggregation thread. Wherein m is a positive integer.

Alternatively, the specific implementation method of this step may be that the result threads sequentially output the aggregation results of the multiple aggregation threads. That is, the aggregation thread is in a passive receiving state, and each result thread sends the obtained aggregation result to the result thread according to a preset sequence (for example, the sequence of each aggregation thread sequence number is from small to large, or the sequence of each aggregation thread sequence number is from large to small). For example, if the current round reaches the mth aggregation thread to output the aggregation result to the result thread, if the mth aggregation thread does not complete the aggregation calculation at this time, the aggregation result is output to the result thread after the mth aggregation thread completes the aggregation calculation. And after the mth aggregation thread outputs the aggregation result to the result thread, the mth+1th aggregation thread outputs the aggregation result to the result thread. Wherein m is a positive integer.

In the above technical solution, the data of the data source is distributed to the plurality of computing threads by the data threads according to the first preset rule, so that whenever the same parallel hash aggregation query instruction is executed on the data of the same data source, the input data (i.e., the original data) and the output data (i.e., the computation result or the intermediate result) of each computing thread are identical; distributing the calculation results obtained by parallel calculation of the plurality of calculation threads to the plurality of aggregation threads through the coordination threads according to a second preset rule, so that the input data (namely the calculation results or the intermediate results) and the output data (namely the aggregation results or the final results) of each aggregation thread are identical whenever the same parallel hash aggregation query instruction is executed on the data of the same data source; and acquiring an aggregation result obtained by parallel aggregation of a plurality of aggregation threads through the result thread according to a third preset rule, so that whenever the same parallel hash aggregation query instruction is executed on the data of the same data source, the sequence of receiving each piece of input data (namely the aggregation result or the final result) by the result thread is consistent. Since in practice, the result thread typically forwards the input data out in the order in which it received it. If the result thread receives the sequence determination of the aggregation results, the sequence of the aggregation results output by the result thread is also certain. Therefore, the technical scheme provided by the disclosure can ensure that the same parallel hash aggregation query instruction is executed on the data of the same data source whenever the summarized aggregation result obtained each time is consistent. Here, the aggregate result agreement includes agreement of the output order of the respective aggregate results.

It should be emphasized that the above aggregation method is not affected by the CPU scheduling manner. That is, no matter how the CPU schedules each thread, the original data input by each calculation thread is consistent, and the output calculation result is consistent; the calculation results input by the aggregation threads are consistent, and the output aggregation results are consistent; and the result thread obtains a consistent summarized aggregation result. Here, the aggregate result agreement includes agreement of the output order of the respective aggregate results.

Fig. 3 is a flowchart of a data parallel aggregation method according to an embodiment of the present disclosure. Referring to fig. 2 and 3, the data parallel aggregation method includes:

s210, distributing data of a data source to a plurality of computing threads through the data threads according to a first preset rule.

Illustratively, assume that table T1 has columns a and b, and that table T1 includes 3n pieces of data, data 1, data 2, data 3, data 4, data 5, data 6, … …, data 3n, respectively. The parallel hash aggregate query instruction select a, sum (b) from t group by a limit 1 is executed. The meaning of the instruction is to group the data of table T by the value of a, sum the b values of each group, and output the first of the aggregated results.

Referring to fig. 3, data 1 is sent to computing thread 1, data 2 is sent to computing thread 2, data 3 is sent to computing thread 3 to enable three computing threads to compute in parallel, data 4 is sent to computing thread 1 after all three threads have been computed, data 5 is sent to computing thread 2, and data 6 is sent to computing thread 3 to enable three computing threads to compute again in parallel. And (3) reversing the process until all 3n pieces of data are distributed.

S220, the received data are calculated in parallel by the plurality of calculation threads, and the calculation results of the plurality of calculation threads are stored in the hash tables, wherein the key words of the hash tables are the sequence numbers of the aggregation threads, and the values of the hash tables are the calculation results.

Optionally, for a calculation result, the aggregate thread number corresponding to the calculation result is a remainder obtained by performing modulo operation on the calculation result and the aggregate thread number.

Illustratively, with continued reference to FIG. 2, the computing thread 1 is operable to initially aggregate data 1, data 4, … …, and data 3n-2 it receives, i.e., sum the b values of each group by the value grouping of a. Similarly, compute thread 2 is configured to initially aggregate data 2, data 5, … …, and data 3n-1 it receives, i.e., sum the b values of each group by the value group of a. The calculation thread 3 is configured to perform a preliminary aggregation of the data 3, the data 6, … … and the data 3n it receives, i.e. to sum the b values of each group according to the value grouping of a. The specific calculation mode of the calculation thread is the prior art, and is not described herein.

Specifically, it is assumed that n data received by the computing thread 1 may be divided into 6 groups according to the value of a, namely, a=1 group, a=2 group, a=3 group, a=4 group, a=5 group, and a=6 group. The calculation thread 1 calculates the sum b11 of the data b belonging to the group of a=1, and takes (1, b 11) as the group calculation result of a=1. Thus, the calculation results of all calculation threads can be obtained. Illustratively, table 1 is the calculation results of three calculation threads.

TABLE 1

When the calculation result (1, b 11) is stored in the hash table of the calculation thread 1, the remainder 1 obtained by performing the modulo operation with 1 (i.e. the value of a) and the number of aggregation threads (in fig. 2, the total number of aggregation threads is 3) is utilized, i.e. the key in the hash table is 1, and the value is (1, b 11). The key is 1, indicating that the calculation result (1, b 11) will be subsequently input into the aggregation thread 1. In this way, all the calculation results can be stored in the hash table. Illustratively, table 2 is a hash table for each computing thread.

TABLE 2

S230, sequentially distributing the calculation results to the corresponding aggregation threads from the hash tables of the plurality of calculation threads through the coordination threads based on the aggregation thread serial numbers.

The implementation method of this step is various, and illustratively, the implementation method of this step includes: after the distribution of all calculation results in one hash table is completed, the coordination thread is used for distributing calculation results in the next hash table. Illustratively, referring to Table 2, the results of computation for computation thread 1 are stored in hash table 1, the results of computation for computation thread 2 are stored in hash table 2, and the results of computation for computation thread 3 are stored in hash table 3. When the step is executed, all calculation results in the hash table 1 are distributed to corresponding aggregation threads from the hash table 1, all calculation results in the hash table 2 are distributed to corresponding aggregation threads, and all calculation results in the hash table 3 are distributed to corresponding aggregation threads.

Alternatively, the implementation method of the step includes: after distribution of all calculation results corresponding to the same aggregation thread number in all hash tables is completed, distribution of all calculation results corresponding to the next aggregation thread number is carried out through the coordination thread. Illustratively, referring to Table 2, the results of calculation for calculation thread 1 are stored in hash table 1, the results of calculation for calculation thread 2 are stored in hash table 2, the results of calculation for calculation thread 3 are stored in hash table 3, and each hash table includes 6 results of calculation. When executing the step, in the first step, all calculation results corresponding to the aggregate thread number 1 (i.e. Key is 1) in the hash table 1 are distributed to the aggregate thread 1, then all calculation results corresponding to the aggregate thread number 1 (i.e. Key is 1) in the hash table 2 are distributed to the aggregate thread 1, and finally all calculation results corresponding to the aggregate thread number 1 (i.e. Key is 1) in the hash table 3 are distributed to the aggregate thread 1. And secondly, distributing all calculation results corresponding to the aggregation thread number 2 (namely Key is 2) in the hash table 1 to the aggregation thread 2, distributing all calculation results corresponding to the aggregation thread number 2 (namely Key is 2) in the hash table 2 to the aggregation thread 2, and finally distributing all calculation results corresponding to the aggregation thread number 2 (namely Key is 2) in the hash table 3 to the aggregation thread 2. Thirdly, distributing all calculation results corresponding to the aggregation thread number 3 (i.e. Key is 3) in the hash table 1 to the aggregation thread 3, distributing all calculation results corresponding to the aggregation thread number 3 (i.e. Key is 3) in the hash table 2 to the aggregation thread 3, and finally distributing all calculation results corresponding to the aggregation thread number 3 (i.e. Key is 3) in the hash table 3 to the aggregation thread 3.

Table 3 shows the data received by each aggregation thread after distributing the data in table 2 to each aggregation thread.

TABLE 3 Table 3

S240, the plurality of aggregation threads aggregate the received calculation results in parallel based on the sequence of the received calculation results.

The specific aggregation manner of the aggregation thread is the prior art, and is not described herein.

S250, acquiring an aggregation result obtained by parallel aggregation of a plurality of aggregation threads according to a third preset rule by the result threads.

Optionally, sequentially acquiring corresponding aggregation results from the plurality of aggregation threads through the result threads; or sequentially outputting the aggregation results of the aggregation threads through the result threads.

According to the technical scheme, the received data are calculated in parallel by the plurality of calculation threads, the calculation results are stored in the hash tables by the plurality of calculation threads, wherein the key words of the hash tables are the sequence numbers of the aggregation threads, the values of the hash tables are the calculation results, the calculation results are distributed to the corresponding aggregation threads from the hash tables of the plurality of calculation threads in sequence by the coordination threads based on the sequence numbers of the aggregation threads, the hash tables are essentially utilized, the purpose that the input data (namely the calculation results or the intermediate results) and the output data (namely the aggregation results or the final results) of all the aggregation threads are identical whenever the same parallel hash aggregation query instruction is executed on the data of the same data source is achieved, and therefore the sequence of all the finally output aggregation results is consistent is ensured.

Optionally, in executing S210, recording the sequence in which the computing thread receives the first data of each packet from the data thread; s230 is replaced with: and distributing the calculation results to the corresponding aggregation threads based on the sequence numbers of the aggregation threads and the sequence order from the hash tables of the plurality of calculation threads through the coordination threads. Illustratively, assuming table T has columns a and b, the table includes (1, 2), (1, 3), (2, 4), and (2, 8), these 4 pieces of data will be sent to the compute thread in sequence. The first data of the packet, in which a is a=1, (1, 2), is grouped with a value of a, and the time when the data (1, 2) is received is recorded. (2, 4) is the first data of the a=2 packet, and the time when the data (2, 4) is received is recorded. It is thus possible to obtain that the first data of the a=1 packet is received before the first data of the a=2 packet. When the calculation result is output to the aggregation thread, the calculation result of the a=1 packet is output first, and then the calculation result of the a=2 packet is output. This arrangement ensures that the order in which the respective aggregate threads receive the respective calculation results is fixed. By the arrangement, when the same parallel hash aggregation query instruction is executed on the data of the same data source, the sequence of the calculation results output by the same calculation thread is consistent, and further, the sequence of the finally output aggregation results is consistent.

Similarly, optionally, in executing S230, the sequencing of the aggregation thread in receiving the first data of each packet from the computing thread is recorded; s250 is replaced with: and acquiring an aggregation result obtained by parallel aggregation of a plurality of aggregation threads according to a third preset rule and the sequence by the result threads. It is assumed that a certain aggregation thread sequentially acquires intermediate results (1, 5), (1, 6), (2, 7) and (2, 8), groups the intermediate results with a value, and obtains an aggregation result after aggregation. Since the first data (1, 5) of the a=1 packet is received prior to the first data (2, 7) of the a=2 packet. When the result thread obtains the aggregation result from the aggregation thread, the aggregation result of the a=1 packet is obtained first, and then the aggregation result of the a=2 packet is obtained. By the arrangement, when the same parallel hash aggregation query instruction is executed on the data of the same data source, the sequence of the aggregation results output by the same aggregation thread is consistent, and further, the sequence of the aggregation results finally output is consistent.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

Fig. 4 is a schematic structural diagram of a data parallel aggregation apparatus according to an embodiment of the present disclosure, as shown in fig. 4, where the apparatus includes:

a first distributing unit 410, configured to distribute, by the data thread, data of the data source to a plurality of computing threads according to a first preset rule;

a second distributing unit 420, configured to distribute, by the coordination thread, a calculation result obtained by parallel calculation of the plurality of calculation threads to a plurality of aggregation threads according to a second preset rule;

and the acquiring unit 430 is configured to acquire, by using the result thread and according to a third preset rule, an aggregate result obtained by parallel aggregation of the multiple aggregation threads.

Further, the first distributing unit 410 is configured to:

acquiring a plurality of pieces of data from a data source through a data thread;

distributing at least one piece of data to each computing thread through the data thread according to a first preset rule so as to enable the computing threads to compute the received data in parallel;

after the parallel computation of the plurality of computation threads is completed, distributing at least one piece of data to each computation thread again through the data threads according to a first preset rule until the distribution of the plurality of pieces of data is completed.

Further, the first preset rule is polling.

Further, the device further comprises a calculating unit, wherein the calculating unit is used for controlling the plurality of calculating threads to calculate the received data in parallel after distributing the data of the data source to the plurality of calculating threads through the data threads according to a first preset rule, and the plurality of calculating threads store respective calculation results into respective hash tables, wherein the key words of the hash tables are aggregation thread serial numbers, and the values of the hash tables are calculation results.

Further, for a calculation result, the aggregate thread number corresponding to the calculation result is the remainder obtained by performing modulo operation on the calculation result and the aggregate thread number.

Further, the second distributing unit 420 is configured to:

and distributing the calculation results to the corresponding aggregation threads based on the aggregation thread serial numbers from the hash tables of the plurality of calculation threads through the coordination threads in sequence.

Further, the second distributing unit 420 is configured to:

after the distribution of all calculation results in one hash table is completed, the coordination thread distributes calculation results in the next hash table.

Further, the second distributing unit 420 is configured to:

after the distribution of all calculation results corresponding to the same aggregation thread number in all hash tables is completed, the coordination thread distributes all calculation results corresponding to the next aggregation thread number.

Further, the device further comprises an aggregation unit, which is used for distributing the calculation results obtained by the parallel calculation of the plurality of calculation threads to the plurality of aggregation threads through the coordination threads according to a second preset rule, and controlling the plurality of aggregation threads to aggregate the calculation results received by the aggregation threads in parallel based on the sequence of the received calculation results.

Further, the obtaining unit 430 is configured to:

sequentially obtaining corresponding aggregation results from the plurality of aggregation threads through the result threads; or alternatively, the process may be performed,

and sequentially outputting the aggregation results of the aggregation threads through the result threads.

The device disclosed in the above embodiment can implement the flow of the method disclosed in the above method embodiments, and has the same or corresponding beneficial effects. In order to avoid repetition, the description is omitted here.

Fig. 5 is a schematic hardware structure of an electronic device provided by an embodiment of the present disclosure, where, as shown in fig. 5, the electronic device may include an intelligent terminal such as a mobile phone, a PAD, and the electronic device includes:

one or more processors 301, one processor 301 being illustrated in fig. 5;

a memory 302;

the electronic device may further include: an input device 303 and an output device 304.

The processor 301, the memory 302, the input means 303 and the output means 304 in the electronic device may be connected by a bus or by other means, in fig. 5 by way of example.

The memory 302 serves as a non-transitory computer readable storage medium that can be used to store software programs, computer executable programs, and modules, such as program instructions/modules corresponding to the data parallel aggregation method in the embodiments of the present disclosure. The processor 301 executes various functional applications of the server and data processing, that is, implements the data parallel aggregation method of the above-described method embodiment, by running software programs, instructions, and modules stored in the memory 302.

Memory 302 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the electronic device, etc. In addition, memory 302 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 302 may optionally include memory located remotely from processor 301, which may be connected to the terminal device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input means 303 may be used to receive entered numeric or character information and to generate key signal inputs related to user settings and function control of the electronic device. The output device 304 may include a display device such as a display screen.

The disclosed embodiments also provide a computer-readable storage medium storing a program or instructions that when executed by a computer cause the computer to perform a data parallel aggregation method, the method comprising:

Optionally, the computer executable instructions, when executed by the computer processor, may also be used to perform the technical solution of the data parallel aggregation method provided by any embodiment of the disclosure.

From the above description of embodiments, it will be apparent to those skilled in the art that the present disclosure may be implemented by means of software and necessary general purpose hardware, but may of course also be implemented by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present disclosure may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments of the present disclosure.

It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data parallel aggregation, comprising:

2. The method of claim 1, wherein distributing the data of the data source to the plurality of computing threads by the data thread with a first preset rule comprises:

3. The method according to claim 1 or 2, wherein the first preset rule is polling.

4. The method of claim 1, wherein after distributing the data of the data source to the plurality of computing threads by the data thread with the first preset rule, the method further comprises:

the plurality of computing threads calculate the received data in parallel, and the plurality of computing threads store respective calculation results into respective hash tables, wherein the key words of the hash tables are aggregation thread serial numbers, and the values of the hash tables are calculation results.

5. The method of claim 4, wherein, for a calculation result, the aggregate thread number corresponding to the calculation result is a remainder obtained by modulo operation of the calculation result and the aggregate thread number.

6. The method of claim 4, wherein distributing the computation results of the parallel computation of the plurality of computation threads to the plurality of aggregation threads by the coordination thread according to a second preset rule comprises:

7. The method of claim 6, wherein distributing, by the coordination thread, the computation results to the corresponding aggregation threads based on the aggregate thread numbers from the hash tables of the respective plurality of computation threads in turn comprises:

8. The method of claim 6, wherein distributing, by the coordination thread, the computation results to the corresponding aggregation threads based on the aggregate thread numbers from the hash tables of the respective plurality of computation threads in turn comprises:

9. The method according to claim 1, wherein after distributing the calculation results obtained by parallel calculation of the plurality of calculation threads to the plurality of aggregation threads by the coordination thread according to a second preset rule, the method further comprises:

and the plurality of aggregation threads aggregate the received calculation results in parallel based on the sequence of the received calculation results.

10. The method of claim 1, wherein the obtaining, by the result thread, an aggregate result obtained by parallel aggregation of the plurality of aggregate threads according to a third preset rule comprises:

11. A data parallel aggregation apparatus, the apparatus comprising:

12. An electronic device, comprising: a processor and a memory;

the processor is adapted to perform the steps of the method according to any of claims 1 to 10 by invoking a program or instruction stored in the memory.

13. A computer readable storage medium storing a program or instructions for causing a computer to perform the steps of the method according to any one of claims 1 to 10.