CN112506991B - Method, system, electronic device and storage medium for parallel processing - Google Patents
Method, system, electronic device and storage medium for parallel processing Download PDFInfo
- Publication number
- CN112506991B CN112506991B CN202011397507.7A CN202011397507A CN112506991B CN 112506991 B CN112506991 B CN 112506991B CN 202011397507 A CN202011397507 A CN 202011397507A CN 112506991 B CN112506991 B CN 112506991B
- Authority
- CN
- China
- Prior art keywords
- calculated
- resource
- resources
- processing
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The application relates to a parallel processing method, a system, an electronic device and a storage medium, wherein the parallel processing method comprises the following steps: acquiring a plurality of resources to be calculated, and determining the dependency relationship among the plurality of resources to be calculated; determining a first resource to be calculated in the plurality of resources to be calculated according to the dependency relationship, and processing the first resource to be calculated in parallel, wherein the first resource to be calculated does not depend on the processing results of other resources to be calculated; and determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, and processing the second resource to be calculated in parallel, wherein the second resource to be calculated depends on the processing results of other resources to be calculated, and the resource to be calculated, which the second resource to be calculated depends on, is processed. Through the data assembling method and device, the problem of low data assembling efficiency is solved, and the beneficial effect of improving the data assembling efficiency is achieved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to methods, systems, electronic devices, and storage media for parallel processing.
Background
With the development of programming languages, the system architecture evolves from the original single service to micro-services, which have great advantages in stability and system flexibility, but bring about data search difficulties. After the micro service is applied, the corresponding data model can be split, and the list search is often the link table query of a plurality of models, so that the database can not be realized after being split. In order to solve the problem, a batch of frameworks for data assembly across a plurality of models are derived, and the plurality of models are assembled into a data wide table for upper-layer application search query. One key metric to consider during data assembly is concurrency handling performance. However, there are complex dependencies between data items, tens of assembly steps are required to complete a complete data, and the assembly sequence needs to be adjusted by considering the dependencies between the data items, which causes problems of high difficulty in data assembly configuration, poor processing performance, low concurrency, and the like, thereby resulting in low data assembly efficiency.
Disclosure of Invention
In the embodiment, a parallel processing method, a system, an electronic device and a storage medium are provided to solve the problem of low efficiency of data assembling configuration in the related art.
In a first aspect, a parallel processing method is provided in this embodiment, and includes:
acquiring a plurality of resources to be calculated, and determining the dependency relationship among the plurality of resources to be calculated;
determining a first resource to be calculated in the plurality of resources to be calculated according to the dependency relationship, and processing the first resource to be calculated in parallel, wherein the first resource to be calculated does not depend on the processing results of other resources to be calculated;
and determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, and processing the second resource to be calculated in parallel, wherein the second resource to be calculated depends on the processing results of other resources to be calculated, and the resource to be calculated, on which the second resource to be calculated depends, has been processed completely.
In some embodiments, determining, according to the dependency relationship, a first to-be-computed resource of the plurality of to-be-computed resources includes: judging whether the processing of each resource to be calculated in a plurality of resources to be calculated depends on the processing results of other resources to be calculated, and determining the resource to be calculated as the first resource to be calculated under the condition that the processing of the resource to be calculated does not depend on the processing results of other resources to be calculated;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, includes: and under the condition that the processing of the resource to be calculated is judged to depend on the processing results of other resources to be calculated, judging whether the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed completely according to the processing results of the resources to be calculated, and under the condition that the processing of the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed completely, determining the current resource to be calculated as the second resource to be calculated.
In some embodiments, after determining whether the processing result depended by the current resource to be computed belongs to the resource to be computed, the method further includes:
and under the condition that the to-be-calculated resource to which the processing result depended by the current to-be-calculated resource belongs is absent, determining that the data validity check of the current to-be-calculated resource fails, wherein the to-be-calculated resource with the failed data validity check is not processed.
In some embodiments, before determining whether the resource to be computed to which the processing result depended by the current resource to be computed belongs is completely processed, the method further includes:
judging whether the current resource to be calculated has cyclic dependence;
under the condition that the current resource to be calculated does not have cyclic dependence, determining to execute the step of judging whether the resource to be calculated to which the processing result depended by the current resource to be calculated belongs is processed completely;
and under the condition that the current resource to be calculated has cyclic dependence, determining that the data validity check of the current resource to be calculated fails, wherein the resource to be calculated which fails in the data validity check is not processed.
In some of these embodiments, determining whether the current resource to be computed has a circular dependency comprises:
and if the dependence times of the current resource to be calculated exceed a preset threshold, judging whether the current resource to be calculated has cyclic dependence.
In some of these embodiments, determining a dependency relationship between a plurality of the resources to be computed comprises: representing a dependency relationship among the plurality of resources to be computed by using a hierarchical dependency tree, wherein the hierarchical dependency tree comprises a plurality of root nodes and a plurality of child nodes, and each child node is connected with a root node at an upper layer or a child node at an upper layer, wherein the first resource to be computed is represented by the root node, and the second resource to be computed is represented by the child node;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, wherein the parallel processing of the second resource to be calculated comprises: and processing the resources to be calculated corresponding to the root nodes and/or the child nodes of each layer of the hierarchical dependency tree layer by layer from top to bottom, wherein the resources to be calculated corresponding to the root nodes and/or the child nodes belonging to the same layer are processed in parallel.
In some embodiments, representing the dependency relationships between the plurality of resources to be computed using a hierarchical dependency tree further comprises:
and if a certain child node depends on a plurality of nodes at different layers, placing the child node into the next layer of the latest construction layer number in the layer numbers of the nodes.
In a second aspect, a parallel processing system is provided in this embodiment, including: a terminal device, a transmission device and a server device; the terminal equipment is connected with the server equipment through the transmission equipment;
the terminal equipment is used for acquiring a plurality of resources to be calculated;
the transmission equipment is used for transmitting a plurality of resources to be calculated;
the server apparatus is used in the parallel processing method according to the first aspect.
In a third aspect, in this embodiment, there is provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the parallel processing method according to the first aspect when executing the computer program.
In a fourth aspect, in the present embodiment, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the parallel processing method of the first aspect described above.
Compared with the related art, the parallel processing method, the parallel processing system, the electronic device and the storage medium provided in the embodiment are used for acquiring a plurality of resources to be calculated and determining the dependency relationship among the plurality of resources to be calculated; according to the dependency relationship, a first resource to be calculated in the plurality of resources to be calculated is determined, and the first resource to be calculated is processed in parallel, wherein the first resource to be calculated does not depend on the processing results of other resources to be calculated. And determining the resource to be calculated which depends on the processing results of other resources to be calculated as a second resource to be calculated according to the dependency relationship and the processing results of the resources to be calculated, and processing the second resource to be calculated in parallel. At the moment, a plurality of resources to be calculated are processed, the problem of low data assembly configuration efficiency in the related technology is solved, and the data assembly efficiency is improved.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more concise and understandable description of the application, and features, objects, and advantages of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a block diagram of a hardware configuration of a parallel processing method according to an embodiment of the present application;
FIG. 2 is a first flowchart of a parallel processing method according to an embodiment of the present application;
FIG. 3 is a flow chart II of a parallel processing method according to an embodiment of the present application;
FIG. 4 is a flow chart diagram of a hierarchical dependency tree construction method of the preferred embodiment of the present application;
FIG. 5 is a diagram illustrating the construction of a hierarchical dependency tree according to the preferred embodiment of the present application.
Detailed Description
For a clearer understanding of the objects, aspects and advantages of the present application, reference is made to the following description and accompanying drawings.
Unless defined otherwise, technical or scientific terms used herein shall have the same general meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The use of the terms "a" and "an" and "the" and similar referents in the context of describing the invention (including a reference to the context of the specification and claims) are to be construed to cover both the singular and the plural, as well as the singular and plural. The terms "comprises," "comprising," "has," "having," and any variations thereof, as referred to in this application, are intended to cover non-exclusive inclusions; for example, a process, method, and system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or modules, but may include other steps or modules (elements) not listed or inherent to such process, method, article, or apparatus. Reference throughout this application to "connected," "coupled," and the like is not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference to "a plurality" in this application means two or more. "and/or" describes the association relationship of the associated object, indicating that there may be three relationships, for example, "a and/or B" may indicate: a exists alone, A and B exist simultaneously, and B exists alone. In general, the character "/" indicates a relationship in which the objects associated before and after are an "or". Reference in the present application to the terms "first," "second," "third," etc., merely distinguish between similar objects and do not denote a particular order or importance to the objects.
The method embodiments provided in the present embodiment may be executed in a terminal, a computer, or a similar computing device. For example, the present invention is executed on a terminal, and fig. 1 is a block diagram of a hardware structure of the terminal according to the parallel processing method of the present embodiment. As shown in fig. 1, the terminal may include one or more processors 102 (only one shown in fig. 1) and a memory 104 for storing data, wherein the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those of ordinary skill in the art that the structure shown in fig. 1 is merely an illustration and is not intended to limit the structure of the terminal described above. For example, the terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 can be used for storing computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the parallel processing method in the embodiment, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. The network described above includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In the present embodiment, a parallel processing method is provided, and fig. 2 is a flowchart of the parallel processing method of the present embodiment, as shown in fig. 2, the flowchart includes the following steps:
step S201, acquiring a plurality of resources to be calculated, and determining a dependency relationship between the plurality of resources to be calculated.
Step S202, according to the dependency relationship, a first resource to be calculated in the plurality of resources to be calculated is determined, and the first resource to be calculated is processed in parallel, wherein the first resource to be calculated does not depend on the processing results of other resources to be calculated.
Step S203, according to the dependency relationship and the processing result of the resource to be calculated, determining a second resource to be calculated in the plurality of resources to be calculated, and processing the second resource to be calculated in parallel, wherein the second resource to be calculated depends on the processing results of other resources to be calculated, and the resource to be calculated, which the second resource to be calculated depends on, has been processed.
When a plurality of resources to be calculated are processed, the dependency relationship among the plurality of resources to be calculated is determined firstly, so that the mutual relation among the plurality of resources to be calculated can be obtained, the complex relation among the plurality of resources to be calculated is favorably combed, and the resources to be calculated are favorably and orderly processed. Among the resources to be calculated, the resource to be calculated, which does not depend on the processing result of the other resource to be calculated, is used as the first resource to be calculated, where the processing result of the other resource to be calculated refers to one or more resources to be calculated, except the first resource to be calculated, among the multiple resources to be calculated. Then the first resource to be calculated is processed in parallel, namely, the resource to be calculated which is only used as a depended party is firstly processed in parallel in the plurality of resources to be calculated. Secondly, after the first resource to be calculated is processed, the resource to be calculated which depends on the processing result of the other resource to be calculated is taken as a second resource to be calculated according to the dependency party which depends on the processing result of the other resource to be calculated in the dependency relationship. And finally, under the condition that the first resource to be calculated is determined to be processed completely, processing the second resource to be calculated in parallel. The second resource to be calculated here includes a resource to be calculated having a dependency relationship, in addition to the first resource to be calculated, among the plurality of resources to be calculated. That is, after a first resource to be calculated that does not depend on the processing results of other resources to be calculated is determined and the first resource to be calculated is parallelized, a second resource to be calculated that can be parallelized is determined. The second resource to be calculated at this time is a resource to be calculated which depends on the processing result of other resources to be calculated.
In the above steps, the dependency relationship among the multiple resources to be calculated is utilized, the first resource to be calculated is determined and processed in parallel, and the second resource to be calculated is determined and processed in parallel according to the dependency relationship and the processing result of the resource to be calculated, so that the multiple resources to be calculated are processed in parallel. In the whole step, only the dependency relationship among the multiple resources to be calculated needs to be judged, and then the first resource to be calculated and the second resource to be calculated are determined, so that parallel processing is realized. No complex and lengthy assembly steps are required for the resources to be computed. Therefore, the steps are adopted, the data assembly configuration is simplified, the concurrent processing performance of the data assembly is improved, and the data assembly efficiency is improved.
In some of these embodiments, determining, from the dependencies, a first to-be-computed resource of the plurality of to-be-computed resources comprises: judging whether the processing of each resource to be calculated in the plurality of resources to be calculated depends on the processing results of other resources to be calculated, and determining the resource to be calculated as a first resource to be calculated under the condition that the processing of the resource to be calculated does not depend on the processing results of other resources to be calculated;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated includes: and under the condition that the processing of the resources to be calculated is judged to depend on the processing results of other resources to be calculated, judging whether the resources to be calculated to which the processing result dependent on the current resources to be calculated belongs are processed or not according to the processing results of the resources to be calculated, and under the condition that the processing of the resources to be calculated to which the processing result dependent on the current resources to be calculated belongs is processed completely, determining the current resources to be calculated to be second resources to be calculated.
In the dependency relationship among a plurality of resources to be calculated, two roles of a relying party and a depended party exist. In the whole dependency relationship, if a certain resource to be computed does not serve as a relying party, only the role of the depended party is possessed, that is, when a certain resource to be computed does not depend on the processing results of other resources to be computed, the resource to be computed is the first resource to be computed. The other to-be-calculated resource herein refers to one or more to-be-calculated resources other than the first to-be-calculated resource among the plurality of to-be-calculated resources.
And under the condition that the processing of the resource to be calculated is judged to depend on the processing results of other resources to be calculated, judging whether the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed or not according to the processing results of the resources to be calculated, and under the condition that the processing of the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed, determining the current resource to be calculated as a second resource to be calculated. That is, when it is determined that a certain resource to be computed depends on the processing results of other resources to be computed, if the resource to be computed to which the processing result to which the current resource to be computed depends belongs is processed, the resource to be computed which depends on the processing results of other resources to be computed, that is, the so-called relying party, may be determined as the second resource to be computed which can perform parallel processing.
In the above steps, the first resource to be calculated and the second resource to be calculated are divided for the plurality of resources to be calculated by using the dependency relationship among the plurality of resources to be calculated, which is helpful for realizing the ordered processing of the plurality of resources to be calculated, thereby improving the efficiency of data assembly.
In some embodiments, after determining whether the resource to be computed to which the processing result depended by the current resource to be computed belongs is processed, the method further includes:
and under the condition that the to-be-calculated resource to which the processing result depended by the current to-be-calculated resource belongs is absent, determining that the data validity check of the current to-be-calculated resource fails, wherein the to-be-calculated resource with the failed data validity check is not processed.
As described above, determining that the current resource to be computed is the second resource to be computed requires determining a processing result that the current resource to be computed depends on other resources to be computed. Therefore, when the relying party of the second resource to be calculated, that is, the resource to be calculated to which the processing result depended by the current resource to be calculated belongs does not exist, the current resource to be calculated is missing at this time, and the validity check fails, and cannot be applied to the next step.
In the above steps, whether the data of the current resource to be calculated has validity is judged by judging whether the resource to be calculated to which the processing result depending on the current resource to be calculated belongs exists, so that the validity of the data of the current resource to be calculated is verified. If the data validity verification fails, the failed data is not processed, so that resource waste in the data assembling process is avoided, and the data assembling efficiency is improved.
In some embodiments, before determining whether the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed, the method further includes:
judging whether the current resource to be calculated has cyclic dependence;
under the condition that the current resource to be calculated does not have circular dependence, determining to execute the step of judging whether the resource to be calculated to which the processing result depended by the current resource to be calculated belongs is processed or not;
and under the condition that the current resource to be calculated has cyclic dependence, determining that the data validity check of the current resource to be calculated fails, wherein the resource to be calculated which fails in the data validity check is not processed.
In the complex dependency relationship of a plurality of resources to be calculated, because the relation among the resources to be calculated is tight, many resources to be calculated are both depended parties and depended parties, and then the phenomenon of circular dependency is easy to occur. But is ineffective for the resources to be calculated with cyclic dependence and does not need to be processed. It is necessary to determine whether the current resource to be calculated has a circular dependency. And when judging that the current resource to be calculated does not have circular dependence, utilizing the resource to be calculated to carry out the next step.
In the steps, whether the current resource to be calculated has cyclic dependence is judged, so that invalid data in the data assembling process can be screened, the data for data assembling can be screened, and the efficiency of data assembling is improved.
In some of these embodiments, determining whether the current resource to be computed has a circular dependency comprises:
and if the dependence times of the current resource to be calculated exceed a preset threshold, judging whether the current resource to be calculated has cyclic dependence.
Because the complicated dependency relationship among a plurality of resources to be calculated is combed in the preamble step, the judgment of whether the current resources to be calculated have loop dependency or not can be carried out by utilizing the loop times of the current resources to be calculated. When the number of times of circulation of the current resource to be calculated exceeds a preset threshold value, determining that the current resource to be calculated has circulation dependence.
In the above steps, whether the current resource to be calculated has the loop dependence is judged by using the loop times of the current resource to be calculated, which is helpful for screening out invalid resources to be calculated. These currently pending computing resources, which are not available, cannot be used for data assembly. Therefore, the invalid resources to be calculated are eliminated in advance, so that the resource waste in the data assembling process can be avoided, and the efficiency of data assembling is improved.
In some of these embodiments, determining the dependency relationships between the plurality of resources to be computed comprises: representing the dependency relationship among a plurality of resources to be calculated by using a hierarchical dependency tree, wherein the hierarchical dependency tree comprises a plurality of root nodes and a plurality of child nodes, and each child node is connected with the root node at the upper layer or the child node at the upper layer, wherein the first resource to be calculated is represented by the root node, and the second resource to be calculated is represented by the child nodes;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, wherein the parallel processing of the second resource to be calculated comprises: and processing the resources to be calculated corresponding to the root nodes and/or the child nodes of each layer of the hierarchical dependency tree layer by layer from top to bottom, wherein the resources to be calculated corresponding to the root nodes and/or the child nodes belonging to the same layer are processed in parallel.
The dependency relationship among the multiple resources to be calculated is expressed in the form of a hierarchical dependency tree, so that depended parties and dependent parties among the multiple resources to be calculated can be embodied more directly and clearly. When constructing the hierarchical dependency tree, firstly, the resource to be calculated which does not depend on the processing result of other resource to be calculated is placed at the position of the root node, and then, from top to bottom, the second resource to be calculated is gradually utilized to construct the child node, and the child node at the moment is connected with the root node at the upper layer or connected with the child node at the upper layer. That is to say, before processing a plurality of resources to be computed, each batch of resources to be computed, which can be processed in parallel, can be sorted in advance in the same hierarchy by using the hierarchical dependency tree, so as to facilitate the subsequent direct parallel processing. Therefore, after the arrangement of the plurality of resources to be calculated is finished, the dependency relationship among the plurality of resources to be calculated can be obtained clearly at a glance. And finally, processing the resources to be calculated which are positioned at the same layer in the hierarchical dependency tree in parallel.
In the above steps, a hierarchical dependency tree is used to comb the dependency relationship among a plurality of resources to be calculated, the resources to be calculated which do not depend on the processing results of other resources to be calculated are used as root nodes, the resources to be calculated which depend on the processing results of other resources to be calculated are used as child nodes, and finally the resources to be calculated which are located on the same layer are processed in parallel. Through the steps, the complex dependency relationships among the multiple resources to be calculated are sorted, so that the clear understanding of the relationship among the multiple resources to be calculated is facilitated; the to-be-computed resources of the nodes on the same layer are processed in parallel, unnecessary waiting and occupation of system resources in data assembly are avoided, and throughput of data assembly is improved, so that efficiency of data assembly is improved.
In some of these embodiments, using the hierarchical dependency tree to represent dependencies between the plurality of resources to be computed further comprises: and if a certain child node depends on a plurality of nodes at different layers, placing the child node into the next layer of the latest construction layer number in the layer numbers to which the plurality of nodes belong.
Because of the close connection among multiple resources to be computed. The same resource to be calculated is possibly dependent on the resource to be calculated at different levels in the hierarchical dependency tree, and then the resource to be calculated is placed in the next layer of the lowest level of the plurality of nodes which are dependent on the resource to be calculated.
The construction of the hierarchical dependency tree is further regulated through the steps, so that the construction of the hierarchical dependency tree which is more clear and ordered is facilitated, and the execution of the subsequent steps is facilitated.
The present embodiment is described and illustrated below by means of preferred embodiments.
Fig. 3 is a preferred flowchart of the parallel processing method of the present embodiment, and as shown in fig. 3, the parallel processing method includes the following steps:
step S301, load a plurality of resources to be calculated.
In some of these embodiments, the list of tasks that need to be performed is retrieved from the configuration database. And according to the task content in the list, one function is used for corresponding to one task. The functions herein include, but are not limited to, specific business logic code.
Step S302, initializing the dependent items and the output items of the plurality of resources to be calculated to obtain the dependent relationship among the plurality of resources to be calculated.
In some embodiments, the dependency items and output items of a plurality of resources to be calculated are found, and the found dependency items and output items are used for establishing a dependency relationship.
Step S303, registering the dependency relationship among the plurality of resources to be calculated to the resource manager, and verifying whether the dependency relationship is valid.
In some of these embodiments, the dependency obtained in step S302 is sent to a resource manager. And verifying whether the dependency relationship conforms to the rule or not by using a preset function.
In some embodiments, the data is checked for validity using a predetermined function, and the check items include, but are not limited to, the following two items:
a. checking the completeness of the dependent term.
And circularly checking a plurality of resources to be calculated, and searching whether the resources to be calculated, to which the processing result depended by each resource to be calculated belongs, are provided by other resources to be calculated. If missing, the check fails.
b. Loop checking for dependent items.
First, an unbounded queue is established. A plurality of resources to be computed are placed in the queue. Then, taking out the resource to be calculated at the head of the queue from the queue, obtaining all the resources to be calculated depending on the output items of the resources to be calculated, putting the resource to be calculated depending on the output items of the resources to be calculated at present into the tail of the unbounded queue, updating the hierarchy to which the resource to be calculated depending on the processing results of other resources to be calculated belongs to the current hierarchy +1 (the next hierarchy), and performing cycle counting. When the cycle count exceeds a preset maximum value, it indicates that cycle dependency exists.
In some embodiments, the rule is a rule inside the resource manager, i.e. the dependency cannot generate a circular dependency, and the sub-layer structure can only depend upwards.
And step S304, under the condition that the dependency relationship among a plurality of resources to be calculated has effectiveness, constructing a hierarchical dependency tree by using the dependency relationship.
In some embodiments, as shown in fig. 4, building a hierarchical dependency tree using dependencies among a plurality of resources to be computed includes the following steps:
step S3041, parsing the dependent items and output items of multiple resources to be computed, and constructing two maps of a dependency-provider and an output-provider:
1. dependency is key, dependency is value;
2. the output is key, and the output side is value.
Step S3042, the resource to be calculated that is not dependent is taken as a root node.
In some embodiments, the resource to be calculated which is not dependent is stored in the list separately, and the resource to be calculated in the list is taken as a root node.
Step S3043, according to the two maps obtained in step S3041, querying an output of the resource to be calculated to which the root node belongs, as a child node.
In some embodiments, a hierarchical dependency tree is constructed layer by layer according to the two maps obtained in step S3041. And taking the resource to be calculated to which the dependency root node belongs as a child node, putting the child node into the hierarchical dependency tree, and constructing the hierarchical dependency tree layer by layer from top to bottom. Until all resources to be computed have been placed in the hierarchical dependency tree. At this time, the hierarchical dependency tree is constructed as shown in fig. 5.
Step S3044, it is determined whether a child node dependent on the node of the previous layer already exists in the hierarchical dependency tree.
In some embodiments, when a resource to be computed to which a processing result depended by a certain resource to be computed belongs already appears in the hierarchical dependency tree, the hierarchy in which the resource to be computed belongs is adjusted to be the next hierarchy of the current hierarchy.
In some embodiments, when the resource to be computed to which the processing result depended by a certain resource to be computed belongs does not appear in the hierarchical dependency tree, the operation is ended.
Step S305, according to the hierarchical dependency tree, a plurality of resources to be calculated are processed from top to bottom. And processing the resources to be calculated at the same level in parallel.
In some embodiments, the independent resources to be calculated, namely the resources to be calculated at the position of the root node of the hierarchical dependency tree, are executed in parallel preferentially according to the hierarchical dependency tree. And executing the resources to be calculated with the dependence in parallel, namely executing the resources to be calculated positioned at the positions of the child nodes of the hierarchical dependence tree in parallel.
And step S306, returning after the plurality of resource tasks to be calculated are calculated.
The method comprises the steps of firstly finding out a dependence item and an output item in a plurality of resources to be calculated to obtain the mutual dependence relationship among the plurality of resources to be calculated, and verifying the dependence relationship. The operation of the step is helpful to eliminate invalid data in a plurality of resources to be calculated, and avoids wasting related resources in the subsequent steps. The resource manager automatically coordinates the assembly sequence among a plurality of resources to be calculated, thereby avoiding unnecessary waiting and occupying system resources. And the resources to be calculated without mutual dependency relationship are automatically executed in parallel, so that the throughput of data assembly is improved. Moreover, the sequence of assembling the link does not need to be considered when assembling the configuration, and the configuration and later maintenance cost are simplified.
In the above steps, the dependency item and the output item are first found out from a plurality of resources to be computed to obtain the mutual dependency relationship among the plurality of resources to be computed, and under the condition that the dependency relationship is verified to be valid, a hierarchical dependency tree is established according to the dependency relationship, and finally the plurality of resources to be computed are processed sequentially from top to bottom according to the hierarchical dependency tree, and the resources to be computed located in the same level are processed in parallel.
Therefore, through the steps, the plurality of resources to be calculated are combed and processed in order, and the data assembling efficiency is improved.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here. For example, step S3041 and step S3042 may exchange positions.
In this embodiment, a parallel processing system is further provided, including: a terminal device, a transmission device and a server device; the terminal device is connected to the server device through the transmission device, and the server device is configured to execute the steps in any one of the method embodiments.
There is also provided in this embodiment an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the method embodiments described above.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, acquiring a plurality of resources to be calculated, and determining the dependency relationship among the plurality of resources to be calculated;
s2, determining a first resource to be calculated in the plurality of resources to be calculated according to the dependency relationship, and processing the first resource to be calculated in parallel, wherein the first resource to be calculated does not depend on the processing results of other resources to be calculated;
and S3, determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, and processing the second resource to be calculated in parallel, wherein the second resource to be calculated depends on the processing results of other resources to be calculated, and the resource to be calculated, which the second resource to be calculated depends on, is processed completely.
It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiment and optional implementation manners, and details are not described in this embodiment again.
In addition, in combination with the parallel processing method provided in the foregoing embodiment, a storage medium may also be provided to implement in this embodiment. The storage medium has a computer program stored thereon; the computer program, when executed by a processor, implements any of the parallel processing methods in the above embodiments.
It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be derived by a person skilled in the art from the examples provided herein without inventive step, shall fall within the scope of protection of the present application.
It is obvious that the drawings are only examples or embodiments of the present application, and it is obvious to those skilled in the art that the present application can be applied to other similar cases according to the drawings without creative efforts. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
The term "embodiment" is used herein to mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly or implicitly understood by one of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the patent protection. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims (8)
1. A parallel processing method, comprising:
acquiring a plurality of resources to be calculated, initializing the dependent items and output items of the plurality of resources to be calculated to obtain the dependent relationship among the plurality of resources to be calculated; registering the dependency relationship among a plurality of resources to be calculated to a resource manager, and verifying whether the dependency relationship is valid or not; determining the dependency relationship among a plurality of resources to be calculated;
determining a first resource to be calculated in the plurality of resources to be calculated according to the dependency relationship, automatically coordinating an assembly sequence among the plurality of resources to be calculated through the resource manager, and processing the first resource to be calculated in parallel, wherein the first resource to be calculated does not depend on processing results of other resources to be calculated;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, and processing the second resource to be calculated in parallel, wherein the second resource to be calculated depends on the processing results of other resources to be calculated, and the resource to be calculated, which the second resource to be calculated depends on, has been processed;
determining a first to-be-computed resource of the plurality of to-be-computed resources according to the dependency comprises:
judging whether the processing of each resource to be calculated in a plurality of resources to be calculated depends on the processing results of other resources to be calculated, and determining the resource to be calculated as the first resource to be calculated under the condition that the processing of the resource to be calculated does not depend on the processing results of other resources to be calculated;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, includes: under the condition that the processing of the resource to be calculated is judged to depend on the processing results of other resources to be calculated, judging whether the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed completely according to the processing results of the resources to be calculated, and under the condition that the processing of the resource to be calculated to which the processing result dependent on the current resource to be calculated belongs is processed completely, determining the current resource to be calculated as the second resource to be calculated;
after judging whether the processing result depended by the current resource to be calculated belongs to the resource to be calculated and is processed, the method further comprises the following steps:
and under the condition that the to-be-calculated resource to which the processing result depended by the current to-be-calculated resource belongs is absent, determining that the data validity check of the current to-be-calculated resource fails, wherein the to-be-calculated resource with the failed data validity check is not processed.
2. The parallel processing method according to claim 1, wherein before determining whether the resource to be computed to which the processing result dependent on the current resource to be computed belongs is processed, the method further comprises:
judging whether the current resource to be calculated has cyclic dependence; under the condition that the current resource to be calculated does not have circular dependence, determining to execute the step of judging whether the resource to be calculated to which the processing result depended by the current resource to be calculated belongs is processed or not; and under the condition that the current resource to be calculated has cyclic dependence, determining that the data validity check of the current resource to be calculated fails, wherein the resource to be calculated which fails in the data validity check is not processed.
3. A parallel processing method according to claim 2, wherein determining whether the current resource to be computed has a loop dependency comprises:
and if the dependence times of the current resource to be calculated exceed a preset threshold, judging whether the current resource to be calculated has cyclic dependence.
4. A parallel processing method according to claim 1, wherein determining dependencies between a plurality of the resources to be computed comprises:
representing a dependency relationship among the plurality of resources to be computed by using a hierarchical dependency tree, wherein the hierarchical dependency tree comprises a plurality of root nodes and a plurality of child nodes, and each child node is connected with a root node at an upper layer or a child node at an upper layer, wherein the first resource to be computed is represented by the root node, and the second resource to be computed is represented by the child node;
determining a second resource to be calculated in the plurality of resources to be calculated according to the dependency relationship and the processing result of the resource to be calculated, wherein the parallel processing of the second resource to be calculated comprises: and processing the resources to be calculated corresponding to the root nodes and/or the child nodes of each layer of the hierarchical dependency tree layer by layer from top to bottom, wherein the resources to be calculated corresponding to the root nodes and/or the child nodes belonging to the same layer are processed in parallel.
5. A parallel processing method according to claim 4, wherein representing dependencies between the plurality of resources to be computed using a hierarchical dependency tree further comprises: and if a certain child node depends on a plurality of nodes at different layers, placing the child node into the next layer of the latest construction layer number in the layer numbers of the nodes.
6. A parallel processing system, comprising: a terminal device, a transmission device and a server device; the terminal equipment is connected with the server equipment through the transmission equipment; the terminal equipment is used for acquiring a plurality of resources to be calculated; the transmission equipment is used for transmitting a plurality of resources to be calculated; the server apparatus is configured to execute the parallel processing method of any one of claims 1 to 5.
7. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is configured to execute the computer program to perform the parallel processing method according to any one of claims 1 to 5.
8. A storage medium having stored thereon a computer program, wherein the computer program is arranged to execute the parallel processing method of any of claims 1 to 5 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011397507.7A CN112506991B (en) | 2020-12-03 | 2020-12-03 | Method, system, electronic device and storage medium for parallel processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011397507.7A CN112506991B (en) | 2020-12-03 | 2020-12-03 | Method, system, electronic device and storage medium for parallel processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112506991A CN112506991A (en) | 2021-03-16 |
| CN112506991B true CN112506991B (en) | 2023-04-07 |
Family
ID=74969684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011397507.7A Active CN112506991B (en) | 2020-12-03 | 2020-12-03 | Method, system, electronic device and storage medium for parallel processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506991B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8073836B2 (en) * | 2004-03-01 | 2011-12-06 | Epicor Software Corporation | System for viewing databases |
| CN102591712B (en) * | 2011-12-30 | 2013-11-20 | 大连理工大学 | Decoupling parallel scheduling method for rely tasks in cloud computing |
| CN109697122B (en) * | 2017-10-20 | 2024-03-15 | 华为技术有限公司 | Task processing method, device and computer storage medium |
| CN110554909A (en) * | 2019-09-06 | 2019-12-10 | 腾讯科技（深圳）有限公司 | task scheduling processing method and device and computer equipment |
| CN110704170A (en) * | 2019-09-09 | 2020-01-17 | 深圳壹账通智能科技有限公司 | Batch task processing method and device, computer equipment and storage medium |
-
2020
- 2020-12-03 CN CN202011397507.7A patent/CN112506991B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112506991A (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112465359B (en) | Calculation force calling method and device | |
| CN110471746B (en) | Distributed transaction callback method, device and system | |
| CN110245029A (en) | A kind of data processing method, device, storage medium and server | |
| CN111932257A (en) | Block chain parallelization processing method and device | |
| CN113467905A (en) | Task processing method and system | |
| CN105955790B (en) | Data processing method and device | |
| CN112506991B (en) | Method, system, electronic device and storage medium for parallel processing | |
| CN111831452A (en) | Task execution method and device, storage medium and electronic device | |
| CN112905318A (en) | Message priority queue based distributed switching network element calling method and system | |
| CN108427599A (en) | Method, apparatus and storage medium is uniformly processed in asynchronous task | |
| CN107168795A (en) | Codon deviation factor model method based on CPU GPU isomery combined type parallel computation frames | |
| CN115328457A (en) | Method and device for realizing form page based on parameter configuration | |
| CN109857533A (en) | A kind of timed task dispatching method, device and intelligent terminal | |
| CN112148333B (en) | Block chain server firmware updating system | |
| CN101141663B (en) | Method and system for improving intelligent network access database efficiency | |
| CN111191103B (en) | Method, device and storage medium for identifying and analyzing enterprise subject information from internet | |
| CN113986495A (en) | Task execution method, device, equipment and storage medium | |
| CN110580172B (en) | Configuration rule verification method and device, storage medium and electronic device | |
| Nanz et al. | Topology-dependent abstractions of broadcast networks | |
| CN109451016A (en) | Data downloading management method, system and relevant device | |
| CN112527486A (en) | Scheduling optimization method and device | |
| CN112148479B (en) | Data processing method, device and computer equipment based on Siddi log audit | |
| CN112612514B (en) | Program development method and device, storage medium and electronic device | |
| CN115525722B (en) | Wide table data synchronization method and device, electronic device and storage medium | |
| CN114490095B (en) | Request result determination method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |