CN111475684A - Data processing network system and calculation chart generation method thereof - Google Patents
Data processing network system and calculation chart generation method thereof Download PDFInfo
- Publication number
- CN111475684A CN111475684A CN202010600035.4A CN202010600035A CN111475684A CN 111475684 A CN111475684 A CN 111475684A CN 202010600035 A CN202010600035 A CN 202010600035A CN 111475684 A CN111475684 A CN 111475684A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- task
- type derivation
- data type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 43
- 238000000034 method Methods 0.000 title claims description 32
- 238000004364 calculation method Methods 0.000 title claims description 14
- 238000009795 derivation Methods 0.000 claims abstract description 197
- 238000004519 manufacturing process Methods 0.000 claims abstract description 10
- 238000010586 diagram Methods 0.000 claims description 20
- 230000008569 process Effects 0.000 claims description 18
- 238000013507 mapping Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 3
- 238000011144 upstream manufacturing Methods 0.000 description 6
- 239000003550 marker Substances 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 239000000872 buffer Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/22—Procedural
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing network system, comprising: a row of data type derivation nodes which are connected in sequence according to the production and consumption relationship; and a row of data operation nodes which are sequentially connected according to the production consumption relation, wherein each data operation node is connected with the consumption of the corresponding data type derivation node of the row where the data operation node is located and is connected with the consumption of the previous data type derivation node, each data type derivation node performs derivation based on the data type derivation result of the previous data type derivation node to obtain a data type derivation result, and each data operation node performs operation by using the data type derivation result of the connected data type derivation node, the data type derivation result of the previous data type derivation node and the operation result of the previous data operation node and outputs the operation result to the next data operation node.
Description
Technical Field
The present disclosure relates to a data processing technology. More particularly, the present disclosure relates to a data processing network system and a computation graph generation method thereof.
Background
With the popularization of distributed computing, a large job is divided to deploy different parts of data to each computing device of different distributed data processing systems for processing, so that a task logic diagram needs to be generated for a job task, a task computation diagram needs to be generated based on the task logic diagram, and finally the computation diagram needs to be actually distributed and deployed to a specified computing resource based on the computing resource.
In actual operation, it is usually involved in determining the type of data and calculating the specific data value. In a dynamic computing environment, an actual data operation typically involves two actions, one is to perform a derivation of the data type, and the other is to perform a calculation of the data value of the data, the derivation of the data type being a prerequisite for performing the numerical calculation. Therefore, the designated tasks of the task nodes contained in the task logic diagram of the job task have data derivation tasks and data calculation tasks. Usually, the data derivation task and the data calculation task are performed sequentially, but in some cases, in order to intuitively understand the need, the type of data may need to be changed in the execution process of the data calculation task, or some data calculation results are needed for derivation of the data type. Therefore, task nodes in the task logic diagram adopt a serial mode based on the task execution sequence, and therefore derivation nodes and operation nodes in the corresponding calculation diagram also adopt a serial structure.
However, the time spent by the data processing network system corresponding to the serial computation graph in the process of executing data processing is the sum of all data type derivation time and all data operation time. With the increasing computing functions of various single computing devices, it is in an extreme state to increase the computing speed of the computing devices. On a single data processing path, because of the data dependency relationship between the preceding and following job tasks, it is not feasible to perform parallel processing on the preceding and following task nodes of the job task to improve the data processing speed, which brings a technical bottleneck for improving the speed of distributed data processing. Therefore, it is desirable to further overcome the current technical bottlenecks.
Disclosure of Invention
It is an object of the present disclosure to provide a solution to at least one of the above problems. Specifically, according to one aspect of the present disclosure, there is provided a data processing network system including: a row of data type derivation nodes which are connected in sequence according to the production and consumption relationship; and a row of data operation nodes which are sequentially connected according to the production consumption relation, wherein each data operation node is connected with the consumption of the corresponding data type derivation node of the row where the data operation node is located and is connected with the consumption of the previous data type derivation node, each data type derivation node performs derivation based on the data type derivation result of the previous data type derivation node to obtain a data type derivation result, and each data operation node performs operation by using the data type derivation result of the connected data type derivation node, the data type derivation result of the previous data type derivation node and the operation result of the previous data operation node and outputs the operation result to the next data operation node.
The data processing network system according to the present disclosure, wherein the data operation node includes a hybrid data operation node, the hybrid data operation node and the data type derivation node in the row of the hybrid data operation node can modify consumption connection and control connection with the next-level data type derivation node corresponding to the next-level data operation node, so that the next-level data type derivation node performs data type derivation after obtaining the message of the hybrid data operation node.
According to another aspect of the present disclosure, there is provided a computation graph generation method including: acquiring a task logic diagram, wherein the task logic diagram comprises a plurality of task nodes which are sequentially connected according to a preset logic sequence and used for completing a specified task, and the specified task of each task node comprises a data type derivation task and a data operation task; analyzing the appointed operation tasks of the task nodes, mapping the data type derivation task of each task node into a data type derivation node one by one, and mapping the data operation task of each task node into a data operation node one by one; and arranging the data type derivation nodes into a row of data type derivation nodes which are sequentially connected according to the production and consumption relationship according to the preset logic sequence of the task nodes, arranging the data operation nodes into a row of data operation nodes which are sequentially connected according to the production and consumption relationship according to the preset logic sequence of the task nodes, establishing consumption connection between the data type derivation nodes related to the same task node and the data operation nodes, and connecting each data operation node with the previous data type derivation node in a consumption manner, thereby forming a calculation graph corresponding to the operation task logic graph.
According to the computational graph generation method of the present disclosure, the analyzing the designated job task of the task node further includes: analyzing and identifying mixed operation tasks in the designated operation tasks, wherein the mixed operation tasks can change data types in the operation process; and mapping the identified hybrid operation task into a hybrid data operation node and further establishing control connection between the hybrid data operation node and a next-level data type derivation node corresponding to the next-level data operation node.
According to the data processing network system and the calculation graph generation method thereof disclosed by the invention, the data derivation process and the data operation process in each task node of all task nodes in the same data processing path are separately mapped to the independent calculation nodes (including the derivation node and the operation node), so that all the data derivation processes and the operation processes in the same data processing path can be processed in parallel, therefore, the unified operation service can be completed, and the time spent in the parallel data processing network system disclosed by the invention can be saved by nearly half compared with the time spent in the conventional serial data processing network system. Furthermore, by determining the hybrid data operation nodes, the data processing network system and the computation graph generating method thereof according to the present disclosure do not impose an additional mental burden on the programmer, that is, the programmer does not need to consider whether to exclude a programming mode of hybrid programming of the data derivation process and the data operation process because the programmer runs the program on the data processing network system according to the present disclosure.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Drawings
Fig. 1 is a schematic diagram illustrating a data processing network system according to a first embodiment of the present disclosure.
Fig. 2 is a schematic diagram of a data processing network system according to a second embodiment of the present disclosure.
Fig. 3 is a schematic diagram of a partial network of a data processing network system according to a second embodiment of the present disclosure.
Detailed Description
The present invention will be described in further detail with reference to the following examples and the accompanying drawings so that those skilled in the art can practice the invention with reference to the description.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, one of the two possible position markers may be referred to hereinafter as a first position marker and may also be referred to as a second position marker, and similarly, the other of the two possible position markers may be referred to as a second position marker and may also be referred to as a first logical position marker, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
For a better understanding of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
For convenience in describing the principles of the present disclosure, the present disclosure develops an illustration based on a deep learning scenario. Fig. 1 is a schematic diagram illustrating a data processing network system according to a first embodiment of the present disclosure. As shown in fig. 1, a data processing network system according to the present disclosure is represented using a task computation graph for convenience of illustration. The data processing network system includes pairs of data type derivation nodes and data operation nodes, such as a data type derivation node 01 and a data operation node 01, a data type derivation node 02 and a data operation node 02, a data type derivation node 03 and a data operation node 03, a data type derivation node 04 and a data operation node 04, a data type derivation node 05 and a data operation node 05 …, a data type derivation node N and a data operation node N, and so on. As shown, the paired data type derivation node and data operation node corresponds to one task node in the task logic diagram. Each task node assumes a specified task. Task nodes, such as task node 01, task node 02, task node 03, task node 04, task node 05, and task node N …, form a data processing path for a given job task. Typically, a task node consumes data output by its previous task node. In the data processing network system corresponding to the task computation graph, one pair of the data type derivation node and the data operation node consumes the results output by the previous pair of the data type derivation node and the data operation node, for example, the data type derivation node 02 and the data operation node 02 use the data output by the data type derivation node 01 and the data operation node 02. Therefore, the task nodes in each task logic graph are mapped to a pair of data type derivation nodes and data operation nodes in the task computation graph. As shown in fig. 1, all data type derivation nodes that complete a given task form a column of data type derivation paths dedicated to data type derivation, and all data operation nodes form a column of data operation paths dedicated to data operation. The latter data type derivation node performs a data type derivation operation based on the output data of the former data type derivation node. For example, the data type derivation node 02 performs a data type derivation operation based on the output data of the data type derivation node 01, and the derivation result data is used or consumed by the data operation node 02. Similarly, the latter data operation node performs data operation based on the output data of the former data operation node. For example, the data operation node 02 performs a data operation based on the output data of the data operation node 01, and the operation result data is used or consumed by the data operation node 03. It should be noted that, to perform the operation, the following data operation node needs not only the output data of the previous data operation node, but also the output data of the previous data type derivation node and the output data of the data type derivation node to which the following data operation node belongs. For example, the data operation node 02 performs a data operation based on the output data of the data operation node 01, the output data of the data type derivation node 01, and the output data of the data type derivation node 02.
Because the data processing network system represented by the task computation graph adopts the data type derivation nodes as shown in the figure to form a row of data derivation paths specially used for data type derivation, and the data operation nodes form a row of data operation paths specially used for data operation, under the condition that the data type derivation does not need a data operation result, the data derivation paths can complete all data type derivation processes in the process of executing operation by the data operation nodes, so that the data type result required by operation is prepared for all operation nodes. Since the time for the arithmetic operation is generally substantially longer than the time taken for the type derivation in a pair of the data type derivation node and the data operation node, the time taken to complete a given job task as a whole will be the sum of the time taken for all the data operation nodes in a row of the data operation path dedicated to the data operation to perform the arithmetic operation, within which the data type derivation has been completed in advance. In some special cases, there may be a case where, in a pair of a data type derivation node and a data operation node, the time taken for the type derivation is longer than the time taken for the data operation, but since the time taken for most of the type derivation is shorter than the time taken for the data operation, the overall time taken for the data operation path dedicated to the data operation is still longer than the data derivation path dedicated to the data type derivation as a whole. Therefore, when the total time spent by the data operation path dedicated to data operation is equal to the data derivation path time dedicated to data type derivation, the time saved is half compared to the solution of performing data type derivation and data operation in a fully serial manner. For the conventional case, the data derivation path time dedicated to data type derivation is shorter than the overall time taken for the data operation path dedicated to data operation, and therefore, a data processing network system employing the present disclosure will be able to save at least one-third of the time for a specified job traffic.
Fig. 2 is a schematic diagram of a data processing network system according to a second embodiment of the present disclosure. In the deep learning system, sometimes the processing of some job tasks running in the deep learning or big data processing system has a case where data operation and data type cannot be separately processed, that is, there is a case where the data type is to be changed during the data operation, or the derivation of the data type is to be dependent on the operation result. At this time, the service logic node in the service logic diagram becomes a mixed service node. It is therefore not possible to distinguish between mapping to a data type derivation node and a data computation node, respectively, when mapping such a mixed service node in a task logic graph to a task computation graph. Generally, such hybrid task nodes are relatively few in practical application, but still are technical problems that the present disclosure needs to face in practice. However, in the case where such a hybrid task node does not exist, a corresponding task computation graph can be obtained by using the embodiment shown in fig. 1. When mixed task nodes exist in the task logic graph, the execution modes of tasks to be executed by the task nodes can be distinguished by analyzing each task node, if data type derivation is carried out firstly and then data operation is carried out, the task nodes are common task nodes, and the task nodes and the data type derivation nodes can be directly and respectively mapped to the data type derivation nodes and the data operation nodes in the task calculation graph. If not, it means a hybrid task node. As described above, such mixed task nodes are usually represented in the code as the code performing the operation modifies the required data type or shape during the operation, or as the result of the required data operation during the data derivation, which results in the entanglement of the two, and the data type derivation and the data operation cannot be directly separated into two independent computing nodes associated with each other. Therefore, in the process of analyzing the task nodes, the mixed task nodes and the common task nodes are identified. The identification mode can be realized by analyzing the code expression of the task node and the parameter reference relation in the code. Alternatively, the identification mode can be realized by directly identifying a section of code for executing a specific task so as to identify whether the section of code representing the task node is a common task node or a mixed task node.
The task node 03 shown in fig. 2 is a hybrid task node. After the mixed task node 03 is analyzed and identified, on one hand, a mixed data operation node 03 is directly mapped in the computational graph corresponding to the mixed task node 03, and on the other hand, a data type derivation node 03 is mapped in the computational graph. The data type derivation node 03 is a stub node, which does not actually perform a data type derivation operation, but delegates its own data type derivation operation to the hybrid data operation node 03. Which specifies the shape of the Tensor object to be passed in. The default parameter of shape is None, which indicates that any shape of Tensor object can be received. The hybrid data operation node 03 performs derivation of data types and data operations in an actual operation process. The hybrid data-computing node 03 will modify the memory allocated by the corresponding data-type derivation node 03, such that a modifiable connection is established between the hybrid data-computing node 03 and the corresponding data-type derivation node 03. In this case, since the data type derivation node 03 does not actually perform the data type operation, the next data type derivation node 04 cannot know the message that the derivation operation of the data type derivation node 03 is completed, and the derivation operation delegated by the data type derivation node 03 is completed by the hybrid data operation node 03, in order to successfully complete the data type derivation operation path, the message that the hybrid data operation node 03 completes needs to be obtained. Therefore, when the task logic graph containing the mixed task nodes is transformed to generate the task computation graph, a message connection edge, that is, a control connection needs to be established between the mixed data computation node 03 and the data type derivation node 04 at the next stage. When the hybrid data operation node 03 completes the operation, the data type derivation operation is also completed, and the data type derivation node 04 is notified of a message that the data type derivation node completes the two operations, so that the data type derivation node 04 knows that the data type result in the cache of the data type derivation node 03 is ready, and thus the data type derivation node 04 can start to execute the data type derivation operation of itself.
Although there is a case where the hybrid task node 03 is mapped to the data type derivation node 03 and the hybrid data operation node 03 as shown in fig. 2, for the case where there are most of the conventional paired data type derivation nodes and data operation nodes in the computation graph, it is also possible to play a role of saving the operation time at partial stages before and after the paired hybrid operation node 03 and data type derivation node 03. Even if the mixed operation node exists, the total time of the whole data processing path for processing the data does not exceed the sum of the actual operation time of the data, so that the time spent by the data type derivation operation is contained in the total operation time, thereby saving the data type derivation time spent in the conventional technical means.
Fig. 3 is a schematic diagram of a partial network of a data processing network system according to a second embodiment of the present disclosure. As shown in fig. 3, each compute node in the task computation graph forms a specific executable after actual deployment on a computing device. For example, the data type derivation computing node may create a data type derivation executable. As shown in fig. 3, each data type derivation executor includes a derivation component that performs derivation operations, a finite state machine that controls the state of the executor, a message bin that performs message communication between the executors, and an output data buffer that buffers the results of the operations. Similarly, each data operation executor includes an operation component that performs operation operations, a finite state machine that controls the state of the executor, a message bin that performs message communication between the executors, and an output data cache that caches the results of the operations. For each conventional data type derivation executor, for example, data type derivation executor 02, when it receives the message sent from the message bin of data type derivation executor 01 during operation, it changes the state of its finite state machine. Generally, the message sent from the message bin of the data type derivation executor 01 triggers the message bin to send a notification message to the message bin of the downstream executor that needs to consume the derivation result data when the data type derivation executor 01 completes its own data type derivation operation and caches the derived data type data in its output data cache, thereby informing the downstream executor that it can use the data type result generated by the downstream executor in its output data cache. After the finite-state machine of the data type derivation executive 02 receives the message change state of the data type derivation executive 01 at the upstream of the finite-state machine in the corresponding message bin, the finite-state machine triggers a derivation component to execute derivation operation based on data type result data in an output data cache of the data type derivation executive 01, and the derivation result is cached in the output data cache of the finite-state machine. And simultaneously triggering the message bin of the self to send the message to the message bin of the corresponding operation executive body 02 and the message bin of the downstream data type derivation executive body 03. The finite state machine of the operation executor 02 changes state based on the received message. When the state of the finite state machine of the operation execution body 02 is sequentially or simultaneously subjected to the messages from the data type derivation execution body 02, the upstream data type derivation execution body 01 and the operation execution body 01, the state is changed to a state which can trigger the operation component of the operation execution body 02 to execute corresponding operation, so that the operation component of the operation execution body 02 deduces data (data type) in the output data cache of the execution body 01 based on the upstream data type, data in the output data cache of the data type derivation execution body 02 and data in the output data cache of the operation execution body 01 to execute operation, and caches the operation result in the corresponding output data cache. Therefore, all data type derivation executors and corresponding operation executors operate in the manner described above.
Optionally, as described above, due to the existence of the hybrid data operation node and its corresponding data type derivation node, there is a modifiable relationship between the type derivation executable and the hybrid data operation executable deployed correspondingly on the actual computing device. Specifically, for example, the data type derivation executor 03 and the hybrid data operation executor 03, the data type derivation executor 03 also receives the message of the data type derivation executor 02, but since the corresponding data type derivation node 03 is a stub node, it does not perform the actual data type derivation process, and therefore, only the figure schematically shows that the derivation component is included, and the derivation component may not be actually included. Whether included or not, the data type derivation executor 03 does not perform any data type derivation operations, and only delegates the data type derivation process to the hybrid data operation executor 03. When the mixed data operation executive 03 receives the messages from the data type derivation executive 03, the upstream data type derivation executive 02 and the operation executive 02 sequentially or simultaneously, the state is changed to the state which can trigger the operation components of the mixed operation executive 03 to execute the corresponding operations,
therefore, the operation component of the hybrid operation executor 03 deduces data (data type) in the output data cache of the executor 02 based on the upstream data type, and the data execution in the output data cache of the executor 02 includes the operation of data type deduction, and caches the operation result in the corresponding output data cache, and caches the data type deduction result in the output data cache of the data type deduction executor 03 as a place-occupying executor. And when the hybrid operation executive 03 caches the operation result and the data type derivation result in the corresponding output data cache, the hybrid operation executive sends a message to the data type derivation executive 04 and the operation executive 04 downstream of the hybrid operation executive.
From the above description, it can be seen that, in the data type derivation path, before encountering the mix operation executors or after encountering the mix operation executors, all the data type derivation executors perform the data type derivation operation based on the derivation result of the upstream data type derivation executors thereof, and thus, the time taken for the data type derivation executors to perform the derivation operation at all the consecutive non-mix task nodes is included in the time taken for the corresponding operation executors to perform the operation. For the data type derivation executor and the hybrid operation executor corresponding to the hybrid task node, it can be regarded that the data type derivation operation executor and the operation executor operate synchronously, and the time spent is the same. Therefore, according to the present disclosure, regardless of whether the hybrid operation executors and the data type derivation executors corresponding thereto are included in the data processing network system, the total time consumption thereof will not exceed the time taken by all the operation executors (including the hybrid operation executors) to perform the operation, thereby objectively saving the time taken by the data type derivation corresponding to all the non-hybrid task computing nodes.
A data processing network system and a method of generating a computation graph of the data processing network according to the present disclosure are described in detail above with reference to the accompanying drawings. It should be noted that although the present disclosure has been described with respect to user-level threads that operate only for two types of operations, operations that operate on a disk (e.g., read and write to a disk), network communications operations, parameter update operations, and the like are actually included. This can be achieved by the operating units of the user-level threads issuing operating instructions or by inserting operating tasks into the task flow managed by the elements of the corresponding user space. How to implement these data operations, reading and writing of the disk, and updating of the parameters in the user space is not part of the disclosure, and therefore, will not be described in detail here.
Although the order in which the computation diagrams are generated is described above in a certain order, the order described above is not intended to limit the control order, because the order does not have a clear precedence order. For example, the data type derivation node and the data operation node generation process are generally performed simultaneously. Therefore, although there must be a chronological order in the written description, the actual execution of the steps is not limited to the order of the order defined by this written description.
The basic principles of the present disclosure have been described in connection with specific embodiments, but it should be noted that it will be understood by those skilled in the art that all or any of the steps or components of the method and apparatus of the present disclosure may be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof, which can be implemented by those skilled in the art using their basic programming skills after reading the description of the present disclosure.
Thus, the objects of the present disclosure may also be achieved by running a program or a set of programs on any computing device. The computing device may be a general purpose device as is well known. Thus, the object of the present disclosure can also be achieved merely by providing a program product containing program code for implementing the method or apparatus. That is, such a program product also constitutes the present disclosure, and a storage medium storing such a program product also constitutes the present disclosure. It is to be understood that the storage medium may be any known storage medium or any storage medium developed in the future.
It is also noted that in the apparatus and methods of the present disclosure, it is apparent that individual components or steps may be disassembled and/or re-assembled. These decompositions and/or recombinations are to be considered equivalents of the present disclosure. Also, the steps of executing the series of processes described above may naturally be executed chronologically in the order described, but need not necessarily be executed chronologically. Some steps may be performed in parallel or independently of each other.
The above detailed description should not be construed as limiting the scope of the disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (4)
1. A data processing network system, comprising:
a row of data type derivation nodes which are connected in sequence according to the production and consumption relationship; and
a row of data operation nodes which are connected in sequence according to the production consumption relation, each data operation node is connected with the consumption of the data type derivation node corresponding to the row where the data operation node is positioned and is connected with the consumption of the previous stage data type derivation node,
and each data operation node executes operation by using the data type derivation result of the data type derivation node connected with the data operation node, the data type derivation result of the previous data type derivation node and the operation result of the previous data operation node and outputs the operation result to the next data operation node.
2. The data processing network system of claim 1, wherein the data-manipulating nodes comprise hybrid data-manipulating nodes that have a consumption-modifiable connection with a data-type derivation node in its row and a subsequent data-type derivation node-controlled connection corresponding to a subsequent data-manipulating node, such that the subsequent data-type derivation node performs data-type derivation after obtaining the message for the hybrid data-manipulating node.
3. A computation graph generation method, comprising:
acquiring a task logic diagram, wherein the task logic diagram comprises a plurality of task nodes which are sequentially connected according to a preset logic sequence and used for completing a specified task, and the specified task of each task node comprises a data type derivation task and a data operation task;
analyzing the appointed operation tasks of the task nodes, mapping the data type derivation task of each task node into a data type derivation node one by one, and mapping the data operation task of each task node into a data operation node one by one; and
arranging the data type derivation nodes into a row of data type derivation nodes which are sequentially connected according to the production consumption relation according to the preset logic sequence of the task nodes, arranging the data operation nodes into a row of data operation nodes which are sequentially connected according to the production consumption relation according to the preset logic sequence of the task nodes, establishing consumption connection between the data type derivation nodes related to the same task node and the data operation nodes, and connecting each data operation node with the previous data type derivation node in a consumption way, thereby forming a calculation graph corresponding to the operation task logic graph.
4. The computational graph generation method of claim 3, the parsing the designated job task of the task node further comprising:
analyzing and identifying mixed operation tasks in the designated operation tasks, wherein the mixed operation tasks can change data types in the operation process; and
and mapping the identified mixed operation task into a mixed data operation node and further establishing control connection between the mixed data operation node and a next-stage data type derivation node corresponding to the next-stage data operation node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010600035.4A CN111475684B (en) | 2020-06-29 | 2020-06-29 | Data processing network system and calculation chart generation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010600035.4A CN111475684B (en) | 2020-06-29 | 2020-06-29 | Data processing network system and calculation chart generation method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111475684A true CN111475684A (en) | 2020-07-31 |
| CN111475684B CN111475684B (en) | 2020-09-22 |
Family
ID=71764021
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010600035.4A Active CN111475684B (en) | 2020-06-29 | 2020-06-29 | Data processing network system and calculation chart generation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111475684B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862109A (en) * | 2021-02-09 | 2021-05-28 | 上海商汤智能科技有限公司 | Deep learning model execution method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102957622A (en) * | 2011-08-16 | 2013-03-06 | 阿里巴巴集团控股有限公司 | Method, device and system for data processing |
| US20160098662A1 (en) * | 2014-10-03 | 2016-04-07 | Datameer, Inc. | Apparatus and Method for Scheduling Distributed Workflow Tasks |
| CN106611037A (en) * | 2016-09-12 | 2017-05-03 | 星环信息科技（上海）有限公司 | Method and device for distributed diagram calculation |
| CN109669772A (en) * | 2018-12-28 | 2019-04-23 | 第四范式（北京）技术有限公司 | Calculate the parallel execution method and apparatus of figure |
| CN111309479A (en) * | 2020-02-14 | 2020-06-19 | 北京百度网讯科技有限公司 | Method, device, equipment and medium for realizing task parallel processing |
-
2020
- 2020-06-29 CN CN202010600035.4A patent/CN111475684B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102957622A (en) * | 2011-08-16 | 2013-03-06 | 阿里巴巴集团控股有限公司 | Method, device and system for data processing |
| US20160098662A1 (en) * | 2014-10-03 | 2016-04-07 | Datameer, Inc. | Apparatus and Method for Scheduling Distributed Workflow Tasks |
| CN106611037A (en) * | 2016-09-12 | 2017-05-03 | 星环信息科技（上海）有限公司 | Method and device for distributed diagram calculation |
| CN109669772A (en) * | 2018-12-28 | 2019-04-23 | 第四范式（北京）技术有限公司 | Calculate the parallel execution method and apparatus of figure |
| CN111309479A (en) * | 2020-02-14 | 2020-06-19 | 北京百度网讯科技有限公司 | Method, device, equipment and medium for realizing task parallel processing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862109A (en) * | 2021-02-09 | 2021-05-28 | 上海商汤智能科技有限公司 | Deep learning model execution method and device, electronic equipment and storage medium |
| CN112862109B (en) * | 2021-02-09 | 2024-05-24 | 上海商汤智能科技有限公司 | Deep learning model execution method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111475684B (en) | 2020-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2341434B1 (en) | Method and apparatus for performing root cause analysis | |
| US6334193B1 (en) | Method and apparatus for implementing user-definable error handling processes | |
| US20120324454A1 (en) | Control Flow Graph Driven Operating System | |
| US8201142B2 (en) | Description language for structured graphs | |
| US20080098375A1 (en) | Runtime optimization of distributed execution graph | |
| CN111078323B (en) | Data processing method and device based on coroutine, computer equipment and storage medium | |
| CN105164639A (en) | Controlling tasks performed by computing system | |
| US20080082644A1 (en) | Distributed parallel computing | |
| CN110806923A (en) | Parallel processing method and device for block chain tasks, electronic equipment and medium | |
| JP2009532754A (en) | Abstract execution model for continuation-based meta-runtime | |
| CN109308213B (en) | Multi-task breakpoint debugging method based on improved task scheduling mechanism | |
| US5386558A (en) | Method and apparatus for executing control system functions in a computer system | |
| CN110569113A (en) | Method and system for scheduling distributed tasks and computer readable storage medium | |
| US20070022424A1 (en) | Technique for processing a computer program | |
| JP6400296B2 (en) | Multi-mode support processor and processing method for supporting multi-mode | |
| CN111475684B (en) | Data processing network system and calculation chart generation method thereof | |
| CN118245032B (en) | Attribute linkage engine method and system for customer relationship management | |
| US20180113729A1 (en) | Category based execution scheduling | |
| CN111158919B (en) | Memory resource in-place sharing decision system and method thereof | |
| US20190213041A1 (en) | Progress visualization of computational job | |
| WO2013165460A1 (en) | Control flow graph driven operating system | |
| US20180101384A1 (en) | Morphed instruction according to configuration update | |
| CN116107728A (en) | Task execution method and device, storage medium and electronic equipment | |
| CN112445607B (en) | Method and device for executing method function by application program | |
| CN114020373A (en) | Method for realizing flow defined based on function formula |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |