CN113760501A - Control method, device and system for data processing link - Google Patents
Control method, device and system for data processing link Download PDFInfo
- Publication number
- CN113760501A CN113760501A CN202110128935.8A CN202110128935A CN113760501A CN 113760501 A CN113760501 A CN 113760501A CN 202110128935 A CN202110128935 A CN 202110128935A CN 113760501 A CN113760501 A CN 113760501A
- Authority
- CN
- China
- Prior art keywords
- task node
- data
- link
- current task
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 124
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000000903 blocking effect Effects 0.000 claims abstract description 104
- 238000001514 detection method Methods 0.000 claims abstract description 39
- 230000008569 process Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 abstract description 7
- 238000007405 data analysis Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 239000013256 coordination polymer Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6209—Protecting access to data via a platform, e.g. using keys or access control rules to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method, a device and a system for controlling a data processing link, and relates to the technical field of computers. One embodiment of the method comprises: for each current task node of a plurality of current task nodes in a data processing link as a current task node, performing: detecting the data quality of data generated by a current task node; and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies. The implementation mode can meet different requirements of different data analysis and data calculation links on data quality.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for controlling a data processing link.
Background
In the field of big data, data is generally analyzed and calculated based on a data processing link formed by a plurality of associated task nodes. And providing reference for enterprise decision-making, product strategies and artificial intelligence technologies according to the analysis and calculation results. Since the quality of the data received by each of the associated task nodes will directly affect the analysis and calculation results of the link, how to block the link based on the data quality has become a problem to be solved in the current big data processing process.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, and a system for controlling a data processing link, which can block the data processing link based on data quality and the requirement of each task node in the data processing link on the data quality, and can meet different requirements of different data analysis and data calculation links on the data quality.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method for controlling a data processing link, including:
for each of a plurality of current task nodes in the data processing link, performing, as a current task node:
detecting the data quality of the data generated by the current task node;
and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
Preferably, the link blocking policy configured for the current task node in advance includes:
and when the detection result indicates that the data quality has a quality problem, directly blocking the next task node of the current task node.
Preferably, the controlling the data processing link where the current task node is located includes:
and setting the task of the current task node as the execution failure so as to block the next task node of the current task node.
Preferably, the link blocking policy configured for the current task node in advance includes:
sending the detection result to a client and receiving a link execution instruction sent by the client;
blocking a task node next to the current task node when the link execution instruction indicates blocking of the data processing link;
and when the link execution instruction indicates to continue executing the data processing link, controlling a task node next to the current task node to continue executing.
Preferably, when the detected result indicates that the quality of the data has a quality problem, the data generated by the current task node is locked;
judging whether a next task node of the current task node has the authority of accessing the locked data, if so, controlling the next task node to process the locked data; otherwise, blocking the next task node from executing.
Preferably, the data generated by the current task node belongs to a plurality of data files;
locking data generated by the current task node, comprising:
locking data files with quality problems in the plurality of data files;
the controlling the next task node to process the locked data comprises: and controlling the next task node to process the locked data file with quality problem.
Preferably, the method for controlling a data processing link further includes: setting a white list for the generated data, wherein the white list maintains at least one next task node corresponding to the current task node, and the next task node belonging to the white list has the authority of processing the locked data;
determining whether the next task node has permission to access the locked data, including:
and judging whether the next task node belongs to a white list of the locked data, if so, determining that the next task node has the authority of accessing the locked data, and otherwise, determining that the next task node has no authority of accessing the locked data.
Preferably, the method for controlling a data processing link further includes:
and when the data required to be processed by the current task node does not comprise the data file with the quality problem, controlling the next task node to execute.
In a second aspect, an embodiment of the present invention provides a control device for a data processing link, including: a quality detection unit and a blocking control unit, wherein,
the quality detection unit is configured to, for each of a plurality of task nodes in the data processing link, as a current task node, execute: detecting the data quality of the data generated by the current task node;
the blocking control unit is configured to, for each task node of a plurality of task nodes in the data processing link as a current task node, execute: and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
In a third aspect, an embodiment of the present invention provides a control system for a data processing link, including: a plurality of task nodes in a data processing link and a control device for the data processing link.
One embodiment of the above invention has the following advantages or benefits: the link blocking strategy configured for the task node in advance is one of multiple stored link blocking strategies, namely different task nodes can be controlled based on the result of detected data quality and different link blocking strategies, the data processing link is blocked based on the data quality and the requirement of each task node on the data quality, and different requirements of different data analysis and data calculation on the data quality can be met.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a plurality of data processing links formed by various task nodes according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a main flow of a control method of a data processing link according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a link blocking manner implemented by a plurality of data processing links based on different link blocking policies according to an embodiment of the present invention;
FIG. 4 is a schematic illustration of a main flow of determining a task node's right to access locked data according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the main devices or systems of a control system of a data processing link according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of the main elements of a control device of a data processing link according to an embodiment of the present invention;
FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the big data processing scheme provided in the embodiment of the present invention, a plurality of data processing links may be formed by different current task nodes to process big data through the plurality of data processing links, where a task node in the plurality of data processing links invokes a result generated by a previous task node of the task node, and the invoking may be completed by an invoking system or may be directly implemented by the task node. As shown in FIG. 1, the task nodes task0, task01, task02, task1, task11, task2, task21, task22, task xxx 1, etc. The data processing link formed by these task nodes may include, as shown in fig. 1: task0 → task01 → …, task0 → task02 → …, task1 → task11 → …, task1 → task2 → task21 → …, task1 → task2 → task22 → …, task1 → task2 → task x 1 → …, etc. The control of each data processing link shown in fig. 1 can be achieved by the method for controlling a data processing link provided in the embodiment of the present invention.
Fig. 2 is a control method of a data processing link according to an embodiment of the present invention, as shown in fig. 1, the control method of the data processing link may include the following:
for each of a plurality of current task nodes in the data processing link as a current task node, performing the following steps S201 to S202:
step S201: detecting the data quality of the data generated by the current task node;
the current task node can be a data cleaning node, a data processing node, a service analysis node and the like. When the current task node is the first task node, the data source on which the data generated by the current task node depends may be original data acquired from other systems such as an order management system, an e-commerce platform, a payment system, and the like or data primarily processed by other systems; when the current task node is a task node other than the first task node, the data source on which the data generated by the current task node depends may be part of or all of the data generated from the last task node corresponding to the current task node. For example, the data sources on which the data generated for task nodes task0 and task1 shown in fig. 1 depend are raw data acquired from other systems or data after being subjected to preliminary processing by other systems; for example, the data source on which the data generated by the task node task01 and task node task02 shown in fig. 1 depends is all or part of the data generated from the task node task0, the data source on which the data generated by the task node task11 and task node task2 depends is all or part of the data generated from the task node task1, the data source on which the data generated by the task nodes task21, task22, and task xx generated is all or part of the data generated from the task node task2, and the data source on which the data generated by the task node task xx 1 depends is all or part of the data generated from the task node task xx 1.
Step S202: and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
The detection result may indicate that the data quality of the data source is not qualified, or a partial data error of the data source, or a qualified rate of the data quality of the data source is lower than a preset qualified threshold, or the like.
The stored multiple link blocking policies may include the following link blocking policies.
The first link blocking strategy is an active blocking mechanism, that is, when the detected result indicates that the data quality has a quality problem, the next task node of the current task node is directly blocked. The quality problem of the data quality refers to that the data quality is unqualified, partial data of a data source is wrong, or the qualified rate of the data quality of the data source is lower than a preset qualified threshold value.
The second link blocking strategy is a passive blocking mechanism, namely, a detection result is sent to a client, and a link execution instruction sent by the client is received; blocking a task node next to the current task node when the link execution instruction indicates blocking of the data processing link; and when the link execution instruction indicates to continue executing the data processing link, controlling a task node next to the current task node to continue executing.
The third link blocking strategy is a file locking mechanism/data authority endowing mechanism, namely when the detection result indicates that the data quality has a quality problem, the data generated by the current task node is locked; judging whether a next task node of the current task node has the authority of accessing the locked data, if so, controlling the next task node to process the locked data; otherwise, the next task node is blocked from executing. The file locking mechanism/data authority endowing mechanism can solve the problem of depending on a certain scheduling system firstly, and once a file with quality problem is locked, the data processing process that no matter the next task node in the scheduling system or other task nodes directly depend on the data file can be blocked. Secondly, the mechanism can also provide a white list, and task nodes or executing users of the data processing process which have low requirements on the quality of the data file can be configured in the white list, so that the task nodes and the data processing process can be continued without being blocked, and on the contrary, other data processing processes or current task nodes which have high requirements on the quality of the data file can be blocked because the data file cannot be read.
The next task node of the current task node is a task node that has execution logic with the current task node and uses data generated by the current task node as a data source on a data processing link, and it should be noted that the number of the next task node of the current task node may be multiple. For example, for the data processing link shown in fig. 1, the current task node is task0, and the next task node corresponding to this task0 has two tasks, namely task01 and task 02; the current task node is task1, and the next task node corresponding to the task1 is two, namely, task11 and task 2; the current task node is task2, and the next task node corresponding to this task2 has two nodes, which are task21, task22, and task xx, respectively.
For the embodiment shown in fig. 2, since the link blocking policy configured for the task node in advance is one of the stored multiple link blocking policies, that is, different task nodes can be controlled based on the result of the detected data quality and different link blocking policies, the data processing link is blocked based on the data quality and the requirement of each task node on the data quality, and different requirements of different data analysis and data calculation on the data quality can be met.
As shown in fig. 3, link blocking policies are configured for task nodes in the data processing links shown in fig. 1, for example, the task node task0 is configured with the first link blocking policy; configuring a second link blocking strategy for task node task 1; a third link blocking policy is configured for task node tas k2 (where task node task22 after task node task2 has the right to acquire problematic data or belongs to a white list in the third link blocking policy, and neither task21 nor task x has the right to acquire problematic data or is not on the white list of the third link blocking policy).
The first link blocking strategy and the second link blocking strategy are generally completed by a calling system, and the third link blocking strategy can be directly realized by communication between current task nodes or realized by the calling system.
For the case that the current task node is the task node task0 shown in fig. 3, the link blocking policy configured by the task node task0 is the first link blocking policy described above. the control mode of the data processing link task0 → task01 → … and task0 → task02 → … where the task0 is located is specifically described by taking the task0 as the current task node: detecting the data quality of the data generated by the current task node task 0; when the detection result indicates that the data quality has quality problems, the next task nodes task01 and task02 of the task0 are directly blocked. More specifically, the specific implementation of controlling the data processing link where the current task node is located may include: setting a task of a current task node (such as task0) to fail execution, and blocking a next task node (task01 and task02) of the current task node. Specifically, the current task node (such as task0) is blocked from being called by the calling system. Namely: after the task0 finishes processing data, a data file task0_ file is generated, data quality detection task0_ file is performed, a detection result is returned, if the data quality does not meet the requirement, the task0 is actively set to fail, the task01 and the task02 of the downstream task of the task0 cannot continue to calculate data because the dependency is not established, and a data processing link is blocked.
For the case that the current task node is the task node task1 shown in fig. 3, the link blocking policy configured by the task node task1 is the second link blocking policy. the control mode of the data processing link task1 → task11 → … and task1 → task2 → … where the task1 is located is specifically described by taking the task1 as the current task node: detecting the data quality of the data generated by the current task node task 1; sending the detection result to a client and receiving a link execution instruction sent by the client; blocking a next task node t ask11 of the current task node task1 when the link execution instruction indicates blocking the data processing link; and when the link execution instruction indicates that the data processing link is continuously executed, controlling the next task node task11 of the current task node to continuously execute. Namely: after the task1 finishes processing the data, a data file task1_ file is generated, and the data quality detection task1_ file returns the detection result. And after the task1 finishes processing the data logic, the client used by the user acquires the result of data quality detection through the gateway interface, and the client used by the user decides whether to block the task11 of the downstream task node of the task1 according to the detection result.
For the case that the current task node is the task node task2 shown in fig. 3, the link blocking policy configured by the task node task2 is the third link blocking policy described above. The control method of the data processing link Task2 → Task21 → …, Task2 → Task22 → … and Task2 → Task xx → 1 → … of the data processing link where the Task2 is located is described by taking the current Task node Task2 as an example: detecting the data quality of data generated by a current task node (such as task2 shown in FIG. 3); when the detection result indicates that the quality of the data has a quality problem, locking the data generated by the current task node (for example, the data generated by the task2 shown in fig. 3 are task2_ file1 and task2_ file 2); judging whether a next task node (such as task21, task22 and task xxx respectively) of the current task node (such as the next task node of task2 shown in FIG. 3) has the right to access the locked data, and if so, controlling the next task node to process the locked data; otherwise, blocking the next task node from executing. For example, if task node task22 has access to the locked data, then task22 is controlled to process the locked data; task nodes task21 and task xx block task21 and task xx execution. Namely: when the task2 finishes processing data to generate a file task2_ file1 and a file task2_ f file2, data quality detection is carried out on the task2_ file1 and the file task2_ file2, and a detection result is returned, if the task2_ file1 does not meet the quality requirement, a link blocking mechanism locks the task2_ file1 in a file permission control mode, all task nodes except a set task node white list which is not blocked do not have permission to read the data file task 24 _ file1 with quality problems, at the moment, the task21 in a downstream task node of the task2 has a higher quality dependence requirement on the task2_ file1, and the task2_ file1 is locked, the task21 does not have execution permission, the task21 can execute the permission, and the downstream task21 cannot execute the task21 which fails to process the data of the task 21; the task22 in the downstream task node of the task2 has low quality requirement on the task2_ file1, and the task node task22 exists in the white list, because the task22 still has the reading right of the task2_ file1, the task22 can be continuously executed, and the data processing link where the task22 is located can be continuously executed. In addition, the data processing procedure of the non-scheduling system, task xx, directly depends on the task2_ file1, and since the task2_ file1 already locks the authority, the task xx may fail to execute, thereby also blocking the data processing link where the task xx is located.
Specifically, data generated by a current task node belongs to a plurality of data files; accordingly, locking data generated by the current task node may include: locking data files with quality problems in the plurality of data files; the controlling the next task node to process the locked data may include: and controlling the next task node to process the locked data file with quality problem.
In this embodiment of the present invention, as shown in fig. 4, the third link blocking policy may further include the following steps:
step S401: setting a white list for the generated data, wherein the white list maintains at least one next task node corresponding to the current task node, and the next task node belonging to the white list has the authority of processing the locked data;
step S402: judging whether the next task node belongs to a white list of the locked data, if so, executing a step S403; otherwise, go to step S404;
step S403; determining that the next task node has the right to access the locked data, and ending the current process;
step S404: determining that the next task node has no authority to access the locked data.
In an embodiment of the present invention, the method for controlling the data processing link may further include: and when the data required to be processed by the current task node does not comprise the data file with the quality problem, controlling the next task node to execute.
As shown in fig. 5, an embodiment of the present invention provides a control apparatus 500 for a data processing link, where the control apparatus 500 for the data processing link may include: a quality detection unit 501 and a blocking control unit 502, wherein,
the quality detection unit 501 is configured to, for each task node in the plurality of task nodes in the data processing link as a current task node, perform: detecting the data quality of the data generated by the current task node;
the blocking control unit 502 is configured to, for each task node in the plurality of task nodes in the data processing link as a current task node, perform: and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
In this embodiment of the present invention, the link blocking policy that the blocking control unit 502 configures for the current task node in advance may include: and when the detection result indicates that the data quality has a quality problem, directly blocking the next task node of the current task node.
In this embodiment of the present invention, the blocking control unit 502 is configured to set the task of the current task node as an execution failure, so as to block a task node next to the current task node.
In this embodiment of the present invention, the link blocking policy that the blocking control unit 502 configures for the current task node in advance may include: sending the detection result to a client and receiving a link execution instruction sent by the client; blocking a task node next to the current task node when the link execution instruction indicates blocking of the data processing link; and when the link execution instruction indicates to continue executing the data processing link, controlling a task node next to the current task node to continue executing.
In this embodiment of the present invention, the link blocking policy that the blocking control unit 502 configures for the current task node in advance may include: when the detection result indicates that the quality of the data has a quality problem, locking the data generated by the current task node; judging whether a next task node of the current task node has the authority of accessing the locked data, if so, controlling the next task node to process the locked data; otherwise, blocking the next task node from executing.
In the embodiment of the invention, the data generated by the current task node belongs to a plurality of data files; the blocking control unit 502 is configured to lock a data file with a quality problem among the plurality of data files; and controlling the next task node to process the locked data file with quality problem.
In this embodiment of the present invention, the blocking control unit 502 is further configured to set a white list for the generated data, where the white list maintains at least one next task node corresponding to the current task node, and the next task node belonging to the white list has an authority to process the locked data; and judging whether the next task node belongs to a white list of the locked data, if so, determining that the next task node has the authority of accessing the locked data, and otherwise, determining that the next task node has no authority of accessing the locked data.
In this embodiment of the present invention, the blocking control unit 502 is further configured to control the next task node to execute when the data that needs to be processed by the current task node does not include a data file with a quality problem.
As shown in fig. 6, an embodiment of the present invention provides a control system 600 for a data processing link, where the control system 600 for the data processing link may include: a plurality of task nodes 601 in a data processing link and the control device 500 of the data processing link provided in the above embodiment.
The control device 500 of the data processing link may configure a corresponding link blocking policy for each task node, where the configured link blocking policy is one of the stored multiple link blocking policies.
In the embodiment of the present invention, as shown in fig. 6, the control system 600 of the data processing link may further include: a scheduling system 602, wherein the scheduling system 602 can schedule each task node according to execution logic between each task node.
Fig. 7 shows an exemplary system architecture 700 of a control method of a data processing link or a control device of a data processing link to which embodiments of the present invention may be applied.
As shown in fig. 7, the system architecture 700 may include end devices 701, 702, 703, a network 704, a server 705 controlling data processing links, servers 706, 707, 708 as task nodes, and a server 709 scheduling task nodes. The network 704 is arranged to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705 controlling the data processing links, between the servers 706, 707, 708 of the task nodes and the server 709 scheduling the task nodes. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 701, 702, 703 to interact with a server 705 controlling the data processing link through a network 704, to receive or send messages or the like. The terminal devices 701, 702, 703 may have installed thereon various communication client applications, such as a 7-web browser application, a search-type application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only). The terminal devices 701, 702, and 703 may receive the result of the detection of the data quality sent by the server 705 controlling the data processing link, and receive a link execution instruction sent by the user through the terminal devices 701, 702, and 703, so that the server 705 controlling the data processing link controls the servers 706, 707, and 708 of the task node to block or continue execution according to the link execution instruction.
The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 705 controlling the data processing link may be a server providing various services, for example, detecting data quality of data generated by the servers 706, 707, 708 of the task node, and controlling the servers 706, 707, 708 of the task node according to the detected result and the link blocking policy.
In addition, the server 705 for controlling the data processing link may also call the server 709 for scheduling task nodes according to the link blocking policy, so that the server 709 for scheduling task nodes sequentially schedules the servers 706, 707, 708 for each task node according to the execution logic among the servers 706, 707, 708 for each task node.
It should be noted that the method for controlling the data processing link provided in the embodiment of the present invention is generally executed by the server 705 that controls the data processing link, and accordingly, the control device of the data processing link is generally disposed in the server 705 that controls the data processing link.
It should be understood that the numbers of terminal devices, networks, servers controlling data processing links, servers of task nodes, and servers scheduling task nodes in fig. 7 are merely illustrative. There may be any number of terminal devices, networks, servers controlling data processing links, servers of task nodes, and servers of scheduling task nodes, as desired for the implementation.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program performs the above-described functions defined in the system of the present invention when executed by the central processing unit (CP U) 801.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a quality detection unit and a blocking control unit. The names of these units do not in some cases constitute a limitation to the unit itself, and for example, the quality detection unit may also be described as a "unit that detects the data quality of data generated by the current task node".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: for each of a plurality of current task nodes in the data processing link, performing, as a current task node: detecting the data quality of the data generated by the current task node; and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
According to the technical scheme of the embodiment of the invention, the link blocking strategy configured for the task node in advance is one of the stored multiple link blocking strategies, namely different task nodes can be controlled based on the result of the detected data quality and different link blocking strategies, the data processing link is blocked based on the data quality and the requirement of each task node on the data quality, and different requirements of different data analysis and data calculation on the data quality can be met.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (12)
1. A method for controlling a data processing link, comprising:
for each of a plurality of current task nodes in the data processing link, performing, as a current task node:
detecting the data quality of the data generated by the current task node;
and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
2. The method according to claim 1, wherein the pre-configuring of the link blocking policy for the current task node comprises:
and when the detection result indicates that the data quality has a quality problem, directly blocking the next task node of the current task node.
3. The method according to claim 2, wherein controlling the data processing link in which the current task node is located comprises:
and setting the task of the current task node as the execution failure so as to block the next task node of the current task node.
4. The method according to claim 1, wherein the pre-configuring of the link blocking policy for the current task node comprises:
sending the detection result to a client and receiving a link execution instruction sent by the client;
blocking a task node next to the current task node when the link execution instruction indicates blocking of the data processing link;
and when the link execution instruction indicates to continue executing the data processing link, controlling a task node next to the current task node to continue executing.
5. The method of claim 1,
when the detection result indicates that the quality of the data has a quality problem, locking the data generated by the current task node;
judging whether a next task node of the current task node has the authority of accessing the locked data, if so, controlling the next task node to process the locked data; otherwise, blocking the next task node from executing.
6. The method of claim 5, wherein the data processing link further comprises a data processing link,
the data generated by the current task node belongs to a plurality of data files;
locking data generated by the current task node, comprising:
locking data files with quality problems in the plurality of data files;
the controlling the next task node to process the locked data comprises: and controlling the next task node to process the locked data file with quality problem.
7. The method of controlling a data processing link according to claim 5 or 6,
further comprising: setting a white list for the generated data, wherein the white list maintains at least one next task node corresponding to the current task node, and the next task node belonging to the white list has the authority of processing the locked data;
determining whether the next task node has permission to access the locked data, including:
and judging whether the next task node belongs to a white list of the locked data, if so, determining that the next task node has the authority of accessing the locked data, and otherwise, determining that the next task node has no authority of accessing the locked data.
8. The method of claim 6, further comprising:
and when the data required to be processed by the current task node does not comprise the data file with the quality problem, controlling the next task node to execute.
9. A control device for a data processing link, comprising: a quality detection unit and a blocking control unit, wherein,
the quality detection unit is configured to, for each of a plurality of task nodes in the data processing link, as a current task node, execute: detecting the data quality of the data generated by the current task node;
the blocking control unit is configured to, for each task node of a plurality of task nodes in the data processing link as a current task node, execute: and controlling a data processing link where the current task node is located based on the detection result and a link blocking strategy configured for the current task node in advance, wherein the link blocking strategy configured for the current task node in advance is one of a plurality of stored link blocking strategies.
10. A control system for a data processing link, comprising: a plurality of task nodes in a data processing link and a control device for a data processing link as claimed in claim 9.
11. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
12. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110128935.8A CN113760501A (en) | 2021-01-29 | 2021-01-29 | Control method, device and system for data processing link |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110128935.8A CN113760501A (en) | 2021-01-29 | 2021-01-29 | Control method, device and system for data processing link |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760501A true CN113760501A (en) | 2021-12-07 |
Family
ID=78786492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110128935.8A Pending CN113760501A (en) | 2021-01-29 | 2021-01-29 | Control method, device and system for data processing link |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760501A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8108715B1 (en) * | 2010-07-02 | 2012-01-31 | Symantec Corporation | Systems and methods for resolving split-brain scenarios in computer clusters |
| CN103037373A (en) * | 2012-12-21 | 2013-04-10 | 成都科来软件有限公司 | Wireless node blocking system |
| CN108521398A (en) * | 2018-02-24 | 2018-09-11 | 浙江远望通信技术有限公司 | A kind of video monitoring safety cut-in method based on equipment feature recognition, white list and constraint set flow control |
| US20180293395A1 (en) * | 2017-04-11 | 2018-10-11 | International Business Machines Corporation | File sharing and policy control based on file link mechanism |
| CN111787073A (en) * | 2020-06-18 | 2020-10-16 | 多加网络科技(北京)有限公司 | Current-limiting fusing platform and method for unified service |
| CN112187511A (en) * | 2020-08-28 | 2021-01-05 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for positioning micro-service fusing abnormity |
-
2021
- 2021-01-29 CN CN202110128935.8A patent/CN113760501A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8108715B1 (en) * | 2010-07-02 | 2012-01-31 | Symantec Corporation | Systems and methods for resolving split-brain scenarios in computer clusters |
| CN103037373A (en) * | 2012-12-21 | 2013-04-10 | 成都科来软件有限公司 | Wireless node blocking system |
| US20180293395A1 (en) * | 2017-04-11 | 2018-10-11 | International Business Machines Corporation | File sharing and policy control based on file link mechanism |
| CN108521398A (en) * | 2018-02-24 | 2018-09-11 | 浙江远望通信技术有限公司 | A kind of video monitoring safety cut-in method based on equipment feature recognition, white list and constraint set flow control |
| CN111787073A (en) * | 2020-06-18 | 2020-10-16 | 多加网络科技(北京)有限公司 | Current-limiting fusing platform and method for unified service |
| CN112187511A (en) * | 2020-08-28 | 2021-01-05 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for positioning micro-service fusing abnormity |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111198813B (en) | Interface testing method and device | |
| CN110825535B (en) | Job scheduling method and system | |
| CN110944046B (en) | Control method of consensus mechanism and related equipment | |
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN109828830B (en) | Method and apparatus for managing containers | |
| CN113517985B (en) | File data processing method and device, electronic equipment and computer readable medium | |
| CN111831461A (en) | Method and device for processing business process | |
| US12079651B2 (en) | Serverless application function execution | |
| US11792015B2 (en) | System and method for electronic signatures as a service | |
| CN113010238A (en) | Permission determination method, device and system for micro application call interface | |
| CN113760503A (en) | Task migration method and device, electronic equipment and computer readable medium | |
| CN112953719B (en) | Token authentication method and device | |
| CN113760501A (en) | Control method, device and system for data processing link | |
| CN112463616A (en) | Chaos testing method and device for Kubernetes container platform | |
| US20220276901A1 (en) | Batch processing management | |
| US11431711B2 (en) | Method, device and computer program product for service access | |
| CN112182003A (en) | Data synchronization method and device | |
| CN113096295A (en) | Numbering method, numbering device, electronic equipment and computer readable storage medium | |
| CN113760487A (en) | Service processing method and device | |
| CN111984613A (en) | Method, device and system for sharing files | |
| CN112860422A (en) | Method, apparatus and computer program product for job processing | |
| CN111147470A (en) | Account authorization method and device and electronic equipment | |
| CN110262756B (en) | Method and device for caching data | |
| CN115484149B (en) | Network switching method, network switching device, electronic equipment and storage medium | |
| CN111179097B (en) | Method, device, electronic equipment and storage medium for modifying warranty |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |