CN112131297A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112131297A CN112131297A CN202011042041.9A CN202011042041A CN112131297A CN 112131297 A CN112131297 A CN 112131297A CN 202011042041 A CN202011042041 A CN 202011042041A CN 112131297 A CN112131297 A CN 112131297A
- Authority
- CN
- China
- Prior art keywords
- task
- running
- data processing
- diagram
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000010586 diagram Methods 0.000 claims abstract description 83
- 230000004048 modification Effects 0.000 claims abstract description 40
- 238000006011 modification reaction Methods 0.000 claims abstract description 40
- 230000004044 response Effects 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 11
- 238000002360 preparation method Methods 0.000 abstract description 3
- 238000011144 upstream manufacturing Methods 0.000 description 9
- 230000003287 optical Effects 0.000 description 4
- 230000000737 periodic Effects 0.000 description 4
- 230000000875 corresponding Effects 0.000 description 2
- 230000001419 dependent Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000000034 method Methods 0.000 description 2
- 230000003068 static Effects 0.000 description 2
- 240000000800 Allium ursinum Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000002093 peripheral Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Abstract
The application provides a data processing method, a data processing device, data processing equipment and a storage medium. The data processing method comprises the following steps: responding to the set task range, cycle type and service date range of the batch tasks, and generating a task operation diagram; the task operation diagram comprises the following steps: the hierarchical relationship of each task in the batch tasks, and the cycle type and the service date range of each task in the batch tasks; updating the task execution diagram in response to a task modification instruction for at least one task in the task execution diagram; and running the batch tasks according to the updated task running chart, and displaying the updated task running chart. The method and the device can improve the efficiency and the preparation of running the batch tasks and the efficiency of maintaining and managing the batch tasks, are more suitable for the application scene of the rerun of the ETL task historical data, and can improve the overall operation and maintenance efficiency.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
In the existing scheme of ETL (Extract-Transform-Load) task historical data rerun, a manual mode is mostly adopted, managers manually comb rerun tasks and upstream and downstream relations, the current service date of the tasks is recorded, the efficiency is low, the phenomena of wrong task relation combing, task omission and the like are easy to occur, and the data accuracy cannot be guaranteed.
Disclosure of Invention
The application provides a data processing method, a device, equipment and a storage medium aiming at the defects of the existing mode, and aims to solve the technical problems that the manual mode is low in efficiency and the accuracy is difficult to guarantee in the prior art.
In a first aspect, an embodiment of the present application provides a data processing method, including:
responding to the set task range, cycle type and service date range of the batch tasks, and generating a task operation diagram; the task operation diagram comprises the following steps: the hierarchical relationship of each task in the batch tasks, and the cycle type and the service date range of each task in the batch tasks;
updating the task execution diagram in response to a task modification instruction for at least one task in the task execution diagram;
and running the batch tasks according to the updated task running chart, and displaying the updated task running chart.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the running chart generating module is used for responding to the set task range, cycle type and service date range of the batch tasks and generating a task running chart; the task operation diagram comprises the following steps: the hierarchical relationship of each task in the batch tasks, and the cycle type and the service date range of each task in the batch tasks;
the task execution diagram updating module is used for responding to a task modification instruction aiming at least one task in the task execution diagram and updating the task execution diagram;
and the information display module is used for running the batch tasks according to the updated task running chart and displaying the updated task running chart.
In a third aspect, an embodiment of the present application provides a data processing apparatus, including:
a memory;
a processor electrically connected to the memory;
the memory stores a computer program which is executed by the processor to implement the data processing method provided by the first aspect of the embodiments of the present application.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the data processing method provided in the first aspect of the embodiment of the present application.
The technical scheme provided by the embodiment of the application at least has the following beneficial effects:
according to the embodiment of the application, the hierarchical relationship among the tasks can be automatically sorted based on the set task range, the period type and the service date range of the batch tasks, the task operation diagram is automatically generated by combining the received modification instruction, then the tasks in the task range are operated in batches according to the task operation diagram, the task operation diagram is displayed for a manager to check and manage, the manager does not need to manually sort and control, the efficiency and the preparation of operating the batch tasks can be improved, the convenience and the efficiency of maintaining and managing the batch tasks can be improved, the application scene of rerun of ETL task historical data is more suitable, and the whole operation and maintenance efficiency can be improved.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;
FIG. 2 is a diagram illustrating an example of a portion of tasks in a task execution diagram according to an embodiment of the present disclosure;
FIG. 3 is another exemplary diagram of a portion of tasks in a task execution diagram according to an embodiment of the present disclosure;
FIG. 4 is a diagram illustrating another example of a portion of tasks in a task execution diagram according to an embodiment of the present disclosure;
fig. 5 is a schematic structural framework diagram of a data processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural framework diagram of a data processing device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar parts or parts having the same or similar functions throughout. In addition, if a detailed description of the known art is not necessary for illustrating the features of the present application, it is omitted. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments.
An embodiment of the present application provides a data processing method, as shown in fig. 1, the method includes:
and S101, responding to the set task range, cycle type and service date range of the batch tasks, and generating a task operation diagram.
The task operation diagram comprises the following steps: the hierarchical relationship of the tasks in the batch (e.g., upstream-downstream relationship), and the cycle type and business date range for each task in the batch.
According to the method and the device, the hierarchical relationship of each task in the task range can be automatically combed by combining the cycle type and the service date range, so that a task operation diagram is generated, and manual combing is not needed.
The task range, the cycle type and the service date range of the batch tasks in the embodiment of the application can be set by a manager on the displayed setting interface. The cycle type of the batch task may be at least one of year, month, day, hour (i.e., hour), minute, and second, and the cycle of each task in the batch task may be the same or different.
In one example, the administrator can select an interface task to be rerun as task a through the setting interface, and set the task range of the batch task as task a and all the downstream tasks thereof, where when the cycle type of task a is "yes", the service date range of the batch task is 2019070201 (indicating 2019, 7, 2, 1 st point) -2019070202 (indicating 2019, 7, 2 nd point).
The task execution graph in this embodiment may be a DAG (Directed acyclic graph) graph.
S102, responding to a task modification instruction aiming at least one task in the task running chart, and updating the task running chart.
The task modification instruction in the embodiment of the application may include at least one of a date modification instruction, a period modification instruction, a task addition instruction, and a task cancellation instruction.
In an optional embodiment, in response to a date modification instruction for at least one task in the task execution diagram, updating a business date range of the first task and a business date range of a task downstream of the first task in the task execution diagram; the first task includes: at least one task for which the date modification instruction is directed.
In the embodiment of the application, an upstream task of a certain task refers to another task on which the task depends, and the task can only be run after the upstream task of the task is run; similarly, a task downstream of a certain task refers to another task that depends on the task, and the task can only be executed after the task is executed.
In one example, as shown in fig. 2, for tasks a-C, task a is the root node, task B is dependent on task a, and task C is dependent on task B, then task a is an upstream task of task B, task B is a downstream task of task a, task B is an upstream task of task C, and task C is a downstream task of task B.
Referring to the example shown in fig. 2, if the date modification instruction is directed to the task B, the task B is the first task, the service date range of the task B in the DAG diagram is updated from the original date range (2019070100-2019070123) to the new date range (2019070122-2019070123) in the date modification instruction, the original date range (2019070100-2019070123) of the task a is maintained unchanged, and the service date range of the task C is updated from the original date range (2019070100-2019070123) to the service date range (2019070122-2019070123) consistent with the task B according to the update of the service date range of the task B, that is, the modification of the service date range of the task B only affects the downstream task, that is, the task C, and does not affect the upstream task a.
In another example, if the date modification instructions are for task A and task D, the business date ranges for task A and task D are updated, respectively, and the business date ranges for tasks downstream of task A and downstream of task B are updated, respectively.
In another optional embodiment, the cycle type of the second task is updated in response to a cycle modification instruction for at least one task in the task execution diagram; when the period type of the second task conflicts with the period type of the task at the adjacent level of the second task in the task operation diagram, updating the level of the second task or the adjacent level task; the second task includes: the cycle modifies at least one task for which the instruction is directed.
The adjacent hierarchical task of the second task may be at least one of an upstream task and a downstream task of the second task.
Optionally, the conflict between the cycle type of the second task and the cycle type of the task in the adjacent hierarchy may be determined according to a set granularity dependency relationship of the cycle type. In one example, if the granularity dependency of the cycle type is set to: tasks of a coarse-grained type may depend on tasks of a fine-grained type (e.g., a task of a periodic type of day may depend on a task of a periodic type of hour), or tasks of the same-grained type may depend on each other (e.g., a task of a periodic type of hour may depend on another task of a periodic type of hour).
Optionally, when the hierarchy of the second task or the adjacent hierarchy task is updated, the updating may be performed according to a preset updating policy, or a conflict situation between the cycle type of the second task and the cycle type of the adjacent hierarchy task may be displayed through an error reporting interface, and the updating may be performed in response to a received updating instruction for the conflict situation.
In yet another alternative embodiment, at least one of a third task and a downstream task that depends only on the third task is cancelled in the task execution graph in response to a task cancellation instruction for at least one task in the task execution graph; responding to a task adding instruction aiming at least one task in the task running graph, adding at least one task in the task running graph, and updating the hierarchical relation among the tasks in the added task running graph; the third task includes: at least one task for which the task cancel instruction is directed.
In one example, as shown in fig. 3, for tasks a-C, task a depends on both task B and task C, and if the task targeted by the task cancel instruction is task C, task C is the third task, and task C is cancelled in the DAG graph, since this example does not have any task that depends only on task C, it is sufficient to cancel task C only.
In another example, as shown in fig. 2, if the task targeted by the task cancel instruction is task B, task B is a third task, and task B is cancelled in the DAG map, and task C needs to be cancelled because task C only depends on task B in this example.
In yet another example, as shown in fig. 4, for tasks a-E, task a and task B both depend on task C, task D depends on both task a and task B, task E depends only on task a, if the task for which the task cancel instruction is directed is task a, task a is a third task, and task a and task E are cancelled in the DAG graph, since task D depends not only on task a but also on task B, after task a is cancelled, task D can also run after task B is completed, and thus task D does not need to be cancelled.
The above three implementation manners may be executed simultaneously or sequentially, and this is not limited in this application.
When a certain task is cancelled based on a task cancelling instruction, the embodiment of the application can delete the task or change the display mark of the task in the task running diagram (for example, graying out the display item of the task).
And S103, running the batch tasks according to the updated task running chart, and displaying the updated task running chart.
Specifically, the tasks of each hierarchy are sequentially run according to the hierarchical relationship in the task running graph.
Optionally, the updated task running diagram may be displayed in the task running monitoring page, and the administrator may view information of each task through the task running monitoring page.
In an optional embodiment, at least one level task in the updated task operation graph, for example, a certain level task, may be displayed through the task operation monitoring page, and the administrator may view the task and select whether to view an upstream task or a downstream task of the task, and display the upstream task or the downstream task of the task in response to a view instruction generated by the administrator by selecting. Through the mode, tasks of different levels can be displayed layer by layer according to requirements.
Optionally, when the updated task running diagram is displayed, task instance information and real-time running state information of each task in the task running diagram can be displayed; the task instance information can comprise information such as task content, cycle type, service date range and the like, and can be established according to a task operation diagram; the real-time running state information may be generated based on the state of the task detected in real-time, whether running has been started, whether running has been interrupted (indicating stopping when the task content is not completed), or finished (indicating stopping when the task content has been completed).
The data processing method provided by the embodiment of the application can be applied to a Business Intelligence (BI) system, and the step S103 and the optional implementation manner thereof can be applied to an ETL system in the BI system to implement the rerun of ETL task history data.
The task modification instruction in the embodiment of the present application may be an instruction generated based on a corresponding operation of the administrator.
In an optional implementation manner, the data processing method provided in this embodiment of the present application, on the basis of the above steps S101 to S103, further includes:
and for each task in the updated task operation diagram, in response to the operation mode of the task being the service date asynchronous mode, resetting the starting date of the service date range of the task to be the current date when the task is finished to operate.
In the asynchronous mode of the service date, the starting date of the service date range is reset in the above mode, so that the independence of the task in each operation can be ensured, and for the same task, the operation in the ETL system does not influence the next operation in the ETL system.
In another optional implementation manner, the data processing method provided in this embodiment of the present application, on the basis of the above steps S101 to S103, further includes:
for each task which is started to run, detecting whether the task is interrupted to run; when the task is detected to have interrupted running, showing a running log of the task which has interrupted running; and when the task restarting instruction which is interrupted to run is received, restarting the task according to the cycle type and the service date range of the task in the task running graph.
And the administrator can analyze the reason of the task interrupting the operation through the displayed operation log and restart the task through the displayed restart setting option when the task needs to be restarted. The task restart instruction may be an instruction generated according to a restart operation of a manager.
In yet another optional implementation manner, the data processing method provided in this embodiment of the present application, on the basis of the steps S101 to S103, further includes:
when at least one task in the task operation diagram starts to operate, shielding the received task modification instruction until all tasks in the task operation diagram finish operating; and in response to the received batch cancel instruction, stopping running the started tasks in the task running diagram, and canceling the running plan of the tasks which are not started in the task running diagram.
By the method, the batch tasks can be frozen during the running of the batch tasks, and data abnormity caused by modification of any one of the batch tasks is prevented, so that the running stability of the batch tasks and the motion isolation of the batch tasks are improved.
Based on the same inventive concept, an embodiment of the present application provides a data processing apparatus, as shown in fig. 5, the data processing apparatus 500 includes: an operation diagram generating module 501, an operation diagram updating module 502 and an information display module 503.
The running chart generating module 501 is configured to generate a task running chart in response to a task range, a cycle type, and a service date range of set batch tasks; the task operation diagram comprises the following steps: the hierarchical relationship of the tasks in the batch task, and the cycle type and business date range of each task in the batch task.
The operation diagram updating module 502 is configured to update the task operation diagram in response to a task modification instruction for at least one task in the task operation diagram.
And the information display module 503 is configured to run the batch tasks according to the updated task running diagram, and display the updated task running diagram.
In an optional embodiment, the operation diagram updating module 502 is specifically configured to: updating the business date range of the first task and the business date range of the downstream tasks of the first task in the task operation diagram in response to the date modification instruction aiming at least one task in the task operation diagram; the first task includes: at least one task for which the date modification instruction is directed.
In another optional implementation, the operation diagram updating module 502 is specifically configured to: updating the cycle type of the second task in response to a cycle modification instruction for at least one task in the task execution diagram; when the period type of the second task conflicts with the period type of the task of the adjacent level of the second task in the task operation diagram, updating the level of the second task or the task of the adjacent level; the second task includes: the cycle modifies at least one task for which the instruction is directed.
In yet another optional implementation, the operation diagram updating module 502 is specifically configured to: in response to a task cancel instruction for a third task in the task execution diagram, canceling at least one of the third task and a downstream task that depends only on the third task in the task execution diagram; responding to a task adding instruction aiming at least one task in the task running graph, adding at least one task in the task running graph, and updating the hierarchical relation among the tasks in the added task running graph; the third task includes: at least one task for which the task cancel instruction is directed.
In some embodiments, the runtime map update module 502 can perform the above three embodiments simultaneously.
Optionally, the data processing apparatus 500 provided in this embodiment of the present application further includes: and a date resetting module.
The date resetting module is used for: and for each task in the updated task operation diagram, in response to the fact that the operation mode of the task is the service date asynchronous mode, resetting the starting date of the service date range of the task to be the current date when the task is finished to operate.
Optionally, the data processing apparatus 500 provided in this embodiment of the present application further includes: and a task restarting module.
The information presentation module 503 is further configured to: for each task which is started to run, detecting whether the task is interrupted to run; and when the task interrupted running is detected, showing a running log of the task interrupted running.
The task restart module is used for: and when the task restarting instruction which is interrupted to run is received, restarting the task according to the cycle type and the service date range of the task in the task running graph.
Optionally, the data processing apparatus 500 provided in this embodiment of the present application further includes: and the task canceling module.
The task cancellation module is used for: when at least one task in the task operation diagram starts to operate, shielding the received task modification instruction until all tasks in the task operation diagram finish operating; and in response to the received batch cancel instruction, stopping running the started tasks in the task running diagram, and canceling the running plan of the tasks which are not started in the task running diagram.
The data processing apparatus 500 of the present embodiment may execute any data processing method provided in the embodiments of the present application, and the implementation principles thereof are similar, and details not shown in the present embodiment may refer to the foregoing method embodiments, and are not repeated herein.
Based on the same inventive concept, an embodiment of the present application provides a data processing apparatus, including: the storage and the processor are electrically connected.
The memory stores a computer program which is executed by the processor to realize any data processing method provided by the embodiment of the application.
Those skilled in the art will appreciate that the data processing apparatus provided in the embodiments of the present application may be specially designed and manufactured for the required purposes, or may comprise known apparatus found in general purpose computers. These devices have stored therein computer programs that are selectively activated or reconfigured. Such a computer program may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and respectively coupled to a bus.
The present application provides in an alternative embodiment a data processing apparatus, as shown in fig. 6, the data processing apparatus 600 comprising: the memory 601 and the processor 602 are electrically connected, such as by a bus 603.
The Memory 601 may be a ROM (Read-Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disk storage, optical disk storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.
The Processor 602 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other Programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 602 may also be a combination of computing functions, e.g., comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
Bus 603 may include a path that transfers information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
Optionally, the data processing device 600 may also include a transceiver 604. The transceiver 604 may be used for reception and transmission of signals. The transceiver 604 may allow the electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. It should be noted that the transceiver 604 is not limited to one in practical applications.
Optionally, the data processing device 600 may further comprise an input unit 605. The input unit 605 may be used to receive input numeric, character, image, and/or sound information or to generate key signal inputs related to user settings and function control of the electronic apparatus 600. The input unit 605 may include, but is not limited to, one or more of a touch screen, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, a camera, a microphone, and the like.
Optionally, the data processing device 600 may further comprise an output unit 606. The output unit 606 may be used to output or present information processed by the processor 602. The output unit 606 may include, but is not limited to, one or more of a display device, a speaker, a vibration device, and the like.
While fig. 6 illustrates a data processing apparatus 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
Optionally, the memory 601 is used for storing application program codes for executing the scheme of the present application, and the processor 602 controls the execution. The processor 602 is configured to execute the application program code stored in the memory 601 to implement any one of the data processing methods provided by the embodiments of the present application.
Based on the same inventive concept, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements any one of the data processing methods provided by the present application.
The computer readable medium includes, but is not limited to, any type of disk including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs, flash Memory, magnetic cards, or fiber optic cards. That is, a readable medium includes any medium that stores or transmits information in a form readable by a device (e.g., a computer).
The embodiment of the application provides various optional embodiments of a computer-readable storage medium suitable for any one of the data processing methods. And will not be described in detail herein.
By applying the embodiment of the application, at least the following beneficial effects can be realized:
1) according to the method and the device, the hierarchical relationship among the tasks can be automatically sorted based on the set task range, the period type and the service date range of the batch tasks, the task operation diagram is automatically generated by combining the received modification instruction, then the tasks in the task range are operated in batches according to the task operation diagram, the task operation diagram is displayed for a manager to check and manage, the manager does not need to manually sort and control, the efficiency and the preparation of operating the batch tasks and the efficiency of maintaining and managing the batch tasks can be improved, the method and the device are more suitable for the application scene of rerun of ETL task historical data, and the overall operation and maintenance efficiency can be improved.
2) According to the embodiment of the application, at least one parameter in the current task hierarchical relationship, period type and service date range can be modified according to the received task modification instruction, or corresponding tasks are added and cancelled, so that the use requirement is met, and the modification efficiency and accuracy are high.
3) According to the method and the device, the tasks can be run in batches in the mode of service date asynchronization, and the requirement for re-running of historical data of the tasks is met.
4) According to the embodiment of the application, the running log of the task can be displayed when the task is interrupted to analyze the interruption reason, restart the interrupted task in time and promote the normal running of the batch tasks.
5) In order to ensure the isolation of the batch task operating environment, each task in the batch tasks can be automatically frozen in a mode of shielding task modification instructions during the batch task operation, so that data abnormity caused by task modification is prevented, and the task operating stability and the data accuracy are ensured; and the running plan for canceling the tasks which are not started to run can be taken out according to the batch canceling instruction, so that the function of batch canceling is realized, and the canceling requirement of special conditions is met.
Those of skill in the art will appreciate that the various operations, methods, steps in the processes, acts, or solutions discussed in this application can be interchanged, modified, combined, or eliminated. Further, other steps, measures, or schemes in various operations, methods, or flows that have been discussed in this application can be alternated, altered, rearranged, broken down, combined, or deleted. Further, steps, measures, schemes in the prior art having various operations, methods, procedures disclosed in the present application may also be alternated, modified, rearranged, decomposed, combined, or deleted.
In the description of the present application, it is to be understood that the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless otherwise specified.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.
Claims (10)
1. A data processing method, comprising:
responding to the set task range, cycle type and service date range of the batch tasks, and generating a task operation diagram; the task operation diagram comprises the following steps: the hierarchical relationship of each task in the batch tasks, and the cycle type and the service date range of each task in the batch tasks;
updating the task execution diagram in response to a task modification instruction for at least one task in the task execution diagram;
and running the batch tasks according to the updated task running chart, and displaying the updated task running chart.
2. The data processing method of claim 1, wherein updating the task execution graph in response to a task modification instruction for at least one task in the task execution graph comprises:
updating the business date range of a first task and the business date range of a task downstream of the first task in the task execution diagram in response to a date modification instruction for at least one task in the task execution diagram;
the first task includes: at least one task for which the date modification instruction is directed.
3. The data processing method of claim 1, wherein updating the task execution graph in response to a task modification instruction for at least one task in the task execution graph comprises:
updating the cycle type of the second task in response to a cycle modification instruction for at least one task in the task execution diagram; the second task includes: at least one task for which the cycle modification instruction is directed;
when the period type of the second task conflicts with the period type of the adjacent level task of the second task in the task operation graph, updating the second task or the level of the adjacent level task.
4. The data processing method of claim 1, wherein updating the task execution graph in response to a task modification instruction for at least one task in the task execution graph comprises:
in response to a task cancel instruction for at least one task in the task execution graph, canceling at least one task of a third task and a downstream task that depends only on the third task in the task execution graph; the third task includes: at least one task for which the task cancel instruction is directed;
responding to a task adding instruction aiming at least one task in the task running graph, adding at least one task in the task running graph, and updating the hierarchical relation between the tasks in the added task running graph.
5. The data processing method according to any one of claims 1 to 4, further comprising:
for each task in the updated task operation diagram, in response to that the operation mode of the task is a service date asynchronous mode, resetting the start date of the service date range of the task to be the current date when the task is finished to operate.
6. The data processing method according to any one of claims 1 to 4, further comprising:
for each task which is started to run, detecting whether the task is interrupted to run;
when the task is detected to have interrupted running, showing a running log of the task;
and restarting the task according to the cycle type and the service date range of the task in the task operation diagram when the received task restarting instruction is received.
7. The data processing method according to any one of claims 1 to 4, further comprising:
when at least one task in the task operation diagram starts to operate, shielding the received task modification instruction until all tasks in the task operation diagram finish operating;
and in response to the received batch cancel instruction, stopping running the started tasks in the task running diagram, and canceling the running plan of the tasks which are not started in the task running diagram.
8. A data processing apparatus, comprising:
the running chart generating module is used for responding to the set task range, cycle type and service date range of the batch tasks and generating a task running chart; the task operation diagram comprises the following steps: the hierarchical relationship of each task in the batch tasks, and the cycle type and the service date range of each task in the batch tasks;
the task execution diagram updating module is used for responding to a task modification instruction aiming at least one task in the task execution diagram and updating the task execution diagram;
and the information display module is used for operating the batch tasks according to the updated task running chart and displaying the updated task running chart.
9. A data processing apparatus, characterized by comprising:
a memory;
a processor electrically connected with the memory;
the memory stores a computer program for execution by the processor to implement the data processing method of any one of claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011042041.9A CN112131297A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011042041.9A CN112131297A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112131297A true CN112131297A (en) | 2020-12-25 |
Family
ID=73844281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011042041.9A Pending CN112131297A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131297A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113326397A (en) * | 2021-08-04 | 2021-08-31 | 北京达佳互联信息技术有限公司 | Service data processing method and device |
-
2020
- 2020-09-28 CN CN202011042041.9A patent/CN112131297A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113326397A (en) * | 2021-08-04 | 2021-08-31 | 北京达佳互联信息技术有限公司 | Service data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10778548B2 (en) | Method and system for determining workload availability in a multi-tenant environment | |
| US10430735B2 (en) | Designer tool for managing cloud computing services | |
| US11042533B2 (en) | Distributed system, computer program product and method | |
| US9710322B2 (en) | Component dependency mapping service | |
| US10157052B2 (en) | Software change management | |
| EP2851799B1 (en) | Fault tolerant batch processing | |
| US20220005027A1 (en) | Smart contract regulation | |
| US10346381B2 (en) | Atomic update operations in a data storage system | |
| CN103380423A (en) | Systems and methods for private cloud computing | |
| Wozniak et al. | An optimization approach for the synthesis of autosar architectures | |
| CN109117141B (en) | Method, device, electronic equipment and computer readable storage medium for simplifying programming | |
| CN104216802A (en) | Memory database recovery method and equipment | |
| CN109885565B (en) | Data table cleaning method and device | |
| EP3869377A1 (en) | Method and apparatus for data processing based on smart contract, device and storage medium | |
| CN112131297A (en) | Data processing method, device, equipment and storage medium | |
| CN112685224A (en) | Method, apparatus and computer program product for task management | |
| CN108733536B (en) | Monitoring management system and method | |
| US20190087238A1 (en) | Incremental workflow execution | |
| JP2016200912A (en) | Computer system, computer, job execution time prediction method and job execution time prediction program | |
| US20200401947A1 (en) | Workload tenure prediction for capacity planning | |
| JP5836316B2 (en) | Fault monitoring system, fault monitoring method, and fault monitoring program | |
| CN112988455A (en) | Method, apparatus and computer program product for data backup | |
| JP6546569B2 (en) | Data processing program and data processing method | |
| CN110609807A (en) | Method, apparatus, and computer-readable storage medium for deleting snapshot data | |
| KR102505002B1 (en) | Code reuse processing method, device, electronic equipment, computer readable storage medium and computer program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |