CN117032922A - Job rerun method, device, equipment, medium and program product - Google Patents

Job rerun method, device, equipment, medium and program product Download PDF

Info

Publication number
CN117032922A
CN117032922A CN202311000290.5A CN202311000290A CN117032922A CN 117032922 A CN117032922 A CN 117032922A CN 202311000290 A CN202311000290 A CN 202311000290A CN 117032922 A CN117032922 A CN 117032922A
Authority
CN
China
Prior art keywords
rerun
task
target
running
statement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311000290.5A
Other languages
Chinese (zh)
Inventor
李永胜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202311000290.5A priority Critical patent/CN117032922A/en
Publication of CN117032922A publication Critical patent/CN117032922A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Abstract

The invention discloses a method, a device, equipment, a medium and a program product for running operation, and relates to the fields of big data technology and financial science and technology. The method comprises the steps of obtaining a target rerun task in a rerun task table; inquiring a blood margin configuration table according to the task identification of the target rerun task to obtain a target operation identification corresponding to the target rerun task; and acquiring a re-running statement from the blood margin configuration table according to the target job identifier, executing the target re-running task according to the re-running statement, and updating a re-running log table according to the re-running task execution information. According to the technical scheme, the automatic matching from the blood margin configuration table to the re-running statement is realized, the blood margin analysis in the re-running process is omitted, the re-running task is automatically executed according to the matched re-running statement, and the problems in the aspects of re-running efficiency and re-running accuracy in the related method are solved.

Description

Job rerun method, device, equipment, medium and program product
Technical Field
The embodiment of the invention relates to the fields of big data technology and financial science and technology, in particular to a method, a device, equipment, a medium and a program product for operation rerun.
Background
In a business system, a scene of job rerun may be required due to adjustment of business data. For example, business personnel frequently change accounts or adjust data content upstream.
When running again, operation staff or developers need to analyze the relationship between the blood edges of the operation, comb the operation information needed to run again and the related levels. The operation and maintenance personnel use the temporary operation rerun function of big data cloud platform, need through manual operation, accomplish the operation and rerun step by step according to blood relationship.
However, the above-mentioned re-running method requires manual analysis of the relationship between the blood edges, and requires manual operation, which consumes a lot of manpower and affects the re-running efficiency. There is a large risk of manual operation, such as running wrong operation, running wrong time batch, etc., which affects the accuracy of re-running.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment, a medium and a program for running again, which can solve the problems of running again efficiency and running again accuracy in the related method.
In a first aspect, an embodiment of the present invention provides a job rerun method, including:
acquiring a target rerun task in a rerun task table, wherein the rerun task table is used for storing rerun task information recorded through a front-end page;
Inquiring a blood margin configuration table according to the task identifier of the target rerun task to obtain a target operation identifier corresponding to the target rerun task, wherein the blood margin configuration table is used for recording the dependency relationship among the operation flow, the operation, the index and the report;
and acquiring a re-running statement from the blood margin configuration table according to the target job identifier, executing the target re-running task according to the re-running statement, and updating a re-running log table according to the re-running task execution information.
In a second aspect, an embodiment of the present invention further provides a running device, where the running device includes:
the task acquisition module is used for acquiring a target rerun task in the rerun task table, wherein the rerun task table is used for storing rerun task information recorded through a front-end page;
the identification inquiry module is used for inquiring a blood margin configuration table according to the task identification of the target rerun task to obtain a target operation identification corresponding to the target rerun task, wherein the blood margin configuration table is used for recording the dependency relationship among the operation flow, the operation, the index and the report;
and the rerun module is used for acquiring rerun sentences from the blood margin configuration table according to the target operation identification, executing the target rerun task according to the rerun sentences, and updating a rerun log table according to rerun task execution information.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the job rerun method according to any one of the embodiments of the present invention when executing the program.
In a fourth aspect, embodiments of the present invention further provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a job rerun method according to any one of the embodiments of the present invention.
In a fifth aspect, embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements a job rerun method according to any one of the embodiments of the present invention.
The embodiment of the invention provides a method, a device, equipment, a medium and a program product for running a running re-running, wherein a target running re-task is configured in a running re-task table, a blood edge configuration table is queried according to a task identifier of the target running re-task to obtain a target job identifier, then a re-running sentence is acquired from the blood edge configuration table according to the target job identifier, the target running re-task is executed according to the re-running sentence, the automatic execution of the running re-task is realized, and the problems in the aspects of re-running efficiency and re-running accuracy in the related method are solved. Because the dependency relationship between the operations is configured in the blood-edge configuration table before the re-running task is executed, the matching from the blood-edge configuration table to the re-running statement can be automatically realized during the re-running operation, the blood-edge analysis in the re-running process is omitted, the automatic execution of the re-running task according to the matched re-running statement is realized, and the re-running efficiency and accuracy are improved. The running task execution information is inserted into the running log table, so that the operation and maintenance personnel can know the running execution condition by inquiring the running log table.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for running a job re-running according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a front-end page according to an embodiment of the present invention;
FIG. 3 is a flowchart of another method for running a task in accordance with an embodiment of the present invention;
FIG. 4 is a flowchart of another method for running a task in accordance with an embodiment of the present invention;
FIG. 5 is a flowchart of another method for running a task in accordance with an embodiment of the present invention;
FIG. 6 is a flowchart of another method for running a task in accordance with an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a device for running a heavy job according to an embodiment of the present invention;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance. The technical scheme of the application obtains, stores, uses, processes and the like the data, which all meet the relevant regulations of national laws and regulations.
Fig. 1 is a flowchart of a method for running a heavy job according to an embodiment of the present application, where the method is applicable to a heavy job running situation in a big data cloud platform. The method may be performed by a job rerun device, which may be implemented in software and/or hardware and is typically configured in an electronic device. For example, the electronic device may be a server or a terminal device.
As shown in fig. 1, the method includes:
s110, acquiring a target rerun task in the rerun task table.
The rerun task table is used for storing rerun task information input by a user through a front-end page, and the front-end page can be a page displayed on the terminal equipment. The re-running task can be understood as a link re-running task, and can be a process of re-running by taking a certain workflow, operation or index as an initial node and selecting a specific workflow, specific operation or specific index according to a dependency relationship.
In the embodiment of the invention, the job flow can be understood as an entity for organizing and managing a plurality of jobs with business relation in the system. I.e. a job flow is a running set of jobs. Each job node may implement certain processing logic according to configuration. The job nodes can be uncorrelated, or can be vertices of a directed acyclic graph. It is understood that the job flow may be a scheduling unit, and the job node therein may be a minimum granularity execution unit.
A job is a basic configuration unit of a system, may be a logical unit defined by a person using the system to perform a certain job, and may include executed programs and parameters. The same program, different parameter forms, can be different job configurations for the system. The job is a basic unit of scheduling, and the job flow is an organization unit of the job, and a group of jobs with similar functions and similar frequencies are generally organized under the same job flow.
The front page may be a page presented to the operation and maintenance personnel through the terminal device. Fig. 2 is a schematic diagram of a front-end page according to an embodiment of the present invention. As shown in fig. 2, the front page includes an information configuration table, a new button, a modification button, and a delete button. The information configuration table includes a task name, a re-running task, a re-running data date, and a task execution time. The operation and maintenance personnel can configure corresponding re-running task information according to the fields in the information configuration table. And checking that the operation and maintenance personnel inputs the rerun task information on the front-end page, and inserting the rerun task information which is newly input into the rerun task table. In addition, the operation and maintenance personnel can also modify or delete the rerun task information in the rerun task list through the front end page.
Optionally, under the condition that the task configuration operation of the front-end page is detected, updating the re-running task table according to the re-running task information corresponding to the task configuration operation. The rerun task information may be content information of the rerun task. For example, the rerun task information includes a task name, a task identification, a rerun data date, and a task execution time.
Table 1 is a schematic representation of the structure of a re-running task table.
S120, inquiring a blood margin configuration table according to the task identification of the target rerun task to obtain a target operation identification corresponding to the target rerun task.
The blood margin configuration table is used for recording the dependency relationship among the operation flow, the operation, the index and the report. Business personnel can pre-arrange the blood margin relation among the operations in the system, initialize the blood margin configuration table according to the blood margin relation, and update the blood margin configuration table in real time according to the blood margin change condition among the operations. Optionally, the data content of the blood-edge configuration table can be automatically obtained from the big data cloud platform according to the field of the blood-edge configuration table through the interface of the big data cloud platform so as to perfect the blood-edge configuration table.
Table 2 is a schematic representation of the structure of a blood-margin configuration table.
In the embodiment of the invention, the task identifier can be a unique identifier of the target rerun task. A target rerun task may be uniquely identified by a task identification. For example, the task identifier may be an english name corresponding to the rerun workflow/job/indicator.
The querying the blood-margin configuration table according to the task identifier of the target rerun task to obtain the target task identifier corresponding to the target rerun task includes: acquiring a task identifier of the target rerun task in the rerun task table; and acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
And matching the task identifier of the target rerun task with the job identifier in the blood edge configuration table to obtain a target job identifier corresponding to the target rerun task. Wherein the job identifier is the identification information in the blood-source configuration table for uniquely identifying the job flow/job/index.
For example, the run_job of the target rerun task is matched with the job_id in the blood-edge configuration table, and the corresponding target job_id is obtained.
S130, acquiring a re-running statement from the blood margin configuration table according to the target job identification, executing the target re-running task according to the re-running statement, and updating a re-running log table according to the re-running task execution information.
Wherein, the re-run sentence may be a sentence for re-running. For example, the re-run statement may be a big data cloud platform job statement or an index statement, or the like.
The running task execution information may be used to indicate the execution of the job running again. The re-running task execution information comprises whether data before and after re-running are different, the number of difference records, the absence of the re-running task, the presence of errors in re-running sentences, the fact that the date of the data is not standard and the like.
The rerun log table may be a table for recording execution information of the rerun task, and the rerun event, execution information, etc. of each job rerun may be queried through the rerun log table. The running re-log table can be queried through the front page to obtain the running re-execution condition.
Table 3 is a schematic representation of the structure of a rerun log table.
For example, the run_job of the target rerun task is matched with the job_id in the blood-edge configuration table, and the corresponding target job_id is obtained. Then, the corresponding rerun statement, table name, english name corresponding to the dependent workflow/job/index, report id, and the like in the blood-source configuration table may be acquired based on the target job_id. And acquiring a re-running statement, a table name, an English name corresponding to a dependent workflow/operation/index, a report id and the like corresponding to the target job_id in the blood-margin configuration table, and executing the re-running task to obtain re-running task execution information. The corresponding content in the rerun task execution information may be obtained based on the fields in table 3 and inserted into the rerun log table.
In the embodiment of the invention, the target rerun task is configured in the rerun task table, the target operation identification is obtained by inquiring the blood-margin configuration table according to the task identification of the target rerun task, the rerun sentence is obtained from the blood-margin configuration table according to the target operation identification, and the target rerun task is executed according to the rerun sentence, so that the rerun task is automatically executed, and the problems of rerun efficiency and rerun accuracy in the related method are solved. Because the dependency relationship between the operations is configured in the blood-edge configuration table before the re-running task is executed, the matching from the blood-edge configuration table to the re-running statement can be automatically realized during the re-running operation, the blood-edge analysis in the re-running process is omitted, the automatic execution of the re-running task according to the matched re-running statement is realized, and the re-running efficiency and accuracy are improved. The running task execution information is inserted into the running log table, so that the operation and maintenance personnel can know the running execution condition by inquiring the running log table.
Fig. 3 is a flowchart of another operation rerun method according to an embodiment of the present invention, where the embodiment further defines that a rerun sentence is obtained from the blood-margin configuration table according to the target operation identifier, the target rerun task is executed according to the rerun sentence, and a rerun log table is updated according to rerun task execution information based on the above embodiment.
As shown in fig. 3, the method includes:
s310, acquiring a target rerun task in the rerun task table.
Illustratively, the acquiring the target rerun task in the rerun task table includes: triggering a re-running event at fixed time; under the condition of triggering a re-running event, acquiring task execution time in the re-running task table; and acquiring a target rerun task to be executed in the rerun task table according to the task execution time.
For example, a timed task is preset to trigger a re-run event when a time specified by the timed task is reached. If the rerun event is triggered, all run_time in the rerun task list is obtained, and run_time later than the current moment is screened out from the run_time, so that the rerun task which is not executed is obtained. And regarding the unexecuted rerun task, taking the unexecuted rerun task as a target rerun task when the task execution time is reached.
Optionally, before acquiring the task execution time in the rerun task table, the method further includes: in the event of a re-run event trigger, the re-run task table and the blood-edge configuration table are synchronized to a massively parallel processing architecture MPP (Massively Parallel Processing) database.
The MPP database may be a database of a massively parallel processing architecture. The MPP distributes tasks to a plurality of servers and nodes in parallel, and after calculation is completed on each node, the results of the respective parts are summarized together to obtain a final result. The linear expansion of the system is achieved by increasing the number of nodes.
Before executing the job rerun, the rerun task table and the blood-margin configuration table are synchronized from the ORACLE database to the MPP database to facilitate parallel execution of the rerun statement.
S320, acquiring a task identification of the target rerun task in the rerun task table.
S330, acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
S340, inquiring the blood margin configuration table according to the target job identification to obtain a re-running statement and a target table name corresponding to the target re-running task.
The target table name may be the name of the data table to be re-run. Target table names table_name and report_name and the like can be acquired from the blood-edge configuration table by the job_id corresponding to the run_job.
The querying the blood margin configuration table according to the target job identifier to obtain a re-running statement and a target table name corresponding to the target re-running task includes: acquiring a re-running statement and a target table name corresponding to the target operation identifier in the blood margin configuration table; and taking the re-running statement and the target table name corresponding to the target job identifier as the re-running statement and the target table name of the target re-running task.
For example, the run_job of the target rerun task is matched with the job_id in the blood-edge configuration table, and the corresponding target job_id is obtained. Then, the corresponding run_sql, table_name, report_id, report_name, and the like in the blood-edge configuration table can be acquired based on the target job_id.
S350, acquiring the dependency information corresponding to the target operation identifier in the blood margin configuration table.
The dependency information may be identification information of a workflow/job/index, etc. on which the job is re-run. For example, the run_job of the target rerun task is matched with the job_id in the blood-edge configuration table, and the corresponding target job_id is obtained. The corresponding rel_id in the blood-edge configuration table may then be obtained based on the target job_id. Wherein rel_id is the corresponding English name of the dependent workflow/operation/index.
S360, acquiring a rerun data date in the rerun task list, and determining a first rerun data range according to the rerun data date and a target list name.
The date of the rerun data may be used to define the period of the data for which the rerun is performed, i.e., which date of the data is to be rerun.
The rerun task list is queried according to the task identification of the target rerun task, and the rerun data date corresponding to the target rerun task is obtained. And according to the rerun data date and the target table name, acquiring the data of the target table name on the corresponding rerun data date from the MPP database as a first rerun data range.
And S370, backing up the target data in the first rerun data range according to the set backup statement to obtain backup data.
For example, before executing the rerun sentence, the target data of the first rerun data range needs to be backed up for comparison with the rerun result data after the rerun of the job.
And S380, executing the target rerun task based on the target data, the dependency information and the rerun statement in the first rerun data range.
For example, based on the target data and the dependency information in the first rerun data range, executing the big data cloud operation statement or the index statement, and implementing the target rerun task to obtain the rerun result data.
S390, comparing the backup data with the rerun result data through setting a comparison statement.
The re-running result data are data generated by executing the target re-running task.
S3100, determining the running task execution information according to the comparison result and the running statement execution information, and storing the running task execution information into a running log table.
Illustratively, when the target rerun task of the job rerun is executed each time, the rerun sentence execution result and the comparison result are used as rerun task execution information, and the rerun task execution information is inserted into the rerun log table. The task_id in the re-running log table is a task identifier, and corresponds to the re-running task table id field.
In the embodiment of the invention, the first rerun data range is determined by combining the rerun data date in the rerun task table and the rerun statement and the target table name corresponding to the target operation mark in the blood margin configuration table, and the target rerun task is executed based on the target data and the rerun statement in the first rerun data range, so that the dependency information and the data range of the operation rerun are automatically determined, and the rerun statement is executed based on the data range and the dependency information, thereby improving the operation rerun efficiency.
Fig. 4 is a flowchart of another operation rerun method according to an embodiment of the present invention, where a target rerun task is executed according to a blood-margin hierarchy relationship is added on the basis of the above embodiments. As shown in fig. 4, the method includes:
s410, acquiring a target rerun task in the rerun task table.
S420, acquiring a task identification of the target rerun task in the rerun task table.
S430, acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
S440, inquiring the blood margin configuration table according to the target job identification to obtain a re-running statement and a target table name corresponding to the target re-running task.
S450, acquiring a blood margin hierarchy relation according to the target operation identifier, and determining an operation re-running hierarchy according to the blood margin hierarchy relation.
The blood-edge hierarchical relationship may be an upstream-downstream dependency relationship of the job. For example, the execution condition of the target job needs to depend on the upstream job, and the target job can be executed only after the execution of the upstream job is completed. Dependency relationships between upstream and downstream jobs are represented by blood-lineage hierarchical relationships.
The job rerun hierarchy may be an execution order of the job rerun. For example, the blood-margin hierarchy relationship is: the execution condition of the job a needs to be completed depending on the execution of the job B, and the execution condition of the job B needs to be completed depending on the execution of the job C, the job re-running level may be the job c→the job b→the job a, which means that the job C is executed first, then the job B is executed, and then the job a is executed.
Illustratively, since the downstream job can be executed after the upstream job is executed, the job re-run level can be obtained from the blood-lineage hierarchy relationship.
S460, executing the target rerun task according to the target table name and the rerun statement and the job rerun level.
The method includes the steps that a rerun data date in a rerun task table is obtained, a first rerun data range is determined according to the rerun data date and a target table name, a rerun sentence is operated according to a job rerun level based on target data in the first rerun data range, and a target rerun task is implemented.
S470, updating the rerun log table according to the rerun task execution information.
According to the embodiment of the invention, the target rerun task is executed according to the target table name and the rerun sentence and the target rerun stage by setting the blood margin hierarchy relation and determining the operation rerun stage based on the blood margin hierarchy relation, so that the rerun is realized automatically according to the hierarchy blood margin, the operation execution sequence is met, and the rerun efficiency is further improved.
Fig. 5 is a flowchart of another method for running a running operation according to an embodiment of the present invention, where a running data range is determined according to report information is added on the basis of the above embodiments. As shown in fig. 5, the method includes:
s510, acquiring a target rerun task in the rerun task table.
S520, acquiring a task identification of the target rerun task in the rerun task table.
S530, acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
S540, inquiring the blood margin configuration table according to the target job identification to obtain a re-running statement and a target table name corresponding to the target re-running task.
S550, acquiring report information in the front-end page.
The report is data generated by a report system. The contents of the report can be used as report information. Alternatively, the relevant report information can be entered through a front-end page.
S560, acquiring the rerun data date in the rerun task table, and determining a second rerun data range according to the rerun data date and report information.
The rerun task list is queried according to the task identification of the target rerun task, and the rerun data date corresponding to the target rerun task is obtained. According to the re-running data date and the report information, the report information of which date is executed with the operation re-running can be determined, namely, a second re-running data range is determined.
S570, executing the target rerun task based on the target data in the second data range and the rerun statement.
Illustratively, the re-run statement is run according to the target data in the second data range in the report to execute the target re-run task.
S580, updating the rerun log table according to the rerun task execution information.
In the embodiment of the invention, report information is input through the front-end page, the second rerun data range is determined according to the report information and the rerun data date, and the target rerun task is executed based on the target data and the rerun sentence in the second data range, so that the reconfigurability of the rerun range is realized, and the flexibility of operation rerun is improved.
Fig. 6 is a flowchart of another operation re-running method according to an embodiment of the present invention, where the embodiment provides a detailed flow of the operation re-running method based on the above embodiments. As shown in fig. 6, the method includes:
s601, starting.
S602, acquiring a re-running task table and a blood edge configuration table in an ORACLE database.
S603, data synchronization.
Illustratively, the re-running task table and the blood-source configuration table are synchronized from the ORACLE database to the MPP database.
S604, storing the synchronized rerun task list and the blood-margin configuration list through an MPP database.
S605, triggering a re-running event and running a re-running program.
S606, acquiring a main key id of the rerun task list, and acquiring a new task in the rerun task list.
For example, a new task may be obtained From the re-running task table by Select From re-running task table Where run_time=mm, where mm represents the date.
S607, judging whether a new task is inquired, if so, executing S608, otherwise executing S612.
S608, acquiring an English name (run_id) of the rerun operation flow/operation/index, and reading a rerun statement corresponding to the heme configuration table according to the job_id corresponding to the run_id.
Exemplary, field content matching is performed according to a run_id of a new task in the rerun task table and a job_id in the heme configuration table, so as to obtain a job_id corresponding to the run_id, and then, according to a Select x From heme_id=rerun task table, a rerun statement corresponding to the heme configuration table is obtained.
S609, judging whether the rerun sentence is inquired, if yes, executing S610, otherwise, executing S612.
And S610, executing the backup sentence and the rerun sentence on the rerun data in sequence to obtain backup data and rerun result data, and executing the comparison sentence on the backup data and the rerun result data to obtain the comparison result.
S611, determining the execution information of the rerun task according to the comparison result and the execution information of the rerun sentence, and storing the execution information of the rerun task into a rerun log table.
S612, ending.
Fig. 7 is a schematic structural diagram of a device for running a task in accordance with an embodiment of the present invention, where the device may be implemented in software and/or hardware and is generally configured in an electronic device. For example, the electronic device may be a server or a terminal device.
As shown in fig. 7, the apparatus includes: task acquisition module 710, identification query module 720, and rerun module 730.
The task acquisition module 710 is configured to acquire a target rerun task in a rerun task table, where the rerun task table is used to store rerun task information that is input by a user through a front end page;
the identifier query module 720 is configured to query a blood-edge configuration table according to the task identifier of the target rerun task, to obtain a target job identifier corresponding to the target rerun task, where the blood-edge configuration table is used to record a dependency relationship among a job flow, a job, an index and a report;
And a re-running module 730, configured to obtain a re-running statement from the blood-edge configuration table according to the target job identifier, execute the target re-running task according to the re-running statement, and update a re-running log table according to the re-running task execution information.
Optionally, the task obtaining module 710 is specifically configured to:
triggering a re-running event at fixed time;
under the condition of triggering a re-running event, acquiring task execution time in the re-running task table;
and acquiring a target rerun task to be executed in the rerun task table according to the task execution time.
Optionally, the apparatus further comprises:
and the data synchronization module is used for synchronizing the rerun task table and the blood margin configuration table to the MPP database of the massive parallel processing architecture under the condition of triggering a rerun event before acquiring the task execution time in the rerun task table.
Optionally, the apparatus further comprises:
the task configuration module is used for updating the re-running task list according to the re-running task information corresponding to the task configuration operation under the condition that the task configuration operation of the front end page is detected before the target re-running task in the re-running task list is acquired, wherein the re-running task information comprises a task name, a task identification, a re-running data date and a task execution time.
Optionally, the identification query module 720 is specifically configured to:
acquiring a task identifier of the target rerun task in the rerun task table;
and acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
Optionally, the re-running module 730 is specifically configured to:
inquiring the blood margin configuration table according to the target job identification to obtain a re-running statement and a target table name corresponding to the target re-running task.
Further, the re-running module 730 is specifically configured to:
acquiring a re-running statement and a target table name corresponding to the target operation identifier in the blood margin configuration table;
and taking the re-running statement and the target table name corresponding to the target job identifier as the re-running statement and the target table name of the target re-running task.
Further, the re-running module 730 is specifically further configured to:
acquiring dependency information corresponding to the target operation identifier in the blood margin configuration table;
acquiring a rerun data date in the rerun task table, and determining a first rerun data range according to the rerun data date and a target table name;
and executing the target rerun task based on the target data, the dependency information and the rerun statement in the first rerun data range.
Optionally, the apparatus further comprises:
the data backup module is used for backing up the target data in the first rerun data range according to a set backup statement before executing the target rerun task to obtain backup data;
and the data comparison module is used for comparing the backup data with the rerun result data through a set comparison statement after the target rerun task is executed, wherein the rerun result data is the data generated by executing the target rerun task.
Optionally, the re-running module 730 is specifically further configured to:
and determining the running task execution information according to the comparison result and the running statement execution information, and storing the running task execution information into a running log table.
Optionally, the re-running module 730 is specifically further configured to:
acquiring a blood margin hierarchy relation according to the target operation identifier, and determining an operation re-running hierarchy according to the blood margin hierarchy relation;
and executing the target rerun task according to the task rerun level according to the target table name and the rerun statement.
Optionally, the re-running module 730 is specifically further configured to:
acquiring report information in a front-end page;
acquiring a rerun data date in the rerun task table, and determining a second rerun data range according to the rerun data date and report information;
And executing the target rerun task based on the target data in the second data range and the rerun statement.
The operation re-running device provided by the embodiment of the invention can execute the operation re-running method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention. Electronic devices (servers or terminal devices) are intended to represent various forms of digital computers, such as laptops, desktops, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 8, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the respective methods and processes described above, such as a job rerun method.
In some embodiments, the job rerun method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the job rerun method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the job rerun method in any other suitable manner (e.g., by means of firmware).
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements a job rerun method as provided by any of the embodiments of the present application.
Computer program product in the implementation, the computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims (16)

1. A method of operation rerun, comprising:
acquiring a target rerun task in a rerun task table, wherein the rerun task table is used for storing rerun task information recorded through a front-end page;
inquiring a blood margin configuration table according to the task identifier of the target rerun task to obtain a target operation identifier corresponding to the target rerun task, wherein the blood margin configuration table is used for recording the dependency relationship among the operation flow, the operation, the index and the report;
and acquiring a re-running statement from the blood margin configuration table according to the target job identifier, executing the target re-running task according to the re-running statement, and updating a re-running log table according to the re-running task execution information.
2. The method of claim 1, wherein the obtaining the target rerun task in the rerun task table comprises:
triggering a re-running event at fixed time;
under the condition of triggering a re-running event, acquiring task execution time in the re-running task table;
and acquiring a target rerun task to be executed in the rerun task table according to the task execution time.
3. The method of claim 2, further comprising, prior to obtaining the task execution time in the rerun task table:
and under the condition of triggering a re-running event, synchronizing the re-running task table and the blood-edge configuration table to an MPP database of the large-scale parallel processing architecture.
4. The method of claim 1, further comprising, prior to obtaining the target rerun task in the rerun task table:
under the condition that task configuration operation of a front-end page is detected, updating the re-running task table according to re-running task information corresponding to the task configuration operation, wherein the re-running task information comprises a task name, a task identifier, a re-running data date and task execution time.
5. The method of claim 1, wherein the querying the blood-source configuration table according to the task identifier of the target rerun task to obtain the target task identifier corresponding to the target rerun task includes:
Acquiring a task identifier of the target rerun task in the rerun task table;
and acquiring a target job identifier corresponding to the target rerun task from the blood margin configuration table according to the task identifier of the target rerun task.
6. The method of claim 1, wherein the obtaining the re-run statement from the blood-edge configuration table according to the target job identification comprises:
inquiring the blood margin configuration table according to the target job identification to obtain a re-running statement and a target table name corresponding to the target re-running task.
7. The method of claim 6, wherein the querying the blood-margin configuration table according to the target job identifier to obtain a re-running sentence and a target table name corresponding to the target re-running task comprises:
acquiring a re-running statement and a target table name corresponding to the target operation identifier in the blood margin configuration table;
and taking the re-running statement and the target table name corresponding to the target job identifier as the re-running statement and the target table name of the target re-running task.
8. The method of claim 7, wherein the performing the target rerun task according to the rerun statement comprises:
Acquiring dependency information corresponding to the target operation identifier in the blood margin configuration table;
acquiring a rerun data date in the rerun task table, and determining a first rerun data range according to the rerun data date and a target table name;
and executing the target rerun task based on the target data, the dependency information and the rerun statement in the first rerun data range.
9. The method of claim 8, further comprising, prior to performing the target rerun task:
backing up target data in the first rerun data range according to a set backup statement to obtain backup data;
after the target rerun task is executed, the backup data and the rerun result data are compared through a set comparison statement, wherein the rerun result data are data generated by executing the target rerun task.
10. The method of claim 9, wherein updating the rerun log table based on the rerun task performance information comprises:
and determining the running task execution information according to the comparison result and the running statement execution information, and storing the running task execution information into a running log table.
11. The method of claim 6, wherein the performing the target rerun task according to the rerun statement comprises:
acquiring a blood margin hierarchy relation according to the target operation identifier, and determining an operation re-running hierarchy according to the blood margin hierarchy relation;
and executing the target rerun task according to the task rerun level according to the target table name and the rerun statement.
12. The method of claim 6, wherein the performing the target rerun task according to the rerun statement comprises:
acquiring report information in a front-end page;
acquiring a rerun data date in the rerun task table, and determining a second rerun data range according to the rerun data date and report information;
and executing the target rerun task based on the target data in the second data range and the rerun statement.
13. A work rerun device, comprising:
the task acquisition module is used for acquiring a target rerun task in the rerun task table, wherein the rerun task table is used for storing rerun task information recorded through a front-end page;
the identification inquiry module is used for inquiring a blood margin configuration table according to the task identification of the target rerun task to obtain a target operation identification corresponding to the target rerun task, wherein the blood margin configuration table is used for recording the dependency relationship among the operation flow, the operation, the index and the report;
And the rerun module is used for acquiring rerun sentences from the blood margin configuration table according to the target operation identification, executing the target rerun task according to the rerun sentences, and updating a rerun log table according to rerun task execution information.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the method of re-running jobs as claimed in any one of claims 1 to 12 when the computer program is executed by the processor.
15. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a job rerun method as claimed in any one of claims 1 to 12.
16. A computer program product comprising a computer program which, when executed by a processor, implements the job rerun method of any one of claims 1-12.
CN202311000290.5A 2023-08-09 2023-08-09 Job rerun method, device, equipment, medium and program product Pending CN117032922A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311000290.5A CN117032922A (en) 2023-08-09 2023-08-09 Job rerun method, device, equipment, medium and program product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311000290.5A CN117032922A (en) 2023-08-09 2023-08-09 Job rerun method, device, equipment, medium and program product

Publications (1)

Publication Number Publication Date
CN117032922A true CN117032922A (en) 2023-11-10

Family

ID=88625717

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311000290.5A Pending CN117032922A (en) 2023-08-09 2023-08-09 Job rerun method, device, equipment, medium and program product

Country Status (1)

Country Link
CN (1) CN117032922A (en)

Similar Documents

Publication Publication Date Title
CN111061833A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN108647357B (en) Data query method and device
US9760592B2 (en) Metrics management and monitoring system for service transition and delivery management
EP4332785A2 (en) Rules-based dataset cleaning
CN110134663B (en) Organization structure data processing method and device and electronic equipment
US20200210427A1 (en) Column lineage and metadata propagation
CN112818048A (en) Hierarchical construction method and device of data warehouse, electronic equipment and storage medium
CN114461644A (en) Data acquisition method and device, electronic equipment and storage medium
CN113656407A (en) Data topology generation method and device, electronic equipment and storage medium
CN116414814A (en) Data checking method, device, equipment, storage medium and program product
CN116028517A (en) Fusion database system and electronic equipment
CN117032922A (en) Job rerun method, device, equipment, medium and program product
CN115048352A (en) Log field extraction method, device, equipment and storage medium
CN115438056A (en) Data acquisition method, device, equipment and storage medium
CN115757304A (en) Log storage method, device and system, electronic equipment and storage medium
CN114860346A (en) Management method of modeling task and operation method of modeling task
CN114169776A (en) Task processing method and device
CN108764607B (en) User monthly data review method, device, equipment and storage medium
CN112988507B (en) Service monitoring method, device, equipment, storage medium and computer program product
US11816112B1 (en) Systems and methods for automated process discovery
US20230409595A1 (en) Synchronising datasets updates
CN117873990A (en) DDL analysis method, device, equipment and storage medium based on WAL log
CN116841982A (en) Database migration method, device, equipment, storage medium and program product
CN117076577A (en) Method, device, equipment, storage medium and program product for determining data blood edges
CN117008963A (en) Method and device for estimating software function points, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination