CN116382871A - Information change identification and task rerun method, device, equipment, medium and product - Google Patents

Information change identification and task rerun method, device, equipment, medium and product Download PDF

Info

Publication number
CN116382871A
CN116382871A CN202310357986.7A CN202310357986A CN116382871A CN 116382871 A CN116382871 A CN 116382871A CN 202310357986 A CN202310357986 A CN 202310357986A CN 116382871 A CN116382871 A CN 116382871A
Authority
CN
China
Prior art keywords
task
result
information
attribute
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310357986.7A
Other languages
Chinese (zh)
Inventor
张胜坤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202310357986.7A priority Critical patent/CN116382871A/en
Publication of CN116382871A publication Critical patent/CN116382871A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/04Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • General Engineering & Computer Science (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Technology Law (AREA)
  • General Business, Economics & Management (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to the technical field of big data, in particular to the field of financial science and technology or other related fields, and provides an information change identification and task rerun method, device, equipment, medium and product, which comprises the following steps: acquiring a target task from a task scheduling system; if the attribute information of the target task is determined to change, identifying the change date of the change of the attribute information, and setting the last day of the change date as the starting date; rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one updating result; at least one task sub-result corresponding to the update result in the task result of the target task is identified, and the identified task sub-result is replaced by the update result, so that the task result is converted into a re-running result. The final rerun result obtained by the method can be matched with the latest attribute information, so that the accuracy of the result is ensured, the change and rerun of the automatic identification attribute information are realized, and the processing efficiency and accuracy are improved.

Description

Information change identification and task rerun method, device, equipment, medium and product
Technical Field
The present disclosure relates to the field of big data technologies, and to the field of financial science and technology, or other related fields, and in particular, to a method, apparatus, device, medium, and product for information change identification and task rerun.
Background
Current financial institutions typically employ task scheduling systems to automate task information and obtain task results, such as: calculating the bond task every day to obtain daily interest of the bond; the financial institutions generally have a large amount of task information, a large variety, and a large amount of task operations (e.g., a large number of holding bonds, a large variety of bonds, and a large amount of bond transactions).
The inventor finds that once the attribute information of one or more task information changes, management personnel are required to rerun the task information according to the changed attribute information, so that the obtained rerun result is easy to be wrong, and the production efficiency of the rerun result is low due to a large amount of manual work of the management personnel.
Disclosure of Invention
The application provides an information change identification and task rerun method, device, equipment, medium and product, which are used for solving the problems that a current software system needs to construct pages or components with corresponding functions of adding, modifying, deleting, rechecking, inquiring and the like according to the inquiring authority and the operating authority of each data, so that the data maintenance process is complex, and the data maintenance operation efficiency is low.
In a first aspect, the present application provides a method for identifying information change and restarting a task, including:
acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result;
if the attribute information of the target task is determined to change, identifying the change date of the change of the attribute information, and setting the last day of the change date as the starting date;
rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task performing data processing on task data from the starting date to the current date;
and identifying at least one task sub-result corresponding to the updating result in the task results of the target task, and replacing the identified task sub-result with the updating result to enable the task result to be converted into a re-running result.
In the above scheme, the obtaining a target task from a preset task scheduling system includes:
accessing a task library of the task scheduling system, and identifying the running state of each task information in the task library;
Acquiring task information corresponding to an idle running state, and loading at least one piece of acquired task information into a preset stack;
sorting task information in the stack according to the category or weight of the task information in the stack to obtain a task queue, wherein the category reflects the service type of the task information, and the weight reflects the importance degree of the task information in a task scheduling system;
and taking the task information positioned at the first position in the task queue as a target task.
In the above solution, if it is determined that the attribute information of the target task changes, identifying a change date on which the attribute information changes includes:
extracting attribute information of the target task, wherein the attribute information records the current attribute characteristics of the target task;
if the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and acquiring changed object information in the attribute information, wherein the snapshot information records the attribute characteristics of the target task updated last time, and the object information is the attribute characteristics of the attribute information changed;
And acquiring an object text corresponding to the object information from a preset database, extracting an effective date from the object text, and setting the effective date as the change date.
In the above solution, if it is determined that the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and obtaining object information in which the change occurs in the attribute information includes:
extracting attribute metadata and attribute data in the attribute information into a preset attribute table, and extracting attribute metadata and attribute data in the snapshot information into a preset snapshot table;
the attribute metadata in the attribute table are in one-to-one correspondence with the attribute metadata in the snapshot table;
if the attribute metadata in the attribute table are determined to be completely consistent with the attribute metadata in the snapshot table, comparing the attribute data in the attribute table with the attribute data in the snapshot table one by one; if the attribute data in the attribute table are completely consistent with the attribute data in the snapshot table, determining that the attribute information is unchanged; if the attribute data in the attribute table is determined to be inconsistent with the attribute data in the snapshot table, determining that the attribute information changes, and taking the attribute data in the attribute table inconsistent with the snapshot table as the object information;
If the attribute metadata in the attribute table are inconsistent with the attribute metadata in the snapshot table, determining that the attribute information changes, extracting the attribute data corresponding to the attribute metadata inconsistent with the snapshot table in the attribute table, and taking the extracted attribute data as the object information.
In the above solution, the rolling back the target task to the start date, and executing the target task according to the attribute information to obtain at least one updated result includes:
at least one task data of the target task from the starting date to the current date is obtained, rollback operation is sequentially carried out on at least one task data, each task data is restored to an unexecuted state until rollback operation of the task data of the target task on the starting date is completed, the target task is rolled back to the starting date, wherein the task data is a task process for generating a task sub-result by carrying out operation according to snapshot information of the target task, and the task sub-result is a result part corresponding to the task data in the task result of the target task;
and allocating an operation resource to the target task, and operating the target task through the operation resource, so that at least one task data from the starting date to the current date in the target task is operated according to the attribute information, and at least one updating result is obtained.
In the above scheme, allocating an operation resource to the target task includes:
acquiring the data volume of task data of the target task from the starting date to the current date;
determining CPU occupation amount required by running the target task according to the data amount, and/or determining memory occupation amount required by running the target task according to the data amount, and/or determining hard disk occupation amount required by running the target task according to the data amount, and/or determining network occupation amount required by running the target task according to the data amount;
according to the CPU occupation amount, the memory occupation amount, the hard disk occupation amount and/or the network occupation amount, obtaining resource occupation information, and distributing operation resources to the target task, wherein the operation resources comprise: CPU resources corresponding to the CPU occupation amount, and/or memory resources corresponding to the memory occupation amount, and/or hard disk resources corresponding to the hard disk occupation amount, and/or network resources corresponding to the network occupation amount.
In the above solution, the identifying at least one task sub-result corresponding to the update result in the task results of the target task, replacing the identified task sub-result with the update result, and converting the task result into a re-running result includes:
Extracting a task ID corresponding to each update result, wherein the task ID is a unique identifier of task data for generating the update result;
sequentially identifying task sub-results corresponding to each task ID in the target task, and sequentially setting at least one identified task sub-result as a result to be replaced;
and respectively replacing at least one to-be-replaced result with an update result of the task ID corresponding to the to-be-replaced result.
In the above solution, after the rolling back the target task to the start date and performing the target task according to the attribute information to obtain at least one updated result, the method further includes:
if at least one update failure condition exists in the update results, rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one retry update result; if at least one update failure result in the retry update results is determined, generating error reporting information; wherein, the update failure condition includes: and the updated result with null value and/or abnormal value in the content represents that the task data can not generate error information of the updated result according to the attribute information.
In a second aspect, the present application provides an information change recognition and task rerun device, including:
the input module is used for acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result;
the processing module is used for identifying the change date of the change of the attribute information if the change of the attribute information of the target task is determined, and setting the last day of the change date as the starting date;
the execution module is used for rolling back the target task to the starting date and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task for performing data processing on task data from the starting date to the current date;
and the output module is used for identifying at least one task sub-result corresponding to the updating result in the task results of the target task, replacing the identified task sub-result with the updating result, and converting the task result into a re-running result.
In a third aspect, the present application provides a computer device comprising: a processor and a memory communicatively coupled to the processor;
The memory stores computer-executable instructions;
the processor executes the computer-executable instructions stored in the memory to implement the information change identification and task re-running methods as described in the claims.
In a fourth aspect, the present application provides a computer readable storage medium having stored therein computer executable instructions that when executed by a processor are configured to implement the above-described information change identification and task re-running method.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the above-described information change identification and task re-running method.
According to the information change identification and task rerun method, device, equipment, medium and product, the target task is obtained from the preset task scheduling system, the change date of the change of the attribute information is identified, the last date of the change date is set as the starting date, the rerun task data are kept in a necessary range, rerun of unnecessary task data is avoided, and rerun efficiency is ensured.
By rolling back the target task to the starting date and executing the target task according to the attribute information to obtain at least one updated result, identifying at least one task sub-result corresponding to the updated result in the task results of the target task, replacing the identified task sub-result with the updated result, converting the task result into a rerun result, enabling the rerun result finally obtained by the target task to be matched with the latest attribute information, ensuring the accuracy of the result, simultaneously realizing the change and rerun of the automatically identified attribute information, and improving the processing efficiency and accuracy.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;
FIG. 2 is a flowchart of an embodiment 1 of a method for information change identification and task rerun according to an embodiment of the present application;
FIG. 3 is a flowchart of an embodiment 2 of a method for identifying information changes and re-running tasks according to an embodiment of the present application;
FIG. 4 is a block diagram of an embodiment 3 of an information change recognition and task rerun device according to an embodiment of the present application;
fig. 5 is a schematic diagram of a hardware structure of a computer device in the computer device according to the present invention.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
Referring to fig. 1, the specific application scenario in the present application is: the financial institution holds the bond quantity more, and the bond kind is many and the bond transaction volume is big to the bond is calculated complicatly, carries out account position processing to the bond product, probably because reasons such as bond reset interest rate is not in time updated and bond return factor and desk calendar change and basic information of bond are absent or inaccurate, lead to the processing inaccuracy such as bond account position profit and loss, correspond the historical erroneous data that produces and also can not automatic correction, can only carry out technical change repair data by technological personnel, consuming time and effort has influenced the accuracy of bond account position and supervision data report.
Aiming at the defects existing in the prior art, in order to reduce the labor input cost and improve the accuracy of the position of the bond account book, a server 11 running with an information change identification and task rerun method is connected with a task scheduling system 12, and the server 11 acquires a target task from a preset task scheduling system;
if the server 11 determines that the attribute information of the target task changes, identifying a change date when the attribute information changes, and setting the last date of the change date as a start date;
the server 11 rolls back the target task to the starting date, and executes the target task according to the attribute information to obtain at least one updating result;
the server 11 identifies at least one task sub-result corresponding to the update result from among the task results of the target task, and replaces the identified task sub-result with the update result, thereby converting the task result into a re-run result.
Therefore, the server 11 automatically rolls back the position of the bond account book to the day before the day of heavy running, then automatically performs daily transaction buying and selling calculation, lifetime processing and the like of the dimension of the bond account book according to the date, and then automatically rolls back to the current latest date, thereby automatically restoring the position data of the historical account book, greatly improving the efficiency and accuracy of restoring the position data of the account book, reducing the input of human resources, and simultaneously realizing the function of automatically re-running the position of the abnormal bond account book.
It should be noted that the method, device, equipment, medium and product for identifying information change and re-running tasks can be used in the financial field. But also can be used in any fields other than the financial field. The information change identification and task rerun method, device, equipment, medium and product application fields are not limited.
The following describes the technical solutions of the present application and how the technical solutions of the present application solve the prior art problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Example 1:
referring to fig. 2, the present application provides a method for identifying information change and re-running tasks, including:
s201: and acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result.
In the step, xxl-job and elastic-job are adopted as task scheduling systems, xxl-job is a distributed task scheduling platform, and the core design aims of the method are rapid development, simple learning, lightweight and easy expansion. The Elastic-Job is a distributed scheduling solution after the secondary development of the current network based on quatz, consisting of two relatively independent sub-projects, elastic-Job-Lite and Elastic-Job-Cloud.
Acquiring a target task from a task scheduling system through a preset calling rule, wherein the calling rule comprises the following steps: and acquiring task information with idle running state as a target task, and/or acquiring task information with business category belonging to a key category as a target task, and/or acquiring task information with weight reaching a preset weight threshold as a target task.
In a preferred embodiment, obtaining a target task from a preset task scheduling system includes:
accessing a task library of a task scheduling system, and identifying the running state of each task information in the task library;
acquiring task information corresponding to an idle running state, and loading at least one piece of acquired task information into a preset stack;
according to the category or weight of the task information in the stack, sequencing the task information in the stack to obtain a task queue, wherein the category reflects the service type of the task information, and the weight reflects the importance degree of the task information in a task scheduling system;
and taking the task information positioned at the first position in the task queue as a target task.
Illustratively, the categories of task information in the database of the task scheduling system include: the number of species a, species B,
The task information of category a includes: task information 1, weight 5; task information 2, weight 4;
the task information of category B includes: task information 3, weight 3; task information 4, weight 6;
if the task information 1 is in an operation state, the task information 2, the task information 3 and the task information 4 are in an idle state, and the task information 2, the task information 3 and the task information 4 are input into a stack;
if the importance of category A is higher than that of category B, the task information 2 is arranged before the task information 3 and the task information 4, and the obtained task queue is: task information 2-task information 4-task information 3. The task information 2 at the first position is taken as a target task.
The task queues obtained by sequencing the task information according to the weights are as follows: task information 4-task information 2-task information 3. The task 4 at the first position is taken as a target task.
S202: if the attribute information of the target task is determined to change, the change date of the change of the attribute information is identified, and the last date of the change date is set as the starting date.
In the step, whether the content of the attribute information is changed is determined by comparing the current attribute information of the target task with the previously stored snapshot information, if the current attribute information and the snapshot information are different, the change date of the change of the attribute information is obtained, and the last date of the change date is set as the starting date, so that the rerun task data are kept in a necessary range, the rerun of unnecessary task data is avoided, and the rerun efficiency is ensured.
In this embodiment, the date of validity of the attribute data in which the change occurs in the attribute information is obtained from the task scheduling platform or the transaction system connected to the task scheduling platform, for example: the attribute data where the change occurs is the interest rate, the interest rate in the current attribute information is 2%, and the interest rate in the snapshot information is 3%, then the effective date of the interest rate of 2% is obtained from the task scheduling platform or the transaction system, such as: effective for 2022, 1 and 12, then 2022, 1 and 11 are taken as change dates.
In a preferred embodiment, if it is determined that the attribute information of the target task changes, identifying a change date on which the attribute information changes includes:
extracting attribute information of a target task, wherein the attribute information records the current attribute characteristics of the target task;
if the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and acquiring changed object information in the attribute information, wherein the snapshot information records the attribute characteristics of the target task updated last time, and the object information is the attribute characteristics of the change in the attribute information;
and acquiring an object text corresponding to the object information from a preset database, extracting an effective date from the object text, and setting the effective date as a change date.
Illustratively, assuming that the target task is an a bond, the attribute information of the target task includes: bond face value: 5 tens of millions; ticket face interest rate: 2%; and (3) a payoff period: paying for half a year;
the snapshot information of the bond a is: bond face value: 5 tens of millions; ticket face interest rate: 3%; and (3) a payoff period: paying for half a year;
object information of the occurrence of the change can be obtained: ticket face interest rate: 2%; obtaining the ticket interest rate from a task scheduling system or a transaction platform: 2% of corresponding object text, for example: a bond interest rate change notification file, identifying the effective date in the object text through a preset natural language model, for example: such as: the effect was obtained on day 1 and day 12 of 2022, and day 1 and day 11 of 2022 were set as the date of change.
Further, if it is determined that the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and acquiring object information in which the change occurs in the attribute information includes:
extracting attribute metadata and attribute data in the attribute information into a preset attribute table, and extracting the attribute metadata and the attribute data in the snapshot information into a preset snapshot table;
the attribute metadata in the attribute table are in one-to-one correspondence with the attribute metadata in the snapshot table;
If the attribute metadata in the attribute table are completely consistent with the attribute metadata in the snapshot table, comparing the attribute data in the attribute table with the attribute data in the snapshot table one by one; if the attribute data in the attribute table is completely consistent with the attribute data in the snapshot table, determining that the attribute information is unchanged; if the attribute data in the attribute table is not completely consistent with the attribute data in the snapshot table, determining that the attribute information changes, and taking the attribute data inconsistent with the snapshot table in the attribute table as object information;
if the attribute metadata in the attribute table are inconsistent with the attribute metadata in the snapshot table, determining that the attribute information changes, extracting the attribute data corresponding to the attribute metadata inconsistent with the snapshot table in the attribute table, and taking the extracted attribute data as object information.
Illustratively, assuming that the target task is an a bond, the attribute information of the target task includes: bond face value: 5 tens of millions; ticket face interest rate: 2%; and (3) a payoff period: paying for half a year; additional handling fees: 1 million;
the snapshot information of the bond a is: bond face value: 5 tens of millions; ticket face interest rate: 3%; and (3) a payoff period: paying for half a year; object information of the occurrence of the change can be obtained: ticket face interest rate: 2%; additional handling fees: 1 million; obtaining the ticket interest rate from a task scheduling system or a transaction platform: 2% of corresponding object text, for example: a bond interest rate change notification file, identifying the effective date in the object text through a preset natural language model, for example: such as: the effect was obtained on day 1 and day 12 of 2022, and day 1 and day 11 of 2022 were set as the date of change. Obtaining additional commission from a task scheduling system or a transaction platform: 1 million corresponding object text, for example: a bond commission management notification file, identifying the effective date in the object text through a preset natural language model, for example: such as: the effect is taken on day 1 and 2 of 2022, and day 1 and 1 of 2022 are set as change dates.
Thus, in implementing the above steps, it may be necessary to determine the change in attribute information using different alignment methods in consideration of different types of objects, such as data tables, fields, tasks, and the like. In addition, in order to ensure the correctness and consistency of the data, the problem of multi-version data management needs to be considered, and the data snapshot is backed up in time to prevent the misoperation of the data.
S203: and rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task performing data processing on task data from the starting date to the current date.
In this step, by determining the task object and the start date that need to be rolled back, all task data of the target task after the start date are queried, and the rolling operation is performed on the task data one by one, that is, the data is modified according to the opposite operation, so as to restore to the state of the start date.
For task data that have already been processed, it is necessary to set their states to processed, avoiding repetitive processing. After processing all the task data needing to be rolled back, the target task rolls back to the state of the starting date.
It should be noted that the integrity and correctness of the data needs to be ensured during the rollback process, so as to avoid adverse effects on the submitted or processed data. In addition, rollback operations may have an impact on other related tasks, requiring comprehensive consideration of processing.
If there are two or more pieces of object information, the earliest time of the change date is set as the start date to ensure the integrity and comprehensiveness of rolling back the target task and running again.
In a preferred embodiment, rolling back the target task to the start date and executing the target task according to the attribute information to obtain at least one updated result includes:
at least one task data of a target task from a starting date to a current date is obtained, rollback operation is sequentially carried out on the at least one task data, each task data is recovered to an unexecuted state until rollback operation of the task data of the target task on the starting date is completed, the target task is rolled back to the starting date, the task data is a task process for generating a task sub-result according to operation of snapshot information of the target task, and the task sub-result is a result part corresponding to the task data in a task result of the target task;
And allocating operation resources to the target task, and operating the target task through the operation resources, so that at least one task data from the starting date to the current date in the target task is operated according to the attribute information to obtain at least one updating result.
Illustratively, if the current date is 2022, 1, 20, and the starting date is 2022, 1, 11, and if the target task is to be performed once every day, then the task data for 2022, 1, 11, to 2022, 1, 20 is rolled back; and then according to [ bond denomination: 5 tens of millions; ticket face interest rate: 2%; and (3) a payoff period: paying for half a year; sequentially executing task data of 2022, 1 month, 11 days, 2022, 1 month, and 20 days, obtaining task sub-results of 2022, 1 month, 11 days, 2022, 1 month, and 20 days, and respectively taking the task sub-results as updated results.
Further, allocating an operation resource to the target task includes:
acquiring the data volume of task data of a target task from a starting date to a current date;
determining CPU occupation amount required by the operation target task according to the data amount, and/or determining memory occupation amount required by the operation target task according to the data amount, and/or determining hard disk occupation amount required by the operation target task according to the data amount, and/or determining network occupation amount required by the operation target task according to the data amount;
According to the CPU occupation amount, the memory occupation amount, the hard disk occupation amount and/or the network occupation amount, obtaining resource occupation information, and distributing computing resources to the target task, wherein the computing resources comprise: CPU resources corresponding to the CPU occupation amount, and/or memory resources corresponding to the memory occupation amount, and/or hard disk resources corresponding to the hard disk occupation amount, and/or network resources corresponding to the network occupation amount.
The allocation of computing resources is illustratively carried out by calling a Hadoop component or a YARN component according to the data quantity, wherein the data quantity of task data is the data input quantity, the data calculation quantity and the data output quantity of task data operated according to attribute information, the Hadoop component and the YARN component support the scheduling of two resources of a memory and a CPU, and in the YARN component, the resource management is jointly completed by a resource manager and a NodeManager, wherein a scheduler in the resource manager is responsible for the allocation of the resources, and the NodeManager is responsible for the supply and isolation of the resources. After resource manager allocates resources on a certain NodeManager to tasks (which is called resource scheduling), the NodeManager needs to provide corresponding resources for the tasks according to requirements, even guarantee that the resources should have exclusivity, and provide basic guarantee for task operation, which is called resource isolation.
The Kafka system is called to distribute the occupation amount of the hard disk according to the data amount, kafka is a message system originally developed from LinkedIn and used as an active stream of LinkedIn, and the active stream data is the most conventional part of data which is used by almost all sites when reporting the use condition of the website. The activity data includes content such as Page View, information on viewed content, and search conditions.
The NetFlow Analyzer is called to distribute network occupation according to data volume, and the NetFlow Analyzer network flow monitoring software is special for monitoring network flow, and helps users to know flow composition, protocol distribution and user activities. The NetFlow Analyzer utilizes Flow technology to collect information about Flow in a network, integrates collection, analysis and reporting of the Flow, and further gets knowledge of the most concerned problems of users such as occupation conditions of Flow and bandwidth, and the like, thereby providing scientific basis for comprehensively knowing network activities of enterprises, reasonably and effectively distributing and planning network bandwidth and ensuring smooth operation of key business applications of the enterprises.
S204: at least one task sub-result corresponding to the update result in the task result of the target task is identified, and the identified task sub-result is replaced by the update result, so that the task result is converted into a re-running result.
In this step, at least one task sub-result corresponding to the update result in the task result of the target task is replaced by the update result, so that the task result is converted into a rerun result, the final result obtained by implementing the target task can be matched with the latest attribute information, the accuracy of the result is ensured, and the whole process of the embodiment realizes the automatic identification of the change and rerun of the attribute information, and improves the processing efficiency and accuracy.
In a preferred embodiment, identifying at least one task sub-result corresponding to the update result in the task results of the target task, replacing the identified task sub-result with the update result, and converting the task result into a re-run result, including:
extracting a task ID corresponding to each update result, wherein the task ID is a unique identifier of task data for generating the update result;
sequentially identifying task sub-results corresponding to each task ID in the target task, and sequentially setting at least one identified task sub-result as a result to be replaced;
and respectively replacing at least one to-be-replaced result with the update result of the task ID corresponding to the to-be-replaced result.
By way of example, the number, name, or date of the task data may be employed as the task ID, ensuring that the update result replaces the task sub-result corresponding thereto by making the update result correspond to the task sub-result based on the task ID, ensuring the reliability and accuracy of the replacement.
Example 2:
referring to fig. 3, the present application provides a method for identifying information change and re-running tasks, including:
s301: and acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result.
This step corresponds to S201 in embodiment 1, and thus will not be described here.
S302: if the attribute information of the target task is determined to change, the change date of the change of the attribute information is identified, and the last date of the change date is set as the starting date.
This step corresponds to S202 in embodiment 1, and thus will not be described here.
S303: and rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task performing data processing on task data from the starting date to the current date.
This step corresponds to S203 in embodiment 1, and thus will not be described here.
S304: at least one task sub-result corresponding to the update result in the task result of the target task is identified, and the identified task sub-result is replaced by the update result, so that the task result is converted into a re-running result.
This step corresponds to S204 in embodiment 1, and thus will not be described here.
S305: if at least one update failure condition exists in the obtained update results, rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one retry update result; if at least one update failure result in the retry update result is determined, generating error reporting information; the update failure condition comprises: and the content has an updating result of a null value and/or an abnormal value, and the task data cannot generate error information of the updating result according to the attribute information.
In this step, the content in the update result is identified by a preset natural language processing model, and if it is determined that the content contains error information indicating that the task data cannot generate the update result according to the attribute information, for example: and if the task fails, a result cannot be generated, and the like, determining that the update result has update failure.
Whether at least one update result obtained by using the isnull function has a null value or not can be identified, and whether the update result obtained by using the countif function has an abnormal value belonging to the repeated value or not is identified.
And identifying whether the obtained updated result has an abnormal value or not through a preset clustering model. The clustering model performs clustering operation on at least one updating result to obtain a data cluster, wherein the data cluster is provided with data points corresponding to each updating result; recognizing data points scattered in the data cluster, calculating Euclidean distance between the data points and the central point of the data cluster, and if the Euclidean distance is determined to be larger than a preset abnormal threshold value, confirming that the obtained updated result has an abnormal value; if the Euclidean distance is not larger than the preset abnormal threshold value, confirming that the obtained updated result does not have an abnormal value.
Example 3:
referring to fig. 4, the present application provides an information change recognition and task rerun device 4, which includes:
the input module 41 is configured to obtain a target task from a preset task scheduling system, where the target task is configured to perform data processing on task data to obtain a task result;
the processing module 42 is configured to identify a change date when the attribute information of the target task changes, and set a last day of the change date as a start date;
the execution module 43 is configured to roll back the target task to the start date, and execute the target task according to the attribute information to obtain at least one update result, where the update result is a task data processing result obtained by performing data processing on task data from the start date to the current date by the target task;
and the output module 44 is configured to identify at least one task sub-result corresponding to the update result in the task results of the target task, replace the identified task sub-result with the update result, and convert the task result into a re-running result.
Optionally, the information change recognition and task rerun device 4 further includes:
the anomaly identification module 45 is configured to roll back the target task to the start date if it is determined that at least one update failure condition exists in the obtained update results, and execute the target task according to the attribute information to obtain at least one retry update result; if at least one update failure result in the retry update result is determined, generating error reporting information; the update failure condition comprises: and the content has an updating result of a null value and/or an abnormal value, and the task data cannot generate error information of the updating result according to the attribute information.
Example 4:
to achieve the above object, the present application further provides a computer device 5, including: a processor and a memory communicatively coupled to the processor; the memory stores computer-executable instructions;
the processor executes computer execution instructions stored in the memory to implement the above information change identification and task re-running method, where the components of the information change identification and task re-running device may be dispersed in different computer devices, and the computer device 5 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack server (including an independent server, or a server cluster formed by multiple application servers) that execute a program, and so on. The computer device of the present embodiment includes at least, but is not limited to: a memory 51, a processor 52, which may be communicatively coupled to each other via a system bus, as shown in fig. 5. It should be noted that fig. 5 only shows a computer device with components-but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead. In the present embodiment, the memory 51 (i.e., readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 51 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 51 may also be an external storage device of a computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like. Of course, the memory 51 may also include both internal storage units of the computer device and external storage devices. In this embodiment, the memory 51 is generally used for storing an operating system installed in a computer device and various application software, such as program codes of the information change recognition and task rerun device of the third embodiment. Further, the memory 51 may also be used to temporarily store various types of data that have been output or are to be output. Processor 52 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 52 is typically used to control the overall operation of the computer device. In this embodiment, the processor 52 is configured to execute the program code or process data stored in the memory 51, for example, to execute the information change recognition and task rerun device, so as to implement the information change recognition and task rerun method of the above embodiment.
The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform some steps of the methods of the various embodiments of the present application. It should be appreciated that the processor may be a central processing unit (Central Processing Unit, CPU for short), other general purpose processors, digital signal processor (Digital Signal Processor, DSP for short), application specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution. The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile memory NVM, such as at least one magnetic disk memory, and may also be a U-disk, a removable hard disk, a read-only memory, a magnetic disk or optical disk, etc.
To achieve the above object, the present application further provides a computer readable storage medium such as a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which computer-executable instructions are stored, which when executed by the processor 52, perform the corresponding functions. The computer readable storage medium of the present embodiment is used for storing computer execution instructions for implementing the information change identification and task rerun method, which when executed by the processor 52 implement the information change identification and task rerun method of the above embodiment.
The storage medium may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short). It is also possible that the processor and the storage medium reside as discrete components in an electronic device or a master device.
The application provides a computer program product, comprising a computer program, wherein the computer program realizes the information change identification and task rerun method when being executed by a processor.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (12)

1. An information change identification and task rerun method is characterized by comprising the following steps:
acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result;
if the attribute information of the target task is determined to change, identifying the change date of the change of the attribute information, and setting the last day of the change date as the starting date;
Rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task performing data processing on task data from the starting date to the current date;
and identifying at least one task sub-result corresponding to the updating result in the task results of the target task, and replacing the identified task sub-result with the updating result to enable the task result to be converted into a re-running result.
2. The method for identifying information changes and restarting a task according to claim 1, wherein the step of acquiring a target task from a preset task scheduling system includes:
accessing a task library of the task scheduling system, and identifying the running state of each task information in the task library;
acquiring task information corresponding to an idle running state, and loading at least one piece of acquired task information into a preset stack;
sorting task information in the stack according to the category or weight of the task information in the stack to obtain a task queue, wherein the category reflects the service type of the task information, and the weight reflects the importance degree of the task information in a task scheduling system;
And taking the task information positioned at the first position in the task queue as a target task.
3. The information change recognition and task rerun method of claim 1, wherein if it is determined that the attribute information of the target task changes, recognizing a change date on which the attribute information changes includes:
extracting attribute information of the target task, wherein the attribute information records the current attribute characteristics of the target task;
if the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and acquiring changed object information in the attribute information, wherein the snapshot information records the attribute characteristics of the target task updated last time, and the object information is the attribute characteristics of the attribute information changed;
and acquiring an object text corresponding to the object information from a preset database, extracting an effective date from the object text, and setting the effective date as the change date.
4. The method for identifying information change and re-running task according to claim 3, wherein if it is determined that the attribute information is different from the snapshot information of the target task, determining that the attribute information changes, and acquiring the object information in which the change occurs in the attribute information, includes:
Extracting attribute metadata and attribute data in the attribute information into a preset attribute table, and extracting attribute metadata and attribute data in the snapshot information into a preset snapshot table;
the attribute metadata in the attribute table are in one-to-one correspondence with the attribute metadata in the snapshot table;
if the attribute metadata in the attribute table are determined to be completely consistent with the attribute metadata in the snapshot table, comparing the attribute data in the attribute table with the attribute data in the snapshot table one by one; if the attribute data in the attribute table are completely consistent with the attribute data in the snapshot table, determining that the attribute information is unchanged; if the attribute data in the attribute table is determined to be inconsistent with the attribute data in the snapshot table, determining that the attribute information changes, and taking the attribute data in the attribute table inconsistent with the snapshot table as the object information;
if the attribute metadata in the attribute table are inconsistent with the attribute metadata in the snapshot table, determining that the attribute information changes, extracting the attribute data corresponding to the attribute metadata inconsistent with the snapshot table in the attribute table, and taking the extracted attribute data as the object information.
5. The method for information change identification and task rerun according to claim 1, wherein the rolling back the target task to the start date and executing the target task according to the attribute information to obtain at least one updated result includes:
at least one task data of the target task from the starting date to the current date is obtained, rollback operation is sequentially carried out on at least one task data, each task data is restored to an unexecuted state until rollback operation of the task data of the target task on the starting date is completed, the target task is rolled back to the starting date, wherein the task data is a task process for generating a task sub-result by carrying out operation according to snapshot information of the target task, and the task sub-result is a result part corresponding to the task data in the task result of the target task;
and allocating an operation resource to the target task, and operating the target task through the operation resource, so that at least one task data from the starting date to the current date in the target task is operated according to the attribute information, and at least one updating result is obtained.
6. The method for information change identification and task rerun according to claim 5, wherein the allocating the computing resource to the target task includes:
acquiring the data volume of task data of the target task from the starting date to the current date;
determining CPU occupation amount required by running the target task according to the data amount, and/or determining memory occupation amount required by running the target task according to the data amount, and/or determining hard disk occupation amount required by running the target task according to the data amount, and/or determining network occupation amount required by running the target task according to the data amount;
according to the CPU occupation amount, the memory occupation amount, the hard disk occupation amount and/or the network occupation amount, obtaining resource occupation information, and distributing operation resources to the target task, wherein the operation resources comprise: CPU resources corresponding to the CPU occupation amount, and/or memory resources corresponding to the memory occupation amount, and/or hard disk resources corresponding to the hard disk occupation amount, and/or network resources corresponding to the network occupation amount.
7. The method for identifying information change and restarting a task according to claim 1, wherein the identifying at least one task sub-result corresponding to the updated result in the task results of the target task, replacing the identified task sub-result with the updated result, and converting the task result into a restarting result, includes:
Extracting a task ID corresponding to each update result, wherein the task ID is a unique identifier of task data for generating the update result;
sequentially identifying task sub-results corresponding to each task ID in the target task, and sequentially setting at least one identified task sub-result as a result to be replaced;
and respectively replacing at least one to-be-replaced result with an update result of the task ID corresponding to the to-be-replaced result.
8. The method for information change identification and task rerun according to claim 1, wherein after the rolling back the target task to the start date and performing the target task according to the attribute information to obtain at least one updated result, the method further comprises:
if at least one update failure condition exists in the update results, rolling back the target task to the starting date, and executing the target task according to the attribute information to obtain at least one retry update result; if at least one update failure result in the retry update results is determined, generating error reporting information; wherein, the update failure condition includes: and the updated result with null value and/or abnormal value in the content represents that the task data can not generate error information of the updated result according to the attribute information.
9. An information change recognition and task rerun device, comprising:
the input module is used for acquiring a target task from a preset task scheduling system, wherein the target task is used for carrying out data processing on task data to obtain a task result;
the processing module is used for identifying the change date of the change of the attribute information if the change of the attribute information of the target task is determined, and setting the last day of the change date as the starting date;
the execution module is used for rolling back the target task to the starting date and executing the target task according to the attribute information to obtain at least one updating result, wherein the updating result is a task data processing result obtained by the target task for performing data processing on task data from the starting date to the current date;
and the output module is used for identifying at least one task sub-result corresponding to the updating result in the task results of the target task, replacing the identified task sub-result with the updating result, and converting the task result into a re-running result.
10. A computer device, comprising: a processor and a memory communicatively coupled to the processor;
The memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the information change identification and task re-running method of any one of claims 1 to 8.
11. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to implement the information change identification and task re-running method of any one of claims 1 to 8.
12. A computer program product comprising a computer program which when executed by a processor implements the information change recognition and task re-running method of any one of claims 1-8.
CN202310357986.7A 2023-04-04 2023-04-04 Information change identification and task rerun method, device, equipment, medium and product Pending CN116382871A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310357986.7A CN116382871A (en) 2023-04-04 2023-04-04 Information change identification and task rerun method, device, equipment, medium and product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310357986.7A CN116382871A (en) 2023-04-04 2023-04-04 Information change identification and task rerun method, device, equipment, medium and product

Publications (1)

Publication Number Publication Date
CN116382871A true CN116382871A (en) 2023-07-04

Family

ID=86980225

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310357986.7A Pending CN116382871A (en) 2023-04-04 2023-04-04 Information change identification and task rerun method, device, equipment, medium and product

Country Status (1)

Country Link
CN (1) CN116382871A (en)

Similar Documents

Publication Publication Date Title
US20050165668A1 (en) Multi-processing financial transaction processing system
CN112882699B (en) Service processing method, device, equipment and medium based on flow configuration engine
CN108647357B (en) Data query method and device
CN112363902A (en) Business process monitoring method and device, computer equipment and storage medium
CN110751376B (en) Work order distribution scheduling method and device, computer equipment and storage medium
US20220019566A1 (en) System and method for integrating systems to implement data quality processing
CN110781235A (en) Big data based purchase data processing method and device, terminal and storage medium
CA3182205A1 (en) Financial risk assessment
CN111353763B (en) Method, device, server and storage medium for processing data
CN116382871A (en) Information change identification and task rerun method, device, equipment, medium and product
CN110503477B (en) Zxfoom zxfoom Muli (Maoli) abnormality of a system(s) apparatus and storage medium
CN114691768A (en) Data processing method, accounting system and related equipment
CN114896955A (en) Data report processing method and device, computer equipment and storage medium
CN114511314A (en) Payment account management method and device, computer equipment and storage medium
CN111131393B (en) User activity data statistical method, electronic device and storage medium
CN115082179A (en) Data processing method, device, equipment and storage medium
JP2007041828A (en) Sla achieved situation determining method
CN110888917A (en) Batch running task execution method, device, server and storage medium
CN112632169B (en) Automatic financial data reporting method and device and computer equipment
US11831490B1 (en) Systems, methods, and media for performing information technology service management correlation for infrastructure environment functions
CN116433197B (en) Information reporting method, device, reporting end and storage medium
CN115017875B (en) Enterprise information processing method, device, system, equipment and medium
CN112347095B (en) Data table processing method, device and server
CN116382924B (en) Recommendation method and device for resource allocation, electronic equipment and storage medium
CN118193016A (en) Method, system, electronic equipment and medium for auditing upgrade data of futures system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination