CN112926877A - Health condition judgment method, device, equipment and storage medium for batch processing business - Google Patents

Health condition judgment method, device, equipment and storage medium for batch processing business Download PDF

Info

Publication number
CN112926877A
CN112926877A CN202110325002.8A CN202110325002A CN112926877A CN 112926877 A CN112926877 A CN 112926877A CN 202110325002 A CN202110325002 A CN 202110325002A CN 112926877 A CN112926877 A CN 112926877A
Authority
CN
China
Prior art keywords
file
batch processing
batch
files
current
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110325002.8A
Other languages
Chinese (zh)
Inventor
吴辰康
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp filed Critical China Construction Bank Corp
Priority to CN202110325002.8A priority Critical patent/CN112926877A/en
Publication of CN112926877A publication Critical patent/CN112926877A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0639Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/10Office automation; Time management
    • G06Q10/103Workflow collaboration or project management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/02Banking, e.g. interest calculation or account maintenance

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Educational Administration (AREA)
  • Data Mining & Analysis (AREA)
  • Technology Law (AREA)
  • Game Theory and Decision Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present invention relates to the field of automatic programming, and in particular, provides a health condition determination method, apparatus, device and storage medium for batch processing service, wherein the method comprises: before the batch processing service is executed, determining a file link relation for reflecting the dependency relation between batch processing files in the batch processing operation, and identifying abnormal files through the file link relation; in the batch processing service execution, determining the actual execution time of the current function module in the batch processing operation, and identifying the abnormal batch processing operation according to the actual execution time; after the batch processing service is executed, checking the basic information of the batch processing files to obtain a checking result, and identifying abnormal files according to the checking result. By the method, partial problems can be identified in advance or in time so as to be solved in advance or in time.

Description

Health condition judgment method, device, equipment and storage medium for batch processing business
Technical Field
The present invention relates to the field of automatic programming, and in particular, to a method, an apparatus, a device, and a storage medium for determining health status of a batch service.
Background
In order to support mass data processing (such as batch collection and payment service, interest bearing processing, automatic credit card payment, periodic automatic transfer, general ledger update, report generation, data integration, and the like) with different service functions, a core system of a commercial bank usually completes corresponding services in a batch processing mode at the end of a day.
Due to the numerous internal application components of the commercial bank core system, the implementation of a batch processing service is mostly composed of a plurality of batch processing jobs, and in terms of a simple batch processing service, the batch processing service is composed of the following four batch processing jobs (each batch processing job includes a plurality of functional modules): the data in the related database is downloaded to a downloading file, records to be modified in the selected downloading file are output to a screened file, the screened file is processed according to the business processing rule to generate a new file, and the content of the new file is updated to the corresponding database or transmitted to other external systems. It can be seen that the above steps are executed, and data transfer and sharing among different batch jobs are basically realized through batch files.
Therefore, the inspection of the batch files is one of the important means for judging the health condition of the batch processing business, and in the prior art, the inspection of the batch files is mostly arranged that after the batch processing business is finished at the end of the day, the important files of the important business are extracted by the staff for manual inspection. In the prior art, only the checking process after the batch processing service is completed exists, the time of the checking mode is lagged, and the problem is found afterwards, so that the problem cannot be solved in advance or in time by taking measures.
There is a need for a method for determining health status of batch services, which can overcome the disadvantages of the prior art and identify some problems in advance or in time, so as to solve the problems in advance or in time.
Disclosure of Invention
An object of the embodiments herein is to provide a method, an apparatus, a device and a storage medium for determining health status of a batch service, so as to identify a part of problems in advance or in time, so as to solve the problems in advance or in time.
In order to achieve the above object, in one aspect, an embodiment herein provides a method for determining health condition of a batch service, including:
before the batch processing service is executed, determining a file link relation for reflecting the dependency relation between batch processing files in the batch processing operation, and identifying abnormal files through the file link relation;
in the batch processing service execution, determining the actual execution time of the current function module in the batch processing operation, and identifying the abnormal batch processing operation according to the actual execution time;
after the batch processing service is executed, checking the basic information of the batch processing files to obtain a checking result, and identifying abnormal files according to the checking result.
Preferably, the determining a file link relationship for embodying a dependency relationship between batch files in a batch job includes:
generating an operation link and an operation and file mapping table according to the operation scheduling configuration and the operation configuration of the batch processing service, wherein the execution sequence of each batch processing operation in the batch processing service is recorded in the operation link, and the input files and the output files of each functional module in the batch processing operation are recorded in the operation and file mapping table;
and determining the file link relation according to the operation link and the operation and file mapping table.
Preferably, the identifying an abnormal file through the file link relationship includes:
determining whether the batch processing file without precursor or successor is determined according to the file link relation;
identifying the non-predecessor or non-successor batch file as the exception file.
Preferably, the identifying the predecessor-free or successor-free batch file as the exception file includes:
and identifying batch processing files which do not belong to special files in the non-predecessor or non-successor batch processing files as the abnormal files, wherein the special files comprise batch processing files for environment initialization or batch processing files transmitted to an external system by a core system.
Preferably, the identifying an abnormal batch processing job by the actual execution time includes:
judging whether the actual execution time of the current function module is greater than the predicted execution time or not;
and if the actual execution time is greater than the predicted execution time, the batch processing job of the current functional module is an abnormal batch processing job.
Preferably, in the batch processing service execution, the method further includes:
comparing a first data volume and a second data volume of the current functional module to obtain a comparison result, wherein the first data volume is a historical average data volume of the input file, and the second data volume is a unit time data volume of the input file;
judging whether the data volume of the current functional module exceeds the limit or not according to the comparison result;
and if the data volume of the current functional module exceeds the limit, alarming.
Preferably, the comparing the first data size and the second data size of the current functional module to obtain a comparison result includes:
judging whether the second data volume is larger than the product of the first data volume and a set multiple;
correspondingly, the determining whether the data volume of the current functional module is over-limit according to the comparison result includes:
and if the second data volume is larger than the product of the first data volume and a set multiple, the data volume of the current functional module is out of limit.
Preferably, the checking the basic information of the batch file to obtain a checking result includes:
judging whether the current batch processing file is stored in a core system or not and whether the timestamp of the current batch processing file is correct or not;
correspondingly, identifying an abnormal file according to the checking result comprises the following steps:
and if the current batch processing file is not stored in the core system or the timestamp of the current batch processing file is wrong, identifying the current batch processing file as an abnormal file.
Preferably, the determining whether the timestamp of the current batch file is correct includes:
judging whether a first timestamp of the current batch file recorded in the batch running log is consistent with a second timestamp of the current batch file recorded in the core system or not;
if the first timestamp is consistent with the second timestamp, the timestamp of the current batch file is correct;
and if the first timestamp is not consistent with the second timestamp, the timestamp of the current batch file is wrong.
Preferably, after the batch processing service is executed, the method further includes:
judging whether the current batch processing file is an important batch processing file;
if the current batch processing file is an important batch processing file, determining the processing failure rate of the important batch processing file;
judging whether the processing failure rate of the important batch processing files is greater than an alarm threshold value or not;
and if the failure rate of the important batch processing files is greater than an alarm threshold value, alarming.
Preferably, the method for determining the processing failure rate of the important batch files comprises:
acquiring the processing result state of each record corresponding to the important batch processing file;
and dividing the number of records with the processing result state as failure by the total number of records corresponding to all the processing result states to obtain the processing failure rate of the important batch processing files.
Preferably, after the batch job is executed, the method further includes:
and storing the data volume of the input file per unit time and the execution time of each functional module into corresponding fields of a history table.
In another aspect, an embodiment herein provides a health condition determination apparatus for batch processing service, where the apparatus includes:
the system comprises a front judgment module, a front judgment module and a front judgment module, wherein the front judgment module is used for determining a file link relation used for reflecting the dependency relation between batch-processed files in batch processing operation before the batch processing service is executed, and identifying abnormal files through the file link relation;
the middle judgment module is used for determining the actual execution time of the current function module in the batch processing operation during the execution of the batch processing service and identifying the abnormal batch processing operation according to the actual execution time;
and the rear judgment module checks the basic information of the batch files to obtain a check result after the batch processing service is executed, and identifies the abnormal files according to the check result.
In yet another aspect, embodiments herein also provide a computer device comprising a memory, a processor, and a computer program stored on the memory, the computer program, when executed by the processor, executing instructions according to any one of the methods described above.
In yet another aspect, embodiments herein also provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor of a computer device, executes instructions according to any one of the methods described above.
According to the technical scheme provided by the embodiment, the method for identifying the abnormal files is arranged before the batch processing service is executed, so that some abnormal files in the batch processing service can be identified in advance, problems can be found in time, and measures can be taken to solve the problems in time. By setting the method for identifying the abnormal batch processing operation in the batch processing service execution, the abnormal operation can be identified in time in the batch processing service process, and then measures are taken in time to solve the problem, so that the problem is prevented from being derived and expanded. In the process of executing the batch processing service, all-around covering of each stage before, during and after the execution of the service can realize full-covering health condition examination and judgment, and timely find and solve problems.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating a health condition determination method for a batch service according to an embodiment of the present disclosure;
fig. 2 is a flowchart illustrating a method for determining a file link relationship provided in an embodiment of the present disclosure;
FIG. 3 illustrates a flowchart of identifying an abnormal batch job by actual execution time provided by embodiments herein;
FIG. 4 illustrates another flow diagram in batch business execution provided by embodiments herein;
FIG. 5 is a schematic flow chart illustrating a process for determining whether a timestamp of a current batch file is correct according to an embodiment of the present disclosure;
FIG. 6 is a schematic flow chart illustrating a batch process performed by the present embodiment;
FIG. 7 is a flowchart illustrating a method for determining a processing failure rate of a critical batch file according to an embodiment of the present disclosure;
fig. 8 is a schematic block diagram illustrating a health condition determining apparatus of a batch service according to an embodiment of the present disclosure;
fig. 9 shows a schematic structural diagram of a computer device provided in an embodiment herein.
Description of the symbols of the drawings:
100. a front judgment module;
200. a middle judgment module;
300. a rear judgment module;
902. a computer device;
904. a processor;
906. a memory;
908. a drive mechanism;
910. an input/output module;
912. an input device;
914. an output device;
916. a presentation device;
918. a graphical user interface;
920. a network interface;
922. a communication link;
924. a communication bus.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments herein without making any creative effort, shall fall within the scope of protection.
In the prior art, the inspection of batch files is mostly arranged that after the batch processing business is finished at the end of a day, the important files of the important business are extracted by workers for manual inspection. In the prior art, only the checking process after the batch processing service is completed exists, the time of the checking mode is lagged, and the problem is found afterwards, so that the problem cannot be solved in advance or in time by taking measures.
In order to solve the above problem, embodiments herein provide a method for determining health condition of a batch service, which can identify some problems in advance or in time, so as to solve the problems in advance or in time. Fig. 1 is a schematic diagram of steps of a health condition determination method for batch processing services provided in an embodiment herein, and the present specification provides the method operation steps as described in the embodiment or the flowchart, but more or less operation steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual system or apparatus product executes, it can execute sequentially or in parallel according to the method shown in the embodiment or the figures.
Referring to fig. 1, a method for determining a health condition of a batch service may include the following steps:
s101: before the batch processing service is executed, determining a file link relation for reflecting the dependency relation between batch processing files in the batch processing operation, and identifying abnormal files through the file link relation.
S102: and in the batch processing service execution, determining the actual execution time of the current function module in the batch processing operation, and identifying the abnormal batch processing operation according to the actual execution time.
S103: after the batch processing service is executed, checking the basic information of the batch processing files to obtain a checking result, and identifying abnormal files according to the checking result.
The method for identifying the abnormal files is arranged before the batch processing service is executed, so that some abnormal files in the batch processing service can be identified in advance, problems can be found in time, and measures can be taken to solve the problems in time. By setting the method for identifying the abnormal batch processing operation in the batch processing service execution, the abnormal operation can be identified in time in the batch processing service process, and then measures are taken in time to solve the problem, so that the problem is prevented from being derived and expanded. In the process of executing the batch processing service, all-around covering of each stage before, during and after the execution of the service can realize full-covering health condition examination and judgment, and timely find and solve problems.
Batch processing business is a corresponding business which is completed by a bank in a batch processing mode at the end of every day, for example: settlement processing, automatic payment of credit cards, automatic transfer at regular intervals and the like. The implementation of a complete batch service consists of a plurality of batch jobs, and each batch job is divided into a plurality of functional modules according to the specific job tasks of the batch job. Taking a simple interest-bearing batch processing service as an example, the batch processing service consists of the following four batch processing jobs: the data in the database related to the interest is downloaded to a downloading file, records to be modified in the selected downloading file are output to a screened file, the screened file is processed according to the business processing rule to generate a new file, and the content of the new file is updated to a corresponding database or transmitted to other external systems.
As can be seen from the above-described business process, data transfer and sharing between different batch jobs is realized by each batch file as an input file or an output file. The input file is input into the functional module A, the output file is generated after the corresponding functional steps are executed, the output file of the functional module A can be used as the input file of the functional module B, one batch processing operation is realized through the combination of a plurality of functional modules, and then the whole batch processing service is completed through the mutual cooperation of a plurality of batch processing operations.
Referring to fig. 2, in this embodiment, the determining a file link relationship for embodying a dependency relationship between batch files in a batch job includes:
s1011: and generating an operation link and an operation and file mapping table according to the operation scheduling configuration and the operation configuration of the batch processing service, wherein the execution sequence of each batch processing operation in the batch processing service is recorded in the operation link, and the input file and the output file of each functional module in the batch processing operation are recorded in the operation and file mapping table.
S1012: and determining the file link relation according to the operation link and the operation and file mapping table.
Specifically, the job scheduling configuration and the job configuration of the batch processing service are configured in advance, wherein a job link can be obtained according to the job scheduling configuration, and a job and file mapping table can be obtained according to the job configuration. The operation link can embody the execution sequence of each batch operation, and the operation and file mapping table can embody the input files and the output files of each functional module in the batch operation. Through the combination of the operation link, the operation and the file mapping table, the dependency relationship among the batch files can be obtained, namely, for example, among three batch files A-B-C, the file A is a precursor of the file B, and the file C is a successor of the file B, so that the dependency relationship among the three files A-B-C, namely, the file link relationship, is obtained.
Referring to fig. 3, in this embodiment, the identifying an abnormal file through the file link relationship includes the following two steps:
and determining whether the batch processing files without predecessors or successors are available according to the file link relation.
Identifying the non-predecessor or non-successor batch file as the exception file.
Specifically, the file link relationship is a dependency relationship between batch files, and more intuitively, the file link relationship can be regarded as a tree structure composed of batch files. If there is a batch file with no predecessor or successor in the file link relationship, it represents that there may be a transmission error for the file, or the file may be a redundant obsolete file, for which an exception file may be identified. For the abnormal file, a subsequent warning measure can be taken to remind relevant working personnel to check the abnormal file.
In one embodiment, the identifying the predecessor-free or successor-free batch file as the exception file includes:
and identifying batch processing files which do not belong to special files in the non-predecessor or non-successor batch processing files as the abnormal files, wherein the special files comprise batch processing files for environment initialization or batch processing files transmitted to an external system by a core system.
Particularly, the batch processing file for environment initialization only needs to be created once when the environment is built, no precursor exists, and the corresponding batch processing job only reads the content of the environment initialization file. For batch files transmitted by the core system to external systems, there is no successor, and such files are only used for cross-system interaction. The two types of batch processing files belong to special files, so that for a batch processing file without a precursor or a subsequent, the possibility that the batch processing file is a special file is eliminated, and the batch processing file is identified as an abnormal file.
Referring to fig. 3, in this embodiment, the identifying an abnormal batch processing job by the actual execution time includes:
s1021: and judging whether the actual execution time of the current function module is greater than the predicted execution time.
S1022: and if the actual execution time is greater than the predicted execution time, the batch processing job of the current functional module is an abnormal batch processing job.
During the execution of the batch processing service and before the steps are executed, the input file and the output file of the current function module are acquired according to the job and file mapping table.
Specifically, the determination method of the predicted execution time of the current functional module is as follows: the method is obtained by dividing the data amount of the input file per unit time by the historical execution efficiency, wherein the historical execution efficiency is as follows: the execution efficiency of the input file in the past means any day before the current day of execution. As is well known, the efficiency is total amount/time, in this embodiment, the total amount refers to the total data amount of the input file in the past, the time is the execution time, and refers to the total time the current function module spends in the execution process in the past, and the execution process refers to the process from the input of the input file to the output of the output file. The total data volume and the total time can be obtained through a history table, and the history table is updated along with the end of batch processing service every day.
The data volume of the input file per unit time is as follows: the data volume of the input file of the current functional module on the current execution day can be determined according to the operation and file mapping table, and then the data volume of the file can be obtained. The predicted execution time, which is the time when execution is performed at the past execution efficiency, can be obtained by dividing the data amount of the input file per unit time by the historical execution efficiency.
The actual execution time is the time that the current function module actually runs in the batch processing service execution, if the actual execution time is greater than the predicted execution time, the batch processing job where the current function module is located may have a condition of overtime job, that is, the batch processing job can be identified as an abnormal batch processing job, and subsequently, the abnormal batch processing job can be alarmed to remind a worker to analyze.
Referring to fig. 4, in the embodiment of this document, in the batch processing service execution, the method further includes:
s1023: and comparing the first data size and the second data size of the current functional module to obtain a comparison result, wherein the first data size is the historical average data size of the input file, and the second data size is the unit time data size of the input file.
S1024: and judging whether the data volume of the current functional module exceeds the limit or not according to the comparison result.
S1025: and if the data volume of the current functional module exceeds the limit, alarming.
Specifically, taking the end-of-day batch processing every day as an example, taking one day as a unit, the historical average data volume of the input file is as follows: the total data size of past input files divided by the total number of days in the past, where past refers to: executing any day before the current day, wherein the total data volume can be obtained through a history table, and the total days are the number of data volumes of the input files in the past. If 20 days are available in the past, the data volume of 20 input files is recorded, the sum of the data volume of 20 input files is the total data volume, and 20 is the total days.
The unit time data volume of the input file is as follows: the amount of input file data for the day of execution. If the data volume of the input file is larger than the historical average data volume of the input file, the data volume of the current functional module is out of limit, the functional module with the out-of-limit data volume is alarmed, workers are reminded to pay attention, follow-up workers can further confirm the reason of the sharp increase of the data volume, and the expectation of prolonging the running time of the whole batch is made.
In this embodiment, the checking the basic information of the batch file to obtain a checking result includes:
and judging whether the current batch processing file is stored in the core system or not and whether the timestamp of the current batch processing file is correct or not.
Correspondingly, identifying an abnormal file according to the checking result comprises the following steps:
and if the current batch processing file is not stored in the core system or the timestamp of the current batch processing file is wrong, identifying the current batch processing file as an abnormal file.
After the batch processing service is executed and before the steps are executed, the execution sequence of each batch processing job needs to be obtained according to the job link, and the input file and the output file of each functional module are obtained according to the job and file mapping table. And further obtaining the function modules, input files and output files thereof which are mutually connected, and judging whether the batch processing files are abnormal one by one according to the function modules, the input files and the output files, and further identifying the abnormal files.
In the above-mentioned execution step, the basic information includes whether the current batch file is stored in the core system, and whether the timestamp of the current batch file is correct. Judging whether the current batch file is stored in the core system, specifically: the current batch file can be first searched on the NAS storage, and if the current batch file can be found, the current batch file is proved to be stored in the core system.
Referring to fig. 5, further, the determining whether the timestamp of the current batch file is correct includes:
s1031: and judging whether the first timestamp of the current batch processing file recorded in the batch operation log is consistent with the second timestamp of the current batch processing file recorded in the core system or not.
S1032: and if the first timestamp is consistent with the second timestamp, the timestamp of the current batch file is correct.
S1033: and if the first timestamp is not consistent with the second timestamp, the timestamp of the current batch file is wrong.
Specifically, when the current batch file is generated, the current batch file generation time is automatically recorded once, and the generation time is a first time stamp of the current batch file, and the first time stamp is recorded into the batch operation log. Similarly, the current batch file is stored in the NAS when being generated, and the NAS records the generation time of the current batch file, where the generation time is the second timestamp of the current batch file. The current batch file may have a timestamp generation error due to name error and the like, and the batch file can be proved to be a correct file only when the first timestamp is consistent with the second timestamp.
Referring to fig. 6, in this embodiment, after the batch processing service is executed, the method further includes:
s1034: and judging whether the current batch processing file is an important batch processing file.
S1035: and if the current batch processing file is the important batch processing file, determining the processing failure rate of the important batch processing file.
S1036: and judging whether the processing failure rate of the important batch processing files is greater than an alarm threshold value.
S1037: and if the failure rate of the important batch processing files is greater than an alarm threshold value, alarming.
The important batch processing files are defined in advance according to actual business needs, can be distinguished by adopting special naming specifications, are preset with a specific file format, and are provided with a processing result state which can be used for marking the processing result of each record as success or failure. In the process of checking the basic information of the batch files one by one in the steps, if the file name of the current batch file conforms to a special naming standard set in advance, the current batch file is determined as an important batch file.
Referring to fig. 7, a processing failure rate needs to be determined for an important batch file, and specifically, the method for determining the processing failure rate of the important batch file includes:
S1035A: and acquiring the processing result state of each record corresponding to the important batch processing file.
S1035B: and dividing the number of records with the processing result state as failure by the total number of records corresponding to all the processing result states to obtain the processing failure rate of the important batch processing files.
Specifically, for the important batch processing file, the processing failure rate can be obtained by obtaining the processing result state of each record corresponding to the important batch processing file. And if the failure rate of the current important batch processing files is greater than a preset alarm threshold value, giving an alarm to remind a worker to take corresponding measures.
In this embodiment, after the batch job is executed, the method further includes:
and storing the data volume of the input file per unit time and the execution time of each functional module into corresponding fields of a history table.
Wherein, the history table stores the following batch processing jobs: the data volume per unit time of the input file of each function module, namely the data volume of the input file on the current day of execution, and the execution time, namely the total time spent by the current function module on the current day in the execution process from the input of the input file to the output of the output file. The information can be used for judging the abnormality in the next batch processing operation.
In addition, information such as execution efficiency can be stored for direct use when abnormality judgment is carried out in the next batch processing operation execution, and the information does not need to be acquired for calculation. For the execution efficiency, the execution efficiency is total amount/time, the total amount refers to total data amount of the input files (i.e. sum of data amount of the input files from the first batch job to the current batch job), the time, i.e. the execution time, refers to total time spent by the current function module in the execution process (i.e. sum of execution time from the first batch job to the current batch job), and the execution process refers to starting from input of the input files to output of the output files.
Based on the above method for judging the health condition of the batch service, the embodiments herein further provide a device for judging the health condition of the batch service. The apparatus may include systems (including distributed systems), software (applications), modules, components, servers, clients, etc. that employ the methods described herein in embodiments, in conjunction with any necessary apparatus to implement the hardware. Based on the same innovative concepts, embodiments herein provide an apparatus as described in the following embodiments. Since the implementation scheme of the apparatus for solving the problem is similar to that of the method, the specific apparatus implementation in the embodiment of the present disclosure may refer to the implementation of the foregoing method, and repeated details are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Specifically, fig. 8 is a schematic block structure diagram of an embodiment of a health condition determining apparatus for batch processing services provided in the embodiment of the present disclosure, and referring to fig. 8, the health condition determining apparatus for batch processing services provided in the embodiment of the present disclosure includes: the device comprises a front judgment module 100, a middle judgment module 200 and a rear judgment module 300.
The pre-determination module 100 determines a file link relationship for representing a dependency relationship between batch-processed files in a batch processing job before the batch processing service is executed, and identifies an abnormal file through the file link relationship.
The middle judgment module 200 determines the actual execution time of the current function module in the batch processing job during the execution of the batch processing service, and identifies the abnormal batch processing job according to the actual execution time.
And the post-judgment module 300 is used for checking the basic information of the batch files to obtain a checking result after the batch processing service is executed, and identifying abnormal files according to the checking result.
In an embodiment herein, referring to fig. 9, there is also provided a computer device 902. Computer device 902 may include one or more processors 904, such as one or more Central Processing Units (CPUs) or Graphics Processors (GPUs), each of which may implement one or more hardware threads. The computer device 902 may also comprise any memory 906 for storing any kind of information, such as code, settings, data, etc., and in a particular embodiment a computer program on the memory 906 and executable on the processor 904, which computer program, when executed by the processor 904, may perform the instructions according to the above-described method. For example, and without limitation, memory 906 may include any one or more of the following in combination: any type of RAM, any type of ROM, flash memory devices, hard disks, optical disks, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent fixed or removable components of computer device 902. In one case, when the processor 904 executes the associated instructions, which are stored in any memory or combination of memories, the computer device 902 can perform any of the operations of the associated instructions. The computer device 902 also includes one or more drive mechanisms 908, such as a hard disk drive mechanism, an optical disk drive mechanism, etc., for interacting with any memory.
Computer device 902 may also include an input/output module 910(I/O) for receiving various inputs (via input device 912) and for providing various outputs (via output device 914). One particular output mechanism may include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, input/output module 910(I/O), input device 912, and output device 914 may also be excluded, acting as only one computer device in a network. Computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the above-described components together.
Communication link 922 may be implemented in any manner, such as over a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. Communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
Corresponding to the methods in fig. 1-7, the embodiments herein also provide a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, performs the steps of the above-described method.
Embodiments herein also provide computer readable instructions, wherein when executed by a processor, a program thereof causes the processor to perform the method as shown in fig. 1-7.
It should be understood that, in various embodiments herein, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments herein.
It should also be understood that, in the embodiments herein, the term "and/or" is only one kind of association relation describing an associated object, meaning that three kinds of relations may exist. For example, a and/or B, may represent: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided herein, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments herein.
In addition, functional units in the embodiments herein may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present invention may be implemented in a form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The principles and embodiments of this document are explained herein using specific examples, which are presented only to aid in understanding the methods and their core concepts; meanwhile, for the general technical personnel in the field, according to the idea of this document, there may be changes in the concrete implementation and the application scope, in summary, this description should not be understood as the limitation of this document.

Claims (15)

1. A health condition judgment method for batch processing service is characterized by comprising the following steps:
before the batch processing service is executed, determining a file link relation for reflecting the dependency relation between batch processing files in the batch processing operation, and identifying abnormal files through the file link relation;
in the batch processing service execution, determining the actual execution time of the current function module in the batch processing operation, and identifying the abnormal batch processing operation according to the actual execution time;
after the batch processing service is executed, checking the basic information of the batch processing files to obtain a checking result, and identifying abnormal files according to the checking result.
2. The method of claim 1, wherein determining a file link relationship that embodies dependencies between batch files in a batch job comprises:
generating an operation link and an operation and file mapping table according to the operation scheduling configuration and the operation configuration of the batch processing service, wherein the execution sequence of each batch processing operation in the batch processing service is recorded in the operation link, and the input files and the output files of each functional module in the batch processing operation are recorded in the operation and file mapping table;
and determining the file link relation according to the operation link and the operation and file mapping table.
3. The method of claim 1, wherein identifying an anomalous file from the file link relationship comprises:
determining whether the batch processing file without precursor or successor is determined according to the file link relation;
identifying the non-predecessor or non-successor batch file as the exception file.
4. The method of claim 3, wherein identifying the predecessor-free or successor-free batch file as the exception file comprises:
and identifying batch processing files which do not belong to special files in the non-predecessor or non-successor batch processing files as the abnormal files, wherein the special files comprise batch processing files for environment initialization or batch processing files transmitted to an external system by a core system.
5. The method of claim 1, wherein identifying an exception batch job by the actual execution time comprises:
judging whether the actual execution time of the current function module is greater than the predicted execution time or not;
and if the actual execution time is greater than the predicted execution time, the batch processing job of the current functional module is an abnormal batch processing job.
6. The method of claim 1, wherein in batch business execution, further comprising:
comparing a first data volume and a second data volume of the current functional module to obtain a comparison result, wherein the first data volume is a historical average data volume of the input file, and the second data volume is a unit time data volume of the input file;
judging whether the data volume of the current functional module exceeds the limit or not according to the comparison result;
and if the data volume of the current functional module exceeds the limit, alarming.
7. The method of claim 6, wherein comparing the first data amount and the second data amount of the current functional module to obtain a comparison result comprises:
judging whether the second data volume is larger than the product of the first data volume and a set multiple;
correspondingly, the determining whether the data volume of the current functional module is over-limit according to the comparison result includes:
and if the second data volume is larger than the product of the first data volume and a set multiple, the data volume of the current functional module is out of limit.
8. The method according to claim 1, wherein the checking the basic information of the batch file to obtain a checking result comprises:
judging whether the current batch processing file is stored in a core system or not and whether the timestamp of the current batch processing file is correct or not;
correspondingly, identifying an abnormal file according to the checking result comprises the following steps:
and if the current batch processing file is not stored in the core system or the timestamp of the current batch processing file is wrong, identifying the current batch processing file as an abnormal file.
9. The method of claim 8, wherein determining whether the timestamp of the current batch file is correct comprises:
judging whether a first timestamp of the current batch file recorded in the batch running log is consistent with a second timestamp of the current batch file recorded in the core system or not;
if the first timestamp is consistent with the second timestamp, the timestamp of the current batch file is correct;
and if the first timestamp is not consistent with the second timestamp, the timestamp of the current batch file is wrong.
10. The method of claim 1, further comprising, after the batch transaction is performed:
judging whether the current batch processing file is an important batch processing file;
if the current batch processing file is an important batch processing file, determining the processing failure rate of the important batch processing file;
judging whether the processing failure rate of the important batch processing files is greater than an alarm threshold value or not;
and if the failure rate of the important batch processing files is greater than an alarm threshold value, alarming.
11. The method of claim 10, wherein: the method for determining the processing failure rate of the important batch processing files comprises the following steps:
acquiring the processing result state of each record corresponding to the important batch processing file;
and dividing the number of records with the processing result state as failure by the total number of records corresponding to all the processing result states to obtain the processing failure rate of the important batch processing files.
12. The method of claim 1, further comprising, after execution of a batch job:
and storing the data volume of the input file per unit time and the execution time of each functional module into corresponding fields of a history table.
13. A health condition determination apparatus for a batch service, the apparatus comprising:
the system comprises a front judgment module, a front judgment module and a front judgment module, wherein the front judgment module is used for determining a file link relation used for reflecting the dependency relation between batch-processed files in batch processing operation before the batch processing service is executed, and identifying abnormal files through the file link relation;
the middle judgment module is used for determining the actual execution time of the current function module in the batch processing operation during the execution of the batch processing service and identifying the abnormal batch processing operation according to the actual execution time;
and the rear judgment module checks the basic information of the batch files to obtain a check result after the batch processing service is executed, and identifies the abnormal files according to the check result.
14. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, performs the instructions of the method of any one of claims 1-12.
15. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor of a computer device, is adapted to carry out the instructions of the method according to any one of claims 1-12.
CN202110325002.8A 2021-03-26 2021-03-26 Health condition judgment method, device, equipment and storage medium for batch processing business Pending CN112926877A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110325002.8A CN112926877A (en) 2021-03-26 2021-03-26 Health condition judgment method, device, equipment and storage medium for batch processing business

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110325002.8A CN112926877A (en) 2021-03-26 2021-03-26 Health condition judgment method, device, equipment and storage medium for batch processing business

Publications (1)

Publication Number Publication Date
CN112926877A true CN112926877A (en) 2021-06-08

Family

ID=76176167

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110325002.8A Pending CN112926877A (en) 2021-03-26 2021-03-26 Health condition judgment method, device, equipment and storage medium for batch processing business

Country Status (1)

Country Link
CN (1) CN112926877A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114938339A (en) * 2022-05-19 2022-08-23 中国农业银行股份有限公司 Data processing method and related device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011070318A (en) * 2009-09-24 2011-04-07 Hitachi Information Systems Ltd Method of performing batch processing, and system for performing batch processing
CN109669766A (en) * 2018-09-11 2019-04-23 深圳平安财富宝投资咨询有限公司 Processing method, device, equipment and the storage medium of batch processing job
CN110413483A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Monitoring method, device, electronic equipment and the storage medium of batch jobs data
CN111813743A (en) * 2020-06-30 2020-10-23 中国工商银行股份有限公司 Method and device for generating batch file relation chains

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011070318A (en) * 2009-09-24 2011-04-07 Hitachi Information Systems Ltd Method of performing batch processing, and system for performing batch processing
CN109669766A (en) * 2018-09-11 2019-04-23 深圳平安财富宝投资咨询有限公司 Processing method, device, equipment and the storage medium of batch processing job
CN110413483A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Monitoring method, device, electronic equipment and the storage medium of batch jobs data
CN111813743A (en) * 2020-06-30 2020-10-23 中国工商银行股份有限公司 Method and device for generating batch file relation chains

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114938339A (en) * 2022-05-19 2022-08-23 中国农业银行股份有限公司 Data processing method and related device
CN114938339B (en) * 2022-05-19 2024-07-12 中国农业银行股份有限公司 Data processing method and related device

Similar Documents

Publication Publication Date Title
CN113396395A (en) Method for effectively evaluating log mode
CN109685634A (en) Pay account checking method, equipment, storage medium and device
CA2948700A1 (en) Systems and methods for websphere mq performance metrics analysis
US11456932B2 (en) System capacity heatmap
US20210397447A1 (en) Automated compliance and testing framework for software development
CN112527599A (en) Intelligent monitoring method and device, electronic equipment and readable storage medium
US10599476B2 (en) Device and method for acquiring values of counters associated with a computational task
EP3710969A1 (en) Key-based logging for processing of structured data items with executable logic
CN114706827A (en) Automatic generation method and device for transaction link topology
CN112926877A (en) Health condition judgment method, device, equipment and storage medium for batch processing business
CN113504996A (en) Load balance detection method, device, equipment and storage medium
CN110704273A (en) Configuration information processing method and device, electronic equipment and storage medium
CN113485901B (en) System evaluation method, device, equipment and medium based on log and index
CN110309206A (en) Order information acquisition method and system
CN116136801B (en) Cloud platform data processing method and device, electronic equipment and storage medium
CN113051135B (en) Monitoring method of application thread pool and thread pool monitoring device
US9098613B2 (en) Logging of application messages
CN110443560B (en) Protocol data management method, device, computer equipment and storage medium
JP2023507688A (en) edge table representation of the process
CN112967127A (en) Suspicious loan checking method, system, computer equipment and storage medium
JP6920235B2 (en) Failure impact investigation device and failure impact investigation method
CN111339160A (en) Scientific and technological achievement data mining method and system
JP2020134977A (en) Workflow visualization device and control method therefor, and program
US11551308B2 (en) Methods for evaluating and optimizing preferred provider organization (PPO) network stacks and devices thereof
CN114298350A (en) Interface generation method and device for operation and maintenance state of application system and display device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination