CN112416727A - Batch processing operation checking method, device, equipment and medium - Google Patents

Batch processing operation checking method, device, equipment and medium Download PDF

Info

Publication number
CN112416727A
CN112416727A CN202011322642.5A CN202011322642A CN112416727A CN 112416727 A CN112416727 A CN 112416727A CN 202011322642 A CN202011322642 A CN 202011322642A CN 112416727 A CN112416727 A CN 112416727A
Authority
CN
China
Prior art keywords
job
target batch
information
checking
batch processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011322642.5A
Other languages
Chinese (zh)
Inventor
吴辰康
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp filed Critical China Construction Bank Corp
Priority to CN202011322642.5A priority Critical patent/CN112416727A/en
Publication of CN112416727A publication Critical patent/CN112416727A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the invention discloses a batch processing operation checking method, a batch processing operation checking device, batch processing operation checking equipment and a batch processing operation checking medium. The method comprises the following steps: collecting, by an import module, at least one type of information related to a target batch job run; performing data analysis on the running condition of the target batch processing operation according to the at least one type of information through an analysis module; and checking the running result of the target batch processing operation according to the at least one type of information through a checking module. By the technical scheme, the automation of batch processing operation checking is realized, and the checking efficiency is improved.

Description

Batch processing operation checking method, device, equipment and medium
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a batch processing job checking method, device, equipment and medium.
Background
In order to meet part of service functions of banks, different job instances are often required to be developed, the jobs are connected in series to form a job execution sequence, a whole batch test is initiated through a job scheduling tool, and a job execution result needs to be checked according to a job service scene and a service function.
In the batch processing non-functional optimization stage, comparison tests of a large number of function points are often required, such as an optimization test of long-time execution of jobs, a high-frequency task splitting optimization test, a database query optimization test, a file-downloading performance optimization test and the like. After the batch processing operation is finished, a large amount of execution data can be generated, but the data distribution is dispersed, so that the collection, analysis and check of operation and maintenance personnel are inconvenient to develop, the work is complicated during manual operation, and the data support is not favorably provided for the optimization of the execution performance of the batch processing operation.
Disclosure of Invention
The embodiment of the invention provides a batch processing operation checking method, device, equipment and medium, which are used for realizing the automation of batch processing operation checking and improving the checking efficiency.
In a first aspect, an embodiment of the present invention provides a method for checking a batch job, including:
collecting, by an import module, at least one type of information related to a target batch job run;
performing data analysis on the running condition of the target batch processing operation according to the at least one type of information through an analysis module;
and checking the running result of the target batch processing operation according to the at least one type of information through a checking module.
In a second aspect, an embodiment of the present invention further provides an apparatus for checking a batch job, including:
an import module to collect at least one type of information related to a target batch job run;
the analysis module is used for carrying out data analysis on the running condition of the target batch processing operation according to the at least one type of information;
and the checking module is used for checking the running result of the target batch processing operation according to the at least one type of information.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method for checking a batch job according to any embodiment of the present invention.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a method for checking a batch job according to any embodiment of the present invention.
According to the technical scheme provided by the embodiment of the invention, at least one type of information related to the running of the target batch processing operation is collected through the import module, the running condition of the target batch processing operation is subjected to data analysis through the analysis module according to the at least one type of information, and the running result of the target batch processing operation is checked through the check module according to the at least one type of information, so that the automation of checking the batch processing operation is realized, and the check efficiency is improved.
Drawings
FIG. 1 is a block diagram of a system for checking a batch job according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a batch job checking method according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an apparatus for checking batch jobs according to a fourth embodiment of the present invention;
fig. 4 is a schematic hardware structure diagram of a computer device in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a schematic structural diagram of a batch job checking system according to an embodiment of the present invention, which is applicable to data collection, analysis, and checking of a batch job operation result. As shown in fig. 1, the system for checking a batch job provided in this embodiment specifically includes: an import module 110, an analysis module 120, and a check module 130. Wherein,
an import module 110, which is a batch job run state information collection tool, for collecting at least one type of information related to a target batch job run; an analysis module 120, configured to perform data analysis on the operation condition of the target batch job according to the at least one piece of information; and the checking module 130 is configured to check the operation result of the target batch processing job according to the at least one piece of information.
A target batch job refers to any one batch job in a batch environment.
The import module 110 is configured to collect the operation result of the target batch job, and specifically, data collection needs to be performed on information of each job, such as job step information, offset information, file information, and a calling program.
In an alternative embodiment, the import module collects at least scheduling information of the target batch job, job result data of the target batch job, file information related to the target batch job, and related information in a server executing the target batch job.
Optionally, the import module 110 accesses a scheduling tool, a Network Attached Storage (NAS), a database (e.g., a relational database), and a server to obtain various information related to the target batch job operation.
The import module accesses the scheduling tool to collect scheduling information (such as the context of job execution) related to the target batch job, and specifically reads the scheduling report output by the scheduling tool to collect relevant scheduling information; the import module accesses the NAS and can collect file information related to target batch processing job operation, such as file names, file quantity, record number in files, file width and the like; the import module may access the database to collect job result data for the batch jobs stored in the database; the import module may access the server to collect relevant information, such as job run information and server system information.
The server for executing the target batch processing job may include an AP (Application) server and a high frequency server, and the AP server is used to implement the execution of the target batch processing job, and the high frequency server is used to implement the concurrence and load balancing of tasks within the target batch processing job. The collection of the load information (such as the use condition of a CPU and a memory) of the server in the batch processing operation execution process is also important for the analysis of the batch processing operation result, and powerful data analysis is provided for the performance optimization of subsequent developers.
The import module 110 can obtain the relevant running information of the target batch job only by providing the basic information of the target batch job, and uniformly store the collected information in the local database, so that when the analysis module 120 performs the batch job running result data analysis and the checking module 130 performs the batch job running result checking, it is not necessary to separately collect information for each job and access the local database.
The analysis module 120 is configured to perform data analysis on the operation result of the target batch job, for example, analyze a long-time job in the target batch job, a critical path of the target batch job, an efficiency ratio of the target batch job, a processing condition of the target batch job, and a task concurrency and load balancing condition of the target batch job.
The critical path of the target batch processing job refers to a job execution sequence which takes the longest time from the beginning of executing the first job to the completion of the last job when the target batch processing job runs, and can be acquired from the job running information in the running time period of the target batch processing job. The key path is a job execution sequence for determining the total running time of the batch job, and the optimization of the key path is beneficial to improving the running speed of the batch job.
The efficiency ratio of the target batch job refers to a ratio between the number of file runs and the run time of the target batch job. Longitudinal comparison of the efficiency ratios of batch jobs (i.e., comparing the efficiency ratios of different batches of batch jobs) facilitates analysis of the conditions to be optimized for individual jobs and batch jobs.
The checking module 130 is configured to implement checking of a target batch job operation result, and may generally include database result checking and file result checking, and may specifically implement checking of a batch job operation result based on a service scene and a service function range of a batch job, for example, may check a file processing condition corresponding to a target batch job, may check a file cleaning lower-level condition corresponding to a target batch job, may check a file transmission condition corresponding to a target batch job, and may also check a copy synchronization condition corresponding to a target batch job.
Optionally, the checking module 130 checks a file processing condition corresponding to the target batch processing job based on related information in a server executing the target batch processing job and file information related to the target batch processing job.
The related information in the server executing the target batch job refers to job execution information, and the job basic information and the job execution step information of the target batch job may be analyzed according to the job execution information of the target batch job, and then the file processing condition corresponding to the target batch job may be checked based on the job basic information, the job execution step information, and the file information related to the target batch job of the target batch job.
The job basic information refers to information such as job definition, program definition, and the like; the job execution step information refers to information such as job content, job step, and the like. The job basic information and the job running step information can be obtained by analyzing a JAR package running in the server and corresponding to the target batch processing job, wherein each job step exists in the JAR package in an xml format.
For example, the checking module checks whether the file is processed correctly according to the file offset information and the upstream file information in the job configuration.
Optionally, the checking module 130 checks the file cleaning and downloading condition corresponding to the target batch processing job based on the job result data of the target batch processing job and the file information related to the target batch processing job. The file downloading refers to writing information in the database into a file, and the file cleaning refers to deleting corresponding information in the database. For example, the checking module compares the database data information and the file information before and after the target batch processing job runs, and judges whether the file corresponding to the lower file is correct or not and whether the data corresponding to the upper file is cleared or not.
Optionally, before the target batch processing job runs, the database and the NAS are accessed through the import tool to obtain database data information and file information before and after the target batch processing job runs, and further after the target batch processing job runs, whether a file corresponding to the file downloading is correct or not, whether data corresponding to the file cleaning is removed or not, and the like can be judged.
Optionally, the checking module 130 checks a file transmission condition corresponding to the target batch processing job based on job result data of the target batch processing job and file information related to the target batch processing job. For example, the checking module checks whether the file transmission is correct according to the file transmission configuration information, the control file and the return value of the operation step.
Optionally, the checking module 130 is further configured to check a copy synchronization condition corresponding to the target batch processing job. For example, the checking module checks whether the database operation is correct after the copy synchronization according to the upstream copy information and the database data information before the target batch processing operation is run.
According to the technical scheme provided by the embodiment of the invention, at least one type of information related to the running of the target batch processing operation is collected through the import module, the running condition of the target batch processing operation is subjected to data analysis through the analysis module according to the at least one type of information, and the running result of the target batch processing operation is checked through the check module according to the at least one type of information, so that the automation of the check of the batch processing operation is realized, the check efficiency is improved, and the problem of incomplete or inaccurate check caused by personnel negligence is also avoided.
The batch processing operation checking system provided by the embodiment of the invention does not influence the existing batch processing operation flow, and can realize the automatic check of the batch processing operation of the cross-platform component only by providing a small amount of configuration parameters for the access data source (such as a scheduling tool, an NAS, a database, a server and the like).
Example two
Fig. 2 is a flowchart of a batch job checking method according to a second embodiment of the present invention, which is applicable to data collection, analysis, and checking of a batch job running result, and the method can be executed by a batch job checking apparatus according to the second embodiment of the present invention, and the apparatus can be implemented in software and/or hardware, and can be generally integrated in a computer device.
As shown in fig. 2, the method for checking a batch job provided in this embodiment specifically includes:
s210, collecting at least one type of information related to the target batch processing job operation through an import module.
A target batch job refers to any one batch job in a batch environment.
Optionally, the collecting, by the import module, at least one type of information related to the target batch job operation may specifically be: at least scheduling information related to the target batch job, job result data related to the target batch job, file information related to the target batch job, and related information in a server executing the target batch job are collected by the import module.
The scheduling information refers to the dependency relationship before and after the execution of the job; the related information in the server refers to job running information and server system information, the job running information refers to information related to batch job running, such as job definition, program definition, job steps and the like, and the server system information refers to server CPU, memory information and the like.
The import module accesses the scheduling tool, the NAS, the database, and the server to obtain various information related to the execution of the target batch job, such as scheduling information of the target batch job, job result data, file information related to the target batch job, and related information in the server executing the target batch job.
Optionally, the import module accesses the scheduling tool to collect scheduling information (e.g., a context of job execution) related to the target batch job, and may specifically read the scheduling report output by the scheduling tool to collect relevant scheduling information; the import module accesses the NAS and can collect file information related to target batch processing job operation, such as file names, file quantity, record number in files, file width and the like; the import module may access the database to collect job result data for the batch jobs stored in the database; the import module may access the server to collect relevant information, such as job run information and server system information.
And S220, performing data analysis on the running condition of the target batch processing job according to at least one type of information related to the running of the target batch processing job through the analysis module.
The analysis module is used for analyzing the target batch processing job operation condition based on the data related to the target batch processing job operation collected by the import module, such as analyzing a long-time job in the target batch processing job, a critical path of the target batch processing job, an efficiency ratio of the target batch processing job, a processing condition of the target batch processing job, a task concurrency and load balance condition of the target batch processing job, and the like.
The data analysis of the operation condition of the target batch processing job can be divided into static analysis and dynamic analysis. Static analysis, which refers to analysis based on scheduling information collected by the importing tool accessing the scheduling tool and related to the target batch processing job; the dynamic analysis refers to analysis based on information related to target batch job operation, which is collected by an import tool accessing a NAS, a database and a server.
Optionally, the analysis module outputs a data analysis result for the operation condition of the target batch job, for example, the data analysis result may be output in the form of a data analysis report.
And S230, checking the running result of the target batch processing job according to at least one type of information related to the running of the target batch processing job through the checking module.
And the checking module is used for collecting data related to target batch processing job operation, such as job step operation information, NAS file information, database information and the like, based on the import module, and checking target batch processing job operation results, such as checking the correctness of offset during file processing, the correctness of database file clearing and downloading, the correctness of file transmission, the correctness of replica synchronization database operation and the like.
Optionally, the checking module outputs a checking result for the operation result of the target batch processing job, for example, the checking result may be output in the form of a checking report.
For those parts of this embodiment that are not explained in detail, reference is made to the aforementioned embodiments, which are not repeated herein.
According to the technical scheme provided by the embodiment of the invention, at least one type of information related to the running of the target batch processing operation is collected through the import module, the running condition of the target batch processing operation is subjected to data analysis through the analysis module according to the at least one type of information, and the running result of the target batch processing operation is checked through the check module according to the at least one type of information, so that the automation of the check of the batch processing operation is realized, the check efficiency is improved, and the problem of incomplete or inaccurate check caused by personnel negligence is also avoided.
EXAMPLE III
On the basis of the foregoing embodiments, this embodiment provides several specific implementations of the analysis module performing data analysis on the target batch job operation result, and several specific implementations of the checking module performing checking on the target batch job operation result.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to the at least one piece of information may specifically be:
job basic information of the target batch job is analyzed by the analysis module based on related information in a server executing the target batch job.
In the present embodiment, the information on the server that executes the target batch job refers to job execution information on the server, and is in the form of a JAR package. The job basic information refers to information such as a job definition, a program definition, and the like.
And the analysis module analyzes the job operation information which is collected by the import module and exists in the JAR packet form corresponding to the target batch processing job in the server to obtain the job basic information of the target batch processing job.
Optionally, the analysis module analyzes the file receiving upstream and downstream relationship between jobs, the processing accuracy of the job file, the step operation calling program parameters, and the like based on the job step information, the calling program, the step file processing information, and the like obtained by the analysis.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to the at least one piece of information may specifically be:
job step execution information of the target batch job is analyzed by the analysis module based on relevant information in a server executing the target batch job.
In the present embodiment, the information on the server that executes the target batch job refers to job execution information on the server, and is in the form of a JAR package. The job execution step information refers to information such as job contents, job steps, and the like, and exists in xml format in the JAR package.
And the analysis module analyzes the job operation information which is collected by the import module and exists in the JAR packet form corresponding to the target batch processing job in the server to obtain the job step operation information of the target batch processing job.
Optionally, the analysis module analyzes the specific running condition of the long-time job based on the information of the job running time, the step running time, the number of file records and the like obtained by analysis.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to at least one type of information may specifically be:
analyzing a key path of the running of the target batch processing job based on the scheduling information of the target batch processing job through an analysis module; wherein the critical path affects a total length of time that the target batch job execution sequence runs.
The critical path of the target batch job refers to a job execution sequence that takes the longest time from the beginning of execution of the first job to the completion of the last job when the target batch job runs. The key path is a job execution sequence for determining the total running time of the batch job, and the optimization of the key path is beneficial to improving the running speed of the batch job.
The analysis module analyzes the scheduling information of the target batch processing jobs collected by the import module and analyzes the scheduling operation duration of each job so as to determine the key path of the target batch processing job operation.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to at least one type of information may specifically be:
the operating efficiency of the target batch job is analyzed by the analysis module based on the scheduling information of the target batch job and the file information related to the target batch job.
Alternatively, the operating efficiency of the target batch job is represented by an efficiency ratio. The efficiency ratio refers to the ratio between the number of file runs and the run time of the target batch job. Longitudinal comparison of the efficiency ratios of batch jobs (i.e., comparing the efficiency ratios of different batches of batch jobs) facilitates analysis of the conditions to be optimized for individual jobs and batch jobs.
The analysis module analyzes the scheduling information of the target batch processing job collected by the import module, calculates the job running time of the target batch processing job, analyzes the file information related to the target batch processing job to obtain the file record number of the target batch processing job, and further calculates the ratio of the file record number of the target batch processing job to the job running time as the efficiency ratio of the target batch processing job to analyze the running efficiency of the target batch processing job.
Optionally, the analysis module compares the operation efficiency of the target batch job with the operation efficiency of the matched batch job of the previous batch to analyze the operation efficiency of the batch jobs of different batches.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to at least one type of information may specifically be:
the processing condition of the target batch job is analyzed by the analysis module based on the related information in the server executing the target batch job.
In this embodiment, the related information in the server executing the target batch job refers to server system information, and may specifically be system information of the AP server, such as CPU, memory information, and the like.
And the analysis module analyzes the processing speed of the target batch processing job and the like according to the CPU and memory use condition of the AP server in each time interval collected by the import module.
In an alternative embodiment, referring to fig. 1, the performing, by the analysis module, data analysis on the operation condition of the target batch processing job according to at least one type of information may specifically be:
and analyzing the task concurrency and load balancing condition corresponding to the target batch processing job through the analysis module based on the relevant information in the server executing the target batch processing job.
In this embodiment, the related information in the server executing the target batch job refers to server system information, and may specifically be system information of a high-frequency server, such as CPU, memory information, and the like.
And the analysis module analyzes the concurrent execution condition and the load balance condition of the tasks in the target batch processing job according to the CPU and memory use condition of the high-frequency server in each time interval collected by the import module.
In an alternative embodiment, referring to fig. 1, the checking the operation result of the target batch processing job according to the at least one type of information by the checking module includes:
the processing condition of the file corresponding to the target batch processing job is checked through a checking module based on the relevant information in the server executing the target batch processing job and the file information relevant to the target batch processing job.
In the present embodiment, the related information in the server that executes the target batch job refers to job execution information, and the job basic information and the job execution step information of the target batch job may be analyzed based on the job execution information of the target batch job, and then the file processing condition corresponding to the target batch job may be checked based on the job basic information, the job execution step information, and the file information related to the target batch job of the target batch job.
Optionally, the checking module checks the file processing condition corresponding to the target batch processing job according to the job basic information and the job operation step information of the target batch processing job output by the analysis module, and the file information related to the target batch processing job.
For example, the checking module checks whether the file is processed correctly according to the file offset information and the upstream file information in the job configuration.
In an alternative embodiment, referring to fig. 1, the checking the operation result of the target batch processing job according to the at least one type of information by the checking module includes:
and checking the file clearing and downloading condition corresponding to the target batch processing job through a checking module based on the job result data of the target batch processing job and the file information related to the target batch processing job.
The file downloading refers to writing information in the database into a file, and the file cleaning refers to deleting corresponding information in the database.
For example, the checking module compares the database data information and the file information before and after the target batch processing job runs, and judges whether the file corresponding to the lower file is correct or not and whether the data corresponding to the upper file is cleared or not.
Optionally, before the target batch processing job runs, the database and the NAS are accessed through the import tool to obtain database data information and file information before and after the target batch processing job runs, and further after the target batch processing job runs, whether a file corresponding to the file downloading is correct or not, whether data corresponding to the file cleaning is removed or not, and the like can be judged.
In an alternative embodiment, referring to fig. 1, the checking the operation result of the target batch processing job according to the at least one type of information by the checking module includes:
and checking the file transmission condition corresponding to the target batch processing job based on the job result data of the target batch processing job and the file information related to the target batch processing job through a checking module.
For example, the checking module checks whether the file transmission is correct according to the file transmission configuration information, the control file and the return value of the operation step.
In an alternative embodiment, referring to fig. 1, further comprising: and checking the copy synchronization condition corresponding to the target batch processing operation through a checking module.
For example, the checking module checks whether the database operation is correct after the copy synchronization according to the upstream copy information and the database data information before the target batch processing operation is run.
The above embodiments are merely exemplary illustrations of functions of the analysis module and the check module, and are not specific limitations of the analysis module and the check module. In addition to the above embodiments, the analysis module and the check module may also support other data analysis and check on the batch job operation result, which is not listed in this embodiment.
Example four
Fig. 3 is a schematic structural diagram of a batch job checking apparatus according to a fourth embodiment of the present invention, which is applicable to data collection, analysis and checking of a batch job operation result, and the apparatus may be implemented in a software and/or hardware manner, and may be generally integrated in a computer device.
As shown in fig. 3, the checking apparatus for batch processing job specifically includes: an import module 310, an analysis module 320, and a review module 330. Wherein,
an import module 310 to collect at least one type of information related to a target batch job run;
an analysis module 320, configured to perform data analysis on the operation condition of the target batch job according to the at least one piece of information;
and the checking module 330 is configured to check the operation result of the target batch processing job according to the at least one piece of information.
According to the technical scheme provided by the embodiment of the invention, at least one type of information related to the running of the target batch processing operation is collected through the import module, the running condition of the target batch processing operation is subjected to data analysis through the analysis module according to the at least one type of information, and the running result of the target batch processing operation is checked through the check module according to the at least one type of information, so that the automation of checking the batch processing operation is realized, and the check efficiency is improved.
Optionally, the import module 310 is specifically configured to collect at least scheduling information of the target batch job, job result data of the target batch job, file information related to the target batch job, and related information in a server that executes the target batch job.
Optionally, the analysis module 320 is specifically configured to analyze the job basic information of the target batch job based on relevant information in a server that executes the target batch job.
Optionally, the analysis module 320 is specifically configured to analyze job step running information of the target batch job based on relevant information in a server that executes the target batch job.
Optionally, the analysis module 320 is specifically configured to analyze a critical path of the target batch job operation based on the scheduling information of the target batch job; wherein the critical path affects a total length of time that the target batch job execution sequence runs.
Optionally, the analysis module 320 is specifically configured to analyze the operation efficiency of the target batch processing job based on the scheduling information of the target batch processing job and the file information related to the target batch processing job.
Optionally, the analysis module 320 is specifically configured to analyze the processing condition of the target batch job based on relevant information in a server executing the target batch job.
Optionally, the analysis module 320 is specifically configured to analyze the task concurrency and load balancing condition corresponding to the target batch processing job based on the relevant information in the server that executes the target batch processing job.
Optionally, the checking module 330 is specifically configured to check a file processing condition corresponding to the target batch processing job based on relevant information in a server that executes the target batch processing job and file information related to the target batch processing job.
Optionally, the checking module 330 is specifically configured to check a document cleaning and downloading condition corresponding to the target batch processing job based on the job result data of the target batch processing job and the file information related to the target batch processing job.
Optionally, the checking module 330 is specifically configured to check a file transmission condition corresponding to the target batch processing job based on the job result data of the target batch processing job and the file information related to the target batch processing job.
Optionally, the checking module 330 is further specifically configured to check a copy synchronization condition corresponding to the target batch processing job.
The batch job checking device can execute the batch job checking method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the batch job checking method.
Since the above-described checking apparatus for a batch job is an apparatus capable of executing the checking method for a batch job in the embodiment of the present invention, based on the checking method for a batch job described in the embodiment of the present invention, a person skilled in the art can understand the specific implementation and various modifications of the checking apparatus for a batch job in the embodiment of the present invention, and therefore, how to implement the checking method for a batch job in the embodiment of the present invention by the checking apparatus for a batch job is not described in detail herein. The scope of the present application is intended to cover any apparatus for implementing the method for checking a batch job in the embodiments of the present invention.
EXAMPLE five
Fig. 4 is a schematic diagram of a hardware structure of a computer device according to a fifth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 4 is only one example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 4, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with the other modules of computer device 12 via bus 18. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a method for checking a batch job provided by an embodiment of the present invention.
That is, the processing unit implements, when executing the program: collecting, by an import module, at least one type of information related to a target batch job run; performing data analysis on the running condition of the target batch processing operation according to the at least one type of information through an analysis module; and checking the running result of the target batch processing operation according to the at least one type of information through a checking module.
EXAMPLE six
A sixth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a method for checking a batch job, according to all embodiments of the present invention: that is, the program when executed by the processor implements: collecting, by an import module, at least one type of information related to a target batch job run; performing data analysis on the running condition of the target batch processing operation according to the at least one type of information through an analysis module; and checking the running result of the target batch processing operation according to the at least one type of information through a checking module.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (15)

1. A method for checking a batch job, comprising:
collecting, by an import module, at least one type of information related to a target batch job run;
performing data analysis on the running condition of the target batch processing operation according to the at least one type of information through an analysis module;
and checking the running result of the target batch processing operation according to the at least one type of information through a checking module.
2. The method of claim 1, wherein collecting, by the import module, at least one type of information related to the target batch job run comprises:
and at least collecting scheduling information of the target batch processing job, job result data of the target batch processing job, file information related to the target batch processing job and related information in a server for executing the target batch processing job through the import module.
3. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
analyzing, by the analysis module, job essential information of the target batch job based on relevant information in a server executing the target batch job.
4. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
analyzing, by the analysis module, job step execution information of the target batch job based on relevant information in a server executing the target batch job.
5. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
analyzing, by the analysis module, a critical path of the target batch job run based on the scheduling information of the target batch job; wherein the critical path affects a total length of time that the target batch job execution sequence runs.
6. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
analyzing, by the analysis module, the operating efficiency of the target batch job based on the scheduling information of the target batch job and the file information related to the target batch job.
7. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
analyzing, by the analysis module, a processing condition of the target batch job based on relevant information in a server executing the target batch job.
8. The method of claim 2, wherein performing, by an analysis module, data analysis on the operation of the target batch job based on the at least one type of information comprises:
and analyzing the task concurrency and load balancing condition corresponding to the target batch processing job based on the relevant information in the server executing the target batch processing job through the analysis module.
9. The method of claim 2, wherein checking the run result of the target batch job according to the at least one piece of information by a checking module comprises:
and checking, by the checking module, a file processing condition corresponding to the target batch processing job based on the related information in the server executing the target batch processing job and the file information related to the target batch processing job.
10. The method of claim 2, wherein checking the run result of the target batch job according to the at least one piece of information by a checking module comprises:
and checking, by the checking module, a file clearing and downloading condition corresponding to the target batch processing job based on the job result data of the target batch processing job and the file information related to the target batch processing job.
11. The method of claim 2, wherein checking the run result of the target batch job according to the at least one piece of information by a checking module comprises:
and checking, by the checking module, a file transmission condition corresponding to the target batch processing job based on job result data of the target batch processing job and file information related to the target batch processing job.
12. The method of claim 1, further comprising:
and checking the copy synchronization condition corresponding to the target batch processing operation through the checking module.
13. An apparatus for checking a batch job, comprising:
an import module to collect at least one type of information related to a target batch job run;
the analysis module is used for carrying out data analysis on the running condition of the target batch processing operation according to the at least one type of information;
and the checking module is used for checking the running result of the target batch processing operation according to the at least one type of information.
14. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-12 when executing the program.
15. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-12.
CN202011322642.5A 2020-11-23 2020-11-23 Batch processing operation checking method, device, equipment and medium Pending CN112416727A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011322642.5A CN112416727A (en) 2020-11-23 2020-11-23 Batch processing operation checking method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011322642.5A CN112416727A (en) 2020-11-23 2020-11-23 Batch processing operation checking method, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN112416727A true CN112416727A (en) 2021-02-26

Family

ID=74777901

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011322642.5A Pending CN112416727A (en) 2020-11-23 2020-11-23 Batch processing operation checking method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN112416727A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113010566A (en) * 2021-03-31 2021-06-22 建信金融科技有限责任公司 Batch processing result checking method and device

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101017546A (en) * 2007-03-12 2007-08-15 中国建设银行股份有限公司 Method and device for categorical data batch processing
US20120005542A1 (en) * 2010-07-01 2012-01-05 LogRhythm Inc. Log collection, structuring and processing
US20140059561A1 (en) * 2012-08-21 2014-02-27 International Business Machines Corporation Reallocating jobs for checking data quality
US20150120581A1 (en) * 2013-10-25 2015-04-30 Housl Pty Ltd Computer implemented frameworks and methodologies configured to enable processing and management of data relating to lease applications
CN109254835A (en) * 2018-06-27 2019-01-22 阿里巴巴集团控股有限公司 Processing method, device, server and the readable storage medium storing program for executing of batch processing task
CN110413483A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Monitoring method, device, electronic equipment and the storage medium of batch jobs data
CN110806933A (en) * 2019-11-05 2020-02-18 中国建设银行股份有限公司 Batch task processing method, device, equipment and storage medium
CN111241180A (en) * 2020-01-13 2020-06-05 中国建设银行股份有限公司 Information processing method, device, equipment and storage medium
CN111723086A (en) * 2020-07-20 2020-09-29 江苏苏宁银行股份有限公司 Data quality checking method, device and equipment and readable storage medium
CN111767199A (en) * 2020-06-24 2020-10-13 中国工商银行股份有限公司 Resource management method, device, equipment and system based on batch processing operation

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101017546A (en) * 2007-03-12 2007-08-15 中国建设银行股份有限公司 Method and device for categorical data batch processing
US20120005542A1 (en) * 2010-07-01 2012-01-05 LogRhythm Inc. Log collection, structuring and processing
US20140059561A1 (en) * 2012-08-21 2014-02-27 International Business Machines Corporation Reallocating jobs for checking data quality
US20150120581A1 (en) * 2013-10-25 2015-04-30 Housl Pty Ltd Computer implemented frameworks and methodologies configured to enable processing and management of data relating to lease applications
CN109254835A (en) * 2018-06-27 2019-01-22 阿里巴巴集团控股有限公司 Processing method, device, server and the readable storage medium storing program for executing of batch processing task
CN110413483A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Monitoring method, device, electronic equipment and the storage medium of batch jobs data
CN110806933A (en) * 2019-11-05 2020-02-18 中国建设银行股份有限公司 Batch task processing method, device, equipment and storage medium
CN111241180A (en) * 2020-01-13 2020-06-05 中国建设银行股份有限公司 Information processing method, device, equipment and storage medium
CN111767199A (en) * 2020-06-24 2020-10-13 中国工商银行股份有限公司 Resource management method, device, equipment and system based on batch processing operation
CN111723086A (en) * 2020-07-20 2020-09-29 江苏苏宁银行股份有限公司 Data quality checking method, device and equipment and readable storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113010566A (en) * 2021-03-31 2021-06-22 建信金融科技有限责任公司 Batch processing result checking method and device

Similar Documents

Publication Publication Date Title
CN108959564B (en) Data warehouse metadata management method, readable storage medium and computer device
US7792950B2 (en) Coverage analysis of program code that accesses a database
US20110276949A1 (en) Memory leak detection
CN112765017B (en) Data query performance testing method and device based on MySQL database
US20120173498A1 (en) Verifying Correctness of a Database System
US20150081875A1 (en) Method and system for operating system (os) verification
US9213543B2 (en) Software internationalization estimation model
CN111666201A (en) Regression testing method, device, medium and electronic equipment
Zaccarelli et al. Stream2segment: An open‐source tool for downloading, processing, and visualizing massive event‐based seismic waveform datasets
CN111966587A (en) Data acquisition method, device and equipment
CN110597704A (en) Application program pressure testing method, device, server and medium
CN117421328B (en) Data management method, system, equipment and readable medium based on artificial intelligence
CN111274130A (en) Automatic testing method, device, equipment and storage medium
CN112416727A (en) Batch processing operation checking method, device, equipment and medium
CN113656391A (en) Data detection method and device, storage medium and electronic equipment
CN111367813B (en) Automatic testing method and device for decision engine, server and storage medium
US7974973B1 (en) Application object tuning
CN110716859A (en) Method for automatically pushing test cases for modified codes and related device
CN114238048B (en) Automatic testing method and system for Web front-end performance
CN111367791B (en) Method, device, medium and electronic equipment for generating test case
CN114661571A (en) Model evaluation method, model evaluation device, electronic equipment and storage medium
US20070214166A1 (en) Program analysis method and apparatus
CN113360397A (en) Regression testing method, device, equipment and storage medium of system function
CN112597041A (en) Cross-branch merging method, system, equipment and storage medium for code coverage rate
CN109697141B (en) Method and device for visual testing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination