CN116501468A - Batch job processing method and device and electronic equipment - Google Patents
Batch job processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN116501468A CN116501468A CN202310437884.6A CN202310437884A CN116501468A CN 116501468 A CN116501468 A CN 116501468A CN 202310437884 A CN202310437884 A CN 202310437884A CN 116501468 A CN116501468 A CN 116501468A
- Authority
- CN
- China
- Prior art keywords
- time consumption
- group
- jobs
- concurrency
- job
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 55
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000013507 mapping Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 11
- 238000009825 accumulation Methods 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 abstract 1
- 238000004891 communication Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010276 construction Methods 0.000 description 3
- 238000010923 batch production Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002035 prolonged effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000005059 dormancy Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006266 hibernation Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007958 sleep Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Feedback Control In General (AREA)
Abstract
The invention provides a batch job processing method, a device and electronic equipment, and relates to the technical field of data processing, wherein the method comprises the following steps: collecting a current running log of a target batch job, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log; determining reference time consumption data corresponding to each group of operations according to the target operation type to which each group of operations belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type; according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of jobs is adjusted; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs; and processing each group of operations according to the adjusted concurrency number control value. The invention realizes the adaptive adjustment of the concurrency control value of each group of jobs, effectively reduces the resource conflict caused by batch job processing and improves the resource utilization rate.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing batch jobs, and an electronic device.
Background
In the process of different batch flows of different products, because the related batch operations and the number are different, the resource requirement of each batch operation is also different, so how to accurately set the maximum concurrency number of the batch operation to efficiently process the batch operation is an important topic to be solved in the industry at present.
In the prior art, the concurrent execution number of batch jobs is generally set according to manual experience in batch job processing. The number of concurrent execution is greatly influenced by manual experience, and in batch operation processing, if the number of concurrent execution is excessively large, insufficient computing resources, such as insufficient CPU (Central Processing Unit ) resources or insufficient memory resources, are caused, so that the resource conflict is serious, and the operation is blocked; if the number of concurrent execution is too small, the computing resources cannot be fully used, and the time consumption of the batch job processing flow is prolonged.
Therefore, there is a need for a batch job processing method and apparatus to solve the above problems.
Disclosure of Invention
The invention provides a batch job processing method, a batch job processing device and electronic equipment, which are used for solving the defects that the resource conflict of batch job processing is serious or the resource utilization rate is low due to the fact that the concurrent number of batch jobs is set by manual experience in the prior art, and improving the resource utilization rate while reducing the resource conflict caused by batch job processing.
The invention provides a batch job processing method, which comprises the following steps:
collecting a current running log of a target batch job, grouping the target batch job according to a job type, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log;
determining reference time consumption data corresponding to each group of operations according to the target operation type of each group of operations; the reference time consumption data is obtained according to a historical operation log of the sample batch operation under the target operation type;
according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of operations is adjusted; the concurrency number control value is the maximum value for controlling the quantity of the concurrent execution jobs of each group of jobs;
and processing each group of jobs according to the adjusted concurrency number control value.
According to the batch job processing method provided by the invention, the method further comprises the following steps:
acquiring a plurality of historical time consumption data of each sample job in the sample batch job according to the historical operation log of the sample batch job under the target job type;
performing quantile processing on the plurality of historical time consumption data to obtain an upper time consumption quantile value and a lower time consumption quantile value corresponding to the sample batch operation;
And constructing the reference time consumption data according to the time consumption upper quantile value and the time consumption lower quantile value.
According to the batch job processing method provided by the invention, the concurrency control value of each group of jobs is adjusted according to the execution time consumption and the reference time consumption data, and the method comprises the following steps:
comparing the execution time consumption with the reference time consumption data, and determining the time consumption comparison value corresponding to each group of operation;
determining the time consumption fluctuation rate corresponding to each group of operations according to the time consumption comparison value and the operation quantity of each group of operations;
determining the concurrent execution quantity adjustment degree of each group of operations according to the time consumption fluctuation rate;
and adjusting the concurrency number control value of each group of jobs according to the concurrency execution number adjustment degree, the maximum concurrency execution number corresponding to the target job type and the minimum concurrency execution number corresponding to the target job type.
According to the batch job processing method provided by the invention, the comparison of the execution time consumption and the reference time consumption data is carried out, and the time consumption comparison value corresponding to each group of jobs is determined, and the batch job processing method comprises the following steps:
for each target job in the set of jobs, performing the steps of:
If the execution time consumption of the current target job is smaller than or equal to the time consumption lower dividing value, determining that the time consumption comparison value of the current target job is a first preset value;
if the execution time consumption of the current target job is greater than or equal to the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a second preset value;
if the execution time consumption of the current target job is greater than the time consumption lower dividing value and less than the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a third preset value;
the first preset value is smaller than the third preset value, and the third preset value is smaller than the second preset value.
According to the batch job processing method provided by the invention, the time consumption fluctuation rate corresponding to each group of jobs is determined according to the time consumption comparison value and the job number of each group of jobs, and the batch job processing method comprises the following steps:
dividing the sum of the time consumption contrast values of all target operations in each group of operations by the number of operations in each group of operations;
and determining the time consumption fluctuation rate corresponding to each group of operation according to the division result.
According to the batch job processing method provided by the invention, the concurrent execution quantity adjustment degree of each group of jobs is determined according to the time consumption fluctuation rate, and the batch job processing method comprises the following steps:
According to the time consumption fluctuation rate, acquiring a preset fluctuation rate matched with the time consumption fluctuation rate from a target mapping table corresponding to the target operation type;
acquiring the concurrent execution quantity adjustment degree of each group of operations in a target mapping table according to the preset fluctuation rate matched with the time consumption fluctuation rate;
the target mapping table stores a mapping relation between the preset fluctuation rate and the concurrent execution quantity adjustment degree in advance.
According to the batch job processing method provided by the invention, the control value of the concurrency number of each group of jobs is adjusted according to the concurrency execution number adjustment degree, the maximum concurrency execution number corresponding to the target job type and the minimum concurrency execution number corresponding to the target job type, and the method comprises the following steps:
according to the concurrency execution quantity adjustment degree, preliminarily adjusting the concurrency quantity control value of each group of operations to obtain corresponding candidate concurrency quantity control values of each group of operations;
if the candidate concurrency number control value corresponding to any group of operations is larger than the maximum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the maximum concurrency execution number;
If the candidate concurrency number control value corresponding to any group of operations is smaller than the minimum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the minimum concurrency execution number;
and if the candidate concurrency number control value corresponding to any group of jobs is greater than or equal to the minimum concurrency execution number and is less than or equal to the maximum concurrency execution number, keeping the concurrency number control value of any group of jobs to be adjusted to be the same as the candidate concurrency number control value corresponding to any group of jobs.
According to the batch job processing method provided by the invention, each group of jobs is processed according to the adjusted concurrency number control value, and the batch job processing method comprises the following steps:
for each target job of the set of jobs, performing the steps of:
acquiring the current concurrent execution quantity of each group of jobs; the current concurrent execution quantity is the quantity of target jobs in an execution state in each set of jobs before the current target jobs are executed; the current target job is a target job submitted by a job scheduling tool in a current period;
if the current concurrent execution number is smaller than the adjusted concurrent number control value, adding 1 to the current concurrent execution number accumulation, configuring the current target job into an execution state to execute the current target job, and subtracting 1 from the current concurrent execution number accumulation under the condition that the current target job is executed;
And if the current concurrent execution number is greater than or equal to the adjusted concurrent number control value, configuring the current target job into a dormant state until the current concurrent execution number is less than or equal to the adjusted concurrent number control value, and configuring the current target job into an execution state to execute the current target job.
The invention also provides a batch job processing device, which comprises:
the system comprises an acquisition module, a storage module and a control module, wherein the acquisition module is used for acquiring a current running log of a target batch job, grouping the target batch job according to a job type, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log;
the reference data generation module is used for determining reference time consumption data corresponding to each group of jobs according to the target job type to which each group of jobs belongs; the reference time consumption data is obtained according to a historical operation log of the sample batch operation under the target operation type;
the dynamic control module is used for adjusting the concurrency number control value of each group of operations according to the execution time consumption and the reference time consumption data; the concurrency number control value is the maximum value for controlling the quantity of the concurrent execution jobs of each group of jobs;
And the processing module is used for processing each group of jobs according to the adjusted concurrency number control value.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the batch job processing method as described above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a batch job processing method as described in any one of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements a batch job processing method as described in any one of the above.
According to the batch job processing method, the batch job processing device and the electronic equipment, the concurrent number control value of each group of jobs is dynamically adjusted based on the execution time consumption of each group of jobs in the target batch job and the reference time consumption data determined according to the target job type of each group of jobs, so that the setting of artificial experience is avoided, the concurrent number control value of each group of jobs is dynamically adapted to the current resources of the CPU, and therefore resource conflict caused by batch job processing is effectively reduced, and meanwhile, the resource utilization rate is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow diagram of a prior art batch job processing method;
FIG. 2 is a schematic flow chart of a batch job processing method according to the present invention;
FIG. 3 is a schematic diagram of a CPU resource distribution provided by the present invention;
FIG. 4 is a schematic diagram of the structure of a DAG provided by the present invention;
FIG. 5 is a second flow chart of the batch job processing method according to the present invention;
FIG. 6 is a flow chart of the reference time consumption data construction step provided by the invention;
FIG. 7 is a flowchart of a concurrency control value adjustment step according to the present invention;
FIG. 8 is a second flowchart of the concurrency control value adjustment step provided by the present invention;
FIG. 9 is a flow chart of the batch job processing steps provided by the present invention;
FIG. 10 is a third flow chart of the batch job processing method according to the present invention;
FIG. 11 is a schematic diagram of a batch job processing apparatus provided by the present invention;
fig. 12 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, in the prior art, the maximum concurrency of batch jobs is usually set through manual experience, that is, a DAG (Directed Acyclic Graph ) is generated by directly combining a plurality of batch jobs according to an execution sequence through a job scheduling tool, and then a preset concurrency number of batch jobs are submitted according to the DAG sequence, so as to complete a batch flow. In this way, batch jobs are executed according to a fixed sequence, and cannot be dynamically adjusted according to actual conditions of resources, if the concurrent execution quantity is set to be too large, CPU resources are insufficient, so that resource conflict is serious, and the jobs are blocked; if the number of concurrent execution is too small, CPU resources cannot be fully used, and the time consumption of a batch job processing flow is prolonged.
In order to solve the above problems, the batch job processing method provided in this embodiment monitors the time consumption change condition of each group of jobs in the batch job in the batch process, so as to dynamically adjust the concurrency number control value (i.e., the maximum concurrency number) of each group of jobs in the batch job according to the time consumption change condition, reduce the concurrency number control value when resources are tense, and increase the concurrency number control value when resources are abundant, so as to achieve the effects of reducing the time consumption of the batch process, reducing batch faults, and improving the resource utilization rate of the CPU.
It should be noted that, batch faults herein refer to faults caused by resource timeout due to job waiting caused by insufficient resources, and include faults caused by database access timeout, cd or ftp communication link timeout, and the like.
The batch job processing method of the present invention is described below with reference to fig. 2 to 10, and may be implemented on a ZOS platform or a LINUX platform, which is not specifically limited in this embodiment.
As shown in fig. 2, one of the flow charts of the batch job processing method provided in this embodiment includes the following steps:
step 201, collecting a current running log of a target batch job, grouping the target batch job according to the job type, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log;
The target batch job can be applied to products which can realize different functions in different fields such as banking, non-banking financial fields and the like in a batch mode, the jobs which can realize different functions are combined through a job scheduling tool, and the jobs are submitted in a batch mode to realize part of the functions, and the jobs which can realize different functions are collectively called batch jobs. Illustratively, in a banking system, a plurality of functions such as batch account billing, customer fee deduction, customer billing, data collection and the like are realized in a batch manner. The number of jobs per batch job, and the time consumption can be set according to the actual business, e.g., about 8 tens of thousands of jobs need to be run in a test environment for more than 10 hours. The job scheduling tool may be a CONTAB command on the LINUX platform, a TWS tool on the ZOS platform, etc.
TABLE 1 operation log of jobs
Optionally, executing the target batch operation in the current monitoring period, and collecting the current running log of the target batch operation in the current monitoring period; the monitoring duration of the current monitoring period can be set according to actual requirements, such as 5 minutes.
As shown in table 1, the current running log includes at least a job list of the target batch job, and a start time, an end time, and an execution time consumption of each job.
After the current running log of the target batch job is collected, the jobs in the target batch job can be divided into a plurality of groups, and the execution time consumption of each grouped job is analyzed and obtained from the current running log.
The division may be performed according to the job type of each job in the target batch job, for example, a job for transmitting data to other products is set as type 1, a job for accessing the database is set as type 2, and a job for directly performing data processing locally without accessing the database is set as type 3.
Step 202, determining reference time consumption data corresponding to each group of jobs according to the target job type to which each group of jobs belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type;
optionally, prior to performing step 202, a historical running log of sample batch jobs for each job type is pre-built with benchmark run time data, such as by determining the benchmark data from the quartile values; and pre-storing each job type and corresponding reference time consumption data in a mapping relation.
In an actual application scene, the reference time consumption data corresponding to each group of operations can be conveniently searched according to the target operation type and the mapping relation of each group of operations.
After the target batch job is executed, each group of jobs can be used as a sample batch job under the corresponding job type, so that the sample batch job can be expanded, and the reference time consumption data can be updated in real time according to the operation log of each group of jobs, so that the accuracy of batch job processing is improved.
Step 203, according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of jobs is adjusted; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs;
in the execution of different batch flows of different products, the related batch jobs and the number are different, the resource requirement of each batch job is also different, the number of jobs executed in parallel at the same time is also dynamically changed, and therefore, the requirement on resources such as a CPU and the like in the execution of the batch flows is dynamically changed, and the problem of resource requirement conflict easily occurs when the batch is executed. As shown in fig. 3, for the resource usage of the CPU when a product executes a batch job, it can be seen that the CPU resource is dynamically changed. Therefore, the concurrency control value of each group of jobs needs to be dynamically adjusted according to the CPU resource condition, so that the concurrency control value of each group of jobs is matched with the current CPU resource, thereby realizing automatic operation and maintenance, reducing the time consumption of batch flow, reducing batch faults, improving the utilization rate of resources such as the CPU and the like.
Optionally, after the execution time consumption and the reference time consumption data are obtained, the execution time consumption of each group of jobs and the corresponding reference time consumption data can be compared to determine the current resource condition of the CPU according to the comparison result, and then the concurrency number control value of each group of jobs is adjusted, so that the concurrency number control value of each group of jobs is adapted to the current resource of the CPU, that is, the concurrency number control value (i.e., the maximum concurrency number) of batch jobs is reduced when the resources of the CPU are tense, so as to avoid job congestion, and the concurrency number control value of the jobs is increased when the resources of the CPU are sufficient, so that the CPU resources are fully used.
The method of adjusting the concurrency control value of each group of jobs may be to directly adjust the concurrency control value of each group of jobs according to the comparison result of the execution time consumption of each group of jobs and the corresponding reference time consumption data.
The adjustment modes here include: under the condition that the comparison result corresponding to a certain group of jobs is that the execution time consumption is larger than the corresponding reference time consumption data, judging that the current resources are tense, and reducing the concurrency number control value of the group of jobs to a first preset value; under the condition that the comparison result is that the execution time consumption is smaller than the corresponding reference time consumption data, judging that the current resources are sufficient, and increasing the concurrency number control value of the group of jobs to a second preset value; or taking the comparison result of the execution time consumption of each group of jobs and the corresponding reference time consumption data as an intermediate variable, calculating to obtain a concurrency number adjustment value through the intermediate variable, and adjusting the concurrency number control value of each group of jobs according to the concurrency number adjustment value, for example, calculating the time consumption fluctuation rate of each group of jobs according to the comparison result, calculating to obtain a concurrency number adjustment value according to the time consumption fluctuation rate, and adjusting the concurrency number control value of each group of jobs according to the concurrency number adjustment value, which is not particularly limited in this embodiment.
And 204, processing each group of jobs according to the adjusted concurrency number control value.
Optionally, after the adjusted concurrency number control value is obtained, the control value may be compared with the current concurrency execution number of each group of jobs, so as to determine, according to the comparison result and the execution sequence of each group of jobs, the execution state of each job at the current monitoring time, such as execution or dormancy, and further process each job according to the execution state of each job at the current monitoring time.
Wherein the execution order of the groups of jobs may be determined from the DAG of the target batch job.
As shown in FIG. 4, the DAG may describe the order of execution among jobs when executing a batch job, e.g., job 1 is a successor of job 2, job 3, representing a serial relationship; a parallel relationship is formed between the jobs 2 and 3, the job 4 is the successor of the jobs 2 and 3, and the job 5 is the successor of the job 4.
FIG. 5 is a second flow chart of the batch job processing method according to the present embodiment; as can be seen from the figure, compared with the batch job processing method shown in fig. 1, the present embodiment can utilize the job scheduling tool to compare the execution time consumption and the reference time consumption data of each group of jobs in the target batch job, so as to determine the current resource condition of the CPU according to the comparison result, and further adjust the concurrency number control value of each group of jobs, so that the concurrency number control value of each group of jobs is adapted to the current resource of the CPU, that is, the concurrency number control value of the batch job is reduced when the resource of the CPU is tense, so as to avoid job congestion, and the concurrency number control value of the job is increased when the resource of the CPU is tense, so that the CPU resource is fully used, on one hand, the maximum concurrency execution job number of the batch job is automatically adjusted, no manual intervention is required, and the degree of automation is effectively improved; on the other hand, when resources such as a CPU (Central processing Unit) are tense, the maximum concurrent execution job number of batch jobs is reduced, so that job overtime faults caused by resource contention can be effectively reduced, batch interruption is reduced, batch flow is optimized, and batch faults are reduced; and by dynamically adjusting the maximum concurrent execution job number of batch jobs, resources can be fully utilized, the time consumption of batch flow is effectively shortened, and the resource utilization rate is improved.
According to the batch job processing method, the concurrent number control value of each group of jobs is dynamically adjusted based on the execution time consumption of each group of jobs in the target batch job and the reference time consumption data determined according to the target job type to which each group of jobs belong, so that manual experience setting is avoided, the concurrent number control value of each group of jobs is dynamically adapted to the current resources of the CPU, resource conflict caused by batch job processing is effectively reduced, and the resource utilization rate is improved.
In some embodiments, the method further comprises:
acquiring a plurality of historical time consumption data of each sample job in the sample batch job according to a historical operation log of the sample batch job under the target job type;
performing quantile processing on a plurality of historical time consumption data to obtain time consumption upper quantile values and time consumption lower quantile values corresponding to sample batch operation;
and constructing reference time consumption data according to the time consumption upper and lower quantile values.
As shown in fig. 6, a flow chart of a reference time consumption data construction step provided in this embodiment is shown, where the reference time consumption data construction step specifically includes:
firstly, periodically collecting historical operation logs of a plurality of batch flows of sample batch operations of specified products in a certain historical time period; the historical time period can be set according to actual requirements, such as one month.
Table 2 reference time consumption data table
Meanwhile, grouping the historical operation logs according to the job types, and acquiring a plurality of historical time consumption data (namely, execution ending time-execution starting time) of each sample job in the sample batch job under each job type from the historical operation logs of each job type; and sorting according to the time consumption, so as to perform quantile processing, such as quartile processing, according to the sorting result, to respectively obtain a historical time consumption lower quartile value (such as 10 seconds) corresponding to the sample batch operation under each operation type, and also instantly consume the lower quartile value, a median value (such as 20 seconds) and an upper quartile value (such as 30 seconds), and instantly consume the lower quartile value, so as to obtain reference time consumption data of each operation type, and write the reference time consumption data of each operation type into a reference time consumption data table, as shown in table 2.
And then, establishing a mapping relation between each job type and corresponding reference time consumption data so as to conveniently find the reference time consumption data corresponding to each group of jobs according to the job type and the mapping relation in the subsequent actual use.
The reference time consumption data table may be updated in real time according to a new historical operation log periodically.
As shown in fig. 7, in some embodiments, the step of adjusting the concurrency control value of each group of jobs according to the execution time consumption and the reference time consumption data in step 203 further includes:
step 2031, comparing the execution time consumption with the reference time consumption data, and determining a time consumption comparison value corresponding to each group of operations;
optionally, the execution time consumption of different grouping jobs is compared with corresponding reference time consumption data one by one, so as to obtain time consumption comparison values corresponding to different grouping jobs.
Preferably, comparing the execution time consumption with the reference time consumption data, determining a time consumption comparison value corresponding to each group of jobs, including:
the following steps are performed for each target job in the respective sets of jobs:
if the execution time consumption of the current target job is smaller than or equal to the time consumption lower dividing value, determining that the time consumption comparison value of the current target job is a first preset value;
if the execution time consumption of the current target job is greater than or equal to the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a second preset value;
if the execution time consumption of the current target job is greater than the time consumption lower dividing value and less than the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a third preset value;
The first preset value is smaller than the third preset value, and the third preset value is smaller than the second preset value.
The first preset value, the second preset value and the third preset value can be set according to actual requirements, for example, the first preset value is-1, the second preset value is 1 and the third preset value is 0.
The step of acquiring the time consumption contrast value for each target job in each set of jobs is described below with specific examples.
Assuming that the current execution time consumption of the target job is A, the time consumption upper dividing value in the reference time consumption data corresponding to the group to which the target job belongs is C, and the time consumption lower dividing value is B; if the comparison is known that A < =B, the time consumption comparison value of the current target operation is-1; if the comparison is known to be A > =C, determining that the time consumption contrast value of the current target job is 1, and if the comparison is known to be B < A < C, determining that the time consumption contrast value of the current target job is 0; and the time consumption comparison values of the various job types are written into a time consumption comparison information table for subsequent use, as shown in table 3.
Table 3 time consumption comparison information table
According to the method provided by the embodiment, the current resource condition of the CPU can be dynamically represented according to different comparison results, and then the corresponding time consumption comparison value is obtained, so that the concurrent number control value of each group of jobs is dynamically adjusted, the setting of artificial experience is avoided, and the resource utilization rate is improved while the resource conflict caused by batch job processing is effectively reduced.
Step 2032, determining a time consumption fluctuation rate corresponding to each group of operations according to the time consumption comparison value and the operation number of each group of operations;
optionally, the job number of each group of jobs is obtained, so as to calculate and obtain the time consumption fluctuation rate corresponding to each group of jobs according to the time consumption comparison value and the job number of each group of jobs.
Preferably, determining the time consumption fluctuation rate corresponding to each group of jobs according to the time consumption comparison value and the job number of each group of jobs comprises:
dividing the sum of the time consumption contrast values of all target operations in each group of operations by the number of operations in each group of operations;
and determining the time consumption fluctuation rate corresponding to each group of operation according to the division result.
Optionally, the calculation formula of the time consumption fluctuation rate P corresponding to each group of operations is as follows:
wherein d i The time consumption contrast value of the ith operation in each group of operations is obtained; con is the number of jobs for each set of jobs.
The method provided by the embodiment can dynamically acquire the time consumption fluctuation rate of each group of jobs so as to dynamically adjust the concurrency number control value of each group of jobs, and avoid artificial experience setting, thereby effectively reducing resource conflict caused by batch job processing and improving the resource utilization rate.
Step 2033, determining the concurrent execution quantity adjustment degree of each group of jobs according to the time consumption fluctuation rate;
Optionally, after the time consumption fluctuation rate is obtained, searching and obtaining the concurrency execution quantity adjustment degree of each group of jobs according to the mapping relation between the time consumption fluctuation rate and the concurrency execution quantity adjustment degree which are stored in advance in a time consumption concurrency quantity adjustment table (i.e. a target mapping table) under each job type; the time consumption fluctuation rate may be input to a model constructed in advance, and the degree of adjustment of the number of concurrent executions of each group of operations may be calculated, which is not particularly limited in this embodiment.
Preferably, determining the concurrent execution quantity adjustment degree of each group of jobs according to the time consumption fluctuation rate includes:
according to the time consumption fluctuation rate, acquiring a preset fluctuation rate matched with the time consumption fluctuation rate from a target mapping table corresponding to the target operation type;
acquiring the concurrent execution quantity adjustment degree (hereinafter also referred to as concurrent quantity adjustment degree) of each group of operations in a target mapping table according to a preset fluctuation rate matched with the time consumption fluctuation rate;
the target mapping table is pre-stored with a mapping relation between a preset fluctuation rate and a concurrent execution quantity adjustment degree.
It should be noted that, the mapping relationship between the preset fluctuation rate and the concurrent execution number adjustment degree in the target mapping table may be configured and updated according to the actual requirement. As shown in table 4, an exemplary time consumption concurrency count adjustment table, i.e., a target mapping table, for a certain job type.
Alternatively, after the time consumption fluctuation rate of each set of jobs is acquired, a preset fluctuation rate matching the time consumption fluctuation rate may be determined in the target mapping table.
Here, the mode of determining the preset volatility matching with the time consumption volatility may be to perform a downward rounding operation on the time consumption volatility according to the configuration of each preset volatility in the target mapping table, if the time consumption volatility corresponding to a certain grouping operation is-45%, then the corresponding preset volatility is-60% according to the mapping table; or, according to the difference between the time consumption fluctuation rate and each preset fluctuation rate, the corresponding preset fluctuation rate is determined, which is not specifically limited in this embodiment.
TABLE 4 time consumption concurrency number adjustment table
And then, searching and acquiring corresponding concurrency number adjustment degrees in a target mapping table according to the preset fluctuation rate matched with the time consumption fluctuation rate, so as to serve as concurrency execution number adjustment degrees of each group of operations. Illustratively, when a preset volatility matching the time consumption volatility is determined to be-60%, the corresponding concurrent execution number adjustment is 60%.
It should be noted that, the time consumption concurrency number adjustment table corresponding to each job type may be constructed by performing multiple adjustment on the historical jobs to obtain the jobs of different job types, and performing the best adjustment when the fluctuation rate of the time consumption is different.
The method provided by the embodiment can dynamically acquire the optimal concurrency execution quantity adjustment degree of each group of jobs so as to dynamically and accurately adjust the concurrency quantity control value of each group of jobs, thereby improving the resource utilization rate while minimizing resource conflict caused by batch job processing.
Step 2034, adjusting the concurrency control value of each group of jobs according to the concurrency execution amount adjustment degree, the maximum concurrency execution amount corresponding to the target job type, and the minimum concurrency execution amount corresponding to the target job type.
Optionally, after the adjustment degree of the concurrency execution number of each group of jobs is obtained, an adjustment rule or an adjustment model may be adopted, and the concurrency number control value of each group of jobs is adjusted according to the adjustment degree of the concurrency execution number, the maximum concurrency execution number corresponding to the target job type, and the minimum concurrency execution number corresponding to the target job type, so that the adjusted concurrency number control value of each group of jobs is adapted to the current resources of the CPU, that is, the CPU resources are utilized to the greatest extent while avoiding job congestion.
Preferably, the adjusting the control value of the concurrency number of each group of jobs according to the adjustment degree of the concurrency execution number, the maximum concurrency execution number corresponding to the target job type, and the minimum concurrency execution number corresponding to the target job type includes:
According to the concurrency execution quantity adjustment degree, preliminarily adjusting the concurrency quantity control value of each group of operations to obtain corresponding candidate concurrency quantity control values of each group of operations;
if the candidate concurrency number control value corresponding to any group of operations is larger than the maximum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the maximum concurrency execution number;
if the candidate concurrency number control value corresponding to any group of operations is smaller than the minimum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the minimum concurrency execution number;
if the candidate concurrency number control value corresponding to any group of operations is greater than or equal to the minimum concurrency execution number and less than or equal to the maximum concurrency execution number, the concurrency number control value of any group of operations is kept to be the same as the candidate concurrency number control value corresponding to any group of operations.
As shown in fig. 8, for each group of jobs, assuming that the adjusted concurrency control value is M1, the candidate concurrency control value is M2, the concurrency control value before adjustment is M0, the concurrency execution number adjustment degree is N, the maximum concurrency execution number is MAX, and the minimum concurrency execution number is MIN, the following steps are specifically performed for adjusting the concurrency control value of each group of jobs:
Firstly, adopting a concurrency number preliminary adjustment formula to carry out preliminary adjustment on concurrency number control values of each group of operations to obtain candidate concurrency number control values M2, wherein the specific formula is as follows:
M2＝M0*(1+N)；
then, comparing the candidate concurrency number control value M2 with the maximum concurrency execution number MAX, and determining that the adjusted concurrency number control value is m1=max under the condition that M2> MAX;
TABLE 5 concurrent numerical control tables for each set of jobs
In the case of m2< = MAX, the candidate concurrency number control value M2 is compared with the minimum concurrency execution number MIN, in the case of M2< MIN, the adjusted concurrency number control value is determined to be m1=min, and in the case of M2 being greater than or equal to MIN, or less than or equal to MAX, m1=m2 is determined.
After the adjusted concurrency number control value of each group of jobs is obtained, the concurrency number control table of each group of jobs may be updated in real time, as shown in table 5, so that the updated concurrency number control table may be referred to for subsequent job processing.
According to the method provided by the embodiment, the concurrent number control values of each group of jobs are dynamically updated and adjusted, so that artificial experience setting is avoided, the concurrent number control values of each group of jobs are dynamically adapted to the current resources of the CPU, resource conflict caused by batch job processing is effectively reduced, and the resource utilization rate is improved.
In some embodiments, the step of processing each set of jobs in step 204 according to the adjusted concurrency control value further includes:
the following steps are performed for each target job of the respective set of jobs:
acquiring the current concurrent execution quantity of each group of jobs; the current concurrent execution quantity is the quantity of target jobs in an execution state in each group of jobs before the current target jobs are executed; the current target job is the target job submitted by the job scheduling tool in the current period;
if the current concurrent execution number is smaller than the adjusted concurrent number control value, adding 1 to the current concurrent execution number, configuring the current target job as an execution state to execute the current target job, and subtracting 1 from the current concurrent execution number under the condition that the current target job is executed;
if the current concurrent execution number is greater than or equal to the adjusted concurrent number control value, configuring the current target job into a dormant state until the current concurrent execution number is less than or equal to the adjusted concurrent number control value, and configuring the current target job into an execution state to execute the current target job.
It should be noted that, when the batch job is initially executed, the current concurrent execution number of each group of jobs is 0, and in the execution process, the current concurrent execution number of each group of jobs is dynamically updated according to the submitting state of each job submitted by the job scheduling tool and the processing state of each job, so as to realize dynamic release of the jobs according to the adjusted concurrent number control value. Wherein the commit status includes committed or uncommitted; the processing state includes execution or hibernation.
As shown in fig. 9, assuming that the current number of concurrent executions is Q and the adjusted control value of the number of concurrent executions is M1, the processing procedure of each group of jobs specifically includes:
firstly, according to a current target job submitted by a job scheduling tool, namely a submitted job, wherein the submitted state is a concurrency numerical control table corresponding to a group to which the job belongs, then, comparing an adjusted concurrency numerical control value in the concurrency numerical control table with the current concurrency execution quantity, if the current concurrency execution quantity Q < the adjusted concurrency numerical control value M1, adding 1 to the current concurrency execution quantity, configuring the current target job into an execution state to truly execute the current target job, and subtracting 1 from the current concurrency execution quantity when the current target job is executed; if the current concurrent execution number Q > =the adjusted concurrent number control value M1, the current target job is configured to be in a waiting state, that is, in a sleep state, and the checking is triggered at regular time until the current concurrent execution number Q < the adjusted concurrent number control value M1, and then the current target job is configured to be in an execution state, so as to execute the current target job.
As shown in fig. 10, a flow chart of an exemplary batch job processing method provided in this embodiment specifically includes the following steps:
Step 1001, collecting the current running log of the target batch job in execution at fixed time, and acquiring and recording a job list and the execution time consumption of each group of jobs from the current running log;
step 1002, obtaining reference time consumption data corresponding to each group of jobs from a reference time consumption data table, and grouping and comparing time consumption comparison information of jobs of different job types;
step 1003, evaluating the resource usage according to the time consumption comparison information, and adjusting the concurrency control value of each group of jobs, wherein the specific adjustment step is shown in fig. 7. Writing the adjusted concurrent number control value into a concurrent number control table corresponding to each group of operation;
step 1004, reading the concurrency number control value of each group of jobs from the concurrency number control table corresponding to each group of jobs, and releasing the jobs according to the concurrency number control value of each group of jobs.
The batch job processing apparatus provided by the present invention will be described below, and the batch job processing apparatus described below and the batch job processing method described above may be referred to correspondingly to each other.
As shown in fig. 11, a schematic structural diagram of a batch job processing apparatus according to this embodiment is provided, where the apparatus includes:
the acquisition module 1101 is configured to acquire a current running log of a target batch job, and determine execution time consumption of each group of jobs in the target batch job according to the current running log;
The reference data generating module 1102 is configured to determine reference time consumption data corresponding to each group of jobs according to a target job type to which each group of jobs belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type;
the dynamic control module 1103 is configured to adjust the concurrency control value of each group of jobs according to the execution time consumption and the reference time consumption data; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs;
the processing module 1104 is configured to process each group of jobs according to the adjusted concurrency control value.
According to the batch job processing device, the concurrent number control value of each group of jobs is dynamically adjusted based on the execution time consumption of each group of jobs in the target batch job and the reference time consumption data determined according to the target job type to which each group of jobs belong, so that manual experience setting is avoided, the concurrent number control value of each group of jobs is dynamically adapted to the current resources of the CPU, resource conflict caused by batch job processing is effectively reduced, and the resource utilization rate is improved.
Fig. 12 illustrates a physical structure diagram of an electronic device, as shown in fig. 12, which may include: a processor 1201, a communication interface (Communications Interface), a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202 and the memory 1203 perform communication with each other through the communication bus 1204. The processor 1201 may invoke logic instructions in the memory 1203 to perform a batch job processing method comprising: collecting a current running log of a target batch job, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log; determining reference time consumption data corresponding to each group of operations according to the target operation type to which each group of operations belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type; according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of jobs is adjusted; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs; and processing each group of operations according to the adjusted concurrency number control value.
Further, the logic instructions in the memory 1203 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the batch job processing method provided by the above methods, the method comprising: collecting a current running log of a target batch job, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log; determining reference time consumption data corresponding to each group of operations according to the target operation type to which each group of operations belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type; according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of jobs is adjusted; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs; and processing each group of operations according to the adjusted concurrency number control value.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform a batch job processing method provided by the above methods, the method comprising: collecting a current running log of a target batch job, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log; determining reference time consumption data corresponding to each group of operations according to the target operation type to which each group of operations belongs; the reference time consumption data is obtained according to the historical operation log of the sample batch operation under the target operation type; according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of jobs is adjusted; the concurrency number control value is the maximum value for controlling the number of concurrent execution jobs of each group of jobs; and processing each group of operations according to the adjusted concurrency number control value.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A batch job processing method, comprising:
collecting a current running log of a target batch job, grouping the target batch job according to a job type, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log;
determining reference time consumption data corresponding to each group of operations according to the target operation type of each group of operations; the reference time consumption data is obtained according to a historical operation log of the sample batch operation under the target operation type;
according to the execution time consumption and the reference time consumption data, the concurrency number control value of each group of operations is adjusted; the concurrency number control value is the maximum value for controlling the quantity of the concurrent execution jobs of each group of jobs;
and processing each group of jobs according to the adjusted concurrency number control value.
2. The batch job processing method of claim 1, wherein the method further comprises:
acquiring a plurality of historical time consumption data of each sample job in the sample batch job according to the historical operation log of the sample batch job under the target job type;
Performing quantile processing on the plurality of historical time consumption data to obtain an upper time consumption quantile value and a lower time consumption quantile value corresponding to the sample batch operation;
and constructing the reference time consumption data according to the time consumption upper quantile value and the time consumption lower quantile value.
3. The batch job processing method according to claim 2, wherein the adjusting the concurrency control value of each set of jobs based on the execution time consumption and the reference time consumption data includes:
comparing the execution time consumption with the reference time consumption data, and determining the time consumption comparison value corresponding to each group of operation;
determining the time consumption fluctuation rate corresponding to each group of operations according to the time consumption comparison value and the operation quantity of each group of operations;
determining the concurrent execution quantity adjustment degree of each group of operations according to the time consumption fluctuation rate;
and adjusting the concurrency number control value of each group of jobs according to the concurrency execution number adjustment degree, the maximum concurrency execution number corresponding to the target job type and the minimum concurrency execution number corresponding to the target job type.
4. A method of batch job processing according to claim 3, wherein comparing the execution time consumption with the reference time consumption data to determine a time consumption comparison value corresponding to each set of jobs comprises:
For each target job in the set of jobs, performing the steps of:
if the execution time consumption of the current target job is smaller than or equal to the time consumption lower dividing value, determining that the time consumption comparison value of the current target job is a first preset value;
if the execution time consumption of the current target job is greater than or equal to the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a second preset value;
if the execution time consumption of the current target job is greater than the time consumption lower dividing value and less than the time consumption upper dividing value, determining that the time consumption comparison value of the current target job is a third preset value;
the first preset value is smaller than the third preset value, and the third preset value is smaller than the second preset value.
5. The batch job processing method according to claim 4, wherein the determining the time consumption fluctuation rate corresponding to each set of jobs according to the time consumption comparison value and the job number of each set of jobs comprises:
dividing the sum of the time consumption contrast values of all target operations in each group of operations by the number of operations in each group of operations;
and determining the time consumption fluctuation rate corresponding to each group of operation according to the division result.
6. A batch job processing method according to claim 3, wherein the determining the degree of concurrent execution number adjustment of the respective sets of jobs according to the time consumption fluctuation rate includes:
according to the time consumption fluctuation rate, acquiring a preset fluctuation rate matched with the time consumption fluctuation rate from a target mapping table corresponding to the target operation type;
acquiring the concurrent execution quantity adjustment degree of each group of operations in a target mapping table according to the preset fluctuation rate matched with the time consumption fluctuation rate;
the target mapping table stores a mapping relation between the preset fluctuation rate and the concurrent execution quantity adjustment degree in advance.
7. The batch job processing method according to claim 3, wherein the adjusting the concurrency count control value of each set of jobs according to the concurrency execution count adjustment degree, the maximum concurrency execution count corresponding to the target job type, and the minimum concurrency execution count corresponding to the target job type includes:
according to the concurrency execution quantity adjustment degree, preliminarily adjusting the concurrency quantity control value of each group of operations to obtain corresponding candidate concurrency quantity control values of each group of operations;
If the candidate concurrency number control value corresponding to any group of operations is larger than the maximum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the maximum concurrency execution number;
if the candidate concurrency number control value corresponding to any group of operations is smaller than the minimum concurrency execution number, adjusting the concurrency number control value of any group of operations to be the same as the minimum concurrency execution number;
and if the candidate concurrency number control value corresponding to any group of jobs is greater than or equal to the minimum concurrency execution number and is less than or equal to the maximum concurrency execution number, keeping the concurrency number control value of any group of jobs to be adjusted to be the same as the candidate concurrency number control value corresponding to any group of jobs.
8. The batch job processing method according to any one of claims 1 to 7, wherein the processing of the respective sets of jobs according to the adjusted concurrency control value includes:
for each target job of the set of jobs, performing the steps of:
acquiring the current concurrent execution quantity of each group of jobs; the current concurrent execution quantity is the quantity of target jobs in an execution state in each set of jobs before the current target jobs are executed; the current target job is a target job submitted by a job scheduling tool in a current period;
If the current concurrent execution number is smaller than the adjusted concurrent number control value, adding 1 to the current concurrent execution number accumulation, configuring the current target job into an execution state to execute the current target job, and subtracting 1 from the current concurrent execution number accumulation under the condition that the current target job is executed; and if the current concurrent execution number is greater than or equal to the adjusted concurrent number control value, configuring the current target job into a dormant state until the current concurrent execution number is less than or equal to the adjusted concurrent number control value, and configuring the current target job into an execution state to execute the current target job.
9. A batch job processing apparatus, comprising:
the system comprises an acquisition module, a storage module and a control module, wherein the acquisition module is used for acquiring a current running log of a target batch job, grouping the target batch job according to a job type, and determining the execution time consumption of each group of jobs in the target batch job according to the current running log;
the reference data generation module is used for determining reference time consumption data corresponding to each group of jobs according to the target job type to which each group of jobs belongs; the reference time consumption data is obtained according to a historical operation log of the sample batch operation under the target operation type;
The dynamic control module is used for adjusting the concurrency number control value of each group of operations according to the execution time consumption and the reference time consumption data; the concurrency number control value is the maximum value for controlling the quantity of the concurrent execution jobs of each group of jobs;
and the processing module is used for processing each group of jobs according to the adjusted concurrency number control value.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the batch job processing method of any one of claims 1 to 8 when the program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310437884.6A CN116501468A (en) | 2023-04-21 | 2023-04-21 | Batch job processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310437884.6A CN116501468A (en) | 2023-04-21 | 2023-04-21 | Batch job processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116501468A true CN116501468A (en) | 2023-07-28 |
Family
ID=87324127
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310437884.6A Pending CN116501468A (en) | 2023-04-21 | 2023-04-21 | Batch job processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116501468A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117575654A (en) * | 2023-11-27 | 2024-02-20 | 数翊科技(北京)有限公司 | Scheduling method and device for data processing job |
-
2023
- 2023-04-21 CN CN202310437884.6A patent/CN116501468A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117575654A (en) * | 2023-11-27 | 2024-02-20 | 数翊科技(北京)有限公司 | Scheduling method and device for data processing job |
| CN117575654B (en) * | 2023-11-27 | 2024-05-14 | 数翊科技(北京)有限公司 | Scheduling method and device for data processing job |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8200659B2 (en) | Method of incorporating DBMS wizards with analytical models for DBMS servers performance optimization | |
| US9720941B2 (en) | Fully automated SQL tuning | |
| CN112685170B (en) | Dynamic optimization of backup strategies | |
| US10402225B2 (en) | Tuning resources based on queuing network model | |
| US9430288B2 (en) | Job scheduling based on historical job data | |
| Ganapathi et al. | Statistics-driven workload modeling for the cloud | |
| Rosa et al. | Predicting and mitigating jobs failures in big data clusters | |
| CN108595254B (en) | Query scheduling method | |
| US11175951B2 (en) | Resource availability-based workflow execution timing determination | |
| Ouyang et al. | Straggler detection in parallel computing systems through dynamic threshold calculation | |
| CN111026553A (en) | Resource scheduling method for offline mixed part operation and server system | |
| CN115373835A (en) | Task resource adjusting method and device for Flink cluster and electronic equipment | |
| US20230305880A1 (en) | Cluster distributed resource scheduling method, apparatus and device, and storage medium | |
| CN112052082B (en) | Task attribute optimization method, device, server and storage medium | |
| CN116501468A (en) | Batch job processing method and device and electronic equipment | |
| CN110377519B (en) | Performance capacity test method, device and equipment of big data system and storage medium | |
| CN115543577A (en) | Kubernetes resource scheduling optimization method based on covariates, storage medium and equipment | |
| US8510273B2 (en) | System, method, and computer-readable medium to facilitate application of arrival rate qualifications to missed throughput server level goals | |
| Ouyang et al. | An approach for modeling and ranking node-level stragglers in cloud datacenters | |
| CN113467908A (en) | Task execution method and device, computer readable storage medium and terminal equipment | |
| Zhang et al. | Autrascale: an automated and transfer learning solution for streaming system auto-scaling | |
| CN113391911A (en) | Big data resource dynamic scheduling method, device and equipment | |
| US9081605B2 (en) | Conflicting sub-process identification method, apparatus and computer program | |
| CN110033242B (en) | Working time determining method, device, equipment and medium | |
| CN116360921A (en) | Cloud platform resource optimal scheduling method and system for electric power Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |