CN111125097A - Report scheduling method and device - Google Patents
Report scheduling method and device Download PDFInfo
- Publication number
- CN111125097A CN111125097A CN201911197747.XA CN201911197747A CN111125097A CN 111125097 A CN111125097 A CN 111125097A CN 201911197747 A CN201911197747 A CN 201911197747A CN 111125097 A CN111125097 A CN 111125097A
- Authority
- CN
- China
- Prior art keywords
- execution
- report
- time
- historical
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000004422 calculation algorithm Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 13
- 238000013468 resource allocation Methods 0.000 claims description 8
- YHXISWVBGDMDLQ-UHFFFAOYSA-N moclobemide Chemical compound C1=CC(Cl)=CC=C1C(=O)NCCN1CCOCC1 YHXISWVBGDMDLQ-UHFFFAOYSA-N 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 abstract description 11
- 238000010586 diagram Methods 0.000 description 16
- 238000012545 processing Methods 0.000 description 6
- 238000007635 classification algorithm Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000003860 storage Methods 0.000 description 3
- 230000003442 weekly effect Effects 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a report scheduling method and a report scheduling device, wherein the method comprises the following steps: inquiring a report configuration table to obtain a plurality of reports to be executed; the report configuration table comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration; generating expected performance data of the server according to historical performance data of the server corresponding to the historical execution data; determining an allocable time period of the server according to preset execution parameters and expected performance data, and updating an execution starting moment according to the allocable time period; and calling the report to be executed according to the updated execution starting time. The invention relieves the calculation pressure of the server in a high-load period, and fully utilizes the vacant period of the server to create a more reasonable report scheduling plan.
Description
Technical Field
The invention relates to the technical field of information scheduling, in particular to a report scheduling method and device.
Background
In an OLAP (On-Line Analytical Processing) scenario, the calculation pressure when generating a report is large, and there are two main calculation ideas at present: one is that data is put in a database as much as possible, such as Oracle/Mysql, and the calculation performance is optimized by schemes of multilayer advanced collection or data association, sql optimization calculation, database host hardware upgrade and the like; one is that the database only makes simple data query and a small amount of summary, most of the statistics and correlation calculation processes are put on a report server host, and independent scripts or programs are developed for calculation. The calculation performance is improved by improving the performance of the report server and the number of hosts.
When the report is scheduled, the situation that the computing power of the server in some time interval is surplus and the computing power of the server in some time interval is tense exists. The conventional report scheduling scheme usually pre-configures fixed scheduling time, and when operation and maintenance personnel arrange a scheduling plan, because a conventional report scene and a special high-load scene need to be considered at the same time, enough elastic interval time needs to be reserved when configuring reports, on one hand, the reports can not be scheduled smoothly under various scenes, on the other hand, a large amount of server scheduling spare time is available, and server resources are wasted.
Disclosure of Invention
The invention provides a report scheduling method and a report scheduling device, which can relieve the calculation pressure of a server in a high-load time period, make full use of the vacant time period of the server and create a better report scheduling plan on the premise of reducing the load of the server as much as possible.
In a first aspect, an embodiment of the present invention provides a report scheduling method, where the method includes: inquiring a report configuration table to obtain a plurality of reports to be executed; the report configuration table comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration; generating expected performance data of the server according to historical performance data of the server corresponding to the historical execution data; determining distributable time periods of the server according to the preset execution parameters and the expected performance data, and updating the execution starting time according to the distributable time periods; and calling the report to be executed according to the updated execution starting time.
In a second aspect, an embodiment of the present invention further provides a report scheduling apparatus, where the apparatus includes: the scanning acquisition module is used for scanning the report configuration table to acquire a plurality of reports to be executed; the report configuration table at least comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration; the prediction module is used for generating expected performance data of the server according to the historical performance data of the server corresponding to the historical execution data; the distribution module is used for determining the distributable time period of the server according to the preset execution parameters and the expected performance data and updating the execution starting time according to the distributable time period; and the scheduling module is used for calling the report to be executed according to the updated execution starting time.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores a computer program that can run on the processor, and the processor implements the report scheduling method when executing the computer program.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code causes the processor to execute the report scheduling method.
The embodiment of the invention has the following beneficial effects: the embodiment of the invention provides a report scheduling scheme, which obtains a plurality of reports to be executed by inquiring a report configuration table, wherein the report configuration table comprises a plurality of reports and historical execution data and preset execution parameters of each report, the historical execution data at least comprises execution starting time and execution ending time, expected performance data of a server is generated according to the historical performance data of the server corresponding to the historical execution data, then an allocable time period of the server is determined according to the preset execution parameters and the expected performance data, the execution starting time is updated according to the allocable time period, the preset execution parameters at least comprise an execution period, an expected next execution time and an expected execution duration, and finally, the reports to be executed are called according to the updated execution starting time. According to the embodiment of the invention, the expected performance data is predicted according to the historical execution data and the historical performance data, and the execution starting time of the report to be executed is updated according to the expected performance data and the preset execution parameters, so that the calculation pressure of a server in a high-load period can be relieved, the vacant period of the server is fully utilized, and a more reasonable report scheduling plan is created on the premise of reducing the load of the server as much as possible.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flowchart of a report scheduling method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a time series model and a classification algorithm model building process according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating an execution flow of a report scheduling method according to an embodiment of the present invention;
FIG. 4 is a block diagram of a report scheduling apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The performance state of the server is dynamically changed, for example, in special high-load scenes such as backup/cut-over operation of a database, role multiplexing of the server, and simultaneous generation of daily/weekly/monthly report increase report pressure at the beginning of a month, the computing power of the server in certain time periods is surplus, and the computing power in certain time periods is tense.
Based on this, the report scheduling method provided by the embodiment of the invention can dynamically adjust the report scheduling plan according to the load conditions of the database and each report server.
To facilitate understanding of the embodiment, a report scheduling method disclosed in the embodiment of the present invention is first described in detail.
The embodiment of the invention provides a report scheduling method, which is shown in a flow chart of the report scheduling method in figure 1 and comprises the following steps:
step S102, inquiring the report configuration table to obtain a plurality of reports to be executed.
In the embodiment of the present invention, the report configuration table is used for storing basic information of the report, including historical execution data information and preset execution parameter information of the report. The historical execution data is used for recording historical execution conditions of the report, and comprises information such as execution starting time and execution ending time, the preset execution parameters are pre-configured data and are used for recording parameter information related to the subsequent report execution, and the parameter information comprises report task ID, execution period, expected next execution time, expected execution duration and the like.
The report to be executed is the report to be executed in the next stage determined from the report configuration table. And obtaining a plurality of reports to be executed according to the preset execution parameters. The execution period refers to the interval duration of the report execution, for example, for weekly reports, the execution period may be set to be one week, if the last execution time is monday eight-early, the predicted next execution time of the weekly report is next monday eight-early, and the next execution time of the report may be determined according to the execution period and the latest execution starting time. The expected execution time is the time consumed by the completion of the execution of the preset report. The historical execution data includes the time when the report starts to be executed and the time when the report ends to be executed in the past time.
It should be noted that, in the embodiment of the present invention, the expected next execution time may be determined according to the historical execution records, the execution time that may be increased or decreased for each report in different scenarios, and the risk that may cause the report to be executed to fail or overtime.
And step S104, generating expected performance data of the server according to the historical performance data of the server corresponding to the historical execution data.
In the embodiment of the present invention, the performance of the server may be measured by a Central Processing Unit (CPU) index, a memory index, a process number index, and an Input/Output (IO) index. Expected performance data for the server may be generated based on historical performance data for the server for a plurality of different time periods. The server historical performance data is used for describing the server performance situation corresponding to the historical execution data generation period. The expected performance data for the servers is a predicted performance of each server for a future period of time.
And step S106, determining an allocable time period of the server according to the preset execution parameters and the expected performance data, and updating the execution starting time according to the allocable time period.
In the embodiment of the invention, the server can comprise a database server and a plurality of report servers. The method comprises the steps of presetting execution parameters for determining report execution information in a future time period, using expected performance data for determining performance conditions of servers in different future time periods, screening distributable time periods with better server performance based on the preset execution parameters and the expected performance data, and updating the starting execution time of the report in a report configuration table according to the determined distributable time periods so as to execute the report according to the distributable time periods when the report is scheduled.
And step S108, calling the report to be executed according to the updated execution starting time.
In the embodiment of the invention, the updated execution starting time is also stored in the report configuration table, the report to be executed is called according to the updated execution starting time, the report scheduling task in a high-load time period can be adjusted to a server low-load time period, the optimal configuration of server resources is realized, and a more reasonable report scheduling plan is obtained.
The embodiment of the invention provides a report scheduling scheme, which obtains a plurality of reports to be executed by inquiring a report configuration table, wherein the report configuration table comprises a plurality of reports and historical execution data and preset execution parameters of each report, the historical execution data at least comprises execution starting time and execution ending time, expected performance data of a server is generated according to the historical performance data of the server corresponding to the historical execution data, then an allocable time period of the server is determined according to the preset execution parameters and the expected performance data, the execution starting time is updated according to the allocable time period, the preset execution parameters at least comprise an execution period, an expected next execution time and an expected execution duration, and finally, the reports to be executed are called according to the updated execution starting time. According to the embodiment of the invention, the expected performance data is predicted according to the historical execution data and the historical performance data, and the execution starting time of the report to be executed is updated according to the expected performance data and the preset execution parameters, so that the calculation pressure of a server in a high-load period can be relieved, the vacant period of the server is fully utilized, and a more reasonable report scheduling plan is created on the premise of reducing the load of the server as much as possible.
In order to facilitate screening of the report to be executed, the report configuration table is queried to obtain a plurality of reports to be executed, and the method comprises the following steps:
and if the predicted next execution time of the target report belongs to the target time period, taking the target report as a report to be executed.
In the embodiment of the invention, the report in the report configuration table is screened, the report can be judged one by one according to the report ID in the report configuration table, and the target report is the report to be judged selected from the report configuration table. The target time period may be determined according to actual requirements or experience, and the embodiment of the present invention is not particularly limited. For example, when a natural day is selected as the target time period, the report configuration table may be queried at the zero point of the day, and the report that is predicted to be executed at the next time and is 24 o' clock before the day is determined as the report to be executed.
In order to ensure that the report forms can be generated smoothly in time under various scenes, the execution parameters also comprise the latest execution deadline, the distributable time period of the server is determined according to the preset execution parameters and the expected performance data, and the method can be executed according to the following steps:
(1) and determining the execution sequence of the plurality of reports to be executed according to the latest execution deadline, the historical execution data and the MAX-MIN resource allocation algorithm.
In the embodiment of the invention, the latest execution deadline is used for describing that the report needs to be executed before the latest execution deadline. And the MAX-MIN resource allocation algorithm is used for determining higher priority for the report with the maximum historical real execution time and the minimum latest deadline. The historical real execution time of the report can be obtained according to the historical execution data of the report. And determining the priority of the to-be-executed reports one by one to obtain the execution sequence of the multiple to-be-executed reports.
For example, for a plurality of reports, report 1, report 2, and report 3, the latest execution deadline sequentially is: 13, 8 and 20, wherein the historical real execution time length sequentially comprises: 1 hour, 3 hours and 5 minutes, the report 2 with the largest historical real execution time and the smallest latest deadline can be determined as the report scheduled first, the schedule table 1 and the schedule table 3. The execution sequence is determined by a MAX-MIN resource allocation algorithm, large tasks can be scheduled first, and mapping from the tasks to the resources is to select the task with the largest earliest completion time to be mapped to the corresponding machine.
(2) And determining the assignable time period of each report to be executed according to the execution sequence, the historical execution data, the expected execution duration and the expected performance data of the server.
In the embodiment of the invention, after the execution sequence of the report to be executed is obtained, the assignable time period is determined for the report to be executed according to the sequence. When the assignable time period is determined, for a report, the historical performance data of a server executing the report is confirmed according to the time period of historical execution data, an assignable time period is determined in a future time period, the expected performance data of the server of the assignable time period is not lower than the maximum value of the historical performance data of the server of the report, and the duration of the assignable time period is not lower than the expected execution duration.
In order to further ensure that the reports can be successfully generated in various scenes, the assignable time period of each report to be executed is determined according to the execution sequence, the historical execution data, the expected execution duration and the expected performance data of the server, and the method can be executed according to the following steps:
(1) and taking a time period corresponding to the query time to the latest execution deadline as an assignable time range.
In the embodiment of the present invention, any one of the time periods within the allocable time range may be selected as the allocable time period. For example, for a report, the query time is zero, and the latest execution deadline of the report is 8 points, then the table may be arbitrarily selected to be executed in an allocable time range of 0-8 points, for example, if the expected execution duration of the table is 1 hour, 1-2 points may be selected to schedule the table.
(2) And taking the report to be executed in each assignable time range as a group of tasks to be executed to obtain a plurality of groups of tasks to be executed.
In the embodiment of the present invention, each assignable time range may include a plurality of reports, for example, 10 tasks to be executed are included between 0 to 8 points of the assignable time range, and 3 tasks to be executed are included between 0 to 12 points of the assignable time range. 10 tasks to be executed between 0 and 8 points can be scheduled, and then 3 tasks to be executed between 0 and 12 points can be scheduled.
(3) And determining historical performance data of the server corresponding to the historical execution data for each report in each group of tasks to be executed according to the execution sequence, the historical execution data and the predicted execution time.
In the embodiment of the present invention, for example, after the first group includes 10 to-be-executed tasks between 0 and 8 points, and the execution order of the 10 to-be-executed tasks is determined according to the MAX-MIN resource allocation algorithm, for a report with the highest priority, the execution duration of the report may be obtained according to the historical execution data (the execution start time minus the execution end time, and for an overtime report, the execution start time plus the expected execution duration of the report), and then the maximum historical performance data of the server corresponding to the report execution duration is obtained. And respectively determining the maximum historical performance data of the server for the reports in each group according to the execution sequence.
(4) Determining the execution result of each report in an allocable time range according to server historical performance data corresponding to the historical execution data of each report, expected performance data of the server and a GBDT (Gradient Boosting Decision Tree) algorithm; the execution result is success, failure or timeout.
In the embodiment of the invention, the historical report execution information in the configuration table is associated with the server performance information at the corresponding time. And calling a GBDT classification algorithm to construct a report execution result prediction model. And determining the execution result of each report in the assignable time range according to the historical performance data of the server corresponding to the historical execution data of each report, the expected performance data of the server in the same future execution time period and the GBDT algorithm. For example, for a report, when the report is executed at 1-3 points yesterday, the corresponding historical performance data (CPU, memory, process number, IO) of the server is A, B, C and D, and the expected performance data of the server at 1-3 points today is compared with A, B, C and D to calculate, and the result that the report is executed at 1-3 points today may be a failure.
(5) And determining the distributable time period of each report to be executed in the distributable time range according to the preset window duration and the execution result.
In the embodiment of the present invention, the preset window duration may be set according to actual requirements or experience, for example, may be set to 15 minutes, then for the report, the expected performance data of the today 1-3 point server is compared with A, B, C and D, if the execution result of the report at today 1-3 points is a failure, according to the preset window, the current time of: 15-3: 15, the expected performance data of the server is compared with A, B, C and D, and if the report is obtained, the report is obtained in the following steps of today 1: 15-3: 15, the execution result is successful, and 1: 15-3: 15 is determined as an allocable time period within the allocable time range 0-8 points.
In one embodiment, the reporting tasks are distributed using a MAX-MIN resource allocation algorithm. MAX is the maximum task execution time. MIN means that the latest execution deadline of the task is minimum. Defining: m is the set of report tasks needing to be allocated with the scheduling scheme one day after the current round. And C is each server set. And Q is the predicted performance data corresponding to the server C at each time on the next day. T is the executable time period of a certain report form, and is the latest execution deadline from the 0 o' clock of the execution day to the execution day. t is the real execution time of a certain report task, and is the last execution starting time of the report-the last execution ending time of the report. And P is a report task for splitting the executable time period T into equal-length interval sets according to the length T. U is a set of maximum performance data corresponding to each interval of a certain report task in an executable interval P (Qij represents the maximum performance data of the ith server in the jth interval, wherein i belongs to C, and j belongs to P).
Firstly, the task with the minimum deadline for latest execution of the report in M is taken, and the executable time range of the report task is taken as T. And then taking all tasks needing to be executed in the time range T, and storing the tasks in the set N. And calculating the real execution time t of the tasks in the N, and taking out the task with the longest relative execution time in the N. And obtaining a T, T, P and U set of the task. And traversing the data in the U one by one, inputting the obtained performance data and the report execution time into an algorithm GBDT model, if the prediction classification is successful, selecting the interval as the execution interval of the report, and if not, continuously taking the next interval in the P until a proper interval is found. The minimum time of the successful prediction interval is the execution time of the report.
In order to obtain the expected performance data of the server more accurately, the expected performance data of the server is generated according to the historical execution data and the historical performance data of the server corresponding to the historical execution data, the following steps can be executed:
expected performance data for each server is generated based on historical execution data, server historical performance data, and an ARIMA Model (autoregressive Moving Average Model).
In the embodiment of the invention, the ARIMA algorithm can be used for predicting the performance conditions (CPU, memory, process number and IO) of each server in the future one day according to the historical time sequence data of each server in one month.
In order to realize the dynamic adjustment of the report scheduling plan, the historical execution data comprises the report execution ending time; after calling the report to be executed according to the updated report starting execution time, the following steps can be executed:
and updating the starting execution time, the ending execution time, the report execution result and the predicted next execution time.
In the embodiment of the invention, after the report execution is finished, the starting execution time, the finishing execution time and the report execution result data of the report are updated to the historical execution data of the report configuration table, and the predicted next execution time in the preset execution parameters is reconfigured to provide a data basis for the subsequent dynamic adjustment of report scheduling. And when the starting execution time of the report is determined next time, taking the current execution data as historical execution data, and dynamically adjusting the report scheduling plan.
It should be noted that the expected next execution time may be calculated from the end execution time and the execution cycle.
In one embodiment, 1, a report configuration table is queried, and a report set M which needs to be executed in the next day is obtained. And if the next execution time in the report is the time of the next day, the task is a report task which needs to be executed the next day. 2. And calling a time series ARIMA prediction algorithm to predict the performance data of each server in the next day. 3. And calling an MAX-MIN resource allocation algorithm to obtain the calling sequence of each report. And judging the report in the report set M according to the GBDT classification algorithm to obtain the starting execution time of the report, and updating the starting execution time field and the next execution time field in the report configuration table. 4. Repeating steps 1-3 until the set M is empty. 5. And if the report is executed, updating the execution ending time of the report.
Considering that some situations may need to add new reports, the scheme is further executed according to the following steps:
and adding the report form in the report form configuration table, and configuring preset execution parameters for the added report form.
In the embodiment of the present invention, when a report is added to the report configuration table, preset execution parameters need to be configured for the newly added report, for example, the preset execution parameters include a report task ID, a latest execution deadline, an execution period, an expected next execution time, an expected execution duration, and the like. When a report is newly added, the expected next execution time can be calculated by subtracting the expected execution duration from the latest execution deadline.
The embodiment of the invention provides a report scheduling method and a report scheduling device, and the execution flow of the method can refer to the report scheduling method execution flow diagram shown in FIG. 3. The method predicts the expected performance data of each server through a time series model, classifies the report execution result based on the expected performance data in the future period through a GBDT algorithm, predicts the influence of different scenes on the report execution duration and the result by using a decision tree algorithm, and finally dynamically adjusts the report scheduling plan according to the load conditions of a database and each report server by referring to the schematic diagram of the time series model and classification algorithm model building process shown in FIG. 2; the MAX-MIN algorithm is used to schedule the optimal dispatch plan based on the scenario of the future day.
An embodiment of the present invention further provides a report scheduling apparatus, referring to a structural block diagram of the report scheduling apparatus shown in fig. 4, where the apparatus includes:
a scanning obtaining module 71, configured to scan the report configuration table to obtain a plurality of reports to be executed; the report configuration table at least comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration; a prediction module 72, configured to generate expected performance data of the server according to historical performance data of the server corresponding to the historical execution data; the allocation module 73 is used for determining an allocable time period of the server according to the preset execution parameters and the expected performance data, and updating the execution starting time according to the allocable time period; and the scheduling module 74 is configured to call the report to be executed according to the updated execution starting time.
In one embodiment, the scan acquisition module is specifically configured to: and if the predicted next execution time of the target report belongs to the target time period, taking the target report as a report to be executed.
In one embodiment, the preset execution parameters further include a latest execution deadline; the allocation module is specifically configured to: determining the execution sequence of a plurality of reports to be executed according to the latest execution deadline, historical execution data and a MAX-MIN resource allocation algorithm; and determining the assignable time period of each report to be executed according to the execution sequence, the historical execution data, the expected execution duration and the expected performance data of the server.
In one embodiment, the allocation module is specifically configured to: taking a time period corresponding to the query time to the latest execution deadline as an allocable time range; taking the report forms to be executed in each allocable time range as a group of tasks to be executed to obtain a plurality of groups of tasks to be executed; determining historical performance data of the server corresponding to the historical execution data for each report in each group of tasks to be executed according to the execution sequence, the historical execution data and the predicted execution duration; determining the execution result of each report in an allocable time range according to the server historical performance data corresponding to the historical execution data of each report, the expected performance data of the server and the GBDT algorithm; the execution result is success, failure or overtime; and determining the distributable time period of each report to be executed in the distributable time range according to the preset window duration and the execution result.
In one embodiment, the prediction module is specifically configured to: expected performance data for each server is generated based on the historical execution data, the historical performance data for the server, and the ARIMA model.
In one embodiment, the historical execution data includes report execution end time; the apparatus also includes an update module to: and updating the starting execution time, the ending execution time, the report execution result and the predicted next execution time.
In one embodiment, the apparatus further comprises an adding module for: and adding the report form in the report form configuration table, and configuring preset execution parameters for the added report form.
The embodiment of the present invention further provides a computer device, referring to the schematic block diagram of the structure of the computer device shown in fig. 5, the computer device includes a memory 81 and a processor 82, the memory stores a computer program that can be executed on the processor, and the processor implements the steps of any one of the methods when executing the computer program.
It is clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the computer device described above may refer to the corresponding process in the foregoing method embodiments, and no further description is provided herein
Embodiments of the present invention also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform any of the steps of the above-described method.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A report scheduling method is characterized by comprising the following steps:
inquiring a report configuration table to obtain a plurality of reports to be executed; the report configuration table comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration;
generating expected performance data of the server according to historical performance data of the server corresponding to the historical execution data;
determining distributable time periods of the server according to the preset execution parameters and the expected performance data, and updating the execution starting time according to the distributable time periods;
and calling the report to be executed according to the updated execution starting time.
2. The report scheduling method according to claim 1, wherein querying the report configuration table to obtain a plurality of reports to be executed comprises:
and if the predicted next execution time of the target report belongs to the target time period, taking the target report as a report to be executed.
3. The report scheduling method according to claim 1, wherein the preset execution parameters further include a latest execution deadline;
determining an allocable time period of the server according to the preset execution parameters and the expected performance data, wherein the step of determining the allocable time period of the server comprises the following steps:
determining the execution sequence of the plurality of reports to be executed according to the latest execution deadline, the historical execution data and a MAX-MIN resource allocation algorithm;
and determining the assignable time period of each report to be executed according to the execution sequence, the historical execution data, the expected execution duration and the expected performance data of the server.
4. The report scheduling method according to claim 3, wherein determining the assignable time period of each report to be executed according to the execution sequence, the historical execution data, the expected execution duration and the expected performance data of the server comprises:
taking a time period corresponding to the query time to the latest execution deadline as an allocable time range;
taking the report forms to be executed in each allocable time range as a group of tasks to be executed to obtain a plurality of groups of tasks to be executed;
determining historical performance data of the server corresponding to the historical execution data for each report in each group of tasks to be executed according to the execution sequence, the historical execution data and the predicted execution duration;
determining the execution result of each report in the distributable time range according to the server historical performance data corresponding to the historical execution data of each report, the expected performance data of the server and the GBDT algorithm; the execution result is success, failure or overtime;
and determining the distributable time period of each report to be executed in the distributable time range according to the preset window duration and the execution result.
5. The report scheduling method according to claim 1, wherein generating expected performance data of a server according to the historical execution data and historical performance data of the server corresponding to the historical execution data comprises:
and generating expected performance data of each server according to the historical execution data, the historical performance data of the servers and the ARIMA model.
6. The report scheduling method according to any of claims 1-5, wherein the historical execution data includes a report execution end time;
after calling the report to be executed according to the updated report starting execution time, the method further comprises the following steps:
and updating the starting execution time, the ending execution time, the report execution result and the predicted next execution time.
7. The report scheduling method according to any one of claims 1 to 5, further comprising:
and adding a report form in the report form configuration table, and configuring preset execution parameters for the added report form.
8. A report scheduling apparatus, comprising:
the scanning acquisition module is used for scanning the report configuration table to acquire a plurality of reports to be executed; the report configuration table at least comprises historical execution data and preset execution parameters of each report; the historical execution data at least comprises a starting execution time and an ending execution time; the preset execution parameters at least comprise an execution cycle, an expected next execution time and an expected execution duration;
the prediction module is used for generating expected performance data of the server according to the historical performance data of the server corresponding to the historical execution data;
the distribution module is used for determining the distributable time period of the server according to the preset execution parameters and the expected performance data and updating the execution starting time according to the distributable time period;
and the scheduling module is used for calling the report to be executed according to the updated execution starting time.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911197747.XA CN111125097B (en) | 2019-11-29 | 2019-11-29 | Report scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911197747.XA CN111125097B (en) | 2019-11-29 | 2019-11-29 | Report scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125097A true CN111125097A (en) | 2020-05-08 |
| CN111125097B CN111125097B (en) | 2024-03-15 |
Family
ID=70497074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911197747.XA Active CN111125097B (en) | 2019-11-29 | 2019-11-29 | Report scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125097B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813518A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Robot early warning method and device, computer equipment and storage medium |
| CN111984658A (en) * | 2020-09-07 | 2020-11-24 | 中国银行股份有限公司 | Report processing method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101894163A (en) * | 2010-07-23 | 2010-11-24 | 中兴通讯股份有限公司 | Database operating and scheduling method and device for performance data acquisition system |
| CN103606021A (en) * | 2013-12-05 | 2014-02-26 | 中国海洋石油总公司 | Dynamic spot goods forecasting and scheduling method for liquefied natural gas (LNG) receiving station |
| US20150149400A1 (en) * | 2011-04-18 | 2015-05-28 | Sap Ag | Method and Apparatus for Monitoring an In-memory Computer System |
| US20160019094A1 (en) * | 2014-07-18 | 2016-01-21 | Thomson Reuters Global Resources (Trgr) | System and method for electronic work prediction and dynamically adjusting server resources |
| CN106815270A (en) * | 2015-12-02 | 2017-06-09 | 大唐移动通信设备有限公司 | A kind of method and device for servicing dynamic management |
| US20180218038A1 (en) * | 2017-01-30 | 2018-08-02 | International Business Machines Corportation | Database optimization based on forecasting hardware statistics using data mining techniques |
| CN110059858A (en) * | 2019-03-15 | 2019-07-26 | 深圳壹账通智能科技有限公司 | Server resource prediction technique, device, computer equipment and storage medium |
| US20190286478A1 (en) * | 2016-12-07 | 2019-09-19 | Tata Consultancy Services Limited | Systems and methods for scheduling tasks and managing computing resource allocation for closed loop control systems |
| CN110297701A (en) * | 2019-05-16 | 2019-10-01 | 平安科技（深圳）有限公司 | Data processing operation dispatching method, device, computer equipment and storage medium |
-
2019
- 2019-11-29 CN CN201911197747.XA patent/CN111125097B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101894163A (en) * | 2010-07-23 | 2010-11-24 | 中兴通讯股份有限公司 | Database operating and scheduling method and device for performance data acquisition system |
| US20150149400A1 (en) * | 2011-04-18 | 2015-05-28 | Sap Ag | Method and Apparatus for Monitoring an In-memory Computer System |
| CN103606021A (en) * | 2013-12-05 | 2014-02-26 | 中国海洋石油总公司 | Dynamic spot goods forecasting and scheduling method for liquefied natural gas (LNG) receiving station |
| US20160019094A1 (en) * | 2014-07-18 | 2016-01-21 | Thomson Reuters Global Resources (Trgr) | System and method for electronic work prediction and dynamically adjusting server resources |
| CN106815270A (en) * | 2015-12-02 | 2017-06-09 | 大唐移动通信设备有限公司 | A kind of method and device for servicing dynamic management |
| US20190286478A1 (en) * | 2016-12-07 | 2019-09-19 | Tata Consultancy Services Limited | Systems and methods for scheduling tasks and managing computing resource allocation for closed loop control systems |
| US20180218038A1 (en) * | 2017-01-30 | 2018-08-02 | International Business Machines Corportation | Database optimization based on forecasting hardware statistics using data mining techniques |
| CN110059858A (en) * | 2019-03-15 | 2019-07-26 | 深圳壹账通智能科技有限公司 | Server resource prediction technique, device, computer equipment and storage medium |
| CN110297701A (en) * | 2019-05-16 | 2019-10-01 | 平安科技（深圳）有限公司 | Data processing operation dispatching method, device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 刘东;张春元;李瑞;黄影;李毅;: "基于预测表的容错实时调度算法", 西安交通大学学报 * |
| 耿多: "多源数据电网调度报表的研制", 西北电力技术 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813518A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Robot early warning method and device, computer equipment and storage medium |
| CN111984658A (en) * | 2020-09-07 | 2020-11-24 | 中国银行股份有限公司 | Report processing method and device |
| CN111984658B (en) * | 2020-09-07 | 2023-09-22 | 中国银行股份有限公司 | Report processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125097B (en) | 2024-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11392561B2 (en) | Data migration using source classification and mapping | |
| US11847103B2 (en) | Data migration using customizable database consolidation rules | |
| US8473956B2 (en) | Priority based scheduling system for server | |
| US20200104377A1 (en) | Rules Based Scheduling and Migration of Databases Using Complexity and Weight | |
| CN110287029A (en) | A method of it is adjusted based on kubernetes container resource dynamic | |
| EP3920490B1 (en) | Adaptive resource allocation method and apparatus | |
| WO2021159638A1 (en) | Method, apparatus and device for scheduling cluster queue resources, and storage medium | |
| CN107580023A (en) | A kind of the stream process job scheduling method and system of dynamic adjustment task distribution | |
| EP3454210B1 (en) | Prescriptive analytics based activation timetable stack for cloud computing resource scheduling | |
| CN113010260A (en) | Elastic expansion method and system for container quantity | |
| CN102104496B (en) | Fault tolerance optimizing method of intermediate data in cloud computing environment | |
| CN112685153A (en) | Micro-service scheduling method and device and electronic equipment | |
| CN108632365A (en) | Service Source method of adjustment, relevant apparatus and equipment | |
| CN103324534A (en) | Operation scheduling method and operation scheduler | |
| CN111125097B (en) | Report scheduling method and device | |
| CN110602227B (en) | Intelligent contract management method and related device | |
| CN111414070A (en) | Case power consumption management method and system, electronic device and storage medium | |
| CN111382031B (en) | Test method and device | |
| CN110188086A (en) | Database automated tuning method and device based on load automatic Prediction | |
| US20230004440A1 (en) | Allocating of computing resources for applications | |
| CN117076882A (en) | Dynamic prediction management method for cloud service resources | |
| CN116185588A (en) | Task scheduling method and device, electronic equipment and readable storage medium | |
| CN110196773B (en) | Multi-time-scale security check system and method for unified scheduling computing resources | |
| US20190215240A1 (en) | Service network maintenance analysis and control | |
| CN117591285A (en) | Computing power resource scheduling and distributing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |