CN102185761B - Two-layer dynamic scheduling method facing to ensemble prediction applications - Google Patents
Two-layer dynamic scheduling method facing to ensemble prediction applications Download PDFInfo
- Publication number
- CN102185761B CN102185761B CN 201110092776 CN201110092776A CN102185761B CN 102185761 B CN102185761 B CN 102185761B CN 201110092776 CN201110092776 CN 201110092776 CN 201110092776 A CN201110092776 A CN 201110092776A CN 102185761 B CN102185761 B CN 102185761B
- Authority
- CN
- China
- Prior art keywords
- service
- sample data
- model predictions
- initial value
- value sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a two-layer dynamic scheduling method facing to ensemble prediction applications, aiming at providing a two-layer scheduling method based on dynamic selection among mesh nodes and the dynamic designation of the resource number in the node. The technical scheme is as follows: two-layer dynamic scheduling system is established firstly; furthermore, the system is arranged at the server terminal and each mesh node terminal; the two-layer scheduling system initializes the scheduling process of the current ensemble prediction; the mode prediction service at each mesh node dynamically competes the unconsumed initial sample data file to the sample data management service; the mesh nodes optimize the resource number of the appointed mode prediction program and start the mode prediction program; the sample data management service at the server terminal stops the ensemble prediction scheduling process and starts the postprocessing on the ensemble prediction flow. The two-layer dynamic scheduling method has real-time dynamic characteristic. By adopting the two-layer dynamic scheduling method, the time effectiveness of the ensemble prediction with large-scale calculation characteristic can be improved, and the calculation cost for executing the ensemble prediction once can be saved effectively.
Description
Technical field
The present invention relates to a kind of dynamic dispatching method based on grid, particularly a kind of dynamic dispatching method towards the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application.
Background technology
Numerical weather forecast is weather service requisite method in service, and it is under the situation of given initial condition and boundary condition, and according to the physical law of air motion, numerical solution air motion fundamental equation group is predicted following atmospheric condition process constantly.Common numerical weather forecast flow process comprises the observational data preliminary treatment, Variational Data Assimilation for Meteorology, and model predictions, reprocessing, product is visual etc., and these tasks particularly model predictions task need be used a large amount of high-performance resources and carry out numerical model and calculate.DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is to solve the emerging numerical weather forecast technology that proposes because of the initial condition of ordinary numeric value weather forecast approximation problem uncertain and simulation process, compare with the ordinary numeric value weather forecast, DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is by after the Variational Data Assimilation for Meteorology, increase the initial disturbance task, this task produces many to the initial value sample, and every pair of initial value sample all needs to carry out model predictions, and model predictions is the largest task of numerical weather forecast calculating, and each model predictions all needs a large amount of CPU computational resources.Therefore the calculating scale of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM will be dozens or even hundreds of times (being relevant to the initial value number of samples that initial disturbance produces) of ordinary numeric value forecast.
DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is used has following characteristics:
1. it is huge that DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is used amount of calculation, the workflow that DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is used is very complicated, particularly produce the initial value sample that surpasses more than 50 pairs even 100 pairs by initial disturbance, every pair of initial value sample all needs to carry out model predictions, and a model predictions is to have the concurrent program that is on a grand scale, need a large amount of high-performance calculation resources, and computing time is generally very long.
2. can independent concurrent execution between the model predictions in the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, namely each model predictions can be independently in the execution of getting on of different machines;
3. what the single DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM was used is ageing extremely important, otherwise can lose the realistic meaning of weather forecast;
4. the simple linear that is in the degree of parallelism of the model predictions program after the initial disturbance and speed-up ratio and is not with cpu resource number N (N generally gets 2 power number of times) increases.Model predictions program parallelization degree is general at first to be increased with the increase of N, reach some after (with the forecast area size of selecting and give the correct time in advance be shaped on the pass), its degree of parallelism can descend again; Though speed-up ratio enlarges with the N value and constantly increases, there are a kind of relation in the mitigation degree of its increase and degree of parallelism: at the beginning namely, when degree of parallelism increased with the N value, its speed-up ratio speedup was bigger, along with degree of parallelism reaches maximum, the speedup of speed-up ratio is maximum also; When N was increased to certain value, it is maximum that its degree of parallelism reaches,
When this moment, the N value continued to increase, degree of parallelism began to descend, and the speed-up ratio speedup will be tending towards relaxing this moment.
Characteristics according to above DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application, know that easily DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is fit to use distributed grid work flow technology to solve very much, and when the initial value number of samples is very big, single grid node is difficult to satisfy the computation requirement of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, if sequentially carry out these model predictions programs, will greatly influence the ageing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM.Because of the dynamic change of high-performance grid computing node environment and degree of parallelism and the speed-up ratio trend feature of model predictions concurrent program itself, adopt the scheduling means between static or simple grid node can't satisfy the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application demand, the dynamic mode of needs use is rationally arranged the concurrent execution to these model predictions tasks, to improve the ageing of whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM; Need to consider the feature of model predictions concurrent program simultaneously, reasonably specify appropriate C PU number of resources at every turn, as far as possible reduce in ageing and assess the cost not influencing DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM.
In the grid computing field, independent multi-task scheduling method refers to not exist between each task data or control to be correlated with, each task can be assigned in the grid node and carry out separately, traditional method comprises the Min-Min dispatching method, the Max-Min dispatching method, the DupLex dispatching method, Sufferage dispatching method etc.
In the dispatching algorithm of independent multitask, Min-Min dispatching method key step comprises:
Step 1: at each task in using, pre-estimate the time that each grid node is carried out this task, and select the grid node of time of implementation minimum;
Step 2: from each grid node, select the priority of task of expection minimum running time to carry out.
The Max-Min dispatching method is just opposite in step 2 with the Min-Min dispatching method, in each grid node, select the priority of task of expection deadline maximum to carry out, this dispatching method is suitable for each task execution time and differs bigger scene, preferentially will grow task and be assigned to preferably in the grid node and preferentially carry out, and can obtain the deadline preferably for application.
DupLex dispatching party rule is the comprehensive of Min-Min dispatching method and Max-Min dispatching method, it is estimated based on Min-Min dispatching method and Max-Min dispatching method at first respectively and uses the possible time of implementation of dispatching method separately, selects more excellent a kind of dispatching method of time of implementation to carry out then.
The Sufferage dispatching method is preferentially to select the bigger task of possible loss to carry out (if this task is not assigned to this computing node), and its key step comprises:
Step 1: to each task, estimate the time of implementation that it is assigned to each grid node;
Step 2: calculate the corresponding Sufferage value of each task, i.e. the difference of the inferior little time of implementation of being calculated by step 1 and minimum time of implementation;
Step 3: to each grid node, preferentially select the bigger task of Sufferage value to carry out.
Traditional independent multi-task scheduling method is to solve the variety classes independent task to be assigned to each grid computing node operation problem expeditiously substantially, and the most amount of calculation of these tasks is little, and its regulation goal is to optimize the whole application time of implementation.On the other hand, these dispatching methods do not consider that concurrent application itself is to dependence such as the factors such as degree of parallelism, speed-up ratio of cpu resource number, these methods are difficult to solve this special applications problem of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, particularly this class application problem with large-scale calculations amount cause the computing node resource than length because of computing time computing capability generation dynamic change.
Therefore how the special case of using at above DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM provides a kind of dispatching method that is applicable to the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application characteristic, is the technical problem that numerical value meteorological field technical staff very pays close attention to.
Summary of the invention
The technical problem to be solved in the present invention is to use the very difficult problem that solves of traditional independent multi-task scheduling method down towards DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, a kind of two layer scheduling methods based on Dynamic Selection between grid node and the dynamic appointment of intra-node number of resources are proposed, improve the ageing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM with large-scale calculations feature, effectively save and carry out assessing the cost of a DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM.
The present invention includes following steps:
The first step makes up two-layer dynamic scheduling system.This system is made up of service end and node side, the initial value sample data that service end management initial disturbance produces is also controlled the execution of each model predictions service of node side, its deploy four Web services and two databases: four Web services refer to that sample data filing service, sample data management service, service quality control service are QoS(Quality of Service) control serves and estimates service.Two databases are respectively that sample data file metadata storehouse and model predictions program are carried out experience database, the former stores initial value sample data metadata information, support sample data filing service and sample data management service, manage the initial value sample data that each DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM produces, the latter stores the input parameter (forecast area that each model predictions program is carried out, the system of giving the correct time in advance etc.), depend on the historical time of implementation mean value of concrete CPU number (2 integral number powers) and the output file size that produces, degree of parallelism, recorded informations such as speed-up ratio are to support to estimate service estimating the model predictions program execution time; Node side is made up of a plurality of grid nodes, each grid node is deployed with two Web services, comprise that the model predictions service (mainly packed the model predictions program of local DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process, in the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process, the model predictions service is the execution pattern prediction program repeatedly, and each model predictions program is input parameter with a pair of initial value sample data) and the heuristic specified services of intra-node number of resources.Two-layer dynamic scheduling system starts after the initial disturbance of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow.The sample data filing service of service end receives the initial value sample data that initial disturbance produces, and the initial value sample data is filed; This DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM sample data is obtained in model predictions service competition on each grid node of sample data management service notice node side; The model predictions service receives after " competition initial value sample data " notice that the sample data management service sends, and sends " obtaining the initial value sample data " request to the sample data management service; The sample data management service verifies by calling QoS control service whether the model predictions service place grid node of asking satisfies minimum QoS demand, after the success of QoS service verification, each model predictions service is read, is locked in the mode of concurrent control and transmits the initial value sample data, after the model predictions service obtains the initial value sample data, namely finish two-layer dynamic scheduling system to the selection of grid node, finished the ground floor scheduling: to the selection of grid node; The model predictions service starts the heuristic specified services of local node internal resource number, the number of resources carrying out this model predictions and serve (assignment procedure with the ageing executory cost that reduces simultaneously of optimizing whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM as target) is specified on the service of the estimating dynamic optimization ground of service end, model predictions service start-up mode prediction program is finished second layer scheduling: specify the number of resources in the grid node then.All initial value sample datas of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM be scheduled for each model predictions service execution intact after, triggered next task---the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow by the sample data management service of service end.
In second step, two-layer dispatching patcher is that the scheduling process of current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is carried out initialization, comprises following three steps:
Step 2.1, after the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process is finished initial disturbance, DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance (being once specifically carrying out of the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process) ID that is moved with this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is parameter, and " sample data filing " request that sends is filed service to start two-layer dispatching patcher to sample data.DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID is produced at random by this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow example, sign that can this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance of unique identification.
Step 2.2, sample data filing service is filed the initial value sample data.Use document (Scott Jensen etc., A Hybrid XML-Relational Grid Metadata Catalog, Workshop on Web Services-based Grid Applications (WGSA'06) in association with International Conference on Parallel Processing, 2006) metadata approach that proposes in adopts the mode of metadata record to be described to every pair of initial value sample data and metadata record is filed sample data file metadata storehouse.The metadata record of every pair of initial value sample data comprises: DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID, initial value sample data numbering, source grid node (grid node of initial value sample initial storage) storage URL(Uniform Resource Locator, URL(uniform resource locator)), the sample state, target save mesh node URL, the model predictions program start time.Initial value sample data numbering is to be produced at random by sample data filing service, is used for this numbering to initial value sample data file of unique identification.Local storage URL represents this to the local memory location of initial value sample data, and convenient follow-up file moves.The sample state comprises 0,1,2,3, and state 0 is initial value sample data initial condition, represents that this is not also occupied by any model predictions service the initial value sample data; State 1 is initial value sample data lock-out state, represents that this has been distributed on model predictions service on certain grid node to the initial value sample data and has occupied but be not transferred on the grid node; State 2 be the initial value sample data by " consumption " state, represent that this is occupied by certain model predictions service the initial value sample data and successfully is transferred on the grid node; The model predictions program successful execution that state 3 these initial value sample datas of expression have been served by model predictions.Target storage URL represents the address of the grid node that this is transferred to the initial value sample data; Model predictions program start time representation model predictions began with the time of this initial value sample data as input parameter start-up mode forecast application program, and this time is used for controlling a pattern prediction program time in reasonable range.
Step 2.3, sample data management service are notified all registered model predictions service competition " consumption " initial value sample datas.The initial value sample data is archived after the management, message subscribing/issue mechanism (list of references: Humphrey.M etc. based on Web service, State and events for Web services:a comparison of five WS-resource framework and WS-notification implementations, 14th IEEE International Symposium on High Performance Distributed Computing, 2005) notify the model predictions service of all grid nodes (after model predictions is disposed startup, it is from trend sample filing service issue), inform that the present service end of each model predictions service has had the initial value sample data, can compete and obtain to carry out model predictions.
In the 3rd step, the initial value sample data file that the model predictions service of various places grid node is not also consumed to sample data management service dynamic competition according to the Practical Calculation ability of oneself may further comprise the steps:
Step 3.1, the initial value sample data is asked in the model predictions service concomitantly.The grid middleware software supervision service MDS(Monitoring and Discovery Service of grid node is at first used in the model predictions service) obtain self idling-resource number, whether satisfy the minimum QoS constraint demand of model predictions then according to self Practical Calculation proficiency testing.If satisfy, the model predictions service initiatively sends " application initial value sample data " request message to the sample data management service of service end, this request message content comprises: local grid node URL, model predictions service URI(Uniform Resource Identifiers, the unified resource identifier); If do not satisfy, then change step 3.3;
Step 3.2, service end sample data management service are called QoS control service the restrictive condition of QoS of model predictions service place grid node are verified.QoS control service call is deployed in the grid middleware software supervision service MDS on the grid node, judges whether the model predictions service place grid node of application satisfies the minimum constraint requirements of the desired QoS of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM (comprising idle CPU number and memory size).If find that grid node does not satisfy the minimum constraint requirements of QoS, then send " refusal " message to the model predictions service of asking, after the model predictions service receives " refusal " message, time-out changes step 3.3 to the model predictions service of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance (so-called DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance refers to the once execution of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process); If this model predictions service place grid node of service end QoS control service discovery satisfies the minimum constraint requirements of QoS, it then further is a pair of initial value sample data file under the current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance of this model predictions service competition ID, method is to sample data file metadata library searching meta data entries by the sample data management service, if finding not have state is 0 metadata record, then the model predictions service to request sends " not having retrievable initial value sample data file " message.After the model predictions service receives " not having retrievable initial value sample data file " message, stop application and obtain the initial value sample data, change step 4.6; If it is 0 metadata record that state is arranged, then enter step 3.4;
Step 3.3, model predictions service are regularly constantly monitored the local resource idle condition circularly, whether satisfy the minimum constraint requirements of QoS with checking local grid node, if satisfy, change step 3.1, otherwise continue this step of circulation;
Step 3.4, the initial value sample data locking protection of model predictions service to obtaining of request, method is that the sample data management service is carried out transactions access operation, for the model predictions service of asking occupies the corresponding metadata record of this initial value sample data, with this initial value sample data of model predictions service request that prevents other, revising this initial value sample data institute corresponding states then is 1;
Step 3.5, sample data management service start the end-to-end transmission of initial value sample data to the model predictions service of application.Method is: the sample data management service is called grid middleware assembly-reliable file transmission service (RFT) (list of references: B.Sotomayor etc., The Globus Toolkit3Programmer's Tutorial, http://www.casa-sotomayor.net/gt3-tutorial/index.html, 2003), the service node that is started initial value sample data place by RFT transmits to the reliable file the grid node of model predictions place, if initial value sample data file transfer failure, then carry out limited number of time ground and restart transmission, as still bust this, this initial value sample data file access authority of release then, it is 0 that the sample data management service is revised initial value sample data state, allow to be deployed on other grid nodes the model predictions service request this to the initial value sample data; Initial value sample data file is as successfully transmission, and the model predictions service sends initial value sample data " transmission is finished " message to the sample data management service, and it is 2 that the sample data management end is revised this initial value sample data institute corresponding states.So far, finish the selection problem of initial value sample data to grid node, finished the ground floor scheduling.
In the 4th step, grid node is optimized number of resources and the start-up mode prediction program of designated mode prediction program.The model predictions service call is deployed in the local heuristic specified services of intra-node resource to carry out allocated resource number (mainly being the CPU number) for this model predictions program, target is to improve parallel efficiency (assessing the cost) not influencing to take into account under the ageing situation of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, and step is as follows:
Step 4.1, the heuristic specified services of intra-node number of resources starts estimates service valuation, estimates it for present mode forecast service and whether also might obtain down a pair of initial value sample data file, and concrete steps are:
Step 4.1.1, the heuristic specified services of intra-node number of resources estimate self model predictions program deadline and send " estimate request information to estimating service, step is as follows:
Step 4.1.1.1, the heuristic specified services of intra-node number of resources is carried out experience database based on the model predictions program and is estimated present mode forecast this performed model predictions program deadline of service according to the initial value sample data forecast parameter information (forecast area and the system of giving the correct time in advance) of current acquisition, current grid node idle available cpu resource information and memory information;
Step 4.1.1.2, the model predictions service is published to present mode forecast this performed model predictions program deadline of service of estimating the service of estimating of service end;
Step 4.1.1.3, the model predictions service send whether may obtain down a pair of effective initial value sample data " estimate request information to estimating service.
Step 4.1.2 estimates service broadcasting renewal and estimates information, and the model predictions service that obtains other grid node estimates to finish the temporal information of present mode prediction program.Estimate service receive " after estimating request information, dynamically ask other just in the model predictions service of execution pattern prediction program issue " Estimated Time Of Completion " information to estimating service.After broadcasting " Estimated Time Of Completion " message is received in each model predictions service, carry out estimating as step 4.1.1 is described according to the real-time implementation status of the performed model program of this model predictions and self computing capability, after all model predictions services had been estimated, just the result of Estimated Time Of Completion sent to the service of estimating separately;
Step 4.1.3 estimates service execution about the process of estimating of a pair of initial value sample data possibility under the model predictions acquisition of request, and method is:
Step 4.1.3.1, with all model predictions service Estimated Time Of Completions according to by early to late rank order, thereby draw transmission " estimate the order of priority k of the model predictions service of request information; k is positive integer, expression send " the model predictions service center that estimates request information in all model predictions services about time of estimating to finish present mode prediction program separately according to by early to the sequence of positions of arrangement in evening;
Step 4.1.3.2 estimates service call sample data management service and obtains effective initial value sample data.All metadata item records of sample data management service search initial value sample data obtain present initial value sample data file status and are all metadata record numbers of 0, are designated as n, and n is positive integer;
Step 4.1.3.3, if satisfy condition k≤n, then estimation results is returned True, otherwise returns False." estimate the model predictions service of request information, the model predictions service is handed to the heuristic specified services of local node internal resource number with estimation results subsequently estimation results to be sent to transmission with form of message.
Step 4.2, the estimation results that the heuristic specified services of grid node internal resource number sends according to the service estimated selects assignment algorithm to specify the cpu resource number, if the estimation results of returning is True, then changes step 4.2.1; If the estimation results of returning is False, then change step 4.2.2:
Step 4.2.1, the cpu resource number assignment procedure that the heuristic specified services time of implementation of local grid intra-node number of resources is preferential: specify big as far as possible cpu resource number, to improve the ageing of whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM.Suppose that current idle cpu resource number is M, M is positive integer, and then the cpu resource number that obtains according to this assignment procedure is cpu_num=2
m(wherein satisfy 2
m≤ M, 2
M+1M, m is positive integer), the cpu resource number of expression appointment is the integral number power less than the maximum 2 of the idle cpu resource number of current grid node.
Step 4.2.2, the cpu resource number assignment procedure of the heuristic specified services time of implementation-balance of efficiency of local node internal resource number, on the basis of the existing computing capability of present mode forecast service, do not influencing under the ageing prerequisite of whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance, select appropriate C PU number of resources, make and the steps include: the parallel efficiency maximization as far as possible of this model predictions program
Step 4.2.2.1 adopts and estimates the Late Finish t that all initial value sample datas of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM are performed based on the method for history
l, step is as follows:
Step 4.2.2.1.1 to each model predictions service i, according to the order of arranging among the step 4.1.3.1, obtains the obtainable execution initial value sample data umber num of model predictions service
i
Step 4.2.2.1.2 is to each model predictions service i, according to the actual max calculation ability cpu_num=2 of this model predictions place grid node
m(satisfy 2
m≤ M, 2
M+1M), forecast area size, the system of giving the correct time in advance, carry out experience database based on the model predictions program, inquire the scheduled time temp of this model predictions service " consumption " every part of initial value sample data, thereby obtain the fastest Estimated Time Of Completion T of model predictions service i
i=temp * numi;
Step 4.2.2.1.3 obtains each model predictions service maximum t of fast Estimated Time Of Completion
l,
Wherein S represents all model predictions set of services.
Step 4.2.2.2 is according to current time t
c, adopt single model predictions program output file to obtain this model predictions program admissible running time of t to the file transfer time predictor method of reprocessing grid node
Allow=t
l-t
c-t
Tm, t wherein
TmExpression present mode prediction program output file is transferred to estimated time of reprocessing place grid node by current grid node, and single model predictions program output file to the file transfer time predictor method step of reprocessing grid node is:
Step 4.2.2.2.1 with the forecast area of initial value sample data with give the correct time in advance and be made as input, carries out to find this model predictions the experience database and estimate output file size S from the model predictions program;
Step 4.2.2.2.2, the monitor service MDS that calls reprocessing place grid node obtain real-time by model predictions service place grid node transmission speed V during to the network implementation of reprocessing place grid node;
Step 4.2.2.2.3 is by calculating t
Tm=S/V estimates the model predictions program, and this estimates that output file is to the transmission time of reprocessing place grid node.
Step 4.2.2.3, with the area size of current initial value sample data with give the correct time in advance and be made as parameter query model predictions program and carry out experience database, obtain this model predictions program with the record tabulation pmList of CPU number of variations, it is degree of parallelism, speed-up ratio and the Estimated Time Of Operation of the present mode prediction program execution of parameter that each record of pmList comprises with the CPU number;
Step 4.2.2.4, the pmList and the admissible time t that are obtained with step 4.2.2.3
AllowAs parameter, solve cost optimization CPU number cpu_num=2
m, satisfy condition: cpu_num≤M,
Set A S={j|pmList[j] .time≤t
Allow, j is 2 integral number power }.PmList[i] the degree of parallelism field of the i item record that records among the pmList for tabulation of .pr, the degree of parallelism when expression CPU number is i.
Step 4.3, the model predictions service starts the local model predictions program that is installed in, and sending " model predictions program start " message to the sample data management service, field start-up time that the sample data management service is upgraded in these initial value sample data corresponding element data is the current time.
Step 4.4, the event of finishing that each this model predictions program of model predictions service execution is monitored in service end QoS control service also arranges timer for the model predictions service of firm start-up mode prediction program.Step is as follows:
Step 4.4.1 during model predictions service start-up mode prediction program, sends the QoS control service that " model predictions program start " message is given service end simultaneously, and QoS control service starts this timer for this model predictions service;
Step 4.4.2, the model predictions service of start-up mode prediction program of QoS control service monitoring, if in the maximum tolerance time that the user sets in advance, do not receive the complete event of model predictions program of certain model predictions service, then cancel this model predictions service, and this model predictions is served the performed corresponding metadata state of current initial value sample data file put 0, thereby this initial value sample data file has been carried out release, sent the model predictions service that broadcast notifies other that computing capability is arranged; Otherwise QoS control service keeps listening state; If in the not overtime time, listen to " the model predictions program is complete " event of some model predictions services, QoS control service call sample data management service is upgraded " the model predictions program the is complete " event of transmission, with the initial value sample data file of performed this model predictions program of this model predictions service accordingly the metadata state be set to 3, send " the model predictions program successful execution that has the initial value sample data to be served by model predictions " message simultaneously and give the sample data management service, service end sample data management service with current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID as parameter, query sample data file metadatabase, if all sample data file metadata states are 3, changeed for the 5th step, otherwise change step 4.5.
Step 4.5, the model predictions service is after " the model predictions program the is complete " event of generation, finish the model predictions program implementation of current initial value sample data file for input, so far finished the complete two-layer scheduling process of a pair of initial value sample data file, change step 3.1.
Step 4.6 stops the service of this model predictions, and sends " free time " message to the sample data management service, to inform the current run-stopping status that is in of this model predictions of sample data management service service.
In the 5th step, the sample data management service of service end stops this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM scheduling process, calls the reprocessing external call interface of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process to start the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process.So far all initial value sample datas that produce of having finished this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM are scheduled for the two-layer implementation of model predictions effectively.
Adopt the present invention can reach following technique effect:
(1) the present invention the 3rd finishes the initial value sample according to the real-time computing dynamic competition initial value sample data of each grid node the model predictions that is deployed in each grid node is selected in the step, the 4th step is in the assignment procedure to the intra-node number of resources, also be to optimize appointment according to the idle condition of current grid node in real time, so the present invention have real-time dynamic characteristic.
(2) the present invention dynamically carried out the selection of grid node in the 3rd step, when selecting, the grid node internal resource of step 4.2.1 in the 4th step selects maximum idling-resource as far as possible, step 4.2.2 does cost optimization in ageing and specifies not influencing whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, therefore can utilize existing idling-resource, improve the ageing of single DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM.
(3) step 4.2.2 of the present invention adopts does not influence under the ageing prerequisite of whole DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, the model predictions service of each grid node is according to degree of parallelism and the speed-up ratio of this model predictions program, select the higher cpu resource number of degree of parallelism as far as possible, therefore improve the execution efficient of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, saved resources costs.
(4) the present invention uses the Web service technology that is easy to fast integration to design and realizes two-layer dispatching patcher, therefore the integrated and Web service of the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow that realizes of convenient and variety of way itself has good modular design method, therefore is easy to be understood by general software developer and develop realization.
Description of drawings
Fig. 1 ordinary numeric value weather forecast flow chart;
Fig. 2 DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow chart;
Fig. 3 two-layer dynamic scheduling system structure chart of the present invention;
Fig. 4 overview flow chart of the present invention.
Flow chart.
Embodiment
Fig. 1 represents common numerical weather forecast flow chart, and it comprises the observational data preliminary treatment, Variational Data Assimilation for Meteorology, and model predictions, reprocessing, product is visual etc.
Fig. 2 represents the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow chart, compare with common numerical weather forecast flow process, after Variational Data Assimilation for Meteorology, increased the step of initial disturbance, this step will produce the initial value sample data more than 50 pairs even 100 pairs, every pair of initial value sample all needs to carry out model predictions, so its calculating scale is tens times even hundreds of times of ordinary numeric value weather forecast.
Fig. 3 is two-layer dynamic scheduling system structure chart of the present invention, two-layer dynamic scheduling system is arranged between the initial disturbance and reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM workflow, be responsible for the initial value sample data is managed, the approach of providing the dynamic cooperation competition to obtain the initial value sample data of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM by ability for the model predictions on the grid node that is deployed in the strange land that respectively distributes, dynamic number of resources of specifying each intra-node simultaneously.In case all initial value sample datas have all been carried out model predictions, two-layer dynamic scheduling system is responsible for starting the reprocessing of corresponding DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance.Two-layer dynamic scheduling system is realized based on Web service technology and grid, be installed in service end and each grid node end, wherein service end has been disposed four Web services and two databases, and these four Web services comprise sample data filing service, sample data management service, QoS control service and estimate service.Two databases are respectively that sample data file metadata storehouse and model predictions program are carried out experience database, the former stores initial value sample data metadata information, support sample data filing service and sample data management service, manage the initial value sample data that each DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM produces, the latter stores the input parameter (forecast area that each model predictions program is carried out, the system of giving the correct time in advance etc.), depend on the historical time of implementation mean value of concrete CPU number (2 integral number powers) and the output file size that produces, degree of parallelism, recorded informations such as speed-up ratio are to support to estimate service estimating the model predictions program execution time; Node side is made up of a plurality of grid nodes, and each grid node is deployed with two Web services, comprises model predictions service and the heuristic specified services of intra-node number of resources.Two-layer dynamic scheduling system starts after the initial disturbance of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow.The sample data filing service of service end receives the initial value sample data that initial disturbance produces, and the initial value sample data is filed; This DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM sample data is obtained in model predictions service competition on each grid node of sample data management service notice node side; The model predictions service receives after " competition initial value sample data " notice that the sample data management service sends, and sends " obtaining the initial value sample data " request to the sample data management service; The sample data management service verifies by calling QoS control service whether the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM service place grid node of asking satisfies minimum QoS demand, after the success of QoS service verification, each model predictions service is read, is locked in the mode of concurrent control and transmits the initial value sample data, after the model predictions service obtains the initial value sample data, namely finish two-layer dynamic scheduling system to the selection of grid node, finished the ground floor scheduling: to the selection of grid node; The model predictions service starts the heuristic specified services of local node internal resource number, the number of resources of carrying out this model predictions service is specified on the service of the estimating dynamic optimization ground of service end, model predictions service start-up mode prediction program is finished second layer scheduling of the present invention: specify the number of resources in the grid node then.All initial value sample datas of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM be scheduled for each model predictions service execution intact after, triggered next task---the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow by the sample data management service of service end.
Fig. 4 is general flow chart of the present invention, comprises five key steps:
The first step makes up two-layer dynamic scheduling system, and this system is arranged on service end and each grid node end.
In second step, two-layer dispatching patcher is that the scheduling process of current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is carried out initialization.
The 3rd step, the initial value sample data file that the model predictions service of various places grid node is not also consumed to sample data management service dynamic competition according to the Practical Calculation ability of oneself.
In the 4th step, grid node is optimized number of resources and the start-up mode prediction program of designated mode prediction program.
In the 5th step, the sample data management service of service end stops this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM scheduling process, calls the reprocessing external call interface of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process to start the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process.
Claims (2)
1. two-layer dynamic dispatching method of using towards DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is characterized in that may further comprise the steps:
The first step, make up two-layer dynamic scheduling system, this system is made up of service end and node side, the initial value sample data that service end management initial disturbance produces is also controlled the execution of each model predictions service of node side, its deploy four Web services and two databases: four Web services refer to that sample data filing service, sample data management service, service quality control service are that service is served and estimated in QoS control; Two databases are respectively that sample data file metadata storehouse and model predictions program are carried out experience database, sample data file metadata library storage initial value sample data metadata information, the model predictions program is carried out size, degree of parallelism, the speed-up ratio of the input parameter of the each model predictions program execution of empirical data library storage, historical time of implementation mean value and output file, described input parameter refers to forecast area, the system of giving the correct time in advance; Node side is made up of a plurality of grid nodes, and each grid node is deployed with two Web services, comprises model predictions service and the heuristic specified services of intra-node number of resources, and the model predictions program of local DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process has been packed in the model predictions service; Two-layer dynamic scheduling system starts after the initial disturbance of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow; The sample data filing service of service end receives the initial value sample data that initial disturbance produces, and the initial value sample data is filed; This DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM sample data is obtained in model predictions service competition on each grid node of sample data management service notice node side; The model predictions service sends " obtaining the initial value sample data " request to the sample data management service after receiving " competition initial value sample data " notice that the sample data management service sends; The sample data management service verifies by calling QoS control service whether the model predictions service place grid node of asking satisfies minimum QoS demand, after the success of QoS service verification, each model predictions service is read, is locked in the mode of concurrent control and transmits the initial value sample data, after the model predictions service obtains the initial value sample data, namely finish two-layer dynamic scheduling system to the selection of grid node, finished the ground floor scheduling: to the selection of grid node; The model predictions service starts the heuristic specified services of local node internal resource number, the number of resources of carrying out this model predictions service is specified in the service of estimating of service end, model predictions service start-up mode prediction program is finished second layer scheduling: specify the number of resources in the grid node then; All initial value sample datas of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM be scheduled for each model predictions service execution intact after, triggered the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow by the sample data management service of service end;
In second step, two-layer dispatching patcher is that the scheduling process of current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM is carried out initialization, comprises following three steps:
Step 2.1, after the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process was finished initial disturbance, the DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID that is moved with this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM was parameter, " sample data filing " request that sends is filed service to start two-layer dispatching patcher to sample data; The DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance refers to once specifically carrying out of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process; DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID is produced at random by this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM application flow example, sign that can this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance of unique identification;
Step 2.2, sample data filing service are used metadata approach to adopt the mode of metadata record to be described to every pair of initial value sample data and metadata record are filed sample data file metadata storehouse; The metadata record of every pair of initial value sample data comprises: DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID, initial value sample data numbering, the source grid node is the grid node storage URL of initial value sample initial storage, the sample state, target save mesh node URL, the model predictions program start time, wherein URL is the abbreviation of URL(uniform resource locator); Initial value sample data numbering is to be produced at random by sample data filing service, is used for this numbering to initial value sample data file of unique identification; Local storage URL represents that this is to the local memory location of initial value sample data; The sample state comprises 0,1,2,3, and state 0 is initial value sample data initial condition, represents that this is not also occupied by any model predictions service the initial value sample data; State 1 is initial value sample data lock-out state, represents that this has been distributed on model predictions service on certain grid node to the initial value sample data and has occupied but be not transferred on the grid node; State 2 be the initial value sample data by " consumption " state, represent that this is occupied by certain model predictions service the initial value sample data and successfully is transferred on the grid node; The model predictions program successful execution that state 3 these initial value sample datas of expression have been served by model predictions; Target storage URL represents the address of the grid node that this is transferred to the initial value sample data; Model predictions program start time representation model predictions began with the time of this initial value sample data as input parameter start-up mode forecast application program;
Step 2.3, the sample data management service is notified the model predictions service of all grid nodes based on the message subscribing/issue mechanism of Web service, inform that the present service end of each model predictions service has had the initial value sample data, can compete and obtain to carry out model predictions;
In the 3rd step, the initial value sample data file that the model predictions service of various places grid node is not also consumed to sample data management service dynamic competition according to the Practical Calculation ability of oneself may further comprise the steps:
Step 3.1, the initial value sample data is asked in the model predictions service concomitantly: the model predictions service at first uses the grid middleware software supervision service MDS of grid node to obtain self idling-resource number, whether satisfy the minimum QoS constraint demand of model predictions then according to self Practical Calculation proficiency testing, if satisfy, the model predictions service initiatively sends " application initial value sample data " request message to the sample data management service of service end, this request message content comprises: local grid node URL, model predictions service URI, URI is the abbreviation of unified resource identifier; If do not satisfy, then change step 3.3;
Step 3.2, service end sample data management service is called QoS control service the restrictive condition of QoS of model predictions service place grid node is verified: QoS control service call is deployed in the grid middleware software supervision service MDS on the grid node, whether the model predictions service place grid node of judging application satisfies the minimum constraint requirements of the desired QoS of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM, if find that grid node does not satisfy the minimum constraint requirements of QoS, then send " refusal " message to the model predictions service of asking, after the model predictions service receives " refusal " message, time-out changes step 3.3 to the model predictions service of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance; If this model predictions service place grid node of service end QoS control service discovery satisfies the minimum constraint requirements of QoS, it then is a pair of initial value sample data file under the current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance of this model predictions service competition ID, method is to sample data file metadata library searching meta data entries by the sample data management service, if finding not have state is 0 metadata record, then the model predictions service to request sends " not having retrievable initial value sample data file " message, after the model predictions service receives " not having retrievable initial value sample data file " message, stop application and obtain the initial value sample data, change step 4.6; If it is 0 metadata record that state is arranged, then enter step 3.4;
Step 3.3, model predictions service are regularly constantly monitored the local resource idle condition circularly, whether satisfy the minimum constraint requirements of QoS with checking local grid node, if satisfy, change step 3.1, otherwise continue circulation step 3.3;
Step 3.4, the initial value sample data locking protection of model predictions service to obtaining of request, method is that the sample data management service is carried out transactions access operation, for the model predictions service of asking occupies the corresponding metadata record of this initial value sample data, with this initial value sample data of model predictions service request that prevents other, revising this initial value sample data institute corresponding states then is 1;
Step 3.5, the sample data management service starts the end-to-end transmission of initial value sample data to the model predictions service of application, method is: the sample data management service is called grid middleware assembly reliable file transmission service RFT, the service node that is started initial value sample data place by RFT transmits to the reliable file the grid node of model predictions place, if initial value sample data file transfer failure, then carry out limited number of time ground and restart transmission, as still bust this, this initial value sample data file access authority of release then, it is 0 that the sample data management service is revised initial value sample data state, allow to be deployed on other grid nodes the model predictions service request this to the initial value sample data; Initial value sample data file is as successfully transmission, and the model predictions service sends initial value sample data " transmission is finished " message to the sample data management service, and it is 2 that the sample data management end is revised this initial value sample data institute corresponding states; So far, finish the selection problem of initial value sample data to grid node, finished the ground floor scheduling;
In the 4th step, grid node is optimized number of resources and the start-up mode prediction program of designated mode prediction program, and step is as follows:
Step 4.1, the heuristic specified services of intra-node number of resources starts estimates service valuation, estimates it for present mode forecast service and whether also might obtain down a pair of initial value sample data file, and concrete steps are:
Step 4.1.1, the heuristic specified services of intra-node number of resources estimate self model predictions program deadline and send " estimate request information to estimating service, step is as follows:
Step 4.1.1.1, the heuristic specified services of intra-node number of resources is carried out experience database based on the model predictions program and is estimated present mode forecast this performed model predictions program deadline of service according to initial value sample data forecast parameter the information----forecast area of current acquisition and the system of giving the correct time in advance, current grid node idle available cpu resource information and memory information;
Step 4.1.1.2, the model predictions service is published to present mode forecast this performed model predictions program deadline of service of estimating the service of estimating of service end;
Step 4.1.1.3, the model predictions service send whether may obtain down a pair of effective initial value sample data " estimate request information to estimating service;
Step 4.1.2 estimates service broadcasting renewal and estimates information, and the model predictions service that obtains other grid node estimates to finish the temporal information of present mode prediction program; Estimate service receive " after estimating request information, dynamically ask other just in the model predictions service of execution pattern prediction program issue " Estimated Time Of Completion " information to estimating service; After broadcasting " Estimated Time Of Completion " message is received in each model predictions service, carry out estimating as step 4.1.1 is described according to the real-time implementation status of the performed model program of this model predictions and self computing capability, after all model predictions services had been estimated, the result with Estimated Time Of Completion after will estimating separately sent to the service of estimating;
Step 4.1.3 estimates service execution about the process of estimating of a pair of initial value sample data possibility under the model predictions acquisition of request, and method is:
Step 4.1.3.1, with all model predictions service Estimated Time Of Completions according to by early to late rank order, thereby draw transmission " estimate the order of priority k of the model predictions service of request information; k is positive integer, expression send " the model predictions service center that estimates request information in all model predictions services about time of estimating to finish present mode prediction program separately according to by early to the sequence of positions of arrangement in evening;
Step 4.1.3.2, estimate service call sample data management service and obtain effective initial value sample data, all metadata item records of sample data management service search initial value sample data, obtain present initial value sample data file status and be all metadata record numbers of 0, be designated as n, n is positive integer;
Step 4.1.3.3, if satisfy condition k≤n, then estimation results is returned True, otherwise returns False; " estimate the model predictions service of request information, the model predictions service is handed to the heuristic specified services of local node internal resource number with estimation results subsequently estimation results to be sent to transmission with form of message;
Step 4.2, the estimation results that the heuristic specified services of grid node internal resource number sends according to the service estimated selects assignment algorithm to specify the cpu resource number, if the estimation results of returning is True, then changes step 4.2.1; If the estimation results of returning is False, then change step 4.2.2:
Step 4.2.1, the cpu resource number assignment procedure that the heuristic specified services time of implementation of local grid intra-node number of resources is preferential: specify big as far as possible cpu resource number, suppose that current idle cpu resource number is M, M is positive integer, and then the cpu resource number that obtains according to this assignment procedure is cpu_num=2
m, wherein 2
m≤ M, 2
M+1M, m is positive integer;
Step 4.2.2, the cpu resource number assignment procedure of the heuristic specified services time of implementation-balance of efficiency of local node internal resource number the steps include:
Step 4.2.2.1 adopts and estimates the Late Finish t that all initial value sample datas of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM are performed based on the method for history
l, step is as follows:
Step 4.2.2.1.1 to each model predictions service i, according to the order of arranging among the step 4.1.3.1, obtains the obtainable execution initial value sample data umber num of model predictions service
i
Step 4.2.2.1.2 is to each model predictions service i, according to the actual max calculation ability cpu_num=2 of this model predictions place grid node
m, forecast area size, the system of giving the correct time in advance, carry out experience database based on the model predictions program, inquire the scheduled time temp of this model predictions service " consumption " every part of initial value sample data, thereby obtain the fastest Estimated Time Of Completion T that model predictions is served i
i=temp * num
i
Step 4.2.2.1.3 obtains each model predictions service maximum of fast Estimated Time Of Completion
Wherein S represents all model predictions set of services;
Step 4.2.2.2 is according to current time t
c, adopt single model predictions program output file to obtain this model predictions program admissible running time of t to the file transfer time predictor method of reprocessing grid node
Allow=t
l-t
c-t
Tm, t wherein
TmExpression present mode prediction program output file is transferred to estimated time of reprocessing place grid node by current grid node, and single model predictions program output file to the file transfer time predictor method step of reprocessing grid node is:
Step 4.2.2.2.1 with the forecast area of initial value sample data with give the correct time in advance and be made as input, carries out to find this model predictions the experience database and estimate output file size S from the model predictions program;
Step 4.2.2.2.2, the monitor service MDS that calls reprocessing place grid node obtain real-time by model predictions service place grid node transmission speed V during to the network implementation of reprocessing place grid node;
Step 4.2.2.2.3 is by calculating t
Tm=S/V estimates the model predictions program, and this estimates that output file is to the transmission time of reprocessing place grid node;
Step 4.2.2.3, with the area size of current initial value sample data with give the correct time in advance and be made as parameter query model predictions program and carry out experience database, obtain this model predictions program with the record tabulation pmList of CPU number of variations, it is degree of parallelism, speed-up ratio and the Estimated Time Of Operation of the present mode prediction program execution of parameter that each record of pmList comprises with the CPU number;
Step 4.2.2.4 is with pmList and admissible time t
AllowAs parameter, solve cost optimization CPU number cpu_num=2
m, satisfy condition: cpu_num≤M,
Set A S={j|pmList[j] .time≤t
Allow, j is 2 integral number power }, pmList[i] the degree of parallelism field of the i item record that records among the pmList for tabulation of .pr, the degree of parallelism when expression CPU number is i;
Step 4.3, the model predictions service starts the local model predictions program that is installed in, and sending " model predictions program start " message to the sample data management service, field start-up time that the sample data management service is upgraded in these initial value sample data corresponding element data is the current time;
Step 4.4, the event of finishing of each this model predictions program of model predictions service execution is monitored in service end QoS control service and for the model predictions service of firm start-up mode prediction program arranges timer, step is as follows:
Step 4.4.1 during model predictions service start-up mode prediction program, sends the QoS control service that " model predictions program start " message is given service end simultaneously, and QoS control service starts this timer for this model predictions service;
Step 4.4.2, the model predictions service of start-up mode prediction program of QoS control service monitoring, if in the maximum tolerance time that the user sets in advance, do not receive the complete event of model predictions program of certain model predictions service, then cancel this model predictions service, and this model predictions is served the performed corresponding metadata state of current initial value sample data file put 0, thereby this initial value sample data file has been carried out release, sent the model predictions service that broadcast notifies other that computing capability is arranged; Otherwise QoS control service keeps listening state; If in the not overtime time, listen to " the model predictions program is complete " event of some model predictions services, QoS control service call sample data management service is upgraded " the model predictions program the is complete " event of transmission, with the initial value sample data file of performed this model predictions program of this model predictions service accordingly the metadata state be set to 3, send " the model predictions program successful execution that has the initial value sample data to be served by model predictions " message simultaneously and give the sample data management service, service end sample data management service with current DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow instance ID as parameter, query sample data file metadatabase, if all sample data file metadata states are 3, changeed for the 5th step, otherwise change step 4.5;
Step 4.5, the model predictions service is after " the model predictions program the is complete " event of generation, finish the model predictions program implementation of current initial value sample data file for input, so far finished the complete two-layer scheduling process of a pair of initial value sample data file, change step 3.1;
Step 4.6 stops the service of this model predictions, and sends " free time " message to the sample data management service;
In the 5th step, the sample data management service of service end stops this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM scheduling process, calls the reprocessing external call interface of this DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process to start the reprocessing of DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM flow process.
2. a kind of two-layer dynamic dispatching method of using towards DATA PROCESSING IN ENSEMBLE PREDICTION SYSTEM as claimed in claim 1 is characterized in that the minimum constraint requirements of described QoS comprises idle CPU number and memory size.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110092776 CN102185761B (en) | 2011-04-13 | 2011-04-13 | Two-layer dynamic scheduling method facing to ensemble prediction applications |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110092776 CN102185761B (en) | 2011-04-13 | 2011-04-13 | Two-layer dynamic scheduling method facing to ensemble prediction applications |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102185761A CN102185761A (en) | 2011-09-14 |
| CN102185761B true CN102185761B (en) | 2013-08-07 |
Family
ID=44571832
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110092776 Expired - Fee Related CN102185761B (en) | 2011-04-13 | 2011-04-13 | Two-layer dynamic scheduling method facing to ensemble prediction applications |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102185761B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013107012A1 (en) * | 2012-01-18 | 2013-07-25 | 华为技术有限公司 | Task processing system and task processing method for distributed computation |
| CN106878039A (en) * | 2015-12-14 | 2017-06-20 | 中兴通讯股份有限公司 | A kind of version task scheduling and managing method and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101662419A (en) * | 2008-08-27 | 2010-03-03 | 华为技术有限公司 | Method, device and system for realizing resource reservation |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040225736A1 (en) * | 2003-05-06 | 2004-11-11 | Raphael Roger C. | Method and apparatus for providing a dynamic quality of service for serving file/block I/O |
-
2011
- 2011-04-13 CN CN 201110092776 patent/CN102185761B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101662419A (en) * | 2008-08-27 | 2010-03-03 | 华为技术有限公司 | Method, device and system for realizing resource reservation |
Non-Patent Citations (2)
| Title |
|---|
| 《A Scientific-Workflow-Based Execution Environment for Ensemble Prediction》;刘灿灿等;《Grid and Pervasive Computing Workshops, 2008. GPC Workshops 08. The 3rd International Conference on》;20080528;全文 * |
| 刘灿灿等.《A Scientific-Workflow-Based Execution Environment for Ensemble Prediction》.《Grid and Pervasive Computing Workshops, 2008. GPC Workshops 08. The 3rd International Conference on》.2008,全文. |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102185761A (en) | 2011-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104915407B (en) | A kind of resource regulating method based under Hadoop multi-job environment | |
| CN101957780B (en) | Resource state information-based grid task scheduling processor and grid task scheduling processing method | |
| Ge et al. | GA-based task scheduler for the cloud computing systems | |
| CN101458634B (en) | Load equilibration scheduling method and device | |
| CN102508709B (en) | Distributed-cache-based acquisition task scheduling method in purchase, supply and selling integrated electric energy acquiring and monitoring system | |
| CN108021435B (en) | Cloud computing task flow scheduling method with fault tolerance capability based on deadline | |
| CN102591921A (en) | Scheduling and management in a personal datacenter | |
| CN110086855A (en) | Spark task Intellisense dispatching method based on ant group algorithm | |
| CN103999114B (en) | Energy management strategy device and energy management method | |
| CN111427675B (en) | Data processing method and device and computer readable storage medium | |
| CN102624865A (en) | Cluster load prediction method and distributed cluster management system | |
| CN108170530A (en) | A kind of Hadoop Load Balancing Task Scheduling methods based on mixing meta-heuristic algorithm | |
| CN103116525A (en) | Map reduce computing method under internet environment | |
| Wei et al. | Adaptive service workflow configuration and agent-based virtual resource management in the cloud | |
| CN102185761B (en) | Two-layer dynamic scheduling method facing to ensemble prediction applications | |
| Han et al. | EdgeTuner: Fast scheduling algorithm tuning for dynamic edge-cloud workloads and resources | |
| CN113190342B (en) | Method and system architecture for multi-application fine-grained offloading of cloud-edge collaborative networks | |
| CN108234617B (en) | Dynamic resource scheduling method in hybrid cloud mode for power system | |
| Jin-Gang et al. | Research and application of auto-scaling unified communication server based on Docker | |
| CN112486657A (en) | Task scheduling method and system | |
| Zhou et al. | AHPA: adaptive horizontal pod autoscaling systems on alibaba cloud container service for kubernetes | |
| CN102929689A (en) | Method for building virtual machine in cloud computing system | |
| CN102662732A (en) | Double-drive thread pool model | |
| Tang et al. | Dynamic scheduling management method for multi-energy system digital twin simulation computing tasks | |
| CN114240395A (en) | Resource delivery method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130807 Termination date: 20150413 |
|
| EXPY | Termination of patent right or utility model |