CN117724858A - Geospatial analysis model scheduling method, device, equipment and medium - Google Patents

Geospatial analysis model scheduling method, device, equipment and medium Download PDF

Info

Publication number
CN117724858A
CN117724858A CN202410179395.XA CN202410179395A CN117724858A CN 117724858 A CN117724858 A CN 117724858A CN 202410179395 A CN202410179395 A CN 202410179395A CN 117724858 A CN117724858 A CN 117724858A
Authority
CN
China
Prior art keywords
model
scheduling
analysis model
network
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410179395.XA
Other languages
Chinese (zh)
Other versions
CN117724858B (en
Inventor
杨岸然
刘万涛
张志群
李军
王轩
谢婷萱
袁丽红
牛艺伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin Institute Of Advanced Technology
National University of Defense Technology
Phytium Technology Co Ltd
Original Assignee
Tianjin Institute Of Advanced Technology
National University of Defense Technology
Phytium Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin Institute Of Advanced Technology, National University of Defense Technology, Phytium Technology Co Ltd filed Critical Tianjin Institute Of Advanced Technology
Priority to CN202410179395.XA priority Critical patent/CN117724858B/en
Priority claimed from CN202410179395.XA external-priority patent/CN117724858B/en
Publication of CN117724858A publication Critical patent/CN117724858A/en
Application granted granted Critical
Publication of CN117724858B publication Critical patent/CN117724858B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The application relates to a method, a device, equipment and a medium for scheduling a geospatial analysis model, wherein a resource data set occupied by the analysis model is obtained; constructing a model classification training network, and training the model classification training network by analyzing a model occupation resource data set to obtain a trained model classification network; a model scheduling generator is constructed, scheduling combination is carried out on each analysis model through the model scheduling generator, and an analysis model calling strategy state data set is generated; constructing a model calling strategy training network, and training the model calling strategy training network by analyzing a model calling strategy state data set to obtain a trained model calling strategy network; and constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system. The intelligent NUMA parallel scheduling method and the intelligent NUMA parallel scheduling system realize intelligent NUMA parallel scheduling of different geospatial analysis models.

Description

Geospatial analysis model scheduling method, device, equipment and medium
Technical Field
The present disclosure relates to the field of geographic information technologies, and in particular, to a method, an apparatus, a device, and a medium for scheduling a geospatial analysis model.
Background
In the technical field of geographic information, the demand of a geographic space analysis model for computing resources is different, and the traditional manual experience scheduling is difficult to adapt to the diversity of model types, so that the utilization rate of the computing resources is low. And NUMA architecture in the multi-core environment brings new optimization space for the geospatial analysis model scheduling. Therefore, how to implement intelligent adaptive scheduling of the geospatial analysis model according to the geospatial analysis model type and the system resource state is a challenge to be solved.
At present, the method for multi-element model scheduling mainly comprises a model scheduling method based on manual experience, a static analysis method based on a performance model and a rule scheduling method based on a control theory. The scheduling effect of the manual empirical method depends on expert knowledge, but is difficult to adapt to the continuous change of the model type; the static analysis method has the problem of difficult modeling of the complex model; the method based on the control theory is complex in scheduling and difficult to popularize. In general, the existing model scheduling method does not comprehensively consider the model type, the system resource state and the matching association of NUMA parallel strategies, and cannot effectively realize intelligent self-adaptive scheduling of the multi-element model.
Disclosure of Invention
Based on the foregoing, it is necessary to provide a method, an apparatus, a device, and a medium for scheduling a geospatial analysis model, which can automatically schedule the analysis model, and implement efficient processing and accurate analysis of geographic information.
A geospatial analysis model scheduling method, the method comprising:
acquiring an analysis model occupied resource data set;
constructing a model classification training network, and training the model classification training network through the analysis model occupation resource data set to obtain a trained model classification network;
a model scheduling generator is constructed, scheduling combination is carried out on each analysis model through the model scheduling generator, and an analysis model calling strategy state data set is generated;
constructing a model calling strategy training network, and training the model calling strategy training network through the analysis model calling strategy state data set to obtain a trained model calling strategy network;
and constructing a model scheduling controller, wherein the model scheduling controller generates an optimal analysis model scheduling scheme according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
A geospatial analysis model scheduling apparatus, the apparatus comprising:
the data acquisition module is used for acquiring the resource data set occupied by the analysis model;
the model classification network training module is used for constructing a model classification training network, and training the model classification training network through the analysis model occupied resource data set to obtain a trained model classification network;
the model scheduling generator construction module is used for constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set;
the model calling strategy network training module is used for constructing a model calling strategy training network, and training the model calling strategy training network through the analysis model calling strategy state data set to obtain a trained model calling strategy network;
the scheduling scheme generating module is used for constructing a model scheduling controller, and the model scheduling controller generates an optimal analysis model scheduling scheme according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
Step 202, acquiring an analysis model occupied resource data set.
And 204, constructing a model classification training network, and training the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
And 206, constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set.
And step 208, constructing a model calling strategy training network, and training the model calling strategy training network by analyzing the model calling strategy state data set to obtain a trained model calling strategy network.
And step 210, constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
step 202, acquiring an analysis model occupied resource data set.
And 204, constructing a model classification training network, and training the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
And 206, constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set.
And step 208, constructing a model calling strategy training network, and training the model calling strategy training network by analyzing the model calling strategy state data set to obtain a trained model calling strategy network.
And step 210, constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
The method, the device, the equipment and the medium for scheduling the geospatial analysis model occupy a resource data set by acquiring the analysis model; constructing a model classification training network, and training the model classification training network by analyzing a model occupation resource data set to obtain a trained model classification network; a model scheduling generator is constructed, scheduling combination is carried out on each analysis model through the model scheduling generator, and an analysis model calling strategy state data set is generated; constructing a model calling strategy training network, and training the model calling strategy training network by analyzing a model calling strategy state data set to obtain a trained model calling strategy network; and constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
According to the intelligent NUMA parallel scheduling method and the intelligent NUMA parallel scheduling system, intelligent NUMA parallel scheduling of different geospatial analysis models is achieved through combination of a constructed model classification network, a strategy network and a model scheduling controller. In the implementation process, the advantages of scheduling strategy experience are drawn, meanwhile, the method is different from a control theory method, scheduling automation is realized by deep learning, high-efficiency processing and accurate analysis of geographic information are realized, a brand new thought is provided for calling different analysis models, and the method has important application value.
Drawings
FIG. 1 is a flow diagram of a geospatial analysis model scheduling method in one embodiment;
FIG. 2 is a schematic diagram of a geospatial analysis model type evaluation tree structure in one embodiment;
FIG. 3 is a schematic diagram of a framework of a geospatial analysis model scheduling method in one embodiment;
FIG. 4 is a block diagram of a geospatial analysis model scheduling apparatus in one embodiment;
fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
It should be noted that the description herein as relating to "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.
In the technical process of realizing the scheme, the inventor finds that in the existing multi-element model scheduling method, some model type persistence changes are difficult to adapt, some modeling problems exist, and some scheduling modes are complex, so that the existing model scheduling method is not flexible enough in actual operation, and intelligent self-adaptive scheduling of the multi-element model cannot be effectively realized. Based on the above, the inventor provides a geospatial analysis model scheduling method, which comprises the steps of respectively establishing a model classification network and a model calling strategy network, then judging and classifying model types according to the model classification network, and outputting NUMA binding strategies according to the model calling strategy network. And finally, constructing a model scheduling controller, and combining the model classification network and the model calling strategy network through the model scheduling controller to generate different NUMA parallel scheduling schemes. Compared with the existing manual experience scheduling, the method has the advantages that automation and intellectualization are realized, the execution efficiency of different types of geospatial analysis models is greatly improved, the efficient processing and accurate analysis of the geographic information are realized, an effective method is provided for model self-adaptive scheduling in heterogeneous multi-core environments, and the performance and efficiency of multi-core computing platforms in the field of geospatial analysis are expected to be improved.
The embodiments of the present application will be described in detail below with reference to the accompanying drawings in the drawings of the embodiments of the present application.
In one embodiment, as shown in fig. 1, there is provided a geospatial analysis model scheduling method, including the steps of:
step 202, acquiring an analysis model occupied resource data set.
It can be appreciated that by using the obtained analysis model to occupy the resource data set, performance evaluation and comparison can be performed on different analysis models, so as to help effectively allocate resources and predict performance under different hardware configurations.
Specifically, according to the system resource occupation condition of the monitoring geospatial analysis model, the model type is marked by using a model type evaluation tree method, so as to manufacture an analysis model occupation resource data set
The model type evaluation tree method is to count the data of the system resource occupation condition according to a geographic space analysis model, classify the data according to indexes such as CPU occupation rate (divided into a low level and a high level), memory occupation size (divided into a low level and a high level), execution time (divided into a slow level and a fast level) and the like, and combine the classified indexes to form the model type evaluation tree.
And the analysis model type labeling is to label records of the system resource data set occupied by each analysis model according to the model type evaluation tree method. This dataset includes CPU occupancy, memory capacity occupancy, execution time, and model type tags that analyze the system resources occupied by the model process. It is noted that the CPU occupancy rate and the memory capacity occupancy size are the maximum values in the execution period of the process taking the analysis model.
Through the indexes, the performance of various analysis models on resource utilization can be known, so that the most suitable analysis model is selected, resource waste or unnecessary performance loss is avoided, resource management is effectively promoted, and the overall performance of the system is improved.
And 204, constructing a model classification training network, and training the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
It is worth to say that, based on the model classification training network constructed by BP neural network, the complex relation in the data set is understood by extracting the characteristic and mode of the analysis model occupation resource data set, so as to better understand the difference of different analysis model types in the resource occupation to obtain the type classification capability of the geospatial analysis model, provide guidance for the subsequent selection of the proper analysis model, and when the analysis model is scheduled, a more proper calling strategy can be formulated according to the type and characteristic of the analysis model.
And 206, constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set.
It can be understood that by recording the call combination conditions of different analysis models, the resource consumption conditions of the analysis models under various call combinations can be known so as to optimize the allocation and utilization of resources. Based on the obtained calling combination of different analysis models, a strategy sample set is selected through a sample selection rule, data support is provided for training of a strategy network for subsequent model calling, a scheduling strategy can be improved, and system performance is optimized.
And step 208, constructing a model calling strategy training network, and training the model calling strategy training network by analyzing the model calling strategy state data set to obtain a trained model calling strategy network.
It can be understood that the model calling strategy network is mainly used for generating NUMA binding strategies so as to better allocate processors for analysis models and realize parallel computation, and on the aspect of processing a large amount of data of analysis models, the resource competition among the processors can be effectively reduced, and the parallel performance of the system is improved. When complex computing tasks are executed under the NUMA architecture, the NUMA binding strategy is generated through the model calling strategy network, so that the memory access is optimized, the performance is improved, the resource management is improved, and the method is particularly suitable for continuously changing system requirements.
And step 210, constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
It can be understood that the model scheduling controller is equivalent to a decision brain and is mainly responsible for managing and controlling the scheduling of analysis models, so that each analysis model can reasonably use system resources to optimize the system performance and resource utilization to the greatest extent, and after integrating all resource data, the overall performance and efficiency of the system are improved by generating a reasonable task scheduling strategy and resource management.
Therefore, the method generates the optimal analysis model scheduling scheme by acquiring the output of the model classification network, the output of the model calling strategy network and the current state of the system, and each analysis model performs calculation tasks based on the optimal analysis model scheduling scheme so as to complete the analysis of the geographic space.
According to the geospatial analysis model scheduling method, the analysis model occupied resource data set is obtained; constructing a model classification training network, and training the model classification training network by analyzing a model occupation resource data set to obtain a trained model classification network; a model scheduling generator is constructed, scheduling combination is carried out on each analysis model through the model scheduling generator, and an analysis model calling strategy state data set is generated; constructing a model calling strategy training network, and training the model calling strategy training network by analyzing a model calling strategy state data set to obtain a trained model calling strategy network; and constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
According to the intelligent NUMA parallel scheduling method and the intelligent NUMA parallel scheduling system, intelligent NUMA parallel scheduling of different geospatial analysis models is achieved through combination of a constructed model classification network, a strategy network and a model scheduling controller. In the implementation process, the advantages of scheduling strategy experience are drawn, meanwhile, the method is different from a control theory method, scheduling automation is realized by deep learning, high-efficiency processing and accurate analysis of geographic information are realized, a brand new thought is provided for calling different analysis models, and the method has important application value.
In one embodiment, obtaining an analysis model occupancy resource dataset includes:
and acquiring the CPU occupancy rate, the memory occupancy size and the execution time of each geographic space analysis model occupying system resources.
And carrying out clustering statistics on the CPU occupancy rate, the memory occupancy size and the execution time, selecting grading boundary conditions, and constructing an analysis model type evaluation tree.
And marking the CPU occupancy rate, the memory occupancy size and the execution time according to the analysis model type evaluation tree to obtain an analysis model label type.
And constructing an analysis model occupation system resource data set through the CPU occupancy rate, the memory occupation size, the execution time and the analysis model label category.
Specifically, the system sequentially runs different geospatial analysis models under the condition that no other execution tasks exist, and obtains the CPU occupancy rate, the memory capacity occupancy size and the execution time of a single analysis model process occupying system resources in the execution process. The CPU occupancy rate and the memory capacity occupancy rate are the maximum value in the execution period of the process of the analysis model, and the data such as the CPU occupancy rate, the memory capacity occupancy rate and the execution time of each analysis model are recorded.
And respectively carrying out k-means clustering statistics on the recorded CPU occupancy rate, memory capacity occupancy size and execution time data, selecting reasonable grading boundary conditions, and constructing an analysis model type evaluation tree. For example, according to the example shown in fig. 2, the CPU occupancy is divided into two levels of low (+.60%) and high (+.60%), the memory capacity occupancy size is divided into two levels of low (+.8g) and high (> 8g), the execution time is divided into two levels of slow (+.10sec) and fast (> 10sec), and then these performance metrics are combined to build an analytical model type evaluation tree.
According to the analysis model type evaluation tree method, the recorded CPU occupancy rate, memory capacity occupancy size and execution time data are utilized to label the analysis model type, and finally the CPU occupancy rate, memory capacity occupancy size and execution time of system resources occupied by the analysis model process are recorded in an analysis model occupied resource data set by the analysis model label typeIs a kind of medium.
In one embodiment, a model classification training network is constructed, the model classification training network is trained by analyzing a model occupation resource data set, and a trained model classification network is obtained, including:
And carrying out parameter standardization on the analysis model occupied resource data set to obtain a standardized resource data set.
And constructing a model classification training network, wherein the model classification training network comprises an input layer, an implicit layer and an output layer.
Training an input layer, an implicit layer and an output layer through a standardized resource data set to obtain a trained model classification network; after receiving the standardized resource data set, the input layer sends the standardized resource data set to the hidden layer for processing, extracts the characteristics in the standardized resource data set, then sends the extracted characteristics to the output layer, and the output layer classifies each analysis model according to the received characteristics and outputs classification results.
In particular, the analytical model occupies a resource datasetExpressed as:
in the method, in the process of the invention,a model type indicating the scheduled execution; />Wherein->CPU occupancy representing the analytical model process, +.>Representing the memory capacity occupation size +.>Representing execution time; />Representing the analytical model tag class.
Input data in an analysis model footprint resource datasetThe parameter normalization is carried out, and the formula of the parameter normalization is expressed as follows:
in the method, in the process of the invention,representing parameters that need to be standardized, including CPU occupancy of analytical model process >Memory capacity occupancy size->Execution time->;/>Representation->Is not limited to the desired one; />Representation->Is a variance of (c). It will be appreciated that normalization is to ensure that the data meets the conditions for model training.
Constructing a model classification training network based on a BP neural network, wherein the network comprises an input layer, an implicit layer and an output layer; the input layer receives 4 standardized parameters, the hidden layer contains 1024 neuron parameters, and the output layer contains 3 parameters for identifying model categories. The model is trained to have the capability of classifying different models.
It is understood that the 3 parameters identifying the model class herein refer to the single thermal encoding of the model type. It should be noted that, different model types may be selected according to the requirement, where 3 identification model type parameters are selected to represent only 8 types in the geospatial analysis model type evaluation tree structure (as shown in fig. 2), which is only an example, and the user may label the algorithm types according to other methods or ways of expert priori knowledge, so as to set the number of independent heat coding parameters for expressing the algorithm types, which is set correspondingly based on the analysis model occupation resource dataset constructed in step 202.
In one embodiment, a model schedule generator is constructed, and scheduling combination is performed on each analysis model by the model schedule generator, so as to generate an analysis model calling strategy state data set, which includes:
a build model schedule generator, expressed as:
in the method, in the process of the invention,representing different sets of geospatial models, +.>Indicate->Reference number of geospatial analysis model of wheel selection, +.>Indicate->The number of NUMA nodes is round-bound.
And generating a plurality of rounds of analysis model scheduling strategies. Under a first round of analysis model scheduling policies, the model scheduling generator distributes analysis models to NUMA nodes in a serial and/or parallel manner to combine the analysis models.
And recording the CPU occupancy rate, the memory occupancy size, the execution time and the used NUMA nodes of each combination, and taking the CPU occupancy rate, the memory occupancy size, the execution time and the used NUMA nodes of each combination as calling strategy state data of each combination under a first round of analysis model scheduling strategy.
And scoring the calling strategy state data of each combination under the first round of analysis model scheduling strategy, and then selecting a strategy sample set according to a sample selection rule.
And generating a second round of analysis model scheduling strategy based on the strategy sample set, and obtaining the calling strategy state data of each combination under the second round of analysis model scheduling strategy through the second round of analysis model scheduling strategy.
And until the calling strategy state data of each combination under all the analysis model scheduling strategies is obtained, forming an analysis model calling strategy state data set.
It will be appreciated that the analysis model schedule generator will be either serially or in parallelThe individual models are assigned to NUMA nodes to implement a combination of analytical model scheduling policies. Under the same analysis model scheduling condition, it records the next strategy using NUMA node analysis model scheduling strategy and scores each strategy according to the descending order of scheduling execution time>,/>. Where N represents the number of NUMA nodes.
Specifically, at step 302, a model schedule generator is built, expressed as:
in the method, in the process of the invention,representing different sets of geospatial models, +.>Indicate->Reference number of geospatial analysis model of wheel selection, +.>Indicate->The number of NUMA nodes is round-bound. When->When the NUMA node is not bound; when->And when the NUMA node is bound, the CPU and the memory of the NUMA node 1 are bound.
Step 304, generating a first round of analysis model scheduling strategyThe model schedule generator assigns analysis models to NUMA nodes in a serial and/or parallel manner, combines the analysis models, and then enforces the policy.
Step 306, for each combination of analysis models, recording CPU occupancy rate before the last analysis model is scheduled(whereiniNumbering system CPU), current memory occupancy status data of the systemo. Then, after the last analysis model is scheduled, the execution time of the analysis model is recordedtAnd NUMA binding numbering used for the analysis model, and labeling the type of the analysis model according to the model type evaluation tree method, and using the same as state data of model calling strategy +.>
Step 308, designing a sample selection rule, where the sample selection rule is: sequencing the calling strategy state data of each combination according to the execution time, giving a score, then selecting the combination of the first three scores, the combination of the second three scores, randomly selecting the three combinations after removing the combination of the first three scores and the combination of the second three scores, and recording the three combinations as a strategy sample set. The set of policy samples thus selected +.>The method has a certain representativeness, and can reduce the complexity of system calculation as the scheduling of the scheduling strategy of the analysis model of the latter round.
It should be noted that the number of combinations selected in the sample selection rule is not limited to 9 combinations selected in the embodiment, and fewer or more combinations may be selected according to practical situations, which depends on the requirements of the user and the performance of the system.
Step 310, generating a second round of analysis model scheduling policies based on the policy sample setReturning to step 306 and invoking the state data of the policy for the model +.>Add to model invocation policy state dataset +.>Is a kind of medium. Then, step S308 is executed to continue to obtain the next round of analysis model scheduling policy and to add the corresponding state data to the model call policy state data set +.>Is a kind of medium.
In one embodiment, a model call policy training network is constructed, the model call policy training network is trained by analyzing a model call policy state data set, and a trained model call policy network is obtained, which includes:
parameter standardization is carried out on the analysis model calling strategy state data set, and a standardized strategy state data set is obtained;
constructing a model calling strategy training network, and presetting a loss function; the model calling strategy training network comprises an input layer, an implicit layer and an output layer;
training an input layer, an implicit layer and an output layer through a standardized strategy state data set and a preset loss function to obtain a trained model calling strategy network; after receiving the standardized policy state data set, the input layer sends the standardized policy state data set to the hidden layer for processing, extracts the features in the standardized policy state data set, then sends the features to the output layer, and the output layer processes the received features to generate an analysis model NUMA binding strategy under different analysis model scheduling strategies.
Specifically, the analytical model invokes the policy state datasetExpressed as:
in the method, in the process of the invention,mrepresenting the number of samples that implement the model scheduling policy,wherein />Numbering the system CPU>Representing the current occupied state data of each CPU of the system;representing current memory occupancy status data->Representing the last model scheduling type->Representing the next model schedule call policy annotation score,/->Representing the last model scheduling policy binding NUMA node number.
Invoking input data in a policy state dataset to an analytical modelThe parameter normalization is carried out, and the formula of the parameter normalization is expressed as follows:
in the method, in the process of the invention,parameters representing the need for standardization, including the use of occupancy status data by the respective CPU>Current memory occupancy status data +.>;/>Representation->Is not limited to the desired one; />Representation->Is a variance of (c). Then, model type->Pressing the subscript number->Converting to one-hot coding. And tag->One-hot encoding representing NUMA node number translation. It will be appreciated that normalization is to ensure that the data meets the conditions for model training.
Constructing a model calling strategy training network based on a BP neural network, wherein the network comprises an input layer, an implicit layer and an output layer; wherein the input layer receives 3 model type coding information parameters, 128 CPU occupation resource information parameters and 1 memory occupation information parameter, and the hidden layer contains 2048 neuron parameters The output layer contains 17 NUMA node binding policy parameters. In addition, a loss function is preset, and a strategy importance score parameter is introduced into the loss function based on the BP neural networkTo improve the learning ability of the model invocation strategy training network to importance strategies.
The predetermined loss function is expressed as:
wherein,is the number of samples, +.>Representation sample->Is the true value of the tag of +.>Representation model sample->Prediction probability->Representing policy score->Representing the super parameter.
In one embodiment, a model scheduling controller is constructed, and the model scheduling controller generates an optimal analysis model scheduling scheme according to an output of a model classification network, an output of a model call policy network and a current system state, and the method comprises the following steps:
the model scheduling controller acquires the currently executed analysis model CPU occupancy rate, the memory occupancy size and the execution time of the system, sends the currently executed analysis model CPU occupancy rate, the currently executed memory occupancy size and the currently executed execution time to the model classification network, classifies the analysis model through the model classification network, and correspondingly updates the analysis model type record based on the classified analysis model data.
When NUMA parallel scheduling is needed, the model scheduling controller reads updated analysis model classification data, sends the updated analysis model classification data to a model calling strategy network, and outputs a NUMA binding strategy through the model calling strategy network; and the model scheduling controller generates an optimal analysis model scheduling scheme according to the updated analysis model classification data, the NUMA binding strategy and the current state of the system.
When the asynchronous scheduling is needed, a model scheduling controller reads a model to be scheduled from a model scheduling queue and judges a binding strategy of the model to be scheduled; the model scheduling controller adds the model to be scheduled to the model classification network for classification based on the binding strategy of the model to be scheduled, and then correspondingly updates the analysis model type record based on the classified model data to be scheduled so as to generate an optimal analysis model scheduling scheme.
Specifically, as shown in fig. 3, a schematic scheduling policy framework diagram of a model scheduling controller in one embodiment is provided, including: model classification network, model call policy network, model dispatch controller, system resource monitor and model classification database.
The working flow is as follows:
step 312: for the executed analysis model, the background program monitors the CPU occupancy rate, memory occupancy size and execution time parameters of the analysis model executed by the current process, takes the parameters as the input of a model classification network, infers the category of the analysis model through the model classification network, and then adds or updates a corresponding analysis model type record in a model classification database by a model scheduling controller.
Wherein, for models that do not exist in the model classification database, the model scheduling controller performs an added operation; for an existing model, determining whether the model type is a model type of potential NUMA scheduling ,/>,/>,/>If the model type is the model type of the potential NUMA schedule, updating the model classification database, otherwise, not operating. It is worth noting that ++here is named>、/>The model call types are only for convenience of explanation, and can be reset according to the situation, and the evaluation tree structure shown in fig. 2 is only one type, and can be changed according to the increase or decrease of the analysis model.
Step 314: for the analysis model which needs NUMA parallel scheduling, the model scheduling controller acquires the updated analysis model type from the model classification database and converts the analysis model type into binary coding representation. And then, calling a system resource monitoring program to detect the occupancy rate of the CPU and the memory, and providing the data as input to a model calling strategy network to obtain an optimal NUMA binding strategy.
Step 316: for asynchronously invoked geospatial model tasks, the model's invocation policy is determined by the model dispatch controller. The model scheduling controller takes out the model to be scheduled from the model scheduling queue and judges the binding strategy of the model to be scheduled;
the binding strategy of the model to be scheduled is determined as follows:
if the model is not in the model classification database, NUMA nodes are not bound.
If the model is in the model classification database and is a potential NUMA scheduling type, step 314 is performed to obtain the optimal NUMA binding policy.
If the model is in the model classification database but is not a potential NUMA scheduling type, it is checked whether the model scheduling queue length is greater than 5. If the queue length is greater than 5, execute step 314 to obtain the optimal NUMA binding policy; if the queue length is less than 5, NUMA nodes are not bound.
Based on the binding policy of the model to be scheduled, returning to step 312, the model to be scheduled is added to the category of the inference analysis model in the model classification network by the model scheduling controller, and then the model to be scheduled is correspondingly updated into the model classification database.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 1 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor do the order in which the sub-steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of other steps or sub-steps of other steps.
In one embodiment, as shown in fig. 4, there is provided a geospatial analysis model scheduling apparatus, comprising: a data acquisition module 402, a model classification network training module 404, a model schedule generator construction module 406, a model invocation policy network training module 408, and a schedule scheme generation module 410, wherein:
a data acquisition module 402, configured to acquire an analysis model occupation resource data set.
The model classification network training module 404 is configured to construct a model classification training network, and train the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
The model schedule generator construction module 406 is configured to construct a model schedule generator, and schedule and combine each analysis model through the model schedule generator to generate an analysis model call policy state data set.
The model invoking policy network training module 408 is configured to construct a model invoking policy training network, and train the model invoking policy training network by analyzing the model invoking policy state data set to obtain a trained model invoking policy network.
The scheduling scheme generating module 410 is configured to construct a model scheduling controller, and the model scheduling controller generates an optimal analysis model scheduling scheme according to the output of the model classification network, the output of the model call policy network, and the current state of the system.
For specific limitations of the geospatial analysis model scheduling means, reference may be made to the above limitations of the geospatial analysis model scheduling method, and no further description is given here. The various modules in the geospatial analysis model scheduling apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing geospatial analysis model scheduling data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a geospatial analysis model scheduling method.
It will be appreciated by those skilled in the art that the structure shown in fig. 5 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory storing a computer program and a processor that when executing the computer program performs the steps of:
step 202, acquiring an analysis model occupied resource data set.
And 204, constructing a model classification training network, and training the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
And 206, constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set.
And step 208, constructing a model calling strategy training network, and training the model calling strategy training network by analyzing the model calling strategy state data set to obtain a trained model calling strategy network.
And step 210, constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
step 202, acquiring an analysis model occupied resource data set.
And 204, constructing a model classification training network, and training the model classification training network by analyzing the model occupation resource data set to obtain a trained model classification network.
And 206, constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set.
And step 208, constructing a model calling strategy training network, and training the model calling strategy training network by analyzing the model calling strategy state data set to obtain a trained model calling strategy network.
And step 210, constructing a model scheduling controller, and generating an optimal analysis model scheduling scheme by the model scheduling controller according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims (10)

1. A geospatial analysis model scheduling method, the method comprising:
acquiring an analysis model occupied resource data set;
constructing a model classification training network, and training the model classification training network through the analysis model occupation resource data set to obtain a trained model classification network;
a model scheduling generator is constructed, scheduling combination is carried out on each analysis model through the model scheduling generator, and an analysis model calling strategy state data set is generated;
Constructing a model calling strategy training network, and training the model calling strategy training network through the analysis model calling strategy state data set to obtain a trained model calling strategy network;
and constructing a model scheduling controller, wherein the model scheduling controller generates an optimal analysis model scheduling scheme according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
2. The geospatial analysis model scheduling method of claim 1 wherein obtaining an analysis model occupancy resource dataset includes:
acquiring the CPU occupancy rate, the memory occupancy size and the execution time of each geographic space analysis model occupying system resources;
carrying out clustering statistics on the CPU occupancy rate, the memory occupancy size and the execution time, selecting grading boundary conditions, and constructing an analysis model type evaluation tree;
performing type labeling on the CPU occupancy rate, the memory occupancy size and the execution time according to the analysis model type evaluation tree to obtain analysis model label types;
and constructing an analysis model occupation system resource data set through the CPU occupancy rate, the memory occupation size, the execution time and the analysis model label category.
3. The geospatial analysis model scheduling method of claim 1 or 2 wherein constructing a model classification training network, training the model classification training network through the analysis model occupancy resource dataset, and obtaining a trained model classification network, comprises:
carrying out parameter standardization on the analysis model occupied resource data set to obtain a standardized resource data set;
constructing a model classification training network, wherein the model classification training network comprises an input layer, an implicit layer and an output layer;
training the input layer, the hidden layer and the output layer through the standardized resource data set to obtain a trained model classification network; after receiving the standardized resource data set, the input layer sends the standardized resource data set to the hidden layer for processing, extracts the characteristics in the standardized resource data set, then sends the extracted characteristics to the output layer, and the output layer classifies each analysis model according to the received characteristics and outputs classification results.
4. A geospatial analysis model scheduling method according to claim 3 wherein constructing a model schedule generator, scheduling each analysis model by the model schedule generator, generating an analysis model call policy state dataset, comprises:
A build model schedule generator, expressed as:
in the method, in the process of the invention,representing different sets of geospatial models, +.>Indicate->The labels of the geospatial analysis model of the wheel selection,indicate->The number of the NUMA node is bound in a round manner;
generating a plurality of rounds of analysis model scheduling strategies;
under a first round of analysis model scheduling strategy, the model scheduling generator distributes each analysis model to NUMA nodes in a serial and/or parallel mode, and combines each analysis model;
recording the CPU occupancy rate, the memory occupancy size, the execution time and the used NUMA nodes of each combination, and taking the CPU occupancy rate, the memory occupancy size, the execution time and the used NUMA nodes of each combination as calling strategy state data of each combination under a first round of analysis model scheduling strategy;
scoring the calling strategy state data of each combination under the first round of analysis model scheduling strategy, and then selecting a strategy sample set according to a sample selection rule;
generating a second round of analysis model scheduling strategy based on the strategy sample set, and obtaining calling strategy state data of each combination under the second round of analysis model scheduling strategy through the second round of analysis model scheduling strategy;
And until the calling strategy state data of each combination under all the analysis model scheduling strategies is obtained, forming an analysis model calling strategy state data set.
5. The geospatial analysis model scheduling method of claim 4 wherein constructing a model invocation policy training network, training the model invocation policy training network through the analysis model invocation policy state data set, results in a trained model invocation policy network, comprising:
parameter standardization is carried out on the analysis model calling strategy state data set, and a standardized strategy state data set is obtained;
constructing a model calling strategy training network, and presetting a loss function; the model calling strategy training network comprises an input layer, an implicit layer and an output layer;
training the input layer, the hidden layer and the output layer through a standardized strategy state data set and a preset loss function to obtain a trained model calling strategy network; after receiving the standardized policy state data set, the input layer sends the standardized policy state data set to the hidden layer for processing, extracts characteristics in the standardized policy state data set, and then sends the characteristics to the output layer, and the output layer processes the received characteristics to generate an analysis model NUMA binding strategy under different analysis model scheduling strategies.
6. The geospatial analysis model scheduling method of claim 5 wherein the predetermined loss function is expressed as:
wherein,is the number of samples, +.>Representation sample->Is the true value of the tag of +.>Representation model sample->Prediction probability->Representing policy score->Representing the super parameter.
7. The geospatial analysis model scheduling method of claim 5 or 6, wherein constructing a model scheduling controller that generates an optimal analysis model scheduling scheme based on the output of the model classification network, the output of the model invocation policy network, and the current system state, comprises:
the model scheduling controller acquires the currently executed analysis model CPU occupancy rate, memory occupancy size and execution time of the system, sends the currently executed analysis model CPU occupancy rate, memory occupancy size and execution time to the model classification network, classifies the analysis model through the model classification network, and correspondingly updates analysis model type records based on classified analysis model data;
when NUMA parallel scheduling is needed, the model scheduling controller reads updated analysis model classification data, sends the updated analysis model classification data to the model calling strategy network, and outputs a NUMA binding strategy through the model calling strategy network; the model scheduling controller generates an optimal analysis model scheduling scheme according to the updated analysis model classification data, the NUMA binding strategy and the current state of the system;
When the asynchronous scheduling is needed, a model scheduling controller reads a model to be scheduled from a model scheduling queue and judges a binding strategy of the model to be scheduled; and the model scheduling controller adds the model to be scheduled into the model classification network for classification based on the binding strategy of the model to be scheduled, and then correspondingly updates the analysis model type record based on the classified model data to be scheduled.
8. A geospatial analysis model scheduling apparatus, the apparatus comprising:
the data acquisition module is used for acquiring the resource data set occupied by the analysis model;
the model classification network training module is used for constructing a model classification training network, and training the model classification training network through the analysis model occupied resource data set to obtain a trained model classification network;
the model scheduling generator construction module is used for constructing a model scheduling generator, and scheduling and combining all analysis models through the model scheduling generator to generate an analysis model calling strategy state data set;
the model calling strategy network training module is used for constructing a model calling strategy training network, and training the model calling strategy training network through the analysis model calling strategy state data set to obtain a trained model calling strategy network;
The scheduling scheme generating module is used for constructing a model scheduling controller, and the model scheduling controller generates an optimal analysis model scheduling scheme according to the output of the model classification network, the output of the model calling strategy network and the current state of the system.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
CN202410179395.XA 2024-02-18 Geospatial analysis model scheduling method, device, equipment and medium Active CN117724858B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410179395.XA CN117724858B (en) 2024-02-18 Geospatial analysis model scheduling method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410179395.XA CN117724858B (en) 2024-02-18 Geospatial analysis model scheduling method, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN117724858A true CN117724858A (en) 2024-03-19
CN117724858B CN117724858B (en) 2024-05-28

Family

ID=

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112529153A (en) * 2020-12-03 2021-03-19 平安科技(深圳)有限公司 Method and device for fine tuning BERT model based on convolutional neural network
CN115543577A (en) * 2022-08-08 2022-12-30 广东技术师范大学 Kubernetes resource scheduling optimization method based on covariates, storage medium and equipment
CN116185684A (en) * 2022-12-30 2023-05-30 华中科技大学 Construction method and application of server fault root cause analysis model
CN116306324A (en) * 2023-05-25 2023-06-23 安世亚太科技股份有限公司 Distributed resource scheduling method based on multiple agents
CN116737370A (en) * 2023-06-07 2023-09-12 深圳天源迪科信息技术股份有限公司 Multi-resource scheduling method, system, storage medium and terminal

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112529153A (en) * 2020-12-03 2021-03-19 平安科技(深圳)有限公司 Method and device for fine tuning BERT model based on convolutional neural network
CN115543577A (en) * 2022-08-08 2022-12-30 广东技术师范大学 Kubernetes resource scheduling optimization method based on covariates, storage medium and equipment
CN116185684A (en) * 2022-12-30 2023-05-30 华中科技大学 Construction method and application of server fault root cause analysis model
CN116306324A (en) * 2023-05-25 2023-06-23 安世亚太科技股份有限公司 Distributed resource scheduling method based on multiple agents
CN116737370A (en) * 2023-06-07 2023-09-12 深圳天源迪科信息技术股份有限公司 Multi-resource scheduling method, system, storage medium and terminal

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
周成虎, 骆剑承: "遥感影象统计分析及其扩展模型", 中国图象图形学报, no. 12, 25 December 2001 (2001-12-25) *
王丽;郭振华;曹芳;高开;赵雅倩;赵坤;: "面向模型并行训练的模型拆分策略自动生成方法", 计算机工程与科学, no. 09, 15 September 2020 (2020-09-15) *
翁国庆;龚阳光;舒俊鹏;黄飞腾;: "基于聚类LSTM深度学习模型的主动配电网电能质量预测", 高技术通讯, no. 07, 15 July 2020 (2020-07-15) *
高培贤;魏立线;刘佳;刘明明;: "针对图像隐写分析的卷积神经网络结构设计", 中国科技论文, no. 14, 23 July 2018 (2018-07-23) *

Similar Documents

Publication Publication Date Title
US10719767B2 (en) Computer implemented system for automating the generation of a business decision analytic model
CN111258767B (en) Cloud computing resource intelligent distribution method and device for complex system simulation application
JP2010157214A (en) Gene clustering program, gene clustering method, and gene cluster analyzing device
CN109586950B (en) Network scene recognition method, network management device, network scene recognition system and storage medium
CN105607952B (en) Method and device for scheduling virtualized resources
CN110969198A (en) Distributed training method, device, equipment and storage medium for deep learning model
CN111160959B (en) User click conversion prediction method and device
CN110796366A (en) Quality difference cell identification method and device
CN116760772A (en) Control system and method for converging flow divider
CN111582341A (en) User abnormal operation prediction method and device
CN113158435B (en) Complex system simulation running time prediction method and device based on ensemble learning
CN113326177A (en) Index anomaly detection method, device, equipment and storage medium
CN117724858B (en) Geospatial analysis model scheduling method, device, equipment and medium
CN117435306A (en) Cluster container expansion and contraction method, device, equipment and storage medium
CN117724858A (en) Geospatial analysis model scheduling method, device, equipment and medium
CN116896591A (en) Scheduling method and device for network data analysis model and computer equipment
CN116541128A (en) Load adjusting method, device, computing equipment and storage medium
Rajbhandari et al. Trust assessment using provenance in service oriented applications
CN115391047A (en) Resource scheduling method and device
CN109978038B (en) Cluster abnormity judgment method and device
CN111144652B (en) Tour comfort algorithm and trend prediction based method, system and device
Balis et al. Improving prediction of computational job execution times with machine learning
Du et al. OctopusKing: A TCT-aware task scheduling on spark platform
CN113935407A (en) Abnormal behavior recognition model determining method and device
CN112631750A (en) Predictive online scheduling and mixed task deployment method based on compressed sensing and oriented to cloud data center

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant