CN113296929A - Resource matching method, device and system based on cloud computing - Google Patents
Resource matching method, device and system based on cloud computing Download PDFInfo
- Publication number
- CN113296929A CN113296929A CN202010603451.XA CN202010603451A CN113296929A CN 113296929 A CN113296929 A CN 113296929A CN 202010603451 A CN202010603451 A CN 202010603451A CN 113296929 A CN113296929 A CN 113296929A
- Authority
- CN
- China
- Prior art keywords
- job
- target cluster
- information
- resource
- computing resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a resource matching method, device and system based on cloud computing, relates to the technical field of computers, and mainly aims to determine resources required by operation through standardizing a format for collecting operation information so as to integrally match computing resources of a cluster. The main technical scheme of the invention is as follows: acquiring job description information for a job from a target cluster; extracting resource configuration information in the job description information by using a preset job model, wherein the resource configuration information at least comprises processing resource demand information and processing process correlation information of the job; calculating the computing resources required by the operation in the target cluster according to the resource configuration information; and determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a resource matching method, device and system based on cloud computing.
Background
In recent years, with the rapid development of cloud computing, the cloud demand in the field of high-performance computing is increasing, the computing task amount is increasing, and an automatic scaling solution using elastic computing resources on the cloud is produced. The automatic scaling solution of the elastic computing resources means that a user can scale system resources according to the utilization rate of a CPU, the usage amount of a memory and the like, and the processing capacity of a cluster is efficiently utilized.
However, for an HPC (High Performance Computing) user using a scheduler to manage jobs, the HPC user often serves multiple HPC users in the same cluster, and the used schedulers may not be uniform according to their own needs and usage habits, but the resource usage situations, such as CPU and memory usage situations, etc., counted by the schedulers of different types regarding jobs in the cluster often cannot accurately reflect the real resource needs of the jobs, for example, jobs with exclusive nodes may cause insufficient resource expansion, and jobs with dependency relationship may cause excessive resource expansion. It can be seen that, in a scenario where multiple schedulers for HPC users are applied, an existing automatic scaling solution for flexible computing resources cannot effectively perform accurate resource matching on computing resources required by jobs, which results in a problem of low cluster job processing efficiency.
Disclosure of Invention
In view of the above problems, the invention provides a resource matching method, device and system based on cloud computing, and mainly aims to determine resources required by a job through standardizing a format of collecting job information, so as to integrally match computing resources of a cluster.
In order to achieve the purpose, the invention mainly provides the following technical scheme:
in one aspect, the present invention provides a resource matching method based on cloud computing, which specifically includes:
acquiring job description information for a job from a target cluster;
extracting resource configuration information in the job description information by using a preset job model, wherein the resource configuration information at least comprises processing resource demand information and processing process correlation information of the job;
calculating the computing resources required by the operation in the target cluster according to the resource configuration information;
and determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation.
In another aspect, the present invention provides a resource matching apparatus based on cloud computing, which specifically includes:
the acquisition unit is used for acquiring job description information aiming at the job from the target cluster;
the extracting unit is used for extracting resource configuration information in the job description information acquired by the acquiring unit by using a preset job model, wherein the resource configuration information at least comprises processing resource demand information and processing process correlation information of the job;
the statistical unit is used for counting the computing resources required by the jobs in the target cluster according to the resource configuration information obtained by the extraction unit;
and the matching unit is used for determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation obtained by the counting unit.
In another aspect, the present invention provides a resource matching system based on cloud computing, the system including a computing node, a management node;
the computing node is used for processing the job in the target cluster and sending the job description information of the job to the management node;
the management node is used for adjusting the currently configured computing resources of the target cluster according to the business description information and a preset strategy and executing the cloud computing-based resource matching method.
In another aspect, the present invention provides a processor, configured to execute a program, where the program executes the above-mentioned resource matching method based on cloud computing.
In another aspect, the present invention provides an electronic device, which includes a processor and a storage medium, where the storage medium stores a computer program, and the computer program, when executed by the processor, implements the above-mentioned resource matching method based on cloud computing.
By means of the technical scheme, the resource matching method, the device and the system based on cloud computing mainly include that when computing resources of a cluster are automatically and telescopically computed, corresponding job description information is obtained from a target cluster by taking a job as a unit, resource configuration information with a unified format is extracted through a preset job model, the resource configuration information comprises processing resource demand information and processing process related information of the job, the processing resource demand information is used for expressing the demand of the job on the computing resources, the processing process related information is used for expressing the demand of the job on the sharing degree of the computing resources, the computing resources required by the target cluster for processing the job can be rapidly and accurately computed from multiple dimensions based on the resource configuration information with the unified format, and then the computing resources required by the target cluster to be executed are determined based on the current computing resources of the target cluster and preset strategies configured by a user And performing capacity expansion or capacity reduction operation on the computing resources so as to improve the rationality of performing telescopic processing on the computing resources of the target cluster, thereby improving the processing efficiency of the target cluster on the operation.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 shows a flowchart of a resource matching method based on cloud computing according to an embodiment of the present invention;
FIG. 2 is a flow chart of another cloud computing-based resource matching method proposed by the embodiment of the invention;
fig. 3 is a block diagram illustrating a resource matching apparatus based on cloud computing according to an embodiment of the present invention;
fig. 4 is a block diagram illustrating another resource matching apparatus based on cloud computing according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating the implementation principle of the cloud computing-based resource matching method according to the embodiment of the present invention;
fig. 6 is a block diagram illustrating steps of a process performed by the cloud computing-based resource matching method according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
The resource matching method based on cloud computing provided by the embodiment of the invention optimizes and improves a contraction and expansion solution of cloud computing resources aiming at a computing cluster, and improves the effectiveness and the rationality of resource matching by accurately representing the computing resources required by operation in a target cluster. The specific steps of the method are shown in fig. 1, and the method comprises the following steps:
The job description information obtained in this step refers to all information submitted by the user for the job when submitting the job, including the requirements of the job processing resources and the associated information of the processing. In the embodiment of the invention, the types of the schedulers used by different users are various, so that the description modes of the schedulers for the jobs are different. The step is to directly acquire all relevant information of the operation, and extract the information required in the embodiment of the invention from the information, so that the condition of information acquisition omission caused by format problems is avoided.
It should be noted that, in the practical application process, the embodiment of the present invention may also limit the description information of the job submitted by the user, and require the user to specify the value of the specified information parameter of the job, so that the comprehensiveness of the obtained job description information may also be ensured. The specific information parameter is set by self-definition according to a specific application scenario, and this embodiment is not limited in this respect.
And 102, extracting resource configuration information in the job description information by using a preset job model.
In this step, the preset job model is preset based on a specific scene applied by the target cluster, and is used for sorting job description information corresponding to jobs of different formats submitted by different users in the cluster and unifying the representation of the jobs on the computing resource requirements. The input is the job description information obtained in the previous step, and the output is the resource configuration information with a uniform format. The resource configuration information at least includes processing resource requirement information of the job and processing process correlation information, where the processing resource requirement information refers to a computing resource required for the job itself in processing, such as a memory required for processing the job, a core number of a CPU, and the like; the processing procedure related information is related to other jobs when the job is processed, for example, the dependency relationship of the job, that is, the processing of job a needs to be performed according to the calculation result of job B, so that job a and job B do not need to be allocated with calculation resources at the same time. That is, when describing the demand of the job for the computing resource, the resource allocation information in this step is at least expressed from different dimensions such as an association relationship between the job itself and the job, that is, the demand of the job for the computing resource is expressed from multiple dimensions, and in addition, a time dimension such as a job waiting time may be included. Through the multi-dimensional representation operation, the requirement of the operation on the computing resource can be more accurately described, and therefore the requirement of the target cluster on the computing resource can be more accurately counted.
The preset job model in this step can be understood as a formatting template of the resource configuration information, and different target clusters can be provided with one or more templates according to their own requirements, so as to analyze jobs submitted by different users and extract information therein, thereby generating the resource configuration information.
And 103, counting the computing resources required by the jobs in the target cluster according to the resource configuration information.
Because each job in the target cluster is represented in the form of resource configuration information, and the resource requirements of the job and the association relationship among the jobs are represented from multiple dimensions, the step can accurately count the requirements of the job on the computing resources based on the resource configuration information of the job in the target cluster.
The jobs in the statistical target cluster in this step may refer to all jobs in the statistical target cluster, or may be jobs that have not been processed in the statistical target cluster and are in a queued state. All the jobs are counted to determine whether to perform the scaling operation or not in order to determine the overall computing resources required by the target cluster to process the jobs, and the calculation needs to be determined by comparing the overall computing resources with the currently configured computing resources. The operation in the queuing state is counted to calculate the increment of the demand of the target cluster for the computing resource, and the method can more quickly judge whether the target cluster needs to execute the capacity expansion operation, but under the condition of no queued operation, the method is difficult to determine whether the target cluster needs to execute the capacity reduction operation.
And step 104, determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation.
In this step, the specific demand condition of the target cluster for the computing resource can be determined by comparing the computing resource required by the job with the currently configured computing resource of the target cluster, and the addition of the preset policy is used for controlling the adjustment degree of the computing resource. The total amount of the target cluster to be subjected to the capacity reduction and expansion can be limited through the control of the preset strategy, and the situation of infinite capacity reduction and expansion is avoided. When the computing resources required for the job are computing resources required for all jobs, the currently configured computing resources are the sum of the used computing resources and the unused computing resources, and when the computing resources required for the job are computing resources required for unprocessed jobs, the currently configured computing resources may be referred to as unused computing resources.
In addition, the preset policy is also used to further check the computing resources required by the job, and the preset policy is used to check whether the computing resources required by the job counted in step 103 are reasonable. If the operation is not reasonable, the calculation resources required by the operation can be corrected according to the preset strategy. Thus, the system administrator can continuously update the specific way of calculating the resource statistics through the preset strategy.
As can be seen from the description of the foregoing embodiment, the resource matching method based on cloud computing provided in the embodiment of the present invention processes jobs in a target cluster in the same representation manner into resource configuration information represented in multiple dimensions, so as to count computing resources required by the target cluster, and then performs controllable scaling adjustment on the required computing resources according to a preset policy and in combination with currently configured computing resources of the target cluster, so as to improve the efficiency of processing jobs by the target cluster.
In practical application, the resource matching method provided by the embodiment of the invention can be used as an independent service to be deployed independently, can be deployed at a cloud end, and can also be deployed in a target cluster.
In correspondence with the above description of the embodiment shown in fig. 1, a cloud computing-based resource matching method proposed by the present invention will be described below with reference to a flow chart shown in fig. 5, specifically, the computing nodes 1-3 and the management node in fig. 5 constitute a target cluster in this embodiment, where the computing nodes are used for processing process jobs submitted by users, such as computing jobs proposed by scientific computing software based on weather, mechanics, molecular dynamics, etc.; the management node is configured to obtain job description information corresponding to a job to be processed by each computing node in the target cluster, where the management node may be an independent node in the target cluster or may be set on a device where a certain computing node is located. A preset operation model is arranged in the management node, resource configuration information in each operation description information is extracted by the preset operation model, and the resource configuration information is reported to an automatic scaling service at a cloud end, the automatic scaling service is used for counting computing resources required by processing operations in the target cluster and determining whether to adjust the computing resources currently configured by the target cluster according to a preset strategy, wherein a computing example in the figure 5 represents the computing resources capable of being provided by the cloud end, and when the automatic scaling service determines that the target cluster needs to be extended, the management node notifies the computing examples provided for the target cluster so that the target cluster can rapidly process the queued operations; when the automatic scaling service determines that the target cluster needs to be scaled down, the use right of part of the computing resources in the target cluster is also obtained through the management node. In fig. 5, the automatic scaling service and the computing instance framed by the dashed line frame belong to cloud resources, and particularly, the computing instance can be flexibly applied and released, and the automatic scaling service provides dynamic allocation of computing resources for the clusters served by the automatic scaling service, so as to realize optimal use of the computing resources.
Further, for the resource matching method based on cloud computing described in fig. 1, an embodiment of the present invention will describe in detail the determination of the demand of the target cluster computing resource and a specific manner how to determine how to reduce or expand the target cluster computing resource, where specific steps of the method are shown in fig. 2, and include:
In this example, the jobs in the target cluster are submitted by users using different schedulers, and the jobs submitted by different users are added to the corresponding schedulers so as to determine the priority level of job processing according to their own requirements. Therefore, to acquire a job in a target cluster, the job can be read from schedulers of different users, and for this purpose, it is necessary to determine to use the scheduler corresponding to the user in the target cluster, acquire a job to be processed in the target cluster from the schedulers, and further acquire corresponding job description information according to the determined job.
This step may be executed periodically and synchronously with step 201, and the load information of the target cluster is obtained while the job description information is obtained. The load information refers to a part of the computing resources currently configured by the target cluster, which have been used for processing the job.
Since the target cluster is generally composed of a plurality of nodes, different nodes have different types according to functions, and the corresponding load information representations of the different nodes are different, the embodiment of the present invention also normalizes the load information by presetting a load model to obtain the resource occupation information corresponding to each node. The load model is similar to the preset operation model, and has the function of representing the load conditions of different nodes in the target cluster in a unified manner, the load information reported by each node is input, and the resource occupation information in a unified format is output. The main contents of the resource occupation information include: the node identification, the node type, the parameters of the node CPU, the memory and the like, the respective use conditions and the like, the operation identification of the node in operation and the like.
The purpose of acquiring the resource occupation information is to determine the currently configured computing resources of the target cluster, and meanwhile, it may also be determined whether the target cluster has releasable computing resources according to the demand of the target cluster for the computing resources, which is counted based on the resource configuration information, that is, the computing resources in the target cluster that do not need to execute job processing temporarily are counted. If the computing resources which can be released exist, a computing resource releasing request can be sent to the cloud end, and the cloud end allocates new jobs for the computing resources to process, so that the computing resources are fully utilized.
And step 203, extracting resource configuration information in the job description information by using a preset job model.
The specific implementation process of the step is as follows: the method includes the steps of firstly determining a preset job model according to an identifier of a target cluster, namely, firstly determining the preset job model, wherein the identifier of the target cluster is determined based on functions of the target cluster, such as an application program installed in the target cluster, a type of a job to be processed and the like. And then, extracting a corresponding information value from the operation description information by using the preset operation model according to preset formatting information, and generating resource configuration information with a uniform format according to the obtained information value.
Wherein, the resource configuration information mainly includes: job identification information, processing resource requirement information, and processing procedure association information. Specifically, the processing resource requirement information may include whether the job exclusively processes the nodes, the number of nodes requested by the job, the number of cores requested by the job at each node, the number of memories requested by the job, and the like, and the processing procedure association information may include job-dependent identification information, job-dependent processing status, and the like.
And step 204, counting the computing resources required by the jobs in the target cluster according to the resource configuration information.
Specifically, when calculating the computing resources required by the job, the computing resources required by the target cluster may be accurately calculated by using a preset policy. For example, by identifying whether a job exclusively occupies the node for processing, the residual computing resources of the node to be exclusively occupied by the job can be effectively prevented from being counted, and for example, by identifying whether a dependent job exists in the job and the processing condition of the dependent job, whether the computing resources need to be expanded for the job can be determined, so that the waste of resources caused by the invalid expansion of the computing resources can be avoided. Therefore, whether each job needs to expand the computing resources and how much computing resources need to be expanded can be accurately judged through the preset strategy.
The preset strategy can be continuously updated, and the resource configuration information corresponding to the operation in the target cluster has a uniform format representation mode, so that the demand condition of the target cluster on the computing resource can be further rapidly and accurately counted.
Firstly, the computing resources required by the target cluster are counted according to the computing resources required by the jobs, that is, the demands of the target cluster on the computing resources are counted by all the jobs in the target cluster. Meanwhile, counting the currently configured computing resources of the target cluster according to the acquired load information.
And then, judging whether the computing resources required by the target cluster are larger than the currently configured computing resources, wherein the purpose of the judgment is to determine whether the computing computation required by the currently unprocessed job in the target cluster can be met by the currently idle computing resources of the target cluster, if the idle computing resources are not enough, namely, the idle computing resources are larger than the currently configured computing resources, determining an expansion request for the target cluster according to a preset strategy, and otherwise, if the idle computing resources are more, determining a reduction request for the target cluster according to the preset strategy.
When determining a capacity reduction and expansion request for a target cluster according to a preset policy, acquiring the preset policy for the target cluster, that is, different target clusters may set corresponding policies, where the preset policies include a maximum capacity reduction and expansion value of the target cluster, and determining whether a difference between a computing resource required by the target cluster and a currently configured computing resource is greater than the maximum capacity reduction and expansion value, if so, the value of the capacity reduction and expansion request may be the maximum capacity reduction and expansion value, otherwise, the value of the capacity reduction and expansion request is the difference. That is to say, the degree of scaling the computing resources of the target cluster can be controlled through the preset strategy, so that the rationality of computing resource matching is improved.
As can be seen from the steps of the foregoing embodiments and the corresponding illustrations, the cloud computing-based resource matching method provided in the embodiments of the present invention realizes accurate determination of computing resources required by the entire target cluster by uniformly and normatively expressing job requirements in the target cluster, and compared with a method for computing resource expansion for queued job requests in the prior art, the cloud computing-based resource matching method provided in the embodiments of the present invention can more effectively integrate computing resources configured by the target cluster, determine the degree of the demands of different jobs on the computing resources, such as exclusive nodes, dependent jobs, and the like, to determine whether to provide a capacity reduction request or not, and determine specific values of capacity reduction and expansion, thereby realizing efficient utilization of cloud computing resources.
According to the embodiment described in fig. 2 and with reference to the flow diagram shown in fig. 5, it is specifically described that specific steps in a target cluster processing job flow of the resource matching method based on cloud computing are shown in fig. 6, and the specific steps include:
first, a job is submitted to a target cluster by a user, assuming that all computing nodes in the target cluster are in a working state, resulting in the job being in a queued pending state.
Second, the automatic scaling service will issue a query request for the job load and the cluster computing resource usage status to the management node of the target cluster periodically.
Thirdly, the management node obtains the job description information and the load information of each computing node of the target cluster according to the query request, namely, steps 201 and 202. The load information is the use state of the currently configured computing resource of the cluster.
Fourthly, extracting resource configuration information in the operation description information by a preset operation model in the management node, and reporting the resource configuration information and the load information to the automatic telescopic service. The computing resources required by the jobs in the target cluster are counted by the automatic scaling service, and whether the computing resources need to be adjusted is determined according to a preset strategy, that is, step 203 and step 205, corresponding capacity reduction and expansion instructions are generated.
Fifthly, the automatic scaling service feeds back the capacity reduction and expansion instruction and sends the capacity reduction and expansion instruction to the management node.
And sixthly, executing the capacity-reducing and expanding instruction by the management node, determining a computing resource, namely a computing instance, provided by the cloud when executing the capacity-expanding instruction, and scheduling the job fingers queued in the target cluster to process the computing instance. When the capacity reduction instruction is executed, the spare computing resources or computing instances in the target cluster are obtained and released to the jobs to be processed in other clusters.
And seventhly, the target cluster distributes the queued jobs to the corresponding calculation examples for processing and obtains the calculation results fed back by the calculation examples.
And finally, the target cluster feeds back the calculation result to the user.
Further, as an implementation of the method shown in fig. 1 and 2, an embodiment of the present invention provides a resource matching device based on cloud computing, where the device mainly aims to determine resources required by a job through a format for standardizing collection of job information, and further integrally match computing resources of a cluster. For convenience of reading, details in the foregoing method embodiments are not described in detail again in this apparatus embodiment, but it should be clear that the apparatus in this embodiment can correspondingly implement all the contents in the foregoing method embodiments. As shown in fig. 3, the apparatus specifically includes:
an acquisition unit 31 configured to acquire job description information for a job from a target cluster;
an extracting unit 32, configured to extract, by using a preset job model, resource configuration information in the job description information acquired by the acquiring unit 31, where the resource configuration information at least includes processing resource requirement information and processing procedure association information of a job;
a counting unit 33, configured to count, according to the resource configuration information obtained by the extracting unit 32, the computing resources required by the jobs in the target cluster;
a matching unit 34, configured to determine whether to adjust the currently configured computing resource of the target cluster according to a preset policy and the computing resource required by the job obtained by the counting unit 33.
Further, as shown in fig. 4, the extracting unit 32 includes:
a determining module 321, configured to determine the preset job model according to the identifier of the target cluster;
an extracting module 322, configured to extract, according to preset formatting information, a corresponding information value from the job description information by using the preset job model determined by the determining module 321;
a generating module 323, configured to generate resource configuration information with a uniform format according to the information value obtained by the extracting module 322.
Further, the resource configuration information specifically includes: job identification information, processing resource requirement information and processing process association information; the processing resource demand information comprises whether to monopolize node processing, the number of requested nodes, the number of cores requested by each node and the number of requested memories; the process-related information includes dependent job identification information.
Further, as shown in fig. 4, the matching unit 34 includes:
a counting module 341, configured to count the computing resources required by the target cluster according to the computing resources required by the job;
a determining module 342, configured to determine whether the computing resource required by the target cluster obtained by the counting module 341 is greater than the currently configured computing resource;
a matching module 343, configured to determine, according to a preset policy, an expansion request for the target cluster when the determining module 342 determines that the current configured computing resource is greater than the current configured computing resource; otherwise, determining the capacity reduction request aiming at the target cluster according to a preset strategy.
Further, the matching module 343 is specifically configured to obtain a preset policy for the target cluster, where the preset policy includes a maximum tolerance value of the target cluster; judging whether the difference value between the computing resource required by the target cluster and the currently configured computing resource is greater than the maximum tolerance value; if so, the value requesting for capacity expansion is the maximum capacity expansion value, otherwise, the value requesting for capacity expansion is the difference value.
Further, as shown in fig. 4, the apparatus further includes:
the obtaining unit 31 is further configured to obtain load information of the target cluster, where the load information includes load information of each node;
the processing unit 35 is configured to process the load information obtained by the obtaining unit 31 by using a preset load model, so as to obtain resource occupation information corresponding to each node;
the statistical unit 33 is further configured to perform statistics on the computing resources releasable by the target cluster according to the resource occupation information obtained by the processing unit 35 and the resource configuration information obtained by the extracting unit 32.
Further, as shown in fig. 4, the apparatus further includes:
a sending unit 36, configured to send a request for releasing computing resources to the cloud when the statistics unit 33 determines that releasable computing resources exist in the target cluster.
Further, the acquiring unit 31 is also configured to acquire job description information of a job in an unprocessed state.
Further, as shown in fig. 4, the acquiring unit 31 further includes:
a determining module 311, configured to determine to use a scheduler corresponding to a user in the target cluster;
an obtaining module 312, configured to obtain job description information corresponding to the job that needs to be processed in the target cluster from the scheduler determined by the determining module 311.
Further, an embodiment of the present invention further provides a resource matching system based on cloud computing, where the system includes a computing node and a management node, where the computing node is configured to process a job in a target cluster, send job description information of the job to the management node, and the management node is configured to adjust a currently configured computing resource of the target cluster according to the job description information and a preset policy.
Specifically, as shown in fig. 5 and 6, the management node is configured to implement statistics and adjustment on the computing resources of the target cluster by calling an automatic scaling service of the cloud, where the automatic scaling service analyzes job description information and load information of the target cluster by using a preset decision to determine whether the computing resources in the target cluster reach an optimal use state, and generates a corresponding capacity reduction instruction based on an analysis result to adjust the computing resources of the target cluster.
In addition, an embodiment of the present invention further provides a processor, where the processor is configured to execute a program, where the program executes the resource matching method based on cloud computing provided in the embodiment shown in fig. 1 or fig. 2 when running.
In addition, an embodiment of the present invention further provides an electronic device, which includes a processor and a storage medium, where the storage medium stores a computer program, and the computer program, when executed by the processor, implements the cloud computing-based resource matching method provided in the embodiment shown in fig. 1 or fig. 2.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be appreciated that the relevant features of the method and apparatus described above are referred to one another. In addition, "first", "second", and the like in the above embodiments are for distinguishing the embodiments, and do not represent merits of the embodiments.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose preferred embodiments of the invention.
In addition, the memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory
(flash RAM) the memory comprises at least one memory chip.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (13)
1. A cloud computing-based resource matching method, the method comprising:
acquiring job description information for a job from a target cluster;
extracting resource configuration information in the job description information by using a preset job model, wherein the resource configuration information at least comprises processing resource demand information and processing process correlation information of the job;
calculating the computing resources required by the operation in the target cluster according to the resource configuration information;
and determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation.
2. The method according to claim 1, wherein extracting resource configuration information in the job description information by using a preset job model comprises:
determining the preset operation model according to the identification of the target cluster;
extracting a corresponding information value from the operation description information according to preset formatting information by using the preset operation model;
and generating resource configuration information with a uniform format according to the obtained information value.
3. The method of claim 2, wherein the resource configuration information comprises: job identification information, processing resource requirement information and processing process association information; wherein the content of the first and second substances,
the processing resource demand information comprises whether to monopolize node processing, the number of requested nodes, the number of cores requested by each node and the number of requested memories;
the process-related information includes dependent job identification information.
4. The method of claim 1, wherein determining whether to adjust the currently configured computing resources of the target cluster according to a preset policy and the computing resources required by the job comprises:
counting the computing resources required by the target cluster according to the computing resources required by the operation;
judging whether the computing resources required by the target cluster are larger than the currently configured computing resources;
if so, determining an expansion request aiming at the target cluster according to a preset strategy;
and if the target cluster size is smaller than the preset size, determining the capacity reduction request aiming at the target cluster according to a preset strategy.
5. The method of claim 4, wherein determining the capacity expansion request for the target cluster according to a preset policy comprises:
acquiring a preset strategy aiming at the target cluster, wherein the preset strategy comprises a maximum capacity value of the target cluster;
judging whether the difference value between the computing resource required by the target cluster and the currently configured computing resource is greater than the maximum tolerance value;
if so, the value requesting for capacity expansion is the maximum capacity expansion value, otherwise, the value requesting for capacity expansion is the difference value.
6. The method of claim 1, wherein when obtaining job description information for a job from a target cluster, the method further comprises:
acquiring load information of the target cluster, wherein the load information comprises load information of each node;
processing the load information by using a preset load model to obtain resource occupation information corresponding to each node;
and counting the computing resources releasable by the target cluster according to the resource occupation information and the resource configuration information.
7. The method of claim 6, further comprising:
and when releasable computing resources exist in the target cluster, sending a computing resource releasing request to the cloud.
8. The method of claim 1, wherein obtaining job description information for a job from a target cluster comprises:
job description information of a job in an unprocessed state is acquired.
9. The method of claim 1, wherein obtaining job description information for a job from a target cluster further comprises:
determining a scheduler corresponding to a user in the target cluster;
and acquiring job description information corresponding to the job needing to be processed in the target cluster from the scheduler.
10. An apparatus for cloud computing-based resource matching, the apparatus comprising:
the acquisition unit is used for acquiring job description information aiming at the job from the target cluster;
the extracting unit is used for extracting resource configuration information in the job description information acquired by the acquiring unit by using a preset job model, wherein the resource configuration information at least comprises processing resource demand information and processing process correlation information of the job;
the statistical unit is used for counting the computing resources required by the jobs in the target cluster according to the resource configuration information obtained by the extraction unit;
and the matching unit is used for determining whether to adjust the currently configured computing resources of the target cluster according to a preset strategy and the computing resources required by the operation obtained by the counting unit.
11. A resource matching system based on cloud computing comprises a computing node and a management node;
the computing node is used for processing the job in the target cluster and sending the job description information of the job to the management node;
the management node is configured to adjust the currently configured computing resources of the target cluster according to the business description information and a preset policy, and execute the resource matching method based on cloud computing according to any one of claims 1 to 9.
12. A processor, wherein the processor is configured to execute a program, and wherein the program is configured to execute the cloud computing-based resource matching method according to any one of claims 1 to 9.
13. An electronic device, characterized in that the electronic device comprises a processor and a storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by the processor, implements the cloud computing-based resource matching method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010603451.XA CN113296929A (en) | 2020-06-29 | 2020-06-29 | Resource matching method, device and system based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010603451.XA CN113296929A (en) | 2020-06-29 | 2020-06-29 | Resource matching method, device and system based on cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113296929A true CN113296929A (en) | 2021-08-24 |
Family
ID=77318046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010603451.XA Pending CN113296929A (en) | 2020-06-29 | 2020-06-29 | Resource matching method, device and system based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113296929A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835953A (en) * | 2021-09-08 | 2021-12-24 | 曙光信息产业股份有限公司 | Statistical method and device of job information, computer equipment and storage medium |
| CN114153518A (en) * | 2021-10-25 | 2022-03-08 | 国网江苏省电力有限公司信息通信分公司 | Autonomous capacity expansion and reduction method for cloud native MySQL cluster |
| CN114205246A (en) * | 2021-12-14 | 2022-03-18 | 中国电信股份有限公司 | Cloud resource planning method and device and storage medium |
| CN114697322A (en) * | 2022-02-17 | 2022-07-01 | 许强 | Data screening method based on cloud service processing |
| CN115454450A (en) * | 2022-09-15 | 2022-12-09 | 北京火山引擎科技有限公司 | Method and device for resource management of data operation, electronic equipment and storage medium |
| CN117112242A (en) * | 2023-10-24 | 2023-11-24 | 纬创软件(武汉)有限公司 | Resource node allocation method and system in cloud computing system |
-
2020
- 2020-06-29 CN CN202010603451.XA patent/CN113296929A/en active Pending
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835953A (en) * | 2021-09-08 | 2021-12-24 | 曙光信息产业股份有限公司 | Statistical method and device of job information, computer equipment and storage medium |
| CN114153518A (en) * | 2021-10-25 | 2022-03-08 | 国网江苏省电力有限公司信息通信分公司 | Autonomous capacity expansion and reduction method for cloud native MySQL cluster |
| CN114205246A (en) * | 2021-12-14 | 2022-03-18 | 中国电信股份有限公司 | Cloud resource planning method and device and storage medium |
| CN114697322A (en) * | 2022-02-17 | 2022-07-01 | 许强 | Data screening method based on cloud service processing |
| CN114697322B (en) * | 2022-02-17 | 2024-03-22 | 上海生慧樘科技有限公司 | Data screening method based on cloud service processing |
| CN115454450A (en) * | 2022-09-15 | 2022-12-09 | 北京火山引擎科技有限公司 | Method and device for resource management of data operation, electronic equipment and storage medium |
| CN115454450B (en) * | 2022-09-15 | 2024-04-30 | 北京火山引擎科技有限公司 | Method and device for resource management of data job, electronic equipment and storage medium |
| CN117112242A (en) * | 2023-10-24 | 2023-11-24 | 纬创软件(武汉)有限公司 | Resource node allocation method and system in cloud computing system |
| CN117112242B (en) * | 2023-10-24 | 2024-01-26 | 纬创软件(武汉)有限公司 | Resource node allocation method and system in cloud computing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113296929A (en) | Resource matching method, device and system based on cloud computing | |
| US11314551B2 (en) | Resource allocation and scheduling for batch jobs | |
| CN107832126B (en) | Thread adjusting method and terminal thereof | |
| WO2021179462A1 (en) | Improved quantum ant colony algorithm-based spark platform task scheduling method | |
| JP5744707B2 (en) | Computer-implemented method, computer program, and system for memory usage query governor (memory usage query governor) | |
| US9870269B1 (en) | Job allocation in a clustered environment | |
| US8547840B1 (en) | Bandwidth allocation of bursty signals | |
| CN111225050B (en) | Cloud computing resource allocation method and device | |
| CN108900626B (en) | Data storage method, device and system in cloud environment | |
| CN114416352A (en) | Computing resource allocation method and device, electronic equipment and storage medium | |
| CN112165436A (en) | Flow control method, device and system | |
| CN111625367B (en) | Method for dynamically adjusting read-write resources of file system | |
| CN112150023A (en) | Task allocation method, device and storage medium | |
| US11765099B2 (en) | Resource allocation using distributed segment processing credits | |
| CN112148468A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN105229608A (en) | Based on the database processing towards array of coprocessor | |
| CN104102646A (en) | Method, device and system for processing data | |
| CN114860449B (en) | Data processing method, device, equipment and storage medium | |
| CN110308991A (en) | A kind of data center's energy conservation optimizing method and system based on Random Task | |
| CN116204311A (en) | Pod cluster capacity expansion and contraction method and device, computer equipment and storage medium | |
| CN110399216B (en) | Method, system and device for distributing power consumption of whole machine box and readable storage medium | |
| CN112000294A (en) | IO queue depth adjusting method and device and related components | |
| CN110569259A (en) | Method and device for processing mass data | |
| CN115981825B (en) | Cluster parallel scheduling system based on hybrid shared state view architecture | |
| CN117519913B (en) | Method and system for elastically telescoping scheduling of container memory resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40058607 Country of ref document: HK |