CN117632370A - Deployment method, device, equipment and storage medium - Google Patents

Deployment method, device, equipment and storage medium Download PDF

Info

Publication number
CN117632370A
CN117632370A CN202311657760.5A CN202311657760A CN117632370A CN 117632370 A CN117632370 A CN 117632370A CN 202311657760 A CN202311657760 A CN 202311657760A CN 117632370 A CN117632370 A CN 117632370A
Authority
CN
China
Prior art keywords
target
cluster
sub
task
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311657760.5A
Other languages
Chinese (zh)
Inventor
狄晓帆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Agricultural Bank of China
Original Assignee
Agricultural Bank of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agricultural Bank of China filed Critical Agricultural Bank of China
Priority to CN202311657760.5A priority Critical patent/CN117632370A/en
Publication of CN117632370A publication Critical patent/CN117632370A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a deployment method, a deployment device, deployment equipment and a storage medium. The method comprises the following steps: determining a sub-cluster to be operated corresponding to a target cluster based on deployment association information of the target task, wherein the deployment association information comprises deployment data quantity of the target task; starting the sub-cluster to be operated so as to execute the target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task; and determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period. The problem of lower deployment efficiency is solved, and the deployment efficiency is improved.

Description

Deployment method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a deployment method, apparatus, device, and storage medium.
Background
The cloud protogenesis technology can help enterprises to construct and operate elastically extensible applications in novel dynamic environments such as public clouds, private clouds and hybrid clouds. Typical technologies of cloud protogenesis include containerization, micro-services, and the like. The definition of cloud protogenesis given by CNCF (cloud protogenesis computing foundation) in 2015 includes application containerization, orchestration scheduling of application support containers. Currently Kubernetes (K8S) is the most widely used container orchestration engine in the industry, and with the rapid development of K8S, the cloud primary projects based on K8S are increasing. Cloud native architecture is computationally decoupled in nature. In the cloud primary architecture, resources can be elastically stretched, and resources can be rapidly increased in a service busy period; and in the idle service period, redundant resources are released in time, so that the resource utilization rate is greatly improved, and the cost is saved.
Furin (Apache link) is an open source stream processing framework developed by the Apache software foundation, the core of which is a distributed stream processing engine written in Java and Scala. The Java is an object-oriented programming language, not only absorbs various advantages of the C++ language, but also abandons concepts such as multiple inheritance, pointers and the like which are difficult to understand in the C++, so that the Java is powerful in function, simple and easy to use. Scala is a multi-paradigm programming language, like Java, the design intent is to implement a scalable language, and to integrate the various features of object-oriented programming and functional programming. The Flink is used for processing real-time and batch data, supporting various data sources and supporting the execution of iterative algorithms. The Flink is used as a big data computing engine of the latest generation, has the advantages of low delay, high throughput and high performance, supports event time concepts, supports stateful computing, supports highly flexible window operation, fault tolerance based on lightweight distributed snapshot realization and independent memory management based on Java virtual machine realization, and supports a preservation point mechanism. The Flink provides a cloud native deployment approach that meets the current cloud requirements of most enterprise applications. Thus, flank is favored by both academia and industry for its superior performance and functionality.
In the technical scheme of relevant deployment, the Flink has only two deployment modes under the Native K8S operation mode, namely a Session mode and an Application mode. Wherein the Session mode uses one already running cluster to execute all submitted jobs. The applied resources are used, so that the execution efficiency is high, but one job fault can influence the operation of other jobs. The method is suitable for submitting more frequent scenes, and small jobs are more; each submitted job in Application mode initiates a separate cluster that is only available to that job. One job failure does not affect the operation of other jobs, but re-applies for resources each submission, which affects execution efficiency and increases additional resource overhead. The method is suitable for scenes with fewer jobs and large jobs. In summary, deployment is inefficient.
Disclosure of Invention
The invention provides a deployment method, a device, equipment and a storage medium, which are used for improving the deployment efficiency.
According to an aspect of the present invention, there is provided a deployment method, the method comprising:
determining a sub-cluster to be operated corresponding to a target cluster based on deployment associated information of the target task, wherein the deployment associated information comprises deployment data quantity of the target task;
Starting the sub-cluster to be operated so as to execute a target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task;
and determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period.
Further, the determining the sub-cluster to be executed corresponding to the target cluster based on the deployment association information of the target task includes:
and determining a sub-cluster to be operated corresponding to the target cluster based on the deployment data volume and the preset data volume.
Further, the executing the target task based on the sub-cluster to be executed to obtain target running time data corresponding to the target task includes:
determining at least one subtask corresponding to the target task;
executing at least one sub-task based on the sub-cluster to be executed to obtain original result data and original running time data corresponding to each sub-task;
Performing data fusion operation on all the original result data to obtain fusion result data and to-be-processed running time data corresponding to the data fusion operation;
and determining target running time data corresponding to a target task based on each piece of original running time data and the running time data to be processed.
Further, the determining, based on each of the raw runtime data and the to-be-processed runtime data, target runtime data corresponding to a target task includes:
determining the maximum runtime data in all the original runtime data;
and summing the maximum operation time data and the operation time data to be processed to obtain target operation time data corresponding to the target task.
Further, the determining, based on the target runtime data and the target number of times of execution, a target sub-cluster corresponding to the target task includes:
and if the target running time data is greater than or equal to a preset time threshold, determining a target sub-cluster corresponding to the target task as the sub-cluster to be run.
Further, the determining, based on the target runtime data and the target number of times of execution, a target sub-cluster corresponding to the target task includes: and if the target task is the first running task, determining that the target sub-cluster is the sub-cluster to be run.
Further, the target sub-cluster comprises a first sub-cluster and a second sub-cluster,
the determining a target sub-cluster corresponding to the target task based on the target runtime data and the target number of runs includes:
if the target running time data is smaller than a preset time threshold value and the target running times are smaller than a preset times threshold value, determining a target sub-cluster corresponding to the target task as a first sub-cluster;
and if the target running time data is smaller than a preset time threshold and the target running times are larger than or equal to a preset times threshold, determining a target sub-cluster corresponding to the target task as a second sub-cluster.
According to another aspect of the invention, there is provided a deployment apparatus comprising:
the sub-cluster determining module is used for determining sub-clusters to be operated corresponding to a target cluster based on deployment associated information of the target task, wherein the deployment associated information comprises deployment data quantity of the target task;
the sub-cluster starting module is used for starting the sub-cluster to be operated so as to execute a target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task;
And the task execution module is used for determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period.
According to another aspect of the present invention, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the deployment method of any of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a deployment method of any embodiment of the present invention.
According to the technical scheme, the sub-cluster to be operated corresponding to the target cluster is determined based on the deployment association information of the target task, wherein the deployment association information comprises the deployment data volume of the target task; starting the sub-cluster to be operated so as to execute the target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task; and determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period. The problem of lower deployment efficiency is solved, and the deployment efficiency is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a deployment method provided in accordance with an embodiment of the present invention;
FIG. 2 is a flow chart of a particular deployment method provided in accordance with an embodiment of the present invention;
FIG. 3 is a block diagram of a deployment device provided in accordance with an embodiment of the present invention;
fig. 4 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It is noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and in the foregoing figures, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
Fig. 1 is a flowchart of a deployment method according to an embodiment of the present invention, where the embodiment is applicable to a scenario of deployment based on a link, and may be executed by a deployment apparatus, where the deployment apparatus may be implemented in a form of hardware and/or software and configured in a processor of an electronic device.
As shown in fig. 1, the deployment method includes the steps of:
s110, determining a sub-cluster to be operated corresponding to the target cluster based on deployment association information of the target task.
The target task is a task to be executed based on the target cluster, and the deployment related information comprises deployment data quantity of the target task. The target cluster is a K8S cluster, is an open source container cluster management system of Google (Google), is a brand-new distributed architecture solution based on container technology, and supports automatic deployment, large-scale scalability and application instantiation management.
The sub-cluster to be executed includes computing resources in the target cluster that do not execute any task, for example, the sub-cluster to be executed may be all computing resources in the target cluster that do not execute any task, or may be part of computing resources in the target cluster that do not execute any task.
In this embodiment, the target task may be a task that is executed based on the target cluster for the first time, and in consideration of the initial operation, the failure rate of the target task may be high, if the target task is directly deployed into the target cluster, once the failure occurs, other jobs in the target cluster may be affected, so that the target task may be executed based on a separate subset group.
It is contemplated that in some scenarios, the runtime of a target task is related to the amount of data of the computing resource and the task itself, and thus, the computing resource it needs may be determined based on the amount of deployment data of the target task to obtain a sub-cluster corresponding to the computing resource.
Specifically, a corresponding relation between the deployment data amount and the computing resource is constructed in advance; determining the deployment data volume of the target task; determining computing resources corresponding to the deployment data volume based on the correspondence; and determining cluster resources corresponding to the computing resources based on the resource configuration information of the target cluster, obtaining a subset group corresponding to the cluster resources, and taking the subset group as a sub-cluster to be operated.
Optionally, cluster resources of the non-executed task in the target cluster may be determined, and a subset group corresponding to the cluster resources of the non-executed task is used as the sub-cluster to be executed.
In this embodiment, determining a sub-cluster to be executed corresponding to a target cluster based on deployment association information of the target task includes: and determining a sub-cluster to be operated, which corresponds to the target cluster, based on the deployment data amount and the preset data amount.
The preset data amount may be determined based on the deployment data amount of the historical tasks performed by the target cluster.
It will be appreciated that between executing the target tasks, at least one historical task is executed based on the target cluster, and a predetermined amount of data is determined based on the execution time corresponding to the at least one historical task and the amount of deployment data corresponding to the at least one historical task, which has the advantage that the runtime of the target task and the cluster resources may be balanced.
Specifically, if the deployment data volume is greater than the preset data volume, taking the sub-clusters corresponding to all cluster resources which do not execute tasks in the target cluster at the current moment as sub-clusters to be operated corresponding to the target cluster.
S120, starting the sub-cluster to be operated so as to execute the target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task.
The target running time data comprises the time from the beginning of the deployment of the target task to the sub-cluster to be run to the time when the execution of the target task is finished.
Specifically, starting a sub-cluster to be operated, deploying a target task into the sub-cluster to be operated, and executing the target task based on the sub-cluster to be operated until the execution of the target task is finished; determining a starting deployment time corresponding to the target task deployment to the sub-cluster to be operated and an ending time corresponding to the ending of the target task execution; and determining a time period corresponding to the ending time and the starting deployment time, and taking the duration corresponding to the time period as target running time data corresponding to the target task.
In this embodiment, based on the sub-cluster to be executed, a target task is executed to obtain target runtime data corresponding to the target task, including: determining at least one subtask corresponding to the target task; executing at least one sub-task based on the sub-cluster to be operated to obtain original result data and original operation time data corresponding to each sub-task; performing data fusion operation on all the original result data to obtain fusion result data and to-be-processed running time data corresponding to the data fusion operation; and determining target running time data corresponding to the target task based on each original running time data and the running time data to be processed.
The sub-cluster to be operated comprises at least one task execution unit, and the task execution unit is used for executing sub-tasks. The original result data comprises a result obtained by the task execution unit executing the subtasks. The raw runtime data is the time spent deploying the subtasks to the task execution units and the time spent executing the subtasks based on the task execution units. The data fusion operation is to splice each original result data to obtain total result data corresponding to the target task, namely fusion result data. The run-time data to be processed is the time consumed for performing the data fusion operation, that is, the time consumed for fusing each original result data to obtain the fused result data. The target runtime may be an average value that determines each raw runtime data, and a sum value of the to-be-processed runtime data.
Specifically, dividing a target task into at least one sub-task based on the computing resources of the sub-cluster to be operated; starting a sub-cluster to be operated, deploying each sub-task into the sub-cluster to be operated, executing all the sub-tasks based on the sub-cluster to be operated, obtaining operation results corresponding to the sub-tasks, and taking the operation results as original result data; meanwhile, for each subtask, determining the time consumed by the subtask when deployed to the sub-cluster to be operated, determining the time consumed by executing the subtask, and carrying out summation processing on the two times to obtain original operation time data; carrying out fusion processing on each piece of original result data to obtain fusion result data, determining the time consumed in the fusion processing process of the fusion result data and the data, and taking the fusion result data as to-be-processed running time data; and determining the average value of each original running time data, carrying out summation processing on the average value and the running time data to be processed, and determining the target running time data corresponding to the target task.
In this embodiment, determining, based on each of the raw runtime data and the to-be-processed runtime data, target runtime data corresponding to a target task includes: determining the maximum runtime data of all the original runtime data; and summing the maximum running time data and the running time data to be processed to obtain target running time data corresponding to the target task.
It will be appreciated that, considering that each sub-cluster to be run may run simultaneously, the target runtime may be the maximum value of each original runtime data determined as the maximum runtime data to obtain the target runtime data.
Specifically, determining the maximum value of all original runtime data, and taking the maximum value as the maximum runtime data; and summing the maximum running time data and the running time data to be processed corresponding to the data fusion operation to obtain target running time data corresponding to the target task.
S130, determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times so as to execute the target task based on the target sub-cluster.
The target sub-clusters are sub-clusters of the target clusters, and the computing capacities of different target sub-clusters are different. Illustratively, the target subset group includes the target cluster and other subsets of the target cluster corresponding to portions of the computing resources. The target number of runs is the historical number of times the target cluster performs the target task.
It may be appreciated that, on the one hand, considering that in some scenarios, the failure rate may be small for tasks that are performed a large number of times, the historical number of times that the target cluster performs the target task may be determined to determine the target sub-cluster corresponding to the target task; on the other hand, considering that in some scenarios, tasks with larger target runtime data require more computing resources, i.e., target sub-clusters corresponding to higher computing power, the target sub-clusters corresponding to the target tasks may be determined based on the target runtime data.
Specifically, for each target task, a frequency variable associated with the target task is set, and the frequency variable is used for recording the frequency of executing the target task based on the target cluster; the corresponding relation between each running time data and the computing resource and the corresponding relation between each running time and the computing resource are preset; determining the target running times of the target cluster for executing the target task based on the corresponding times variable of the target task; determining a computing resource corresponding to the target operation times based on the corresponding relation between the operation times and the computing resource, taking the computing resource as a first computing resource, and determining a computing resource corresponding to the target operation time based on the corresponding relation between the operation time data and the computing resource, and taking the computing resource as a second computing resource; determining the average value of the first computing resource and the second computing resource, and determining a subset group corresponding to the computing resource corresponding to the average value based on the configuration information of each target subset group, wherein the subset group is used as a target subset group; the target task is deployed to the target sub-cluster to execute the target task based on the target sub-cluster.
Alternatively, considering that in some scenarios, the degree of change of the target task is proportional to time, so that the historical task corresponding to the target task that is too far away may not have a reference value, and thus the target running number may be the historical number of times the target cluster executes the target task within the set time range.
In this embodiment, determining the target sub-cluster corresponding to the target task based on the target runtime data and the target number of times of execution includes: and if the target task is the first running task, determining the target sub-cluster as the sub-cluster to be run.
It will be appreciated that for a first running task, the failure rate may be high, and once a failure occurs, it may affect other running tasks in the cluster, and thus, the target task is run using a separate sub-cluster to be run.
Specifically, determining a frequency variable corresponding to a target task; if the number of times variable does not exist or the number of times variable is 1, determining that the target task is the first operation task, and determining that the target sub-cluster is the sub-cluster to be operated.
In this embodiment, determining the target sub-cluster corresponding to the target task based on the target runtime data and the target number of times of execution includes: and if the target running time data is greater than or equal to a preset time threshold, determining the target sub-cluster corresponding to the target task as the sub-cluster to be run.
It can be appreciated that, on one hand, considering that the running time of the target task is too long, other tasks running in the cluster are also affected, and on the other hand, considering that the task deployment needs a long time, therefore, the sub-cluster may be a sub-cluster to be run, and therefore, for the target task with too long running time, the target task is run by adopting the sub-cluster to be run.
Specifically, when the target running time data is greater than or equal to a preset time threshold (for example, 30 minutes), determining that a target sub-cluster corresponding to the target task is a sub-cluster to be run. The method has the advantages of reducing the deployment time of the target task and improving the cluster computing speed.
In this embodiment, the target subset group includes a first sub-group and a second sub-group, and determining, based on the target runtime data and the target runtime number of times, a target subset group corresponding to the target task includes: if the target running time data is smaller than a preset time threshold value and the target running times are smaller than a preset times threshold value, determining a target sub-cluster corresponding to the target task as a first sub-cluster; and if the target running time data is smaller than a preset time threshold and the target running times are larger than or equal to a preset times threshold, determining a target sub-cluster corresponding to the target task as a second sub-cluster.
Wherein the computing power of the first sub-cluster is smaller than the computing power of the second sub-cluster.
Specifically, if the target running time data is smaller than a preset time threshold (for example, 30 minutes) and the target running times are smaller than a preset times threshold, determining that a target sub-cluster corresponding to the target task is a first sub-cluster; if the target running time data is smaller than the preset time threshold and the target running times are larger than or equal to the preset times threshold (for example, 20 times), determining that the target sub-cluster corresponding to the target task is a second sub-cluster.
According to the technical scheme, on one hand, task operation with few recent successful operation times or high failure rate is put into a single cluster for operation; on the other hand, compared with the time for executing the task, the time for deploying the task is longer, and the task with longer running time is operated by adopting the independent cluster, so that the influence on the operation of other tasks in the cluster can be avoided, and the deployment efficiency is improved.
Fig. 2 is a flowchart of a specific deployment method according to an embodiment of the present invention, as shown in fig. 2, where the deployment method includes:
s210, determining whether the target task is a first execution task based on the number of times variable associated with the target task.
Specifically, the number of times the target cluster executes the target task is determined based on a number of times variable associated with the target task; if the number of times variable is greater than 0, determining that the target task is not the first execution task, otherwise, determining that the target task is the first execution task. If the target task is the first execution task, executing S220; if the target task is not the first execution task, S250 is executed.
S220, determining a sub-cluster to be operated corresponding to the target cluster based on the deployment data volume of the target task.
Specifically, if the target task is a first execution task, determining a computing resource corresponding to the deployment data volume according to a pre-constructed corresponding relation between the deployment data volume and the computing resource; and determining cluster resources corresponding to the computing resources based on the resource configuration information of the target cluster, obtaining a subset group corresponding to the cluster resources, and taking the subset group as a sub-cluster to be operated.
And S230, starting the sub-cluster to be operated so as to execute the target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task.
S240, determining a target sub-cluster corresponding to the target task as a sub-cluster to be operated under the condition that the target operation time data is greater than or equal to a preset time threshold.
S250, determining a target sub-cluster corresponding to the target cluster based on the average running time data of the target task and the target running times.
The target sub-cluster is a sub-cluster of the target cluster, and comprises 2 sub-clusters which run for a long time, namely a stable sub-cluster (stablecmaster) and a non-sub-cluster (volatilecrouter), the two sub-clusters run for a short time, the applied resources are used, and the execution efficiency is high.
Wherein determining a target subset group corresponding to the target cluster based on the target number of runs comprises: according to the target operation times, namely the successful operation times corresponding to the set time period, as the conditions of influencing the operation of other jobs due to faults in the stable sub-cluster are very few, the target tasks with the target operation times greater than or equal to the preset time threshold are deployed to the stable sub-cluster in consideration of the low fault rate of more recent successful operation times; the fault rate of few successful operation times is high recently, so that target tasks with the operation target operation times smaller than the preset time threshold are deployed to a non-sub cluster, and the situation that other operations are affected by a certain fault in the cluster occasionally occurs, but the total service time is not affected after resubmitting due to the short operation time.
Wherein determining a target subset group corresponding to the target cluster based on the average runtime data of the target task comprises: determining historical running time data of the target task in a set time period to obtain average running time data of the target task; deploying the target tasks with average running time data smaller than a preset time threshold value to a stable subset group; and deploying the target tasks with the average running time data being greater than or equal to a preset time threshold value to the non-sub-clusters.
In summary, deploying the target tasks with average running time data smaller than a preset time threshold and target running times larger than or equal to a preset times threshold to the stable subset group; and deploying the target tasks with the operation target operation times smaller than a preset time threshold and the average operation time data larger than or equal to the preset time threshold to the non-sub-clusters.
S260, executing the target task based on the target sub-cluster.
Specifically, the target task is executed based on the target sub-cluster, and after the target task is executed, the average running time data and the target running times are updated.
According to the technical scheme, only the advantages of a Session mode and an Application mode are combined, and an Application-like mode is used for starting the operation of a single cluster for tasks with large data volume; the task with small data volume is operated by using a mode similar to a Session mode and using a cluster which exists for a long time, and the operation with low failure rate is put into a stable subset group according to the latest successful operation times, and the operation with high failure rate is put into a non-sub-cluster, so that the influence on other tasks is effectively reduced. Compared with the mode that only a Session mode or an Application mode can be selected to deploy tasks in the prior art, the flexibility of the Flink in Native K8S operation deployment is improved.
Fig. 3 is a block diagram of a deployment apparatus according to an embodiment of the present invention, where the embodiment is applicable to a scenario of deployment based on a link, and the apparatus may be implemented in a form of hardware and/or software, and integrated into a processor of an electronic device with an application development function.
As shown in fig. 3, the deployment apparatus includes: a sub-cluster determining module 301, configured to determine a sub-cluster to be executed corresponding to a target cluster based on deployment association information of the target task, where the deployment association information includes a deployment data amount of the target task; a subset group starting module 302, configured to start the to-be-executed sub-cluster, so as to execute a target task based on the to-be-executed sub-cluster, and obtain target runtime data corresponding to the target task; the task execution module 303 is configured to determine, based on the target runtime data and a target number of times of execution, a target sub-cluster corresponding to the target task, so as to execute the target task based on the target sub-cluster, where the target sub-cluster is a subset of the target cluster, and the target number of times of execution is a number of times of successful execution corresponding to the target task in a set period. The problem of lower deployment efficiency is solved, and the deployment efficiency is improved.
Optionally, the subset group determining module 301 is specifically configured to:
and determining a sub-cluster to be operated corresponding to the target cluster based on the deployment data volume and the preset data volume.
Optionally, the subset group initiation module 302 includes a runtime determination unit, where the runtime determination unit is configured to:
determining at least one subtask corresponding to the target task;
executing at least one sub-task based on the sub-cluster to be executed to obtain original result data and original running time data corresponding to each sub-task;
performing data fusion operation on all original result data to obtain fusion result data and to-be-processed running time data corresponding to the data fusion operation;
and determining target running time data corresponding to a target task based on each piece of original running time data and the running time data to be processed.
Optionally, the runtime determination unit is specifically configured to:
determining the maximum runtime data in all the original runtime data;
and summing the maximum operation time data and the operation time data to be processed to obtain target operation time data corresponding to the target task.
Optionally, the task execution module 303 includes a target sub-cluster determining unit, where the target sub-cluster determining unit is configured to:
and if the target running time data is greater than or equal to a preset time threshold, determining a target sub-cluster corresponding to the target task as the sub-cluster to be run.
Optionally, the target sub-cluster determining unit is specifically configured to:
and if the target task is the first running task, determining that the target sub-cluster is the sub-cluster to be run.
Optionally, the target sub-cluster determining unit is further configured to:
if the target running time data is smaller than a preset time threshold value and the target running times are smaller than a preset times threshold value, determining a target sub-cluster corresponding to the target task as a first sub-cluster;
and if the target running time data is smaller than a preset time threshold and the target running times are larger than or equal to a preset times threshold, determining a target sub-cluster corresponding to the target task as a second sub-cluster.
The deployment device provided by the embodiment of the invention can execute the deployment method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 4 is a block diagram of an electronic device according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the deployment method.
In some embodiments, the deployment method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the deployment method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the deployment method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable deployment apparatus, such that the computer programs, when executed by the processor, cause the functions/operations specified in the flowchart and/or block diagram to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims (10)

1. A deployment method, comprising:
determining a sub-cluster to be operated corresponding to a target cluster based on deployment associated information of the target task, wherein the deployment associated information comprises deployment data quantity of the target task;
starting the sub-cluster to be operated so as to execute a target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task;
And determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period.
2. The method of claim 1, wherein the determining a sub-cluster to be run corresponding to the target cluster based on the deployment-related information of the target task comprises:
and determining a sub-cluster to be operated corresponding to the target cluster based on the deployment data volume and the preset data volume.
3. The method of claim 1, wherein the executing the target task based on the sub-cluster to be executed to obtain target runtime data corresponding to the target task comprises:
determining at least one subtask corresponding to the target task;
executing at least one sub-task based on the sub-cluster to be executed to obtain original result data and original running time data corresponding to each sub-task;
performing data fusion operation on all original result data to obtain fusion result data and to-be-processed running time data corresponding to the data fusion operation;
And determining target running time data corresponding to a target task based on each piece of original running time data and the running time data to be processed.
4. A method according to claim 3, wherein said determining target runtime data corresponding to a target task based on each of said raw runtime data and said pending runtime data comprises:
determining the maximum runtime data in all the original runtime data;
and summing the maximum operation time data and the operation time data to be processed to obtain target operation time data corresponding to the target task.
5. The method of claim 1, wherein the determining a target sub-cluster corresponding to the target task based on the target runtime data and a target number of runs comprises:
and if the target running time data is greater than or equal to a preset time threshold, determining a target sub-cluster corresponding to the target task as the sub-cluster to be run.
6. The method of claim 1 or 5, wherein the determining a target sub-cluster corresponding to the target task based on the target runtime data and a target number of runs comprises:
And if the target task is the first running task, determining that the target sub-cluster is the sub-cluster to be run.
7. The method of claim 1, 5 or 6, wherein the target sub-cluster comprises a first sub-cluster and a second sub-cluster,
the determining a target sub-cluster corresponding to the target task based on the target runtime data and the target number of runs includes:
if the target running time data is smaller than a preset time threshold value and the target running times are smaller than a preset times threshold value, determining a target sub-cluster corresponding to the target task as a first sub-cluster;
and if the target running time data is smaller than a preset time threshold and the target running times are larger than or equal to a preset times threshold, determining a target sub-cluster corresponding to the target task as a second sub-cluster.
8. A deployment apparatus, comprising:
the sub-cluster determining module is used for determining sub-clusters to be operated corresponding to a target cluster based on deployment associated information of the target task, wherein the deployment associated information comprises deployment data quantity of the target task;
the sub-cluster starting module is used for starting the sub-cluster to be operated so as to execute a target task based on the sub-cluster to be operated and obtain target operation time data corresponding to the target task;
And the task execution module is used for determining a target sub-cluster corresponding to the target task based on the target running time data and the target running times, so as to execute the target task based on the target sub-cluster, wherein the target sub-cluster is a sub-cluster of the target cluster, and the target running times are successful running times corresponding to the target task in a set period.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the deployment method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the deployment method of any one of claims 1-7.
CN202311657760.5A 2023-12-05 2023-12-05 Deployment method, device, equipment and storage medium Pending CN117632370A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311657760.5A CN117632370A (en) 2023-12-05 2023-12-05 Deployment method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311657760.5A CN117632370A (en) 2023-12-05 2023-12-05 Deployment method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117632370A true CN117632370A (en) 2024-03-01

Family

ID=90030244

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311657760.5A Pending CN117632370A (en) 2023-12-05 2023-12-05 Deployment method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117632370A (en)

Similar Documents

Publication Publication Date Title
JP7214786B2 (en) Scheduling method, device, device and medium for deep learning inference engine
US11184263B1 (en) Intelligent serverless function scaling
CN114840323A (en) Task processing method, device, system, electronic equipment and storage medium
CN111858040A (en) Resource scheduling method and device
CN114490048A (en) Task execution method and device, electronic equipment and computer storage medium
CN116661960A (en) Batch task processing method, device, equipment and storage medium
CN114816709A (en) Task scheduling method, device, server and readable storage medium
WO2024139754A1 (en) Test node regulation and control method and apparatus, electronic device and storage medium
JP6488910B2 (en) Control method, control program, and information processing apparatus
CN113986497A (en) Queue scheduling method, device and system based on multi-tenant technology
CN112187570A (en) Risk detection method and device, electronic equipment and readable storage medium
CN116521573A (en) Automatic test method, device, equipment and storage medium
CN115756808A (en) Resource adjusting method, device, equipment and storage medium
CN116319758A (en) Data migration method, device, electronic equipment and readable storage medium
CN117632370A (en) Deployment method, device, equipment and storage medium
WO2022111466A1 (en) Task scheduling method, control method, electronic device and computer-readable medium
CN115437709A (en) Method and device for loading application home page splash screen resources
CN115269145A (en) High-energy-efficiency heterogeneous multi-core scheduling method and device for offshore unmanned equipment
CN114385351A (en) Cloud management platform load balancing performance optimization method, device, equipment and medium
CN114416357A (en) Method and device for creating container group, electronic equipment and medium
CN116893893B (en) Virtual machine scheduling method and device, electronic equipment and storage medium
CN117057411B (en) Large language model training method, device, equipment and storage medium
CN117407111A (en) Method and device for starting graphic display service, electronic equipment and storage medium
CN117472570A (en) Method, apparatus, electronic device and medium for scheduling accelerator resources
Righi et al. On providing on-the-fly resizing of the elasticity grain when executing HPC applications in the cloud

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination