CN114827157A - Cluster task processing method, device and system, electronic equipment and readable medium - Google Patents
Cluster task processing method, device and system, electronic equipment and readable medium Download PDFInfo
- Publication number
- CN114827157A CN114827157A CN202210381189.8A CN202210381189A CN114827157A CN 114827157 A CN114827157 A CN 114827157A CN 202210381189 A CN202210381189 A CN 202210381189A CN 114827157 A CN114827157 A CN 114827157A
- Authority
- CN
- China
- Prior art keywords
- server
- task
- cluster
- cluster task
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 92
- 238000000034 method Methods 0.000 claims abstract description 45
- 230000008569 process Effects 0.000 claims description 23
- 230000002159 abnormal effect Effects 0.000 claims description 22
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000012423 maintenance Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1014—Server selection for load balancing based on the content of a request
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
Abstract
The application relates to a cluster task processing method, a cluster task processing device, a cluster task processing system, an electronic device and a computer readable medium. The method comprises the following steps: acquiring cluster tasks to be processed; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; and sending the cluster task to the first server for task processing. The cluster task processing method, the cluster task processing device, the cluster task processing system, the electronic equipment and the computer readable medium can select the server with the best performance to execute the cluster task, comprehensively consider the characteristics of different types of servers, guarantee the normal operation of the cluster task in a flexible and various mode, and improve the stability of the cluster task and the resource utilization rate.
Description
Technical Field
The present application relates to the field of computer information processing, and in particular, to a method, an apparatus, a system, an electronic device, and a computer-readable medium for processing a cluster task.
Background
In the process of cluster task processing, there are often some tasks with short running time but high concurrency, such as timing tasks, and in the running process of the timing tasks, there are the characteristics of short running time, high concurrency, and the like: for example, 1000 timing tasks exist in a cluster, each timing task is executed once every minute for 30s, in this scenario, sufficient resources have to be reserved for the 1000 timing tasks, and only half of the resources are actually used (only 30s of use time and no task is used for the remaining 30s of use every 1 minute).
Meanwhile, in the process of running the timed task, frequent task creation and destruction can bring fragmentation of server resources, such as memory, cgroup, disk and the like; the large amount of fragmented server resources can affect the stability of other services on the server.
In this case, how to ensure the proper execution of the cluster task, when the server is unstable, the problem processing problem is found in time, which is a difficult problem in the cluster task processing.
Therefore, a new cluster task processing method, apparatus, system, electronic device, and computer readable medium are needed.
The above information disclosed in this background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
In view of this, the present application provides a method, an apparatus, a system, an electronic device, and a computer readable medium for processing a cluster task, which can select a server with optimal performance to execute the cluster task, and comprehensively consider the characteristics of different types of servers, and ensure the normal operation of the cluster task in a flexible and diverse manner, thereby improving the stability of the cluster task and the resource utilization rate.
Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.
According to an aspect of the present application, a method for processing a cluster task is provided, where the method includes: acquiring cluster tasks to be processed; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; and sending the cluster task to the first server for task processing.
In an exemplary embodiment of the present application, further comprising: monitoring the abnormal exit probability of the cluster task in the process of task processing of the first server; determining a second server based on a target scheduling policy when the abnormal exit probability is greater than a threshold; and sending the cluster task to the second server for task processing.
In an exemplary embodiment of the present application, determining the second server based on the preset scheduling policy includes: extracting a plurality of server sequences from a target scheduling strategy; the second server is a later group of servers of the first server based on the server ranking.
In an exemplary embodiment of the present application, determining a target service type according to the identification of the cluster task includes: determining service types corresponding to the cluster tasks according to the operation of the user; generating a service type table based on a plurality of cluster tasks and corresponding service types thereof; and determining the target service type in a service type table according to the identification of the cluster task.
In an exemplary embodiment of the present application, determining the first server based on the target service type and the target scheduling policy includes: determining corresponding scheduling strategies for a plurality of service types according to the operation of a user; generating a scheduling policy table based on a plurality of service types and scheduling policies corresponding to the service types; and determining the first server in a scheduling policy table according to the identification of the cluster task.
In an exemplary embodiment of the present application, determining the first server in a scheduling policy table according to the identifier of the cluster task includes: extracting a plurality of servers from a scheduling policy table according to the identification of the cluster task; sequentially extracting the servers according to the sequence of the servers and pre-distributing server resources; and when the pre-allocation of the server resources is successful, taking the current server as the first server.
In an exemplary embodiment of the present application, when pre-allocating server resources is successful, the method includes: determining that pre-allocating server resources is successful when the server resources are unmasked and the server resource quota is not hit.
In an exemplary embodiment of the present application, sending the cluster task to the first server for task processing includes: sending the cluster task to a third-party cloud server for task processing; and/or sending the cluster task to a low-cost server for task processing; and/or sending the cluster task to a common server for task processing.
According to an aspect of the present application, a cluster task processing apparatus is provided, the apparatus including: the task module is used for acquiring cluster tasks to be processed; the type module is used for determining a target service type according to the identification of the cluster task; a policy module to determine a first server based on the target service type and a target scheduling policy; and the processing module is used for sending the cluster task to the first server for task processing.
According to an aspect of the present application, a cluster task processing system is provided, which includes: the scheduling server is used for acquiring cluster tasks to be processed; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; sending the cluster task to the first server for task processing; the third-party cloud server is used for serving as the first server to process the cluster task according to the scheduling of the scheduling server; the low-cost server is used for serving as the first server to process the cluster task according to the scheduling of the scheduling server; and the common server is used as the first server to process the cluster task according to the scheduling of the scheduling server.
According to an aspect of the present application, an electronic device is provided, the electronic device including: one or more processors; storage means for storing one or more programs; when executed by one or more processors, cause the one or more processors to implement a method as above.
According to an aspect of the application, a computer-readable medium is proposed, on which a computer program is stored, which program, when being executed by a processor, carries out the method as above.
According to the cluster task processing method, device, system, electronic equipment and computer readable medium, cluster tasks to be processed are obtained; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; the cluster task is sent to the first server for task processing, so that the server with the best performance can be selected to execute the cluster task, the characteristics of different types of servers are comprehensively considered, the normal operation of the cluster task is ensured by adopting a flexible and various mode, and the stability and the resource utilization rate of the cluster task are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings. The drawings described below are only some embodiments of the present application, and other drawings may be derived from those drawings by those skilled in the art without inventive effort.
FIG. 1 is a block diagram illustrating a clustered task processing system in accordance with an exemplary embodiment.
FIG. 2 is a flow diagram illustrating a method of cluster task processing in accordance with an exemplary embodiment.
Fig. 3 is a flowchart illustrating a cluster task processing method according to another exemplary embodiment.
Fig. 4 is a flowchart illustrating a cluster task processing method according to another exemplary embodiment.
FIG. 5 is a block diagram illustrating a clustered task processing device in accordance with an example embodiment.
FIG. 6 is a block diagram illustrating an electronic device in accordance with an example embodiment.
FIG. 7 is a block diagram illustrating a computer-readable medium in accordance with an example embodiment.
Detailed Description
In describing particular embodiments, specific details of structures, properties, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by one skilled in the art. However, it is not excluded that a person skilled in the art may implement the invention in a specific case without the above-described structures, performances, effects or other features.
The flow chart in the drawings is only an exemplary flow demonstration, and does not represent that all the contents, operations and steps in the flow chart are necessarily included in the scheme of the invention, nor does it represent that the execution is necessarily performed in the order shown in the drawings. For example, some operations/steps in the flowcharts may be divided, some operations/steps may be combined or partially combined, and the like, and the execution order shown in the flowcharts may be changed according to actual situations without departing from the gist of the present invention.
The block diagrams in the figures generally represent functional entities and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and/or processing unit devices and/or microcontroller devices.
The same reference numerals denote the same or similar elements, components, or parts throughout the drawings, and thus, a repetitive description thereof may be omitted hereinafter. It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, or sections, these elements, components, or sections should not be limited by these terms. That is, these phrases are used only to distinguish one from another. For example, a first device may also be referred to as a second device without departing from the spirit of the present invention. Furthermore, the term "and/or", "and/or" is intended to include all combinations of any one or more of the listed items.
In order that the objects, technical solutions and advantages of the present invention will become more apparent, the present invention will be further described in detail with reference to the accompanying drawings in conjunction with the following specific embodiments.
In describing particular embodiments, specific details of structures, properties, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by one skilled in the art. However, it is not excluded that a person skilled in the art may implement the invention in a specific case without the above-described structures, performances, effects or other features.
It will be appreciated by those skilled in the art that the drawings are merely schematic representations of exemplary embodiments, and that the blocks or processes shown in the drawings are not necessarily required to practice the present application and are, therefore, not intended to limit the scope of the present application.
FIG. 1 is a block diagram illustrating a clustered task processing system in accordance with an exemplary embodiment.
As shown in fig. 1, the system architecture 10 may include terminal devices 101, 102, 103, a network 104 and a scheduling server 105, a third party cloud server 106, a low cost server 107, a general server 108. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the scheduling server 105, between the scheduling server 105, the third-party cloud server 106, the low-cost server 107, and the general server 108. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the scheduling server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The scheduling server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the query request, and forward the data to the third-party cloud server 106 and/or the low-cost server 107 and/or the common server 108 for processing.
The third-party cloud server 106 may be a serverless server, and also may be a serverless application. Serverless relies on a third-party cloud service platform, the service end logically runs in a stateless computing container, and the state of the service level is recorded by a database and a storage resource used by a developer.
The low-cost server 107 is a type of server constructed to reduce the manufacturing and usage costs of the server, and is limited in performance according to the target cost.
Among other things, the regular server 108 is one of the computers that runs faster, is more heavily loaded, and is more expensive than regular computers. The server provides calculation or application services for other clients (such as terminals like PC, smart phone, ATM and the like and even large equipment like train system and the like) in the network. The server has high-speed CPU computing capability, long-time reliable operation, strong I/O external data throughput capability and better expansibility.
The scheduling server 105 may, for example, obtain the cluster tasks to be processed; the scheduling server 105 may determine a target service type, for example, from the identification of the cluster task; dispatch server 105 may determine a first server, for example, based on the target service type and a target dispatch policy; the scheduling server 105 may, for example, send the cluster task to the first server for task processing; the third-party cloud server 106 may perform cluster task processing as the first server according to the scheduling of the scheduling server, for example; the low cost server 107 may, for example, perform the processing of the cluster task as the first server according to the scheduling of the scheduling server; the generic server 108 may perform the processing of the cluster task as the first server, e.g., according to the scheduling of the scheduling server.
The scheduling server 105 may be a single entity server, or may be composed of a plurality of servers, for example, it should be noted that the cluster task processing method provided in the embodiment of the present application may be executed by the scheduling server 105, and accordingly, the cluster task processing device may be disposed in the scheduling server 105. And the web page end provided for the user to do quantity browsing is generally located in the terminal equipment 101, 102, 103. The computing side executing the clustering task is generally located in a third-party cloud server 106, a low-cost server 107 and a common server 108.
FIG. 2 is a flow diagram illustrating a method of cluster task processing in accordance with an exemplary embodiment. The cluster task processing method 20 includes at least steps S202 to S208.
As shown in fig. 2, in S202, a cluster task to be processed is acquired. The dispatching server can obtain the cluster task by the client and can also locally call the cluster task for execution.
Without loss of generality, in the embodiments of the present application, the timing cluster task is taken as an example to perform the description of the subsequent embodiments.
In S204, a target service type is determined according to the identifier of the cluster task. Determining the service types corresponding to the plurality of cluster tasks according to the operation of the user; generating a service type table based on a plurality of cluster tasks and corresponding service types thereof; and determining the target service type in a service type table according to the identification of the cluster task.
In one particular embodiment, cluster resource usage rules may be created based on administrator actions, which may, for example, assume that there are serverless, low cost task server, normal server level resources in the cluster.
In S206, a first server is determined based on the target service type and a target scheduling policy. Determining corresponding scheduling strategies for a plurality of service types according to the operation of the user; generating a scheduling policy table based on a plurality of service types and scheduling policies corresponding to the service types; and determining the first server in a scheduling policy table according to the identification of the cluster task.
In a specific embodiment, the available resources of the timing task of the service B in the service line a are "serverless, low-cost task server, normal server", and are degraded according to the sequence.
The server may be determined as the first server for the timed cluster task of service B under service line a according to the above policies.
In S208, the cluster task is sent to the first server for task processing. The cluster task can be sent to a third-party cloud server for task processing; the cluster task can be sent to a low-cost server for task processing; the cluster task can be sent to a common server for task processing.
According to the cluster task processing method, cluster tasks to be processed are obtained; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; the cluster task is sent to the first server for task processing, so that the server with the best performance can be selected to execute the cluster task, the characteristics of different types of servers are comprehensively considered, the normal operation of the cluster task is ensured by adopting a flexible and various mode, and the stability and the resource utilization rate of the cluster task are improved.
In a specific application scenario, the cluster task processing method can be applied to tasks of data search and processing classes, and can be provided for a user APP to operate. And the background server receives processing instructions sent by a large number of users through the APP and processes the processing instructions through the cluster task server.
In the cluster task server, there is a scheduling server, which may be a specific server in the cluster task, or may also be a server whose processing task is idle at the current time in the cluster task. And the scheduling server regularly acquires the resource running state and the task running state of all servers in the whole cluster every 10 seconds. The operating state may include: network, disk capacity, whether the disk is read-only, and the number of running test tasks.
The specific judgment is as follows:
the overtime of the processing time of the three continuous tasks is regarded as the running state of the server resource is abnormal;
the server resource operation state is considered abnormal when the disk capacity exceeds a certain 80% in three continuous operation states;
if the disk state in operation is a read-only state, judging that the server resource operation state is abnormal;
and the failure of the task processing for three times is regarded as the running state exception of the server resource.
And the scheduling server judges whether the servers in the cluster can bear the task state of the next period (10 seconds) according to the monitoring of the resource running state and the task running state of the servers in the cluster, and stores the task state in a resource list.
When a task request sent by an APP of a user is received, a server type name to be processed is allocated to the user according to a service line and a service attribute corresponding to the task request. The server types are arranged from high level to low level one by one, and can be, for example, serverless, low-cost task server, normal server.
Firstly, a server with a service type of a serverless is selected from a resource list, and if the server corresponding to the serverless exists, the scheduling server forwards the task processing request to the serverless for processing. Otherwise, selecting the servers of the next level one by one to distribute the tasks.
In the process of processing by a server type server, a scheduling server detects the task processing state at regular time, and selects the servers of the next level one by one to perform task allocation again when the tasks are not completed on time or the server resources are abnormal.
It should be clearly understood that this application describes how to make and use particular examples, but the principles of this application are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.
Fig. 3 is a flowchart illustrating a cluster task processing method according to another exemplary embodiment. The process 30 shown in fig. 3 is a supplementary description of the process shown in fig. 2.
As shown in fig. 3, in S302, during the task processing performed by the first server, the abnormal exit probability of the cluster task is monitored. Continuing with the above embodiment, a serverless server is determined as the first server for the timing cluster task of service B under service line a. And detecting the abnormal exit probability of the serverless server.
In S304, whether the abnormal exit probability of the cluster task is greater than a threshold.
In S306, when the abnormal exit probability is greater than a threshold, a second server is determined based on a target scheduling policy. For example, a task running on a serverless server may be abnormal by more than 10%, an alarm may be issued to maintenance personnel, the resource may be masked, and the task on the resource may be transferred.
In S308, the cluster task is sent to the second server for task processing. The available resources of the timing task of the service B under the service line a are "serverless, low-cost task server, normal server", and when the serverless server is abnormal, the low-cost server can be selected as the second server to perform task processing.
In S310, when the abnormal exit probability is smaller than the threshold, different warning information is generated according to different abnormal exit probability values.
In one embodiment, the abnormal exit probability of the task running on the first server is less than 0.01%, the probability of accidental phenomenon is considered to be caused by non-failure, and an alarm is sent to the creator check of the part of the abnormal exit task
In one embodiment, the abnormal exit probability of the task running on the first server is 0.01% -0.1%, and an alarm is sent to maintenance personnel to check whether hidden problems exist or not
In one embodiment, the abnormal exit probability of the task running on the first server is 0.1% -10%, an alarm is sent to a maintenance person for inspection, and the scheduling module is notified to reduce the value corresponding to the allocation rate of the resource.
Fig. 4 is a flowchart illustrating a cluster task processing method according to another exemplary embodiment. The process 40 shown in fig. 4 is a detailed description of the step S206 "determining the first server in the scheduling policy table according to the identifier of the clustered task" in the process shown in fig. 2.
As shown in fig. 4, in S402, a plurality of servers are extracted from the scheduling policy table according to the identification of the cluster task. Take the available resources of the timing task of the service B under the service line a as "serverless, low-cost task server, normal server" as an example.
In S404, servers are sequentially extracted and server resources are pre-allocated according to the ranking of the plurality of servers.
In S406, when the server resource is not masked and the server resource quota is not hit, it is determined that pre-allocating the server resource is successful.
In a specific embodiment, it may be determined whether the resource is masked first, and if so, the resource cannot be allocated, and a next resource determination is performed; and judging whether the resource has a decrement, if so, judging by using a random function rand, if the decrement is hit, the resource cannot be distributed, and judging the next resource: for example, the decrement is 10%, the rand (1, 100) is used to determine whether the random number is less than or equal to 10.
In S408, the current server is taken as the first server.
In one embodiment, when pre-allocating the serverless resources is successful, the service copy is scheduled to the serverless, and the decision is ended.
In one embodiment, when the pre-allocation of the server resources is unsuccessful, the low-cost server resources are pre-allocated, and when the allocation of the low-cost server resources is successful, the service copy is dispatched to the low-cost server, and the decision is ended.
In one embodiment, when the pre-allocation of the low-cost server resources is unsuccessful, the normal server resources are pre-allocated, and when the allocation of the normal server resources is successful, the service copy is dispatched to the normal server, and the decision is ended.
In one embodiment, when the resource allocation of the common server is unsuccessful, the scheduling is determined to be failed, and alarm information is generated.
In a specific application, the running state of the resource on the server and the running state of the upper task can be obtained at regular time (the time of the timing can be configured, and for example, the time is once every 10 s). The resource operation state comprises a network, a disk capacity, whether a disk is read only and a test task is operated, the judgment strategy for the resource operation state is configurable, and the specific strategy can be as follows:
1. requesting the overtime of the running state, regarding the overtime of three times as network abnormity, sending an alarm to maintenance personnel and shielding the resource, wherein the resource does not participate in subsequent task allocation any more, transfers the running task on the resource and does not perform subsequent judgment any more;
2. the request operation state is successful, the disk capacity in the operation state for three times exceeds a certain threshold (configurable, for example 80%, three times), and a warning is sent to maintenance personnel;
3. the request operation state is successful, the disk in the operation state is read only, the hardware fault is regarded once, an alarm is sent to maintenance personnel, the resource is shielded, the resource does not participate in subsequent task allocation any more, the task running on the resource is transferred, and subsequent judgment is not performed any more;
4. and if the request operation state is successful and the operation states are normal, and the three continuous operation task failures are regarded as unknown abnormalities, a warning is sent out, the resource is shielded, and the subsequent task allocation is not participated.
According to the cluster task processing method, the cluster servers are divided into different levels, and the resource use rule of the service is defined, so that the multilevel cluster task scheduling of the degradation strategy decision is completed.
By using resources of three levels, namely a server, a low-specification-cost server and a normal server, as used in the present case, and by combining a scheduler decision mechanism and a degradation strategy, when the server fails, the timing task can be ensured to be established on other server resources and can normally run, and the problem that the normal running of the timing task is influenced by the failure of the conventional server is avoided.
Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments are implemented as computer programs executed by a CPU. When executed by the CPU, performs the functions defined by the methods provided herein. The program may be stored in a computer readable storage medium, which may be a read-only memory, a magnetic or optical disk, or the like.
Furthermore, it should be noted that the above-mentioned figures are only schematic illustrations of the processes involved in the method according to exemplary embodiments of the present application, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.
FIG. 5 is a block diagram illustrating a clustered task processing device in accordance with an example embodiment. As shown in fig. 5, the cluster task processing device 50 includes: a task module 502, a type module 504, a policy module 506, and a processing module 508.
The task module 502 is configured to obtain a cluster task to be processed;
the type module 504 is configured to determine a target service type according to the identifier of the cluster task; the type module 504 is further configured to determine service types corresponding to the plurality of cluster tasks according to the operation of the user;
generating a service type table based on a plurality of cluster tasks and corresponding service types thereof; and determining the target service type in a service type table according to the identification of the cluster task.
The policy module 506 is configured to determine a first server based on the target service type and a target scheduling policy; the policy module 506 is further configured to determine corresponding scheduling policies for the plurality of service types according to the operation of the user; generating a scheduling policy table based on a plurality of service types and scheduling policies corresponding to the service types; and determining the first server in a scheduling policy table according to the identification of the cluster task.
The processing module 508 is configured to send the cluster task to the first server for task processing. The processing module 508 is further configured to send the cluster task to a third-party cloud server for task processing; the processing module 508 is further configured to send the cluster task to a low-cost server for task processing; the processing module 508 is further configured to send the cluster task to a common server for task processing.
According to the cluster task processing device, cluster tasks to be processed are obtained; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; the cluster task is sent to the first server for task processing, so that the server with the best performance can be selected to execute the cluster task, the characteristics of different types of servers are comprehensively considered, the normal operation of the cluster task is ensured by adopting a flexible and various mode, and the stability and the resource utilization rate of the cluster task are improved.
FIG. 6 is a block diagram illustrating an electronic device in accordance with an example embodiment.
An electronic device 600 according to this embodiment of the present application is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 that connects the various system components (including the storage unit 620 and the processing unit 610), a display unit 640, and the like.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present application described in the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 2, 3, 4.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 600' (e.g., keyboard, pointing device, bluetooth device, etc.), such that a user can communicate with devices with which the electronic device 600 interacts, and/or any device (e.g., router, modem, etc.) with which the electronic device 600 can communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware. Therefore, as shown in fig. 7, the technical solution according to the embodiment of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present application.
The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The computer readable medium carries one or more programs which, when executed by a device, cause the computer readable medium to perform the functions of: acquiring cluster tasks to be processed; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; and sending the cluster task to the first server for task processing. The computer readable medium may also implement the following functions: monitoring the abnormal exit probability of the cluster task in the process of task processing of the first server; determining a second server based on a target scheduling policy when the abnormal exit probability is greater than a threshold; and sending the cluster task to the second server for task processing.
Those skilled in the art will appreciate that the modules described above may be distributed in the apparatus according to the description of the embodiments, or may be modified accordingly in one or more apparatuses unique from the embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Exemplary embodiments of the present application are specifically illustrated and described above. It is to be understood that the application is not limited to the details of construction, arrangement, or method of implementation described herein; on the contrary, the intention is to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A cluster task processing method is characterized by comprising the following steps:
acquiring cluster tasks to be processed;
determining a target service type according to the identification of the cluster task;
determining a first server based on the target service type and a target scheduling policy;
and sending the cluster task to the first server for task processing.
2. The cluster task processing method of claim 1, further comprising:
monitoring the abnormal exit probability of the cluster task in the process of task processing of the first server;
determining a second server based on a target scheduling policy when the abnormal exit probability is greater than a threshold;
and sending the cluster task to the second server for task processing.
Optionally, determining the second server based on a preset scheduling policy includes:
extracting a plurality of server sequences from a target scheduling strategy;
taking a subsequent set of servers of the first server as the second server based on the server ranking.
3. The method of cluster task processing according to claim 1, wherein determining a target service type according to the identification of the cluster task comprises:
determining service types corresponding to the cluster tasks according to the operation of the user;
generating a service type table based on a plurality of cluster tasks and corresponding service types thereof;
and determining the target service type in a service type table according to the identification of the cluster task.
4. The clustered task processing method of claim 1 wherein determining a first server based on the target service type and a target scheduling policy comprises:
determining corresponding scheduling strategies for a plurality of service types according to the operation of a user;
generating a scheduling policy table based on a plurality of service types and scheduling policies corresponding to the service types;
and determining the first server in a scheduling policy table according to the identification of the cluster task.
5. The method of claim 4, wherein determining the first server in a scheduling policy table based on the identification of the clustered task comprises:
extracting a plurality of servers from a scheduling policy table according to the identification of the cluster task;
sequentially extracting the servers according to the sequence of the servers and pre-distributing server resources;
when the pre-allocation of server resources is successful, taking the current server as the first server;
optionally, when the pre-allocating server resource is successful, the method includes:
determining that pre-allocating server resources is successful when the server resources are unmasked and the server resource quota is not hit.
6. The method for processing the cluster task according to claim 1, wherein sending the cluster task to the first server for task processing comprises:
sending the cluster task to a third-party cloud server for task processing; and/or
Sending the cluster task to a low-cost server for task processing; and/or
And sending the cluster task to a common server for task processing.
7. A cluster task processing apparatus, comprising:
the task module is used for acquiring cluster tasks to be processed;
the type module is used for determining a target service type according to the identification of the cluster task;
a policy module to determine a first server based on the target service type and a target scheduling policy;
and the processing module is used for sending the cluster task to the first server for task processing.
8. A clustered task processing system, comprising:
the scheduling server is used for acquiring cluster tasks to be processed; determining a target service type according to the identification of the cluster task; determining a first server based on the target service type and a target scheduling policy; sending the cluster task to the first server for task processing;
the third-party cloud server is used for serving as the first server to process the cluster task according to the scheduling of the scheduling server;
the low-cost server is used for processing the cluster task as the first server according to the scheduling of the scheduling server;
and the common server is used as the first server to process the cluster task according to the scheduling of the scheduling server.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210381189.8A CN114827157A (en) | 2022-04-12 | 2022-04-12 | Cluster task processing method, device and system, electronic equipment and readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210381189.8A CN114827157A (en) | 2022-04-12 | 2022-04-12 | Cluster task processing method, device and system, electronic equipment and readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114827157A true CN114827157A (en) | 2022-07-29 |
Family
ID=82534521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210381189.8A Pending CN114827157A (en) | 2022-04-12 | 2022-04-12 | Cluster task processing method, device and system, electronic equipment and readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114827157A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117745514A (en) * | 2024-02-20 | 2024-03-22 | 浙江宇视科技有限公司 | Image distribution method, device, equipment and medium of intelligent server cluster |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108737270A (en) * | 2018-05-07 | 2018-11-02 | 北京京东尚科信息技术有限公司 | A kind of method for managing resource and device of server cluster |
| CN110018893A (en) * | 2019-03-12 | 2019-07-16 | 平安普惠企业管理有限公司 | A kind of method for scheduling task and relevant device based on data processing |
| CN110290180A (en) * | 2019-05-31 | 2019-09-27 | 北京大米科技有限公司 | Distributed task dispatching method, apparatus, computer equipment and storage medium |
| CN111597042A (en) * | 2020-05-11 | 2020-08-28 | Oppo广东移动通信有限公司 | Service thread running method and device, storage medium and electronic equipment |
| WO2020238751A1 (en) * | 2019-05-28 | 2020-12-03 | 阿里巴巴集团控股有限公司 | Resource access method under serverless architecture, device, system, and storage medium |
| CN114237886A (en) * | 2021-12-15 | 2022-03-25 | 平安证券股份有限公司 | Task processing method and device, computer equipment and storage medium |
-
2022
- 2022-04-12 CN CN202210381189.8A patent/CN114827157A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108737270A (en) * | 2018-05-07 | 2018-11-02 | 北京京东尚科信息技术有限公司 | A kind of method for managing resource and device of server cluster |
| CN110018893A (en) * | 2019-03-12 | 2019-07-16 | 平安普惠企业管理有限公司 | A kind of method for scheduling task and relevant device based on data processing |
| WO2020238751A1 (en) * | 2019-05-28 | 2020-12-03 | 阿里巴巴集团控股有限公司 | Resource access method under serverless architecture, device, system, and storage medium |
| CN110290180A (en) * | 2019-05-31 | 2019-09-27 | 北京大米科技有限公司 | Distributed task dispatching method, apparatus, computer equipment and storage medium |
| CN111597042A (en) * | 2020-05-11 | 2020-08-28 | Oppo广东移动通信有限公司 | Service thread running method and device, storage medium and electronic equipment |
| CN114237886A (en) * | 2021-12-15 | 2022-03-25 | 平安证券股份有限公司 | Task processing method and device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117745514A (en) * | 2024-02-20 | 2024-03-22 | 浙江宇视科技有限公司 | Image distribution method, device, equipment and medium of intelligent server cluster |
| CN117745514B (en) * | 2024-02-20 | 2024-05-10 | 浙江宇视科技有限公司 | Image distribution method, device, equipment and medium of intelligent server cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10048996B1 (en) | Predicting infrastructure failures in a data center for hosted service mitigation actions | |
| US20060212518A1 (en) | Copying chat data from a chat session already active | |
| US11218386B2 (en) | Service ticket escalation based on interaction patterns | |
| US9361277B2 (en) | Method and apparatus for implementing microblog message pages | |
| US10642585B1 (en) | Enhancing API service schemes | |
| CN114827157A (en) | Cluster task processing method, device and system, electronic equipment and readable medium | |
| CN114070847B (en) | Method, device, equipment and storage medium for limiting current of server | |
| CN113191889A (en) | Wind control configuration method, configuration system, electronic device and readable storage medium | |
| US11243979B1 (en) | Asynchronous propagation of database events | |
| CN111885184A (en) | Method and device for processing hot spot access keywords in high concurrency scene | |
| CN116521639A (en) | Log data processing method, electronic equipment and computer readable medium | |
| CN111010453A (en) | Service request processing method, system, electronic device and computer readable medium | |
| CN112994934B (en) | Data interaction method, device and system | |
| CN112818204B (en) | Service processing method, device, equipment and storage medium | |
| CN111784174B (en) | Method and device for managing risk policy based on user portrait and electronic equipment | |
| CN114449040A (en) | Configuration issuing method and device based on cloud platform | |
| CN113282455A (en) | Monitoring processing method and device | |
| US20080125878A1 (en) | Method and system to detect application non-conformance | |
| CN113282489A (en) | Interface testing method and device | |
| CN112613955A (en) | Order processing method and device, electronic equipment and storage medium | |
| CN111680867B (en) | Resource allocation method and device and electronic equipment | |
| CN115858319B (en) | Stream data processing method and device | |
| US20090125364A1 (en) | Framework for achieving a rewarding relationship | |
| CN110262756B (en) | Method and device for caching data | |
| US20210111941A1 (en) | Scalable subscriptions for virtual collaborative workspaces |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |