CN111190731A - Cluster task scheduling system based on weight - Google Patents
Cluster task scheduling system based on weight Download PDFInfo
- Publication number
- CN111190731A CN111190731A CN201911368322.0A CN201911368322A CN111190731A CN 111190731 A CN111190731 A CN 111190731A CN 201911368322 A CN201911368322 A CN 201911368322A CN 111190731 A CN111190731 A CN 111190731A
- Authority
- CN
- China
- Prior art keywords
- task
- interface
- state
- processing
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 claims abstract description 159
- 230000008569 process Effects 0.000 claims abstract description 155
- 238000007781 pre-processing Methods 0.000 claims description 44
- 238000004140 cleaning Methods 0.000 claims description 32
- 238000007689 inspection Methods 0.000 claims description 20
- 238000004458 analytical method Methods 0.000 claims description 19
- 230000007246 mechanism Effects 0.000 claims description 5
- 230000004083 survival effect Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 description 29
- 238000004891 communication Methods 0.000 description 17
- 238000007726 management method Methods 0.000 description 17
- 238000003860 storage Methods 0.000 description 12
- 238000004590 computer program Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 6
- 239000000872 buffer Substances 0.000 description 4
- 230000003203 everyday effect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The embodiment of the application provides a cluster task scheduling system based on weight, which comprises: the system comprises a database and an embedded framework running on each node of a computer cluster, wherein a task file, task configuration parameters and a task state are stored in the database, and the embedded framework comprises a task flow module, a task processing module and a task process interface; the method and the device can effectively save the management cost and the node resources in the computer cluster, and improve the convenience and the efficiency of scheduling.
Description
Technical Field
The application relates to the technical field of computers, in particular to a cluster task scheduling system based on weight.
Background
With the advent of the big data era, how to implement efficient storage and calculation has become more and more difficult, and under such a background, based on the network and network communication technologies, computers dispersed at different geographic locations are connected to form spatially dispersed and logically unified data storage and calculation clusters, which become the main choices for implementing large-scale data processing at present, and different big data calculation frameworks are endless, such as large data storage calculation hadoop oriented, spark oriented to memory iterative operation, storm specially for streaming calculation, and the like, which have specific optimization performance very high efficiently when processing services in respective fields, but have a problem that the cooperation between tasks depends on processing, when there is a sequential execution dependency between service calculation tasks, the computational framework is often severely constrained, even causing an order of magnitude reduction in efficiency.
A computer cluster is a computer system that is connected by multiple computers (also called nodes) to cooperate in performing computing jobs. The nodes are located in the same administrative domain, have a unified administrative policy, and provide services to users as a whole. The process of distributing a job task on a computer cluster to multiple nodes may be referred to as scheduling of the task.
In the prior art, a task scheduling system is provided, in which a node is set in a computer cluster, and other nodes are slave nodes, wherein a master node is responsible for managing the slave nodes, and the management specifically includes that the master node periodically sends heartbeat information to the slave nodes, and determines the state information of the slave nodes according to the heartbeat information replied by the slave nodes; the master node assigns job tasks to the slave nodes according to the status information of the slave nodes, and the like.
In practice, when a task processing request exists in a user application, a storage file can be sent to the task scheduling system, and the master node can select an optimal slave node to process the task processing request according to the state information of each slave node recorded in the configuration file; the master node returns the IP address and the port of the selected slave node to the user application, and simultaneously informs the slave node of uploading the file and sends information such as the file name, the file size and the like to the slave node.
However, some cluster applications have a scheduling requirement of a light task, and although the related task amount is small and the load is small, the diversity of scheduling trigger, the complexity of scheduling operation and the reliability of scheduling operation and maintenance cannot be saved, and when the task scheduling system is used for scheduling the light task, the following problems exist:
firstly, the master node needs to manage a plurality of slave nodes in the computer cluster, which undoubtedly consumes a great deal of management cost;
secondly, the plurality of slave nodes are too large for light tasks, and excessive node resources are wasted;
thirdly, it needs to rely on the communication interaction between the user application and the task scheduling system, so that when a certain task needs to be executed periodically (for example, every day), the user application needs to send a corresponding task processing request to the task scheduling system every day, and thus the task scheduling system in the prior art has the disadvantages of cumbersome operation and low efficiency.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a cluster task scheduling system based on weight, which can effectively save the management cost and node resources in a computer cluster and improve the convenience and efficiency of scheduling.
In order to solve at least one of the above problems, the present application provides the following technical solutions:
in a first aspect, the present application provides a cluster task scheduling system based on weight, including:
the system comprises a database and an embedded framework running on each node of a computer cluster;
the database stores task files, task configuration parameters and task states;
the embedded framework comprises a task flow module, a task processing module and a task process interface, wherein the task process interface is packaged with a processing function of a preset task, and the task flow module provides a flow interface;
the task processing module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found, and calling the flow interface to execute corresponding task processing;
the task flow module is used for analyzing the task process of the preset task when the flow interface is called by the task processing module, calling the corresponding task process interface according to the analysis result to execute the processing of the corresponding task process, and switching the task state after the processing of the corresponding task process is successful.
Further, the embedded framework further comprises a node management module;
the node management module is used for registering the survival statement of the node where the node is located with the database, and acquiring the failure node from the database so as to take over the task of the failure node.
Further, the task processing module includes:
and the task execution sub-module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found to be in a to-be-processed state, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired.
Further, the task flow module is specifically configured to, when the flow execution interface is called by the task execution sub-module, analyze a task process related to the preset task, call the process service processing interface according to an analysis result to perform processing of a service logic process, and convert a state of the preset task from a to-be-processed state to a processing-successful state after the processing of the service logic process is successful.
Further, the task process interface further comprises a final environment cleaning interface, and a cleaning function of the running environment when the task is terminated is packaged in the final environment cleaning interface;
the task flow module is also used for calling the final environment cleaning interface after the business logic process is successfully processed to clean the running environment when the task is terminated, and converting the state of the preset task from a state to be processed into a state of successful processing after the running environment is successfully cleaned when the task is terminated.
Furthermore, the states of the preset tasks also comprise a breakpoint redo state, the task process interface also comprises a process breakpoint redo environment cleaning interface, and the process breakpoint redo environment cleaning interface is packaged with a cleaning function of a service logic process file;
the task execution sub-module is also used for scanning the database, locking and acquiring a task file of a preset task when the preset task in the breakpoint redoing state is found, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired;
the first task flow module is further configured to, when the flow execution interface is called by the task execution submodule, analyze a task process related to the preset task, call the process breakpoint redo environment cleaning interface according to an analysis result to clean the service logic process file, and convert the task state from the breakpoint redo state to a to-be-processed state after the service logic process file is successfully cleaned.
Furthermore, the state of the preset task also comprises a state to be checked, the process interface also comprises a process check interface, the task process interface comprises a process environment check interface, and a function of environment check before operation is packaged in the process environment check interface before operation;
the task processing module further comprises:
the task checking sub-module is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be checked is found, and calling the flow checking interface to execute corresponding task processing after the task file is successfully acquired;
and the second task flow module is also used for analyzing the task process related to the preset task when the flow inspection interface is called by the task inspection submodule, calling the process environment inspection interface according to the analysis result to carry out the environment inspection before operation, and converting the task state from the state to be inspected into the state to be processed after the environment inspection is successful before operation.
Furthermore, the state of the preset task also comprises a state to be preprocessed, the process interface also comprises a process preprocessing interface, the task process interface also comprises a process preprocessing interface, and a preprocessing function before operation is packaged in the process preprocessing interface;
the task processing module further comprises:
the task preprocessing submodule is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be preprocessed is found, and calling the flow preprocessing interface to execute corresponding task processing after the task file is successfully acquired;
and the third task flow module is also used for analyzing the task process related to the preset task when the flow preprocessing interface is called by the preprocessing submodule, calling the process preprocessing interface according to the analysis result to carry out preprocessing before operation, and converting the task state from the state to be preprocessed into the state to be checked after the preprocessing is successful before the operation.
Further, the task processing module is specifically configured to scan the database, obtain a task file of a task in a preset task state by using an optimistic lock mechanism when the task is found, and call the flow interface to execute corresponding task processing after the task is successfully obtained.
Further, the task configuration parameters stored in the database include a maximum number of processing nodes, and the task file of the task is allowed to be acquired when the actual number of processing nodes of the task does not exceed the maximum number of processing nodes.
According to the technical scheme, the cluster task scheduling system based on the weight is characterized in that a database and embedded frameworks running on all nodes of a computer cluster are arranged, task files, task configuration parameters and task states are stored in the database, the embedded frameworks comprise a task flow module, a task processing module and a task process interface, management cost and node resources in the computer cluster can be effectively saved, and convenience and efficiency of scheduling are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a weight-based cluster task scheduling system in an embodiment of the present application;
fig. 2 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In consideration of the problems of diversity of scheduling trigger, complexity of scheduling operation and reliability of scheduling operation and maintenance in the conventional task scheduling system, the application provides a cluster task scheduling system based on weight, a database and an embedded framework running on each node of a computer cluster are arranged, task files, task configuration parameters and task states are stored in the database, and the embedded framework comprises a task flow module, a task processing module and a task process interface, so that the management cost and node resources in the computer cluster can be effectively saved, and the scheduling convenience and efficiency are improved.
In order to effectively save management cost and node resources in a computer cluster and improve scheduling convenience and efficiency, the present application provides an embodiment of a cluster task scheduling system based on weights, and referring to fig. 1, the cluster task scheduling system based on weights specifically includes the following contents:
the system comprises a database and an embedded framework running on each node of a computer cluster;
the database stores task files, task configuration parameters and task states;
the embedded framework comprises a task flow module, a task processing module and a task process interface, wherein the task process interface is packaged with a processing function of a preset task, and the task flow module provides a flow interface;
the task processing module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found, and calling the flow interface to execute corresponding task processing;
the task flow module is used for analyzing the task process of the preset task when the flow interface is called by the task processing module, calling the corresponding task process interface according to the analysis result to execute the processing of the corresponding task process, and switching the task state after the processing of the corresponding task process is successful.
As can be seen from the above description, the cluster task scheduling system based on weight provided in the embodiment of the present application can set a database and an embedded framework running on each node of a computer cluster, where a task file, a task configuration parameter, and a task state are stored in the database, and the embedded framework includes a task flow module, a task processing module, and a task process interface, so that management cost and node resources in the computer cluster can be effectively saved, and convenience and efficiency of scheduling are improved.
In some embodiments of the present application, the embedded framework further comprises a node management module;
the node management module is used for registering the survival statement of the node where the node is located with the database, and acquiring the failure node from the database so as to take over the task of the failure node.
In some embodiments of the present application, the task processing module comprises:
and the task execution sub-module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found to be in a to-be-processed state, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired.
In some embodiments of the application, the task flow module is specifically configured to, when the flow execution interface is called by the task execution sub-module, analyze a task process related to the preset task, call the process service processing interface according to an analysis result to process a service logic process, and convert a state of the preset task from a to-be-processed state to a processing-successful state after the service logic process is successfully processed.
In some embodiments of the present application, the task process interface further includes a final environment cleaning interface, and a cleaning function of the operation environment at the time of task termination is encapsulated in the final environment cleaning interface;
the task flow module is also used for calling the final environment cleaning interface after the business logic process is successfully processed to clean the running environment when the task is terminated, and converting the state of the preset task from a state to be processed into a state of successful processing after the running environment is successfully cleaned when the task is terminated.
In some embodiments of the present application, the states of the preset tasks further include a breakpoint redo state, the task process interface further includes a process breakpoint redo environment cleaning interface, and the process breakpoint redo environment cleaning interface encapsulates a cleaning function of a service logic process file;
the task execution sub-module is also used for scanning the database, locking and acquiring a task file of a preset task when the preset task in the breakpoint redoing state is found, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired;
the first task flow module is further configured to, when the flow execution interface is called by the task execution submodule, analyze a task process related to the preset task, call the process breakpoint redo environment cleaning interface according to an analysis result to clean the service logic process file, and convert the task state from the breakpoint redo state to a to-be-processed state after the service logic process file is successfully cleaned.
In some embodiments of the present application, the state of the preset task further includes a state to be checked, the process interface further includes a process check interface, the task process interface includes a process environment check interface, and a function of pre-operation environment check is encapsulated in the pre-operation environment check interface of the process;
the task processing module further comprises:
the task checking sub-module is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be checked is found, and calling the flow checking interface to execute corresponding task processing after the task file is successfully acquired;
and the second task flow module is also used for analyzing the task process related to the preset task when the flow inspection interface is called by the task inspection submodule, calling the process environment inspection interface according to the analysis result to carry out the environment inspection before operation, and converting the task state from the state to be inspected into the state to be processed after the environment inspection is successful before operation.
In some embodiments of the present application, the state of the preset task further includes a state to be preprocessed, the process interface further includes a process preprocessing interface, the task process interface further includes a process preprocessing interface, and a function of preprocessing before operation is encapsulated in the process preprocessing interface;
the task processing module further comprises:
the task preprocessing submodule is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be preprocessed is found, and calling the flow preprocessing interface to execute corresponding task processing after the task file is successfully acquired;
and the third task flow module is also used for analyzing the task process related to the preset task when the flow preprocessing interface is called by the preprocessing submodule, calling the process preprocessing interface according to the analysis result to carry out preprocessing before operation, and converting the task state from the state to be preprocessed into the state to be checked after the preprocessing is successful before the operation.
In some embodiments of the present application, the task processing module is specifically configured to scan the database, obtain a task file of a task in a preset task state by using an optimistic lock mechanism when the task is found, and call the flow interface to execute corresponding task processing after the task is successfully obtained.
In some embodiments of the present application, the task configuration parameters stored in the database include a maximum number of processing nodes, and the task file of the task is allowed to be acquired when the actual number of processing nodes of the task does not exceed the maximum number of processing nodes.
To further illustrate the present solution, the present application further provides a specific application example of the above cluster task scheduling system based on weight, which specifically includes the following contents:
the application discloses a task scheduling system, which comprises a database and an embedded framework, wherein the embedded framework runs on each node of a computer cluster;
the task configuration parameters comprise the maximum processing node number, and the task file of the task is allowed to be acquired when the actual processing node number of the task does not exceed the maximum processing node number;
the embedded framework comprises a task flow module, a task processing module and a task process interface; wherein, a processing function of a preset task process is encapsulated in the task process interface; the task flow module provides a flow interface;
the task processing module is used for scanning the database, locking and acquiring a task file of a task when the task in a preset task state is found, and calling the flow interface to execute corresponding task processing after the task file is successfully acquired;
and the task flow module is used for analyzing the task process related to the task when the flow interface is called by the task processing module, calling the corresponding task process interface according to the analysis result to execute the processing of the corresponding task process, and switching the task state after the processing of the corresponding task process is successful.
Preferably, the embedded framework further comprises a node management module;
the node management module is used for registering the survival statement of the node where the node is located with the database, acquiring the failure node from the database and handing over the task of the failure node.
Preferably, the preset task state includes a state to be processed, the process interface includes a process execution interface, the task process interface includes a process service processing interface, and a processing function of a service logic process is encapsulated in the process service processing interface;
the task processing module comprises:
the task execution submodule is used for scanning the database, locking and acquiring a task file of a task when the task in a state to be processed is found, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired;
the task flow module is specifically configured to, when the flow execution interface is called by the task execution submodule, analyze a task process related to the task, call the process service processing interface according to an analysis result to process a service logic process, and convert a task state from a to-be-processed state to a processing-successful state after the service logic process is successfully processed.
Preferably, the task process interface further includes a final environment cleaning interface, and a cleaning function of the operating environment when the task is terminated is encapsulated in the final environment cleaning interface;
the task flow module is further configured to call the final environment cleaning interface after the business logic process is successfully processed to clean the running environment at the task termination time, and convert the task state from the to-be-processed state to the processing successful state after the running environment is successfully cleaned at the task termination time.
Preferably, the preset task state further includes a breakpoint redo state, the task process interface further includes a process breakpoint redo environment cleaning interface, and the process breakpoint redo environment cleaning interface encapsulates a cleaning function of a service logic process file;
the task execution sub-module is also used for scanning the database, locking and acquiring a task file of a task in a breakpoint redo state when the task is found, and calling the flow execution interface to execute corresponding task processing after the task is successfully acquired;
the task flow module is further configured to, when the flow execution interface is called by the task execution submodule, analyze a task process related to the task, call the process breakpoint redo environment cleaning interface according to an analysis result to clean the service logic process file, and convert the task state from the breakpoint redo state to a to-be-processed state after the service logic process file is successfully cleaned.
Preferably, the preset task state further includes a state to be checked, the process interface further includes a process check interface, the task process interface includes a process environment check interface, and a function of environment check before operation is packaged in the process environment check interface before operation;
the task processing module further comprises:
the task checking sub-module is used for scanning the database, locking and acquiring a task file of a task when the task in a state to be checked is found, and calling the flow checking interface to execute corresponding task processing after the task file is successfully acquired;
the task flow module is further configured to, when the flow inspection interface is called by the task inspection submodule, analyze a task process related to the task, call the process environment inspection interface according to an analysis result to perform pre-operation environment inspection, and convert a task state from a state to be inspected to a state to be processed after the pre-operation environment inspection is successful.
Preferably, the preset task state further includes a state to be preprocessed, the process interface further includes a process preprocessing interface, the task process interface further includes a process preprocessing interface, and a function of preprocessing before operation is encapsulated in the process preprocessing interface;
the task processing module further comprises:
the task preprocessing submodule is used for scanning the database, locking and acquiring a task file of a task when the task in a state to be preprocessed is found, and calling the flow preprocessing interface to execute corresponding task processing after the task file is successfully acquired;
the task flow module is also used for analyzing the task process related to the task when the flow preprocessing interface is called by the preprocessing submodule, calling the process preprocessing interface according to the analysis result to carry out preprocessing before operation, and converting the task state from the state to be preprocessed into the state to be checked after the preprocessing is successful before operation.
Preferably, the task processing module is specifically configured to scan the database, obtain a task file of a task in a preset task state by using an optimistic lock mechanism when the task is found, and call the flow interface to execute corresponding task processing after the task is successfully obtained.
As can be seen from the above, the present application can also achieve at least the following advantages:
firstly, just because the embedded framework can run on each node of the computer cluster, all nodes in the computer cluster are equal, unlike the prior art that the main node and the slave node are distinguished, the management cost of the main node for the slave node in the prior art can be saved;
secondly, the tasks can exist in the form of task files, the database is the central position of each node, each node can seize the task files through the database, and in the specific implementation, the preemptive property of a database lock and an optimistic lock mechanism in the system can be used for realizing free task scheduling without a main node;
thirdly, task configuration parameters representing task requirements can be obtained in a configuration mode, for example, the maximum processing node number of a task can be configured through a human-computer interaction interface, and only when the actual processing node number of the task does not exceed the maximum processing node number, a task file of the task is allowed to be obtained; on one hand, the configuration can flexibly configure the number of nodes to be scheduled according to task requirements, so that resource overhead is saved compared with the prior art in which a master node schedules all slave nodes; on the other hand, the configuration can flexibly configure the execution period of the task according to the task requirement without depending on the communication interaction between the user application and the task scheduling system, so that compared with the prior art that the user application needs to send corresponding task processing requests to the task scheduling system every day, the task scheduling system can configure the execution period of the task and periodically execute fr tasks by the preempted nodes, and therefore the task scheduling system further has the advantages of convenience in operation and high efficiency.
In order to effectively save management cost and node resources in a computer cluster and improve scheduling convenience and efficiency, an embodiment of an electronic device for implementing all or part of contents in the weight-based cluster task scheduling system is provided in the present application, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between the weight-based cluster task scheduling system and relevant equipment such as a core service system, a user terminal, a relevant database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the logic controller may refer to an embodiment of the weight-based cluster task scheduling system and an embodiment of the weight-based cluster task scheduling system in the embodiments for implementation, and the contents thereof are incorporated herein, and repeated details are not repeated.
It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the weight-based cluster task scheduling system may be executed on the electronic device side as described above, or all operations may be completed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
Fig. 2 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 2, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 2 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In an embodiment, the weight-based cluster task scheduling system function may be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
step S101:
step S102:
as can be seen from the above description, according to the electronic device provided in the embodiment of the present application, by setting a database and an embedded framework running on each node of a computer cluster, where a task file, a task configuration parameter, and a task state are stored in the database, the embedded framework includes a task flow module, a task processing module, and a task process interface, management cost and node resources in the computer cluster can be effectively saved, and convenience and efficiency of scheduling are improved.
In another embodiment, the weight-based cluster task scheduling system may be configured separately from the central processor 9100, for example, the weight-based cluster task scheduling system may be configured as a chip connected to the central processor 9100, and the function of the weight-based cluster task scheduling system is implemented by the control of the central processor.
As shown in fig. 2, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 2; further, the electronic device 9600 may further include components not shown in fig. 2, which may be referred to in the art.
As shown in fig. 2, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present application further provides a computer-readable storage medium capable of implementing all the steps in the weight-based cluster task scheduling system with a server or a client as an execution subject in the foregoing embodiments, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the weight-based cluster task scheduling system with a server or a client as an execution subject in the foregoing embodiments, for example, when the processor executes the computer program, the processor implements the following steps:
step S101: . . . (ii) a
Step S102: . . . .
As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application, by setting a database and an embedded framework running on each node of a computer cluster, where a task file, a task configuration parameter, and a task state are stored in the database, and the embedded framework includes a task flow module, a task processing module, and a task process interface, management cost and node resources in the computer cluster can be effectively saved, and convenience and efficiency of scheduling are improved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A weight-based cluster task scheduling system, comprising: the system comprises a database and an embedded framework running on each node of a computer cluster;
the database stores task files, task configuration parameters and task states;
the embedded framework comprises a task flow module, a task processing module and a task process interface, wherein the task process interface is packaged with a processing function of a preset task, and the task flow module provides a flow interface;
the task processing module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found, and calling the flow interface to execute corresponding task processing;
the task flow module is used for analyzing the task process of the preset task when the flow interface is called by the task processing module, calling the corresponding task process interface according to the analysis result to execute the processing of the corresponding task process, and switching the task state after the processing of the corresponding task process is successful.
2. The weight-based cluster task scheduling system of claim 1, wherein the embedded framework further comprises a node management module;
the node management module is used for registering the survival statement of the node where the node is located with the database, and acquiring the failure node from the database so as to take over the task of the failure node.
3. The weight-based cluster task scheduling system of claim 1, wherein the task processing module comprises:
and the task execution sub-module is used for scanning the database, locking and acquiring a task file of the preset task when the preset task is found to be in a to-be-processed state, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired.
4. The weight-based cluster task scheduling system of claim 3, wherein the task flow module is specifically configured to, when the flow execution interface is called by the task execution sub-module, parse a task process related to the preset task, call the process service processing interface according to a parsing result to perform processing of a service logic process, and convert a state of the preset task from a to-be-processed state to a processing-successful state after the processing of the service logic process is successful.
5. The weight-based cluster task scheduling system of claim 1, wherein the task process interface further comprises a final environment cleaning interface, wherein a cleaning function of the running environment at the end of the task is encapsulated in the final environment cleaning interface;
the task flow module is also used for calling the final environment cleaning interface after the business logic process is successfully processed to clean the running environment when the task is terminated, and converting the state of the preset task from a state to be processed into a state of successful processing after the running environment is successfully cleaned when the task is terminated.
6. The weight-based cluster task scheduling system of claim 1, wherein the states of the preset tasks further include a breakpoint redo state, the task process interfaces further include a process breakpoint redo environment cleaning interface, and the process breakpoint redo environment cleaning interface encapsulates a cleaning function of a business logic process file;
the task execution sub-module is also used for scanning the database, locking and acquiring a task file of a preset task when the preset task in the breakpoint redoing state is found, and calling the flow execution interface to execute corresponding task processing after the task file is successfully acquired;
the first task flow module is further configured to, when the flow execution interface is called by the task execution submodule, analyze a task process related to the preset task, call the process breakpoint redo environment cleaning interface according to an analysis result to clean the service logic process file, and convert the task state from the breakpoint redo state to a to-be-processed state after the service logic process file is successfully cleaned.
7. The weight-based cluster task scheduling system of claim 1, wherein the status of the preset task further comprises a status to be checked, the process interface further comprises a process check interface, the task process interface comprises a process environment check interface, and a pre-process environment check function is encapsulated in the pre-process environment check interface;
the task processing module further comprises:
the task checking sub-module is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be checked is found, and calling the flow checking interface to execute corresponding task processing after the task file is successfully acquired;
and the second task flow module is also used for analyzing the task process related to the preset task when the flow inspection interface is called by the task inspection submodule, calling the process environment inspection interface according to the analysis result to carry out the environment inspection before operation, and converting the task state from the state to be inspected into the state to be processed after the environment inspection is successful before operation.
8. The weight-based cluster task scheduling system of claim 1, wherein the states of the preset tasks further include a state to be preprocessed, the process interfaces further include a process preprocessing interface, the task process interfaces further include a process preprocessing interface, and a function of preprocessing before running is encapsulated in the process preprocessing interface;
the task processing module further comprises:
the task preprocessing submodule is used for scanning the database, locking and acquiring a task file of a preset task when the preset task in a state to be preprocessed is found, and calling the flow preprocessing interface to execute corresponding task processing after the task file is successfully acquired;
and the third task flow module is also used for analyzing the task process related to the preset task when the flow preprocessing interface is called by the preprocessing submodule, calling the process preprocessing interface according to the analysis result to carry out preprocessing before operation, and converting the task state from the state to be preprocessed into the state to be checked after the preprocessing is successful before the operation.
9. The weight-based cluster task scheduling system of claim 1, wherein the task processing module is specifically configured to scan the database, obtain a task file of a task in a preset task state by using an optimistic lock mechanism when the task is found, and call the flow interface to execute corresponding task processing after the task is successfully obtained.
10. The weight-based cluster task scheduling system of claim 1, wherein the task configuration parameters stored in the database include a maximum number of processing nodes, and wherein the task file for the task is allowed to be retrieved if the actual number of processing nodes for the task does not exceed the maximum number of processing nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911368322.0A CN111190731A (en) | 2019-12-26 | 2019-12-26 | Cluster task scheduling system based on weight |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911368322.0A CN111190731A (en) | 2019-12-26 | 2019-12-26 | Cluster task scheduling system based on weight |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111190731A true CN111190731A (en) | 2020-05-22 |
Family
ID=70707623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911368322.0A Pending CN111190731A (en) | 2019-12-26 | 2019-12-26 | Cluster task scheduling system based on weight |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111190731A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913793A (en) * | 2020-07-31 | 2020-11-10 | 同盾控股有限公司 | Distributed task scheduling method, device, node equipment and system |
| CN112058619A (en) * | 2020-08-31 | 2020-12-11 | 张和昆 | Mining high-efficient dust removal based on environmental protection is carried and is filtered integrated device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104050029A (en) * | 2014-05-30 | 2014-09-17 | 北京先进数通信息技术股份公司 | Task scheduling system |
-
2019
- 2019-12-26 CN CN201911368322.0A patent/CN111190731A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104050029A (en) * | 2014-05-30 | 2014-09-17 | 北京先进数通信息技术股份公司 | Task scheduling system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913793A (en) * | 2020-07-31 | 2020-11-10 | 同盾控股有限公司 | Distributed task scheduling method, device, node equipment and system |
| CN112058619A (en) * | 2020-08-31 | 2020-12-11 | 张和昆 | Mining high-efficient dust removal based on environmental protection is carried and is filtered integrated device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| CA3000422C (en) | Workflow service using state transfer | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN110413346B (en) | Parameter updating method and device | |
| CN110769278A (en) | Distributed video transcoding method and system | |
| CN110413673B (en) | Database data unified acquisition and distribution method and system | |
| CN111897633A (en) | Task processing method and device | |
| CN105975351A (en) | User behavior message reporting method and apparatus | |
| CN110781180B (en) | Data screening method and data screening device | |
| CN112463535A (en) | Multi-cluster exception handling method and device | |
| CN111858007A (en) | Task scheduling method and device based on message middleware | |
| US20230275976A1 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| CN111200606A (en) | Deep learning model task processing method, system, server and storage medium | |
| CN111464352A (en) | Call link data processing method and device | |
| CN111190731A (en) | Cluster task scheduling system based on weight | |
| CN114296953A (en) | Multi-cloud heterogeneous system and task processing method | |
| CN113626002A (en) | Service execution method and device | |
| CN116456496B (en) | Resource scheduling method, storage medium and electronic equipment | |
| CN111240998B (en) | Test case processing method and device | |
| CN115981871A (en) | GPU resource scheduling method, device, equipment and storage medium | |
| CN115914375A (en) | Disaster tolerance processing method and device for distributed message platform | |
| CN110427260B (en) | Host job scheduling method, device and system | |
| CN112559001A (en) | Method and device for updating application | |
| CN117076057B (en) | AI service request scheduling method, device, equipment and medium | |
| CN116594784B (en) | Method, device and system for scheduling edges and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200522 |
|
| RJ01 | Rejection of invention patent application after publication |