CN116483543A - Task processing method, device, equipment and storage medium - Google Patents

Task processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN116483543A
CN116483543A CN202310540287.6A CN202310540287A CN116483543A CN 116483543 A CN116483543 A CN 116483543A CN 202310540287 A CN202310540287 A CN 202310540287A CN 116483543 A CN116483543 A CN 116483543A
Authority
CN
China
Prior art keywords
current
node
task
instruction
child node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310540287.6A
Other languages
Chinese (zh)
Inventor
马聪
谢鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jinan Inspur Data Technology Co Ltd
Original Assignee
Jinan Inspur Data Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jinan Inspur Data Technology Co Ltd filed Critical Jinan Inspur Data Technology Co Ltd
Priority to CN202310540287.6A priority Critical patent/CN116483543A/en
Publication of CN116483543A publication Critical patent/CN116483543A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023—Failover techniques
    • G06F11/203—Failover techniques using migration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application discloses a task processing method, a device, equipment and a storage medium, which relate to the technical field of computers and comprise the following steps: collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type; acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task; determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type; and sending the task to the target child node so that the target child node executes the current instruction. According to the method and the system, each node is divided into operation nodes with different instruction types according to the default strategy, the target child node for executing the task is determined according to the specific instruction type of the task when the task is issued, smooth execution of the task is ensured, and the node utilization rate is improved.

Description

Task processing method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a task processing method, device, apparatus, and storage medium.
Background
The existing mass storage cluster is based on distributed storage, and provides interface access, information interaction and task processing functions through a main management node of management software, and when a cluster main node fails, a new main node is selected to continue service. If there is a task being executed at the time of failure, the failure of the master node may cause a process to be lost, thereby interrupting the task, and there is a possibility that redundant data exists due to abrupt interruption of the task, resulting in that the task being executed cannot be executed smoothly.
Disclosure of Invention
Accordingly, the present invention is directed to a task processing method, apparatus, device, and storage medium, which can ensure smooth execution of tasks and improve node utilization. The specific scheme is as follows:
in a first aspect, the present application discloses a task processing method, applied to a task manager, including:
collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type;
acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task;
determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type;
and sending the task to the target child node so that the target child node executes the current instruction.
Optionally, before collecting the current resource usage information of all the child nodes in the storage cluster, the method further includes:
acquiring the master node and all management nodes in the storage cluster;
and synchronously running the task manager on the master node and all the management nodes.
Optionally, before the task issued by the master node is obtained, the method further includes:
when the main node fails, judging whether the management node which normally operates exists or not;
if the management node which is in normal operation exists, selecting one management node from the management nodes which are in normal operation as a new master node;
and if the management node which normally operates does not exist, terminating the task.
Optionally, the collecting current resource usage information of all the sub-nodes in the storage cluster, and dividing all the sub-nodes based on the current resource usage information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type, includes:
collecting current resource use information of all child nodes in a storage cluster; the current resource use information comprises central processing unit capacity information, memory capacity information, network information and read-write information;
determining priority resource information corresponding to each instruction type;
acquiring current priority resource information corresponding to a current instruction type, and acquiring the current resource use information of which the current priority resource information meets the current task requirement from all the current resource use information to obtain a resource information set;
and determining the sub-node corresponding to each piece of current resource use information in the resource information set as a current sub-node set corresponding to the current instruction type.
Optionally, the sending the task to the target child node so that the target child node executes the current instruction includes:
sending the task to the target child node, and judging whether the current instruction depends on other instructions or not;
if the current instruction depends on the other instructions, waiting, and executing the current instruction when the completion information of the other instructions is received;
and if the current instruction is not dependent on the other instructions, directly executing the current instruction.
Optionally, the sending the task to the target child node, so that after the target child node executes the current instruction, further includes:
when the target child node fails, judging whether other child nodes exist in the current child node set corresponding to the current instruction;
if the other child nodes exist in the current child node set corresponding to the current instruction, determining a first current target child node from the other child nodes;
sending the task to the first current target child node so that the first current target child node executes the current instruction;
if the other child nodes do not exist in the current child node set corresponding to the current instruction, determining a second current target child node from the other child node sets; the other sub-node sets are the sub-node sets which do not contain the current sub-node set in all the sub-nodes;
and sending the task to the second current target sub-node so that the second current target sub-node executes the current instruction.
Optionally, the sending the task to the first current target sub-node so that the first current target sub-node executes the current instruction includes:
the task is sent to the first current target child node, and whether the current instruction contains a rollback instruction or not is judged;
if the current instruction comprises the rollback instruction, executing a preset rollback operation by using the first current target child node;
re-executing the current instruction with the first current target child node;
and if the current instruction does not contain the rollback instruction, continuing to execute the current instruction by using the first current target child node.
In a second aspect, the present application discloses a task processing device, applied to a task manager, including:
the information collection module is used for collecting current resource use information of all child nodes in the storage cluster;
the set dividing module is used for dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method so as to obtain a sub-node set corresponding to each instruction type;
the instruction type determining module is used for acquiring a task issued by the main node and determining a current instruction type corresponding to a current instruction contained in the task;
the child node determining module is used for determining a target child node from the corresponding child node set based on a preset node determining method and the current instruction type;
and the instruction execution module is used for sending the task to the target child node so that the target child node can execute the current instruction.
In a third aspect, the present application discloses an electronic device comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the task processing method as previously disclosed.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the task processing method as previously disclosed.
It can be seen that the present application provides a task processing method, including: collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type; acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task; determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type; and sending the task to the target child node so that the target child node executes the current instruction. Therefore, the method and the device divide each node into operation nodes with different instruction types according to the default strategy, determine the target sub-node for executing the task according to the specific instruction type of the task and the current resource use information when issuing the task, fully utilize cluster resources, ensure smooth execution of the task and improve the utilization rate of the node.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a task processing method disclosed in the present application;
FIG. 2 is a flowchart of a specific task processing method disclosed in the present application;
FIG. 3 is a flowchart of a specific task processing method disclosed in the present application;
FIG. 4 is a schematic diagram of a task processing device provided in the present application;
fig. 5 is a block diagram of an electronic device provided in the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
At present, the existing mass storage cluster is based on distributed storage, and interface access, information interaction and task processing functions are provided for the outside through one main management node of management software, and when a cluster main node fails, a new main node is selected to continue service. If there is a task being executed at the time of failure, the failure of the master node may cause a process to be lost, thereby interrupting the task, and there is a possibility that redundant data exists due to abrupt interruption of the task, resulting in that the task being executed cannot be executed smoothly. Therefore, the task processing method can ensure that the task is successfully executed, and improves the node utilization rate.
The embodiment of the invention discloses a task processing method, which is shown in fig. 1 and is applied to a task manager, wherein the method comprises the following steps:
step S11: collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type.
In this embodiment, current resource usage information of all sub-nodes in a storage cluster is collected, and all the sub-nodes are divided based on the current resource usage information and a preset instruction type determining method, so as to obtain a sub-node set corresponding to each instruction type. It can be understood that before collecting current resource usage information of all child nodes in a storage cluster, the master node and all management nodes in the storage cluster are obtained; and synchronously running the task manager on the master node and all the management nodes.
Step S12: and acquiring a task issued by the master node, and determining the current instruction type corresponding to the current instruction contained in the task.
In this embodiment, all the sub-nodes are divided based on the current resource usage information and a preset instruction type determining method, so that after a sub-node set corresponding to each instruction type is obtained, a task issued by a main node is obtained, and a current instruction type corresponding to a current instruction contained in the task is determined. It can be understood that at any time point of the whole flow, when the master node fails, whether the management node which operates normally exists is judged; if the management node which is in normal operation exists, selecting one management node from the management nodes which are in normal operation as a new master node; and if the management node which normally operates does not exist, terminating the task. After the master node issues a task, the task manager configuration module distributes the task to each node for execution according to the specific instruction type of the task, if the master node fails during the execution of the same task, the master node switches to other management nodes to monitor the task for execution, and if no switchable management node exists, the task is terminated.
Step S13: and determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type.
In this embodiment, after a task issued by a master node is acquired and a current instruction type corresponding to a current instruction included in the task is determined, a target child node is determined from the corresponding child node set based on a preset node determining method and the current instruction type. It can be understood that after the storage cluster deployment is completed, the task manager configuration module collects resource usage conditions of all nodes in real time, and dynamically adjusts each instruction type node according to information such as a CPU (Central Processing Unit ), a memory, an IOPS (Input/Output Operations Per Second), the number of times of performing read/write (I/O) operations per second, a network, and the like of each node. For example, if the current instruction type is a performance data reporting task, determining a sub-node set arranged according to available CPU information, and determining sub-nodes in the sub-node set arranged according to the available CPU information as target sub-nodes according to the sequence; if the current instruction type is a hardware information inquiry task, determining a sub-node set arranged according to the residual memory, and determining the sub-nodes in the sub-node set arranged according to the residual memory as target sub-nodes according to the sequence; if the current instruction type is a read-write task, determining a sub-node set arranged according to the IOPS, and determining the sub-nodes in the sub-node set arranged according to the IOPS as target sub-nodes according to the sequence; if the current instruction type is remote copy, determining a sub-node set arranged according to the network condition, and sequentially determining the sub-nodes in the sub-node set arranged according to the network condition as target sub-nodes. The specific task execution principle is as follows: the CPU can execute the performance data reporting task most preferentially; the node with the largest residual memory preferentially executes the hardware information inquiry task; the node with small IOPS preferably executes the read-write task; the tasks such as remote copy and the like are preferably executed under the best network condition; and under the fault condition of the executing node, the task manager module reselects other nodes of the same type for executing. It should be noted that, according to the task execution principle, the child nodes in the child node set arranged according to the available CPU information are arranged from large to small according to the available CPU information; according to the information of the residual memory, the child nodes in the child node set arranged according to the residual memory are arranged from big to small; the child nodes in the child node set arranged according to the IOPS are arranged from small to large according to the information of the IOPS; the child nodes in the child node set arranged according to the network condition are arranged from good to bad according to the information of the network condition. The child nodes in each of the child node sets are child nodes meeting the current demand, that is, child nodes not meeting the current demand may exist, and the child nodes not meeting the current demand are not included in the child node set. Under the fault condition of the executing node, the task manager module reselects other nodes of the same type for executing, namely selects the current optimal sub-node in the same sub-node set for executing.
Step S14: and sending the task to the target child node so that the target child node executes the current instruction.
In this embodiment, after determining a target child node from the corresponding child node set based on a preset node determining method and the current instruction type, the task is sent to the target child node, so that the target child node executes the current instruction. Specifically, the task is sent to the target child node, and whether the current instruction depends on other instructions or not is judged; if the current instruction depends on the other instructions, waiting, and executing the current instruction when the completion information of the other instructions is received; and if the current instruction is not dependent on the other instructions, directly executing the current instruction.
The application discloses a task cooperative processing method based on a storage cluster, which is characterized in that a task manager configuration module is constructed, wherein the module comprises specific instructions executed by each task, types of the specific instructions, corresponding rollback instructions and other information. The task manager configuration module operates on the cluster master node and the management node, and distributes instruction execution according to the collected node information. All executed instructions report results to task manager configuration modules of the master node and the management nodes, if node faults occur during task execution, the task manager configuration modules select new nodes to roll back and then continue to execute the instructions, and the task manager configuration modules realize whole-course monitoring and scheduling of tasks to ensure that task execution is completed.
It can be seen that the present application provides a task processing method, including: collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type; acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task; determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type; and sending the task to the target child node so that the target child node executes the current instruction. Therefore, the method and the device divide each node into operation nodes with different instruction types according to the default strategy, determine the target sub-node for executing the task according to the specific instruction type of the task and the current resource use information when issuing the task, fully utilize cluster resources, ensure smooth execution of the task and improve the utilization rate of the node.
Referring to fig. 2, an embodiment of the present invention discloses a task processing method, and compared with the previous embodiment, the present embodiment further describes and optimizes a technical solution.
Step S21: and collecting current resource use information of all child nodes in the storage cluster, and determining priority resource information corresponding to each instruction type.
In this embodiment, current resource usage information of all child nodes in the storage cluster is collected, and priority resource information corresponding to each instruction type is determined. The current resource use information comprises CPU capacity information, memory capacity information, network information and read-write information. For example, it is determined that priority resource information corresponding to an instruction type of the performance data reporting task is available cpu capacity information, priority resource information corresponding to an instruction type of the hardware information query task is available memory capacity information, priority resource information corresponding to an instruction type of the read-write task is read-write information, and priority resource information corresponding to an instruction type of the remote copy task is network information.
Step S22: acquiring current priority resource information corresponding to a current instruction type, and acquiring the current resource use information of which the current priority resource information meets the current task requirement from all the current resource use information to obtain a resource information set.
In this embodiment, current resource usage information of all sub-nodes in a storage cluster is collected, after priority resource information corresponding to each instruction type is determined, current priority resource information corresponding to a current instruction type is obtained, and current resource usage information that the current priority resource information meets current task requirements is obtained from all the current resource usage information, so as to obtain a resource information set.
It can be understood that the task manager configuration module constructed in the present application includes specific instructions executed by each task, execution type of each instruction (such as calculation, read-write, storage resource creation, alarm report, etc.), whether each instruction has a dependent instruction, whether rollback is required if this step fails, and a corresponding rollback instruction. When the storage cluster is deployed, a cluster master node and 3 (which can be adjusted according to the cluster scale) management nodes are designated, each node is divided into different instruction type operation nodes according to a default strategy, and a task manager configuration module is operated on the master node and the management nodes.
Step S23: and determining the sub-node corresponding to each piece of current resource use information in the resource information set as a current sub-node set corresponding to the current instruction type.
In this embodiment, after obtaining a resource information set, the child node corresponding to each piece of current resource usage information in the resource information set is determined to be a current child node set corresponding to the current instruction type. It can be understood that the task manager configuration module collects resource usage conditions of all nodes in real time, dynamically adjusts each instruction type node according to information such as a CPU, a memory, an IOPS, a network and the like of each node, and adjusts sub-node information in a corresponding sub-node set according to the adjusted node information.
Step S24: and acquiring a task issued by the master node, and determining the current instruction type corresponding to the current instruction contained in the task.
Step S25: and determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type.
Step S26: and sending the task to the target child node so that the target child node executes the current instruction.
In this embodiment, after determining a target child node from the corresponding child node set based on a preset node determining method and the current instruction type, the task is sent to the target child node, so that the target child node executes the current instruction. Further, when the target child node fails, judging whether other child nodes exist in the current child node set corresponding to the current instruction; if the other child nodes exist in the current child node set corresponding to the current instruction, determining a first current target child node from the other child nodes; sending the task to the first current target child node so that the first current target child node executes the current instruction; if the other child nodes do not exist in the current child node set corresponding to the current instruction, determining a second current target child node from the other child node sets; the other sub-node sets are the sub-node sets which do not contain the current sub-node set in all the sub-nodes; and sending the task to the second current target sub-node so that the second current target sub-node executes the current instruction.
It can be understood that the task is sent to the first current target sub-node, so that when the first current target sub-node executes the current instruction, the task is sent to the first current target sub-node, and whether the current instruction contains a rollback instruction or not is judged; if the current instruction comprises the rollback instruction, executing a preset rollback operation by using the first current target child node; re-executing the current instruction with the first current target child node; and if the current instruction does not contain the rollback instruction, continuing to execute the current instruction by using the first current target child node. For example, if an executing node failure occurs, the task manager configuration module reassigns other nodes, first executing a rollback instruction, and then executing an instruction that is interrupted by the node failure. Therefore, cluster resources are fully utilized, task execution under a fault scene is ensured, and the robustness of the storage system is enhanced.
The specific flow is shown in fig. 3, (1) a task manager configuration module is constructed; (2) The cluster deployment designates a cluster master node and a management node, and the task manager module synchronously operates at the master node and the management node; (3) The task manager configuration module collects resource use conditions of all nodes in real time and dynamically adjusts all instruction type nodes; (4) The master node issues a task, and a task manager configuration module distributes the task to each node of a corresponding type for execution according to the type of an instruction in the task; (5) If the main node fails, judging whether a management node with normal operation exists, if so, selecting a new main node from the management nodes with normal operation, and if not, failing the task; (6) If the node A1 executes the instruction A1, the instruction A1 has a dependent instruction b1, then executing the step (7), and if the instruction A1 has no dependent instruction, then executing the step (8); (7) Waiting for the completion of the execution of the instruction b1 and reporting the completion to a task manager configuration module, and after receiving the completion message of the execution of the instruction b1, issuing a task to enable the node A1 to execute the instruction A1; (8) Judging whether the node A1 fails in the process of executing the instruction A1, if the node A1 fails in the process of executing the instruction A1, executing the step (9), and if the node A1 does not fail in the process of executing the instruction A1, executing the step (15); (9) Judging whether the node B1 of the same type of A1 exists, if the node B1 of the same type of A1 exists, executing the step (10), and if the node B1 of the same type of A1 does not exist, executing the step (11); (10) The task manager configuration module selects node B1 to replace A1 node to execute instructions; (11) Selecting other types of nodes C1 to replace the A1 node to execute instructions; (12) Judging whether a rollback instruction c1 exists in the instruction a1, if the rollback instruction c1 exists in the instruction a1, executing the step (13), and if the rollback instruction c1 does not exist in the instruction a1, executing the step (14); (13) node B1 executing instruction c1; (14) The node B1 executes the instruction a1, and reports an execution result to the task manager configuration module after completion; (15) And reporting the execution results to a task manager configuration module by all nodes, and ending the task. It should be noted that in step (10) or (11), it is only determined whether there is a node B1 or C1 that can be executed instead of the A1 node, and then the step (12) is directly performed without executing the instruction A1, and when the rollback instruction C1 exists in the instruction A1, the node B1 or the node C1 is controlled to execute the rollback instruction C1, and then the node B1 or the node C1 is controlled to execute the instruction A1. In addition, multiple instructions may be executed simultaneously.
For the specific content of the steps S24 and S25, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no detailed description is given here.
Therefore, the embodiment of the application collects the current resource use information of all the child nodes in the storage cluster; determining priority resource information corresponding to each instruction type; acquiring current priority resource information corresponding to a current instruction type, and acquiring the current resource use information of which the current priority resource information meets the current task requirement from all the current resource use information to obtain a resource information set; determining the sub-node corresponding to each piece of current resource use information in the resource information set as a current sub-node set corresponding to the current instruction type; acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task; determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type; and sending the task to the target child node so that the target child node executes the current instruction, cluster resources are fully utilized, smooth execution of the task is ensured, and the node utilization rate is improved.
Referring to fig. 4, the embodiment of the application further correspondingly discloses a task processing device, which is applied to a task manager and includes:
the information collection module 11 is configured to collect current resource usage information of all child nodes in the storage cluster;
the set dividing module 12 is configured to divide all the sub-nodes based on the current resource usage information and a preset instruction type determining method, so as to obtain a sub-node set corresponding to each instruction type;
the instruction type determining module 13 is configured to obtain a task issued by a master node, and determine a current instruction type corresponding to a current instruction included in the task;
a child node determining module 14, configured to determine a target child node from the corresponding child node set based on a preset node determining method and the current instruction type;
and the instruction execution module 15 is configured to send the task to the target child node so that the target child node executes the current instruction.
As can be seen, the present application includes: collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type; acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task; determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type; and sending the task to the target child node so that the target child node executes the current instruction. Therefore, the method and the device divide each node into operation nodes with different instruction types according to the default strategy, determine the target sub-node for executing the task according to the specific instruction type of the task and the current resource use information when issuing the task, fully utilize cluster resources, ensure smooth execution of the task and improve the utilization rate of the node.
In some embodiments, the information collecting module 11 specifically includes:
a node obtaining unit, configured to obtain the master node and all management nodes in the storage cluster;
the task manager running unit is used for synchronously running the task manager on the master node and all the management nodes;
the information collection unit is used for collecting current resource use information of all child nodes in the storage cluster; the current resource use information comprises CPU capacity information, memory capacity information, network information and read-write information.
In some embodiments, the set partitioning module 12 specifically includes:
the priority resource information determining unit is used for determining priority resource information corresponding to each instruction type;
the resource information set acquisition unit is used for acquiring current priority resource information corresponding to the current instruction type, and acquiring the current resource use information of which the current priority resource information meets the current task requirement from all the current resource use information so as to acquire a resource information set;
and the current sub-node set determining unit is used for determining the sub-node corresponding to each piece of current resource use information in the resource information set as a current sub-node set corresponding to the current instruction type.
In some embodiments, the instruction type determining module 13 specifically includes:
a management node judging unit, configured to judge whether the management node that operates normally exists when the master node fails;
a new master node determining unit, configured to select one management node from the management nodes that are in normal operation as a new master node if the management node that is in normal operation exists;
a task termination unit, configured to terminate the task if the management node that is running normally does not exist;
the task acquisition unit is used for acquiring tasks issued by the main node;
and the current instruction type determining unit is used for determining the current instruction type corresponding to the current instruction contained in the task.
In some specific embodiments, the child node determining module 14 specifically includes:
and the target child node determining unit is used for determining a target child node from the corresponding child node set based on a preset node determining method and the current instruction type.
In some embodiments, the instruction execution module 15 specifically includes:
a first task sending unit, configured to send the task to the target child node;
a current instruction judging unit configured to judge whether the current instruction depends on other instructions;
a first current instruction execution unit, configured to wait if the current instruction depends on the other instruction, and execute the current instruction when receiving completion information of the other instruction;
the second current instruction execution unit is used for directly executing the current instruction if the current instruction is not dependent on the other instructions;
the other child node judging unit is used for judging whether other child nodes exist in the current child node set corresponding to the current instruction when the target child node fails;
a first current target sub-node determining unit, configured to determine a first current target sub-node from the other sub-nodes if the other sub-nodes exist in the current sub-node set corresponding to the current instruction;
a second task sending unit, configured to send the task to the first current target child node;
the rollback instruction judging unit is used for judging whether the current instruction contains a rollback instruction or not;
the preset rollback operation execution unit is used for executing preset rollback operation by utilizing the first current target child node if the current instruction contains the rollback instruction;
a third current instruction execution unit, configured to re-execute the current instruction using the first current target child node;
a fourth current instruction execution unit, configured to continue executing the current instruction with the first current target child node if the current instruction does not include the rollback instruction;
a second current target sub-node determining unit, configured to determine a second current target sub-node from other sub-node sets if the other sub-nodes do not exist in the current sub-node set corresponding to the current instruction; the other sub-node sets are the sub-node sets which do not contain the current sub-node set in all the sub-nodes;
and the fifth current instruction execution unit is used for sending the task to the second current target child node so that the second current target child node can execute the current instruction.
Further, the embodiment of the application also provides electronic equipment. Fig. 5 is a block diagram of an electronic device 20, according to an exemplary embodiment, and the contents of the diagram should not be construed as limiting the scope of use of the present application in any way.
Fig. 5 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein the memory 22 is configured to store a computer program that is loaded and executed by the processor 21 to implement relevant steps in the task processing method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and computer programs 222, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the task processing method performed by the electronic device 20 as disclosed in any of the previous embodiments.
Further, the embodiment of the application also discloses a storage medium, and the storage medium stores a computer program, and when the computer program is loaded and executed by a processor, the steps of the task processing method disclosed in any one of the previous embodiments are realized.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The task processing method, device, equipment and storage medium provided by the invention are described in detail, and specific examples are applied to illustrate the principle and implementation of the invention, and the description of the examples is only used for helping to understand the method and core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims (10)

1. A method for processing tasks, applied to a task manager, comprising:
collecting current resource use information of all sub-nodes in a storage cluster, and dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type;
acquiring a task issued by a main node, and determining a current instruction type corresponding to a current instruction contained in the task;
determining a target child node from the corresponding child node set based on a preset node determination method and the current instruction type;
and sending the task to the target child node so that the target child node executes the current instruction.
2. The method according to claim 1, wherein before collecting the current resource usage information of all the child nodes in the storage cluster, further comprises:
acquiring the master node and all management nodes in the storage cluster;
and synchronously running the task manager on the master node and all the management nodes.
3. The task processing method according to claim 2, characterized by further comprising:
when the main node fails, judging whether the management node which normally operates exists or not;
if the management node which is in normal operation exists, selecting one management node from the management nodes which are in normal operation as a new master node;
and if the management node which normally operates does not exist, terminating the task.
4. The task processing method according to claim 1, wherein the collecting current resource usage information of all the sub-nodes in the storage cluster and dividing all the sub-nodes based on the current resource usage information and a preset instruction type determining method to obtain a sub-node set corresponding to each instruction type includes:
collecting current resource use information of all child nodes in a storage cluster; the current resource use information comprises central processing unit capacity information, memory capacity information, network information and read-write information;
determining priority resource information corresponding to each instruction type;
acquiring current priority resource information corresponding to a current instruction type, and acquiring the current resource use information of which the current priority resource information meets the current task requirement from all the current resource use information to obtain a resource information set;
and determining the sub-node corresponding to each piece of current resource use information in the resource information set as a current sub-node set corresponding to the current instruction type.
5. The task processing method according to claim 1, wherein the sending the task to the target child node for the target child node to execute the current instruction includes:
sending the task to the target child node, and judging whether the current instruction depends on other instructions or not;
if the current instruction depends on the other instructions, waiting, and executing the current instruction when the completion information of the other instructions is received;
and if the current instruction is not dependent on the other instructions, directly executing the current instruction.
6. The task processing method according to claim 4, wherein the sending the task to the target child node so that after the target child node executes the current instruction, further comprises:
when the target child node fails, judging whether other child nodes exist in the current child node set corresponding to the current instruction;
if the other child nodes exist in the current child node set corresponding to the current instruction, determining a first current target child node from the other child nodes;
sending the task to the first current target child node so that the first current target child node executes the current instruction;
if the other child nodes do not exist in the current child node set corresponding to the current instruction, determining a second current target child node from the other child node sets; the other sub-node sets are the sub-node sets which do not contain the current sub-node set in all the sub-nodes;
and sending the task to the second current target sub-node so that the second current target sub-node executes the current instruction.
7. The task processing method according to claim 6, wherein the sending the task to the first current target child node so that the first current target child node executes the current instruction includes:
the task is sent to the first current target child node, and whether the current instruction contains a rollback instruction or not is judged;
if the current instruction comprises the rollback instruction, executing a preset rollback operation by using the first current target child node;
re-executing the current instruction with the first current target child node;
and if the current instruction does not contain the rollback instruction, continuing to execute the current instruction by using the first current target child node.
8. A task processing device, applied to a task manager, comprising:
the information collection module is used for collecting current resource use information of all child nodes in the storage cluster;
the set dividing module is used for dividing all the sub-nodes based on the current resource use information and a preset instruction type determining method so as to obtain a sub-node set corresponding to each instruction type;
the instruction type determining module is used for acquiring a task issued by the main node and determining a current instruction type corresponding to a current instruction contained in the task;
the child node determining module is used for determining a target child node from the corresponding child node set based on a preset node determining method and the current instruction type;
and the instruction execution module is used for sending the task to the target child node so that the target child node can execute the current instruction.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the task processing method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor implements the task processing method as claimed in any one of claims 1 to 7.
CN202310540287.6A 2023-05-11 2023-05-11 Task processing method, device, equipment and storage medium Pending CN116483543A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310540287.6A CN116483543A (en) 2023-05-11 2023-05-11 Task processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310540287.6A CN116483543A (en) 2023-05-11 2023-05-11 Task processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116483543A true CN116483543A (en) 2023-07-25

Family

ID=87226881

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310540287.6A Pending CN116483543A (en) 2023-05-11 2023-05-11 Task processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116483543A (en)

Similar Documents

Publication Publication Date Title
CN108847982B (en) Distributed storage cluster and node fault switching method and device thereof
US9002997B2 (en) Instance host configuration
EP3806432A1 (en) Method for changing service on device and service changing system
CN110311831B (en) Container cloud-based system resource monitoring method and related equipment
JP2004227359A (en) Operation management method for storage system based on policy
EP4053699A1 (en) Instance host configuration
CN109582335B (en) Method, device and equipment for on-line upgrading of non-interrupt storage cluster nodes
EP2723017A1 (en) Method, apparatus and system for implementing distributed auto-incrementing counting
US20210065083A1 (en) Method for changing device business and business change system
CN110716793A (en) Execution method, device, equipment and storage medium of distributed transaction
CN111966289A (en) Partition optimization method and system based on Kafka cluster
CN109271172A (en) A kind of host behavior extension method and device of swarm cluster
CN112698952A (en) Unified management method and device for computing resources, computer equipment and storage medium
CN112181627A (en) Timing task scheduling method, device and system
CN111064626B (en) Configuration updating method, device, server and readable storage medium
CN103714059A (en) Data updating method and device
CN114900449B (en) Resource information management method, system and device
CN116483543A (en) Task processing method, device, equipment and storage medium
CN115473858A (en) Data transmission method and streaming data transmission system
JPH08292922A (en) Network management device
CN113032188A (en) Method, device, server and storage medium for determining main server
CN111405313A (en) Method and system for storing streaming media data
CN117376194B (en) Network detection method, system, electronic device and computer readable storage medium
CN112463514A (en) Monitoring method and device for distributed cache cluster
CN110825529B (en) Service message management system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination