CN111290854A - Task management method, device and system, computer storage medium and electronic equipment - Google Patents

Task management method, device and system, computer storage medium and electronic equipment Download PDF

Info

Publication number
CN111290854A
CN111290854A CN202010064754.9A CN202010064754A CN111290854A CN 111290854 A CN111290854 A CN 111290854A CN 202010064754 A CN202010064754 A CN 202010064754A CN 111290854 A CN111290854 A CN 111290854A
Authority
CN
China
Prior art keywords
task
executed
node
service
scheduling service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010064754.9A
Other languages
Chinese (zh)
Other versions
CN111290854B (en
Inventor
曹智颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Cloud Computing Beijing Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202010064754.9A priority Critical patent/CN111290854B/en
Publication of CN111290854A publication Critical patent/CN111290854A/en
Application granted granted Critical
Publication of CN111290854B publication Critical patent/CN111290854B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793—Remedial or corrective actions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The disclosure provides a task management method, a task management device and a distributed task management system. The method is applied to a distributed task management system, wherein the distributed task management system comprises a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server; the method comprises the following steps: each task scheduling service monitors a father node corresponding to the node; when the deleted first node exists in the nodes, a second node corresponding to the first node is created through a target task scheduling service, wherein the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node; and the target task scheduling service acquires the task information corresponding to the first node from the database and adds the task information to a task queue corresponding to the target task scheduling service. According to the task management method and the task management system, the task of the task scheduling service with the fault is taken over by the task scheduling service which normally works, the task is ensured not to be lost, and the usability of the task management system is improved.

Description

Task management method, device and system, computer storage medium and electronic equipment
Technical Field
The present disclosure relates to the field of cloud computing technologies, and in particular, to a task management method, a task management apparatus, a distributed task management system, a computer storage medium, and an electronic device.
Background
The task management system is widely used for computing tasks needing long execution time, such as offline voice recognition, big data statistics tasks, machine learning model training and the like, and manages the computing tasks needing long execution time, and executes task issuing, monitors task progress, stores task states and the like.
The traditional task management system mainly comprises a control console, a task management service and a task execution service, wherein a user issues a task to the task management service through the control console, the task management service issues the task to the task execution service, the task execution service starts to execute the task after receiving the task, and the task execution progress is updated to a database. And the task management service can display the task execution state to the console in real time for the user to inquire. The traditional task management system is simple in structure and only suitable for processing an atomized task, namely the task only relates to one task execution service, when one task is composed of a plurality of sequential subtasks, the subtasks need to be executed by a plurality of different task execution services, if the traditional task management system is adopted, the task execution efficiency is necessarily reduced, and the usability of the task management service is low.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
Embodiments of the present disclosure provide a task management method, a task management apparatus, a task management system, a computer storage medium, and an electronic device, which can ensure that a task is not lost at least to a certain extent, and achieve high availability of the task management system.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the embodiments of the present disclosure, a task management method is provided, where the task management method is applied to a distributed task management system, where the distributed task management system includes a task scheduling server and nodes corresponding to multiple task scheduling services in the task scheduling server, where the task scheduling services are used to manage and schedule tasks of users; the method comprises the following steps: each task scheduling service monitors a father node corresponding to the node; when the deleted first node exists in the nodes, creating a second node corresponding to the first node through a target task scheduling service, wherein the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node; and the target task scheduling service acquires task information corresponding to the first node from a database and adds the task information to a task queue corresponding to the target task scheduling service.
According to an aspect of the embodiments of the present disclosure, a task management device is provided, where the task management device is applied to a distributed task management system, where the distributed task management system includes a task scheduling server and nodes corresponding to multiple task scheduling services in the task scheduling server, where the task scheduling services are used to manage and schedule tasks of users; the device comprises: the monitoring module is used for monitoring a father node corresponding to the node by each task scheduling service; a node creating module, configured to create, when it is monitored that a deleted first node exists in the nodes, a second node corresponding to the first node through a target task scheduling service, where the target task scheduling service corresponding to the second node is different from a task scheduling service corresponding to the first node; and the task taking module is used for acquiring the task information corresponding to the first node from a database by the target task scheduling service and adding the task information into a task queue corresponding to the target task scheduling service.
In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus further includes: and the registration module is used for registering in the distributed coordination service system through each task scheduling service so as to obtain nodes with different node numbers corresponding to each task scheduling service.
In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus further includes: and the identification modification module is used for modifying the task scheduling service identification corresponding to the task information into the task scheduling service identification corresponding to the target task scheduling service.
In some embodiments of the present disclosure, the distributed task management system further comprises a task management server and a task execution server; based on the foregoing solution, the task management apparatus further includes: the request generation module is used for receiving the tasks to be executed sent by the task management server through the task scheduling server and forming task execution requests according to the tasks to be executed; and the task execution module is used for sending the task execution request to the task execution server so as to enable the task execution service in the task execution server to execute the task.
In some embodiments of the present disclosure, based on the foregoing solution, the request generating module includes: the task adding unit is used for adding the task to be executed to a task queue corresponding to the task scheduling service and judging whether an idle thread exists in a thread pool of the task scheduling service or not; the task pulling unit is used for pulling the task to be executed from the task queue corresponding to the task scheduling service and acquiring a subtask in the task to be executed when the idle thread exists; and the request generating unit is used for forming the task execution request according to the attribute information of the subtask and the subtask.
In some embodiments of the present disclosure, based on the foregoing solution, the request generating unit is configured to: judging whether the subtask is a subtask to be executed; and when the subtask is judged to be the subtask to be executed, forming the task execution request according to the subtask to be executed.
In some embodiments of the present disclosure, the number of the task execution services is plural, and each of the task execution services has a different task type tag; based on the foregoing solution, the task execution module is configured to: matching the task type of the task to be executed in the task execution request with the task type label to determine a target task execution service; and sending the task execution request to the target task execution service so that the target task execution service executes the task.
In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: inquiring whether the sub task to be executed is executed or not; if the execution of the subtask to be executed is completed, updating task progress information corresponding to the task to be executed in the database; and if the execution of the subtasks to be executed is not finished, adding the tasks to be executed into the task queues corresponding to the task scheduling services again.
In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: judging whether the tasks to be executed have unexecuted subtasks or not; if the task exists, the unexecuted subtask is executed when the unexecuted subtask is a subtask to be executed until the unexecuted subtask does not exist in the task to be executed; and if not, updating the task state information corresponding to the task to be executed in the database.
In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: each task scheduling service updates the task quantity in the node according to the task quantity in the task queue corresponding to each task scheduling service; and the task management server acquires the task quantity from each node, determines task distribution weight according to the task quantity, and executes task distribution based on the task distribution weight.
In some embodiments of the present disclosure, the distributed task management system further comprises a console; based on the foregoing solution, the task management device is further configured to: the console responds to a first trigger operation of a user to enter the task to be executed; or the console responds to a second trigger operation of the user to generate a task progress query request, and sends the task progress query request to the task management server to obtain task progress information.
According to an aspect of the embodiments of the present disclosure, there is provided a distributed task management system, including: the console is used for responding to the triggering operation of a user to generate a task issuing instruction or a task progress query request; the task management server is connected with the console and used for responding to the task issuing instruction to issue the task or responding to the task progress inquiry request to feed back task progress information to the console; the task scheduling server is connected with the task management server and comprises a plurality of task scheduling services, and each task scheduling service is registered in the distributed coordination service system to form a node and is used for receiving the tasks issued by the task management server and scheduling and managing the tasks through each task scheduling service; the task execution server is connected with the task scheduling server, comprises a plurality of task execution services and is used for responding to a task execution request sent by the task scheduling service so as to execute a task; and the data storage equipment is connected with the task management server and the task scheduling server and is used for storing information related to the tasks.
According to an aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the task management method according to the embodiments described above.
According to an aspect of an embodiment of the present disclosure, there is provided an electronic device including one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to perform the task management method as described in the above embodiments.
In the technical solution provided in the embodiment of the present disclosure, a plurality of task scheduling services in a task management system are registered in a distributed coordination service framework to form nodes corresponding to the task scheduling services, that is, the task management system is a multi-node distributed task management system. Each task scheduling service monitors a father node corresponding to the node, when the deleted first node exists, a second node corresponding to the first node is created through the target task scheduling service, meanwhile, the target task scheduling service acquires task information corresponding to the first node from a database, and the task information is added into a task queue corresponding to the target task scheduling service. According to the technical scheme, after a certain task scheduling service fails, the tasks of the failed task scheduling service can be taken over through other normally-working task scheduling services, the tasks are guaranteed not to be lost, and the usability of the task management system is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:
fig. 1 shows a schematic diagram of an exemplary system architecture to which technical aspects of embodiments of the present disclosure may be applied;
fig. 2 is a schematic diagram showing an architecture of a conventional task management system in the related art;
FIG. 3 schematically shows a flowchart of a task management method according to one embodiment of the present disclosure;
FIG. 4 schematically illustrates an architectural diagram of a distributed task management system according to one embodiment of the present disclosure;
FIG. 5 schematically illustrates a distributed architecture diagram among task scheduling services in a distributed task processing system according to one embodiment of the present disclosure;
FIG. 6 schematically illustrates a structural diagram of task information stored in a data storage device, according to one embodiment of the present disclosure;
FIG. 7 schematically illustrates a relationship diagram between task scheduling services upon failure of the task scheduling services according to one embodiment of the present disclosure;
FIG. 8 schematically illustrates a relationship between task scheduling services after task recovery according to one embodiment of the present disclosure;
FIG. 9 schematically illustrates a structural diagram of recovered task information stored in a data storage device, according to one embodiment of the present disclosure;
FIG. 10 schematically illustrates an execution flow diagram of a task scheduling service according to one embodiment of the present disclosure;
FIG. 11 schematically shows a block diagram of a task management device according to one embodiment of the present disclosure;
FIG. 12 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present disclosure may be applied.
As shown in fig. 1, system architecture 100 may include a task management front end 101, a network 102, and a task management back end 103. The network 102 is used to provide a medium for a communication link between the task management front-end 101 and the task management back-end 103. Network 102 may include various connection types, such as wired communication links, wireless communication links, and so forth.
It should be understood that the number of task management front-ends, networks, and task management back-ends in FIG. 1 is merely illustrative. There may be any number of task management front ends, networks, and task management back ends, as desired. For example, the task management back end 103 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like. The task management front-end 101 may be a terminal device such as, but not limited to, a laptop, a tablet, a desktop computer, etc.
In one embodiment of the present disclosure, a user may enter task information in the task management front end 101 or send a task progress query request through the task management front end 101, and the task information and the task progress query request are sent to the task management back end 103 through the network 102. The task management back end 103 comprises a task management server, a task scheduling server and a task execution server, wherein the task scheduling server comprises a plurality of task scheduling services for managing and scheduling tasks of users; the task execution server comprises a plurality of task execution services for executing the received tasks of the user. After receiving the task issued by the task management front end 101, the task management server issues the task to the task scheduling server, schedules the task by the plurality of task scheduling services, and invokes the corresponding task execution service to execute each task. In the process of scheduling tasks by the task scheduling service, if a certain task scheduling service fails, the task scheduled by the failed task scheduling service can be taken over by the task scheduling service which does not fail based on the distributed coordination service system, so that the task is ensured not to be lost. In addition, when the task management server issues the tasks to the task scheduling server, the weight of the issued tasks can be dynamically adjusted according to the task amount in the task queue corresponding to each task scheduling service, and the tasks are guaranteed to be evenly distributed to each task scheduling service. According to the technical scheme of the embodiment of the invention, a plurality of task scheduling services and nodes corresponding to the task scheduling services can be arranged in the task management system to form the distributed task management system, a plurality of tasks can be executed in the distributed task management system at the same time, and the task execution efficiency is improved.
It should be noted that the task management method provided by the embodiment of the present disclosure is generally executed by a server, and accordingly, the task management device is generally disposed in the server. However, in other embodiments of the present disclosure, the task management method provided in the embodiments of the present disclosure may also be executed by a terminal device.
In the related art in the field, fig. 2 shows an architecture diagram of a conventional task management system, as shown in fig. 2, the conventional task management system includes a console 201, a task management server 202, a task execution server 203, and a database 204, and a specific task execution flow is as follows: a user issues a task to the task management server 202 through the console 201; the task management server 202 stores the task information in the database 204, and simultaneously issues the task to the task execution server 203; after receiving the task, the task execution server 203 starts to execute the task, and periodically updates the task execution progress to the database 204; the user sends a task progress query request to the task management server 202 through the console 201, and the task management server 202 responds to the task progress query request and displays the task execution state in the database 204 to the console 201 in real time for the user to browse.
However, the conventional task management system has a simple architecture, and is suitable for processing an atomic task, and if one task is composed of a plurality of sequential subtasks, and the subtasks are executed by a plurality of different task execution services, in order to make the task management system have high availability, the task management service needs to be set as a plurality of nodes, and the tasks are managed by different nodes. But there are problems accordingly, such as how to coordinate the assignment of tasks among the nodes, how to handle the tasks of a node when the node fails, and the like.
In view of the problems in the related art, the embodiments of the present disclosure provide a task management method, which is based on a Cloud technology, and the Cloud technology (Cloud technology) is based on a generic term of a network technology, an information technology, an integration technology, a management platform technology, an application technology, and the like applied in a Cloud computing business model, and can form a resource pool, be used as needed, and be flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.
Cloud computing (cloud computing) is a computing model that distributes computing tasks over a pool of resources formed by a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the "cloud" appear to the user as being infinitely expandable and available at any time, available on demand, expandable at any time, and paid for on-demand.
As a basic capability provider of cloud computing, a cloud computing resource pool (called as an ifas (Infrastructure as a Service) platform for short is established, and multiple types of virtual resources are deployed in the resource pool and are selectively used by external clients.
According to the logic function division, a PaaS (Platform as a Service) layer can be deployed on an IaaS (Infrastructure as a Service) layer, a SaaS (Software as a Service) layer is deployed on the PaaS layer, and the SaaS can be directly deployed on the IaaS. PaaS is a platform on which software runs, such as a database, a web container, etc. SaaS is a variety of business software, such as web portal, sms, and mass texting. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.
Cloud processing is generally applied to the field of Big data, Big data (Big data) refers to a data set which cannot be captured, managed and processed by a conventional software tool within a certain time range, and is a massive, high-growth-rate and diversified information asset which can have stronger decision-making power, insight discovery power and flow optimization capability only by a new processing mode. With the advent of the cloud era, big data has attracted more and more attention, and the big data needs special technology to effectively process a large amount of data within a tolerance elapsed time. The method is suitable for the technology of big data, and comprises a large-scale parallel processing database, data mining, a distributed file system, a distributed database, a cloud computing platform, the Internet and an extensible storage system.
The disclosed embodiment provides a task management method first, and fig. 3 schematically shows a flowchart of the task management method according to an embodiment of the present disclosure, where the task management method may be executed by a task management front end and a task management back end, and specifically may be executed by a terminal device and a server, and in addition, the task management method is applied to a distributed task management system, where the distributed task management system includes a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, where the task scheduling services are used to manage and schedule tasks of users. Referring to fig. 3, the task management method at least includes steps S310 to S330, which are described in detail as follows:
in step S310, each task scheduling service listens to a parent node corresponding to the node.
In an embodiment of the present disclosure, the task management system is a distributed task management system, and the distributed task management system includes a task scheduling server, where the task scheduling server includes a plurality of task scheduling services, the plurality of task scheduling services form a task scheduling cluster, and each task scheduling service is registered in the distributed coordination service system to form a node. Fig. 4 shows a schematic architecture diagram of a distributed task management system, as shown in fig. 4, a distributed task management system 400 includes a console 401, a task management server 402, a task scheduling server 403, a task execution server 404, and a data storage device 405, where the console 401 may specifically be a Web console, and is configured to respond to a trigger operation of a user to generate a task issuing instruction or a task progress query request, that is, the user may perform operations such as task issuing, task status monitoring, and task execution result query through the console; the task management server 402 is also called a task management background, serves a background of the console 401, is connected to the console 401, and is configured to issue a task in response to a task issuing instruction sent by the console 401 or to respond to a task progress query request sent by the console 401 to feed back task progress information to the console 401; the task scheduling server 403 is connected to the task management server 402, and includes a task scheduling service cluster, where the task scheduling service cluster includes a plurality of task scheduling services, where each task scheduling service is registered in a distributed coordination service framework to form a node, and each task scheduling service receives a task issued by the task management device 402 and is responsible for scheduling the whole life cycle of the task, and in addition, the task scheduling server 403 may also update the task execution progress to the data storage device 405 in real time; the task execution server 404 is connected to the task scheduling server 403, and includes a task execution service cluster, where the task execution service cluster includes a plurality of task execution services, each task execution service is an atomic computation task execution service, receives a sub-task execution request submitted by the task scheduling service, and executes a corresponding computation task; the data storage device 405 may be a hard disk or other media that can be used to store data, and is used to store data submitted by the task management server 402 and the task scheduling server 403. In the embodiment of the present disclosure, each task execution service has a different task type tag, a user may mark the type of each task when entering the task through the console 401, and when receiving the task, the task execution server 404 may match the type of the task with the task type tag of the task execution service to determine a task execution service matched therewith, and execute the task through the matched task execution service.
In an embodiment of the present disclosure, the task management server 402, the task scheduling server 403, and the task execution server 404 may be the same server or different servers, which is not specifically limited in this embodiment of the present disclosure.
In an embodiment of the present disclosure, the distributed task management system is constructed based on a distributed coordination service system, and nodes corresponding to each task scheduling service are formed by registering each task scheduling service in the distributed coordination service system, where each node corresponds to a different node number. In the embodiment of the present disclosure, the distributed coordination service system may specifically be a zookeeper, which is a sub-item of Apache Hadoop and is mainly used to solve some data management problems often encountered in distributed applications, such as: unified naming services, state synchronization services, cluster management, management of distributed application configuration items, and the like. The zookeeper has an event monitoring function, which allows a user to register some monitors on a designated node, and when some specific event is triggered, the zookeeper server will notify the interested client of the event. The structure of the zookeeper data model can be regarded as a tree as a whole, each node is referred to as a Znode, fig. 5 shows a distribution architecture diagram among task scheduling services in the distributed task management system, as shown in fig. 5, N task scheduling services Broker1, Broker2, … … and Broker N exist in the distributed task management system, each task scheduling service is registered in the zookeeper to form nodes corresponding to the respective task scheduling services, the nodes are/Broker/1,/Broker/2, … … and/Broker/N, and the nodes are child nodes of the nodes/brokers, that is, the/brokers is a parent node of the node corresponding to the respective task scheduling services. After the tree structure is formed, each task scheduling service can register a listener on a node of the task scheduling service to monitor a parent node/brokers, and when information in the parent node or a child node changes, a zookeeper can notify the change information to each task scheduling service.
In an embodiment of the present disclosure, the information of the task scheduling service and the corresponding node thereof may be stored in a database, that is, the data storage device 405, so that the task management server 402 can obtain the relevant information from the information, and in addition, when the task scheduling service updates the task progress, the node number of the task scheduling service itself may also be updated into the data storage device 405, as shown in fig. 6, a schematic structural diagram of the task information stored in the data storage device is shown, as shown in fig. 6, where there are a plurality of tasks, each task has a corresponding task ID, a task state, a task scheduling service identifier of the task scheduling service, and other information, for example, a task whose task ID is 1 corresponds to a task state of Step1, and a task scheduling service identifier of the task scheduling service is 1; the task state corresponding to the task with the task ID of 2 is Step2, and the task scheduling service of the task scheduling service is 2; the task state corresponding to the task with the task ID of 3 is Complete, and the task scheduling service of the task scheduling service is N. It should be noted that the task ID, the task status and the task scheduling service identifier may also be in other representations, including but not limited to the above examples, and the disclosed embodiments are not limited thereto.
In an embodiment of the present disclosure, when a task scheduling service is hung up due to a fault, a node of the faulty task scheduling service is automatically deleted, and a corresponding task may not be smoothly executed, thereby causing a task execution failure and reducing user experience.
In step S320, when it is monitored that there is a deleted first node in the nodes, a second node corresponding to the first node is created through a target task scheduling service, where the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node.
In an embodiment of the present disclosure, in order to ensure that all tasks can be executed smoothly, when there is a task scheduling service suspended due to a failure, the task scheduling service working normally can take over its corresponding tasks, and execute all tasks in the task queue in sequence. The specific process of taking over the tasks of the task scheduling service which is hung up due to the fault comprises the following steps: when the deleted first node exists in the nodes, determining a target task scheduling service by the competition of other task scheduling services except the suspended task scheduling service; a second node corresponding to the first node is then created in the zookeeper by the target task scheduling service. The method for determining the target task scheduling service uniquely from the plurality of competing task scheduling services is a characteristic of zookeeper, and as to how to determine the target task scheduling service in a competing manner, details of the embodiment of the disclosure are not repeated herein. Fig. 7 shows a relationship diagram between task scheduling services when the task scheduling services fail, and as shown in fig. 7, the task scheduling service numbered/brokers/2 fails and hangs up, so the node numbered/brokers/2 is deleted. Then a new node may be registered by the target task scheduling service to generate a new node corresponding to the node numbered/brokers/2.
In an embodiment of the present disclosure, the number of the first node and the number of the second node may not be exactly the same, for example, the number of the second node may be set to/brokers/N-r, for example, if the task scheduling service corresponding to the node numbered/brokers/2 fails and is hung up, the node is deleted, then the target task scheduling service may take over the task of the task scheduling service, and register the node corresponding to/brokers/2 on the zookeeper, and the number of the newly registered node may be set to/brokers/2-r; the number of the second node may also be set in other forms based on the number of the first node, and of course, the number of the first node and the number of the second node may also be set in completely different forms, which is not specifically limited by the embodiment of the present disclosure.
In step S330, the target task scheduling service obtains task information corresponding to the first node from a database, and adds the task information to a task queue corresponding to the target task scheduling service.
In one embodiment of the present disclosure, after the second node corresponding to the first node is successfully created, task information corresponding to the first node may be acquired from the data storage device 405 through the target task scheduling service, and the task information is added to a task queue corresponding to the target task scheduling service. In the embodiment of the present disclosure, each task scheduling service corresponds to one task queue, and after receiving the task issued by the task management server 402, each task scheduling service may add the received task to the corresponding task queue, which may improve task processing efficiency and ensure high availability of the task processing system.
Further, after the task information is added to the task queue corresponding to the target task scheduling service, the task scheduling service identifier corresponding to the task information in the data storage device 405 may also be modified to the task scheduling service identifier corresponding to the target task scheduling service.
Continuing with the example in step S320, when other task scheduling services monitor that the node numbered/brokers/2 is deleted, a new node corresponding to the node numbered/brokers/2 is created in a competitive manner, and after the node numbered/brokers/2-r is successfully created, the target task scheduling service may obtain all task information of the Broker2 from the database, place the task information into its own task queue, and modify the task scheduling service identifier corresponding to the task information in the data storage device to the task scheduling service identifier corresponding to the target task scheduling service. FIG. 8 is a diagram illustrating the relationship between task scheduling services after task recovery, and as shown in FIG. 8, the task scheduling service Broker1 is a target task scheduling service determined after competition, and registers in the zookeeper to form a new node, which is numbered/brokers/2-r and corresponds to the deleted node numbered/brokers/2. Meanwhile, fig. 9 shows a schematic structural diagram of the recovered task information stored in the data storage device, and as shown in fig. 9, the task states corresponding to the tasks with the original task IDs of 1 and 3 both become Running, the task scheduling service identifier of the task scheduling service is not changed, the task state corresponding to the task with the original task ID of 2 becomes Running, and the task scheduling service identifier of the task scheduling service is changed from 2 to 1.
In an embodiment of the present disclosure, information such as task state information and a task scheduling service identifier corresponding to a task is updated to the data storage device at regular time, in addition, a new task scheduling service may be further created, and a corresponding node is registered in the zookeeper, and the task scheduling service identifier corresponding to the new task scheduling service and corresponding node information are also updated to the data storage device for the task management server to obtain.
In an embodiment of the present disclosure, based on the system architecture diagram of the distributed task management system shown in fig. 4, the task management server 402 may issue the to-be-executed task sent by the user to each task scheduling service in the task scheduling server 403, and after receiving the to-be-executed task sent by the task management server 402, the task scheduling service may form a task execution request according to the to-be-executed task, and then send the task execution request to a corresponding task execution service in the task execution server 404, so that the task execution service executes the task.
In an embodiment of the present disclosure, the execution flow of each task scheduling service is the same, and in order to make the technical solution of the present disclosure clearer, the following describes the execution flow of the task scheduling service by taking one task scheduling service as an example.
Fig. 10 is a schematic diagram illustrating an execution flow of a task scheduling service, and as shown in fig. 10, in step S1001, a task to be executed is added to a task queue corresponding to the task scheduling service, and it is determined whether an idle thread exists in a thread pool of the task scheduling service; in step S1002, when there is an idle thread, pulling a task to be executed from a task queue corresponding to the task scheduling service, and acquiring a subtask in the task to be executed; in step S1003, it is determined whether the subtask is a subtask to be executed; in step S1004, when it is determined that the subtask is a subtask to be executed, a task execution request is formed according to the subtask to be executed; in step S1005, matching the task type of the task to be executed in the task execution request with the task type tag to determine a target task execution service; in step S1006, a task execution request is sent to the target task execution service to cause the target task execution service to execute the task; in step S1007, it is queried whether the execution of the subtask to be executed is completed; in step S1008, if the execution of the to-be-executed subtask is completed, updating task progress information corresponding to the to-be-executed task in the database; in step S1009, if the execution of the to-be-executed subtask is not completed, the to-be-executed task is added to the task queue corresponding to the task scheduling service again; in step S1010, it is determined whether there is an unexecuted subtask in the task to be executed; in step S1011, if the task exists, the unexecuted subtask is executed when the unexecuted subtask is the to-be-executed subtask, until the unexecuted subtask does not exist in the to-be-executed task; in step S1012, if not, the task state information corresponding to the task to be executed in the data storage device is updated.
Before step S1001, after receiving the to-be-executed task, the task scheduling service may put the to-be-executed task into a task queue corresponding to the task scheduling service, where the to-be-executed tasks in the task queue may be sorted according to the sequence of the receiving time, the to-be-executed task with the earlier receiving time may be located at the front end of the task queue, and the to-be-executed task with the later receiving time may be located at the rear end of the task queue. In step S1003, after the idle thread in the task scheduling service acquires a scheduled to-be-executed task, the task scheduling service needs to acquire attribute information of the subtasks because the to-be-executed task includes a plurality of subtasks, and then forms a task execution request according to the attribute information of the subtasks and the subtasks, acquiring the attribute information of the subtasks specifically includes determining whether the pulled to-be-executed task includes a plurality of subtasks and whether the subtasks are to-be-executed subtasks, for example, a to-be-executed task includes 5 sequentially arranged subtasks, which are respectively No. 1, No. 2, No. 3, No. 4, and No. 5 subtasks, acquiring an execution state of the subtask when the task scheduling service pulls the subtask No. 1, and determining whether the to-be-executed subtask is according to the execution state of the subtask, for example, the subtask No. 1 and the subtask No. 2 in the to-be-executed tasks are, when the query of whether the execution of the subtask 2 is completed is finished, the task to be executed is put into the task queue again when the query result indicates that the execution is not completed, so that the execution state of the subtask 1 acquired by the task scheduling service is that the subtask 1 is executed, and accordingly, the subtask 1 is not the subtask to be executed, and then the execution states of the remaining subtasks can be sequentially judged to determine whether the subtasks are the subtasks to be executed. In step S1007, the query of the execution status of the to-be-executed subtask may be performed at a preset time point, for example, the query is performed immediately after the task execution service receives and executes the to-be-executed subtask, the query is performed at the executed 3S and 5S, and the like, and if the to-be-executed subtask execution status queried at the preset time point is that the execution is not completed, the to-be-executed task is placed back to the tail of the task queue for sorting. In Step S1008, the task progress information is the execution progress of the subtasks to be executed, for example, Step1, Step2, Running, etc. in fig. 5 and fig. 9 are all execution progress, so as to identify where the subtasks to be executed are executed; in step S1012, the task state information is the execution result of the task to be executed, such as Complete in fig. 5, and indicates that the task to be executed is completed, but other indications may also be used, such as successful execution, waiting for re-execution, and the like.
In one embodiment of the present disclosure, after the execution of the entire to-be-executed task is completed, the task execution result, i.e., the task state information, is updated to the data management device 405, and then the steps S1001 to S1012 are repeatedly performed to execute other to-be-executed tasks in the task queue.
In an embodiment of the disclosure, the task management server issues the tasks to the task scheduling services in the task scheduling server, and in order to ensure balanced distribution of the tasks, avoid that some task scheduling services have too high load and some task scheduling services have too low load, the task issuing amount of each task scheduling service can be dynamically adjusted according to the task amount corresponding to the task scheduling service. In order to obtain the task quantity corresponding to each task scheduling service, the task quantity in the node corresponding to each task scheduling service can be periodically updated according to the task quantity in each task queue through each task scheduling service, then the task management server can obtain the task quantity of all the nodes from zookeeper, determines task distribution weight according to the task quantity, and further executes task distribution based on the task distribution weight. For example, the task scheduling service cluster includes three task scheduling services A, B, C, the number of tasks stored in the node corresponding to the task scheduling service a is 10, the number of tasks stored in the node corresponding to the task scheduling service B is 5, and the number of tasks stored in the node corresponding to the task scheduling service C is 15, so that it can be determined that the task allocation weight corresponding to the task scheduling service B is the largest, the task allocation weight corresponding to the task scheduling service a is the second to the smallest, the specific task allocation weights are 5/12, 1/3 and 1/4, if the task management background needs to issue 60 to-be-executed tasks, 25 tasks can be allocated to the task scheduling service B, 20 tasks are allocated to the task scheduling service a, and 15 tasks are allocated to the task scheduling service C, so that it can be ensured that the task scheduling service a, B, C, and C are equal to 5/12, The number of tasks in the task scheduling service B and the task scheduling service C is 30, and balance of task allocation is guaranteed.
In an embodiment of the present disclosure, only the console 401 can be touched by the user, and the internal details of the distributed task management system 400 are invisible to the user, so that the user can only perform different triggering operations on the console to perform task issuing, task status monitoring, task execution result query, and the like. Specifically, the console 401 may enter a task to be executed in response to a first trigger operation of the user; or, the console 401 may generate a task progress query request in response to the second trigger operation of the user, and send the task progress query request to the task management server 402, and after receiving the task progress query request, the task management server 402 may obtain a task state of a task corresponding to a task ID in the task progress query request from the data storage device 405, and feed back and display the task state in a display interface of the console, so as to facilitate browsing by the user.
The task management method in the embodiment of the disclosure can be applied to a plurality of scenarios, in particular to computing tasks requiring longer execution time, such as offline speech recognition, big data statistics tasks, machine learning model training, and the like.
Taking a big data statistics task as an example, assuming that one task is to push advertisements to foreign visitors in a scenic spot, firstly, people in the scenic spot can be obtained, and specifically, the composition of people in the scenic spot, namely the visitors contained in the people, can be determined according to geographic coordinate data submitted by positioning equipment carried by the visitors and stored in a database; then, according to the information such as the page which is clicked and browsed recently by each tourist, the chat content, the position sharing information and the like, judging which tourists are the foreign tourists; then, aiming at the tourists identified as foreign tourists, constructing a user portrait according to the user information, browsing information and other information; finally, related advertisements can be pushed to each tourist based on the user image of each tourist, for example, if some users frequently browse a train and air ticket booking platform, advertisement information of the train and the air ticket can be pushed to the user, and if some users frequently browse a food recommendation platform, food advertisements can be pushed to the user, and the like. In the process of pushing the advertisement, three subtasks exist, wherein the first subtask is used for screening crowd data, the second subtask is used for constructing a user portrait of a foreign tourist, the third subtask is used for advertising based on the user portrait, and the three subtasks are different in type. When the task management system performs task management, even if part of the three subtasks are executed completely and the execution of part of the subtasks is not completed, only the incomplete subtasks can be executed when the task is executed repeatedly, the completed subtasks do not need to be executed repeatedly, and the task execution efficiency is improved.
In an embodiment of the present disclosure, the task management system is a distributed task management system, which can schedule a plurality of tasks through the task scheduling server, wherein each task scheduling service can completely schedule the whole life cycle of the plurality of tasks, thereby ensuring high availability of the task management system.
According to the task management method in the embodiment of the disclosure, the task management system is combined with the distributed coordination service system, the task management system is provided with a plurality of task scheduling services, each task scheduling service is registered in the distributed coordination service system to form a node, and then the multi-node distributed task management system is formed, each task scheduling service in the distributed task management system can completely schedule the whole life cycle of a plurality of tasks, when a fault occurs in an individual task scheduling service and is hung, other task scheduling services which normally work can compete to take over the corresponding task of the task scheduling service which has the fault and is hung, service self-registration and fault self-healing are realized, the task is ensured not to be lost, and the normal service of the whole distributed task management system is ensured. Meanwhile, a plurality of task scheduling services and a plurality of task execution services exist in the distributed task management system, and different task execution services can execute different tasks, so that the task execution efficiency is improved; in addition, the weighted task issuing of the distributed task management system can balance the load of each task scheduling service, prevent the task scheduling service fault caused by overhigh load of the individual task scheduling service, and further ensure the normal service of the distributed task management system.
The following describes an embodiment of an apparatus of the present disclosure, which may be used to perform the task management method in the foregoing embodiment of the present disclosure. For details that are not disclosed in the embodiments of the apparatus of the present disclosure, please refer to the task management method described above in the present disclosure.
FIG. 11 schematically shows a block diagram of a task management device according to one embodiment of the present disclosure.
Referring to fig. 11, a task management apparatus 1100 according to an embodiment of the present disclosure is applied to a distributed task management system including a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, where the task scheduling services are used for managing and scheduling tasks of a user; the task management device 1100 includes: a listening module 1101, a node creation module 1102 and a task takeover module 1103.
The monitoring module 1101 is configured to monitor, by each task scheduling service, a parent node corresponding to the node; a node creating module 1102, configured to create, when it is monitored that a deleted first node exists in the nodes, a second node corresponding to the first node through a target task scheduling service, where the target task scheduling service corresponding to the second node is different from a task scheduling service corresponding to the first node; a task taking-over module 1103, configured to obtain, by the target task scheduling service, task information corresponding to the first node from a database, and add the task information to a task queue corresponding to the target task scheduling service.
In an embodiment of the present disclosure, the task management device 1100 further includes: and the registration module is used for registering in the distributed coordination service system through each task scheduling service so as to obtain nodes with different node numbers corresponding to each task scheduling service.
In an embodiment of the present disclosure, the task management device 1100 further includes: and the identification modification module is used for modifying the task scheduling service identification corresponding to the task information into the task scheduling service identification corresponding to the target task scheduling service.
In one embodiment of the present disclosure, the distributed task management system further includes a task management server and a task execution server; the task management device 1100 further includes: the request generation module is used for receiving the tasks to be executed sent by the task management server through the task scheduling server and forming task execution requests according to the tasks to be executed; and the task execution module is used for sending the task execution request to the task execution server so as to enable the task execution service in the task execution server to execute the task.
In one embodiment of the present disclosure, the request generation module includes: the task adding unit is used for adding the task to be executed to a task queue corresponding to the task scheduling service and judging whether an idle thread exists in a thread pool of the task scheduling service or not; the task pulling unit is used for pulling the task to be executed from the task queue corresponding to the task scheduling service and acquiring a subtask in the task to be executed when the idle thread exists; and the request generating unit is used for forming the task execution request according to the attribute information of the subtask and the subtask.
In one embodiment of the present disclosure, the request generation unit is configured to: judging whether the subtask is a subtask to be executed; and when the subtask is judged to be the subtask to be executed, forming the task execution request according to the subtask to be executed.
In one embodiment of the present disclosure, the number of the task execution services is multiple, and each of the task execution services has a different task type tag; the task execution module is configured to: matching the task type of the task to be executed in the task execution request with the task type label to determine a target task execution service; and sending the task execution request to the target task execution service so that the target task execution service executes the task.
In one embodiment of the present disclosure, the task management device 1100 is further configured to: inquiring whether the sub task to be executed is executed or not; if the execution of the subtask to be executed is completed, updating task progress information corresponding to the task to be executed in the database; and if the execution of the subtasks to be executed is not finished, adding the tasks to be executed into the task queues corresponding to the task scheduling services again.
In one embodiment of the present disclosure, the task management device 1100 is further configured to: judging whether the tasks to be executed have unexecuted subtasks or not; if the task exists, the unexecuted subtask is executed when the unexecuted subtask is a subtask to be executed until the unexecuted subtask does not exist in the task to be executed; and if not, updating the task state information corresponding to the task to be executed in the database.
In one embodiment of the present disclosure, the task management device 1100 is further configured to: each task scheduling service updates the task quantity in the node according to the task quantity in the task queue corresponding to each task scheduling service; and the task management server acquires the task quantity from each node, determines task distribution weight according to the task quantity, and executes task distribution based on the task distribution weight.
In one embodiment of the present disclosure, the distributed task management system further comprises a console; the task management device 1100 is further configured to: the console responds to a first trigger operation of a user to enter the task to be executed; or the console responds to a second trigger operation of the user to generate a task progress query request, and sends the task progress query request to the task management server to obtain task progress information.
FIG. 12 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present disclosure.
It should be noted that the computer system 1200 of the electronic device shown in fig. 12 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 12, the computer system 1200 includes a Central Processing Unit (CPU)1201, which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) 1202 or a program loaded from a storage section 1208 into a Random Access Memory (RAM) 1203, and implements the image labeling method described in the above-described embodiment. In the RAM1203, various programs and data necessary for system operation are also stored. The CPU 1201, ROM 1202, and RAM1203 are connected to each other by a bus 1204. An Input/Output (I/O) interface 1205 is also connected to bus 1204.
The following components are connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including a Display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 1208 including a hard disk and the like; and a communication section 1209 including a network interface card such as a LAN (Local area network) card, a modem, or the like. The communication section 1209 performs communication processing via a network such as the internet. A driver 1210 is also connected to the I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is mounted on the drive 1210 as necessary, so that a computer program read out therefrom is mounted into the storage section 1208 as necessary.
In particular, the processes described below with reference to the flowcharts may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1209, and/or installed from the removable medium 1211. The computer program, when executed by a Central Processing Unit (CPU)1201, performs various functions defined in the system of the present disclosure.
It should be noted that the computer readable medium shown in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
As another aspect, the present disclosure also provides a computer-readable medium, which may be included in the task management device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (15)

1. A task management method is characterized in that the task management method is applied to a distributed task management system, the distributed task management system comprises a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, wherein the task scheduling services are used for managing and scheduling tasks of users; the method comprises the following steps:
each task scheduling service monitors a father node corresponding to the node;
when the deleted first node exists in the nodes, creating a second node corresponding to the first node through a target task scheduling service, wherein the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node;
and the target task scheduling service acquires task information corresponding to the first node from a database and adds the task information to a task queue corresponding to the target task scheduling service.
2. The task management method according to claim 1, characterized in that:
and registering each task scheduling service in the distributed coordination service system to acquire nodes with different node numbers corresponding to each task scheduling service.
3. The method of claim 1, wherein after adding the task information to a task queue corresponding to the target task scheduling service, the method further comprises:
and modifying the task scheduling service identifier corresponding to the task information into a task scheduling service identifier corresponding to the target task scheduling service.
4. The task management method according to claim 1, wherein the distributed task management system further comprises a task management server and a task execution server;
the task scheduling server receives a task to be executed sent by the task management server and forms a task execution request according to the task to be executed;
and sending the task execution request to the task execution server so that the task execution service in the task execution server executes the task.
5. The task management method according to claim 4, wherein the task scheduling server receives the to-be-executed task sent by the task management server and forms a task execution request according to the to-be-executed task, and the task execution request includes:
adding the task to be executed into a task queue corresponding to the task scheduling service, and judging whether an idle thread exists in a thread pool of the task scheduling service;
when the idle thread exists, pulling the task to be executed from a task queue corresponding to the task scheduling service, and acquiring a subtask in the task to be executed;
and forming the task execution request according to the attribute information of the subtasks and the subtasks.
6. The task management method according to claim 5, wherein the forming the task execution request according to the attribute information of the subtask and the subtask includes:
judging whether the subtask is a subtask to be executed;
and when the subtask is judged to be the subtask to be executed, forming the task execution request according to the subtask to be executed.
7. The task management method according to claim 4, wherein the number of the task execution services is plural, and each of the task execution services has a different task type tag;
the sending the task execution request to the task execution server to enable the task execution service in the task execution server to execute the task includes:
matching the task type of the task to be executed in the task execution request with the task type label to determine a target task execution service;
and sending the task execution request to the target task execution service so that the target task execution service executes the task.
8. The task management method according to claim 4, wherein after sending the task execution request to the task execution server, the method further comprises:
inquiring whether the sub task to be executed is executed or not;
if the execution of the subtask to be executed is completed, updating task progress information corresponding to the task to be executed in the database;
and if the execution of the subtasks to be executed is not finished, adding the tasks to be executed into the task queues corresponding to the task scheduling services again.
9. The task management method according to claim 8, wherein after updating the task progress information corresponding to the task to be executed in the database, the method further comprises:
judging whether the tasks to be executed have unexecuted subtasks or not;
if the task exists, the unexecuted subtask is executed when the unexecuted subtask is a subtask to be executed until the unexecuted subtask does not exist in the task to be executed;
and if not, updating the task state information corresponding to the task to be executed in the database.
10. The task management method of claim 1, further comprising:
each task scheduling service updates the task quantity in the node according to the task quantity in the task queue corresponding to each task scheduling service;
and the task management server acquires the task quantity from each node, determines task distribution weight according to the task quantity, and executes task distribution based on the task distribution weight.
11. The task management method according to claim 4, wherein the distributed task management system further comprises a console; the method further comprises the following steps:
the console responds to a first trigger operation of a user to enter the task to be executed; alternatively, the first and second electrodes may be,
and the console responds to a second trigger operation of the user to generate a task progress query request and sends the task progress query request to the task management server so as to acquire task progress information.
12. A task management device is applied to a distributed task management system, the distributed task management system comprises a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, wherein the task scheduling services are used for managing and scheduling tasks of users; the device comprises:
the monitoring module is used for monitoring a father node corresponding to the node by each task scheduling service;
a node creating module, configured to create, when it is monitored that a deleted first node exists in the nodes, a second node corresponding to the first node through a target task scheduling service, where the target task scheduling service corresponding to the second node is different from a task scheduling service corresponding to the first node;
and the task taking module is used for acquiring the task information corresponding to the first node from a database by the target task scheduling service and adding the task information into a task queue corresponding to the target task scheduling service.
13. A distributed task management system, comprising:
the console is used for responding to the triggering operation of a user to generate a task issuing instruction or a task progress query request;
the task management server is connected with the console and used for responding to the task issuing instruction to issue the task or responding to the task progress inquiry request to feed back task progress information to the console;
the task scheduling server is connected with the task management server and comprises a plurality of task scheduling services, and each task scheduling service is registered in the distributed coordination service system to form a node and is used for receiving the tasks issued by the task management server and scheduling and managing the tasks through each task scheduling service;
the task execution server is connected with the task scheduling server, comprises a plurality of task execution services and is used for responding to a task execution request sent by the task scheduling service so as to execute a task;
and the data storage equipment is connected with the task management server and the task scheduling server and is used for storing information related to the tasks.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the task management method of any one of claims 1 to 11.
15. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to perform a task management method as claimed in any one of claims 1-11.
CN202010064754.9A 2020-01-20 2020-01-20 Task management method, device, system, computer storage medium and electronic equipment Active CN111290854B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010064754.9A CN111290854B (en) 2020-01-20 2020-01-20 Task management method, device, system, computer storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010064754.9A CN111290854B (en) 2020-01-20 2020-01-20 Task management method, device, system, computer storage medium and electronic equipment

Publications (2)

Publication Number Publication Date
CN111290854A true CN111290854A (en) 2020-06-16
CN111290854B CN111290854B (en) 2024-03-15

Family

ID=71023320

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010064754.9A Active CN111290854B (en) 2020-01-20 2020-01-20 Task management method, device, system, computer storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN111290854B (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111611066A (en) * 2020-06-30 2020-09-01 平安银行股份有限公司 Task execution method, task execution server, and storage medium
CN112380183A (en) * 2020-11-13 2021-02-19 深圳市和讯华谷信息技术有限公司 Distributed file synchronization method and device, computer equipment and storage medium
CN112463312A (en) * 2020-11-02 2021-03-09 北京健康之家科技有限公司 Dynamic maintenance system and method for timing task, medium and computing equipment
CN112463561A (en) * 2020-11-20 2021-03-09 中国建设银行股份有限公司 Fault positioning method, device, equipment and storage medium
CN112804093A (en) * 2020-12-31 2021-05-14 杭州东方通信软件技术有限公司 Centralized scheduling support method and system based on fault capability center
CN112948068A (en) * 2020-09-16 2021-06-11 深圳市明源云科技有限公司 Task scheduling method and device and electronic equipment
CN113065779A (en) * 2021-04-07 2021-07-02 网易（杭州）网络有限公司 Data processing method and device and electronic equipment
CN113190341A (en) * 2021-05-31 2021-07-30 内蒙古豆蔻网络科技有限公司 Server resource scheduling method and system
CN113553126A (en) * 2021-07-06 2021-10-26 网易（杭州）网络有限公司 Data processing method and device
CN113656157A (en) * 2021-08-10 2021-11-16 北京锐安科技有限公司 Distributed task scheduling method and device, storage medium and electronic equipment
CN113741872A (en) * 2021-09-03 2021-12-03 上海新炬网络信息技术股份有限公司 Software application automatic publishing method based on job scheduling
CN113760485A (en) * 2020-07-16 2021-12-07 北京沃东天骏信息技术有限公司 Scheduling method, device and equipment of timing task and storage medium
CN113760513A (en) * 2021-09-16 2021-12-07 康键信息技术（深圳）有限公司 Distributed task scheduling method, device, equipment and medium
WO2022048329A1 (en) * 2020-09-01 2022-03-10 北京达佳互联信息技术有限公司 Menu display method and apparatus
CN114567625A (en) * 2022-03-01 2022-05-31 上海创远仪器技术股份有限公司 Android Http service-based radio monitoring equipment control processing system, method, device, processor and storage medium thereof
CN115185673A (en) * 2022-05-17 2022-10-14 贝壳找房(北京)科技有限公司 Distributed timed task scheduling method, system, storage medium and program product
CN116680064A (en) * 2023-08-03 2023-09-01 中航信移动科技有限公司 Task node management method, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102209100A (en) * 2011-03-15 2011-10-05 厦门亿力吉奥信息科技有限公司 Task scheduling cloud processing system and method
CN102932210A (en) * 2012-11-23 2013-02-13 北京搜狐新媒体信息技术有限公司 Method and system for monitoring node in PaaS cloud platform
CN104462370A (en) * 2014-12-09 2015-03-25 北京百度网讯科技有限公司 Distributed task scheduling system and method
CN105447097A (en) * 2015-11-10 2016-03-30 北京北信源软件股份有限公司 Data acquisition method and system
CN106909451A (en) * 2017-02-28 2017-06-30 郑州云海信息技术有限公司 A kind of distributed task dispatching system and method
CN108958920A (en) * 2018-07-13 2018-12-07 众安在线财产保险股份有限公司 A kind of distributed task dispatching method and system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102209100A (en) * 2011-03-15 2011-10-05 厦门亿力吉奥信息科技有限公司 Task scheduling cloud processing system and method
CN102932210A (en) * 2012-11-23 2013-02-13 北京搜狐新媒体信息技术有限公司 Method and system for monitoring node in PaaS cloud platform
CN104462370A (en) * 2014-12-09 2015-03-25 北京百度网讯科技有限公司 Distributed task scheduling system and method
CN105447097A (en) * 2015-11-10 2016-03-30 北京北信源软件股份有限公司 Data acquisition method and system
CN106909451A (en) * 2017-02-28 2017-06-30 郑州云海信息技术有限公司 A kind of distributed task dispatching system and method
CN108958920A (en) * 2018-07-13 2018-12-07 众安在线财产保险股份有限公司 A kind of distributed task dispatching method and system

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111611066A (en) * 2020-06-30 2020-09-01 平安银行股份有限公司 Task execution method, task execution server, and storage medium
CN113760485A (en) * 2020-07-16 2021-12-07 北京沃东天骏信息技术有限公司 Scheduling method, device and equipment of timing task and storage medium
WO2022048329A1 (en) * 2020-09-01 2022-03-10 北京达佳互联信息技术有限公司 Menu display method and apparatus
CN112948068B (en) * 2020-09-16 2024-03-12 深圳市明源云科技有限公司 Task scheduling method and device and electronic equipment
CN112948068A (en) * 2020-09-16 2021-06-11 深圳市明源云科技有限公司 Task scheduling method and device and electronic equipment
CN112463312A (en) * 2020-11-02 2021-03-09 北京健康之家科技有限公司 Dynamic maintenance system and method for timing task, medium and computing equipment
CN112380183A (en) * 2020-11-13 2021-02-19 深圳市和讯华谷信息技术有限公司 Distributed file synchronization method and device, computer equipment and storage medium
CN112463561A (en) * 2020-11-20 2021-03-09 中国建设银行股份有限公司 Fault positioning method, device, equipment and storage medium
CN112804093A (en) * 2020-12-31 2021-05-14 杭州东方通信软件技术有限公司 Centralized scheduling support method and system based on fault capability center
CN113065779A (en) * 2021-04-07 2021-07-02 网易（杭州）网络有限公司 Data processing method and device and electronic equipment
CN113065779B (en) * 2021-04-07 2023-08-11 网易（杭州）网络有限公司 Data processing method and device and electronic equipment
CN113190341A (en) * 2021-05-31 2021-07-30 内蒙古豆蔻网络科技有限公司 Server resource scheduling method and system
CN113553126A (en) * 2021-07-06 2021-10-26 网易（杭州）网络有限公司 Data processing method and device
CN113553126B (en) * 2021-07-06 2024-03-22 网易（杭州）网络有限公司 Data processing method and device
CN113656157A (en) * 2021-08-10 2021-11-16 北京锐安科技有限公司 Distributed task scheduling method and device, storage medium and electronic equipment
CN113656157B (en) * 2021-08-10 2024-04-23 北京锐安科技有限公司 Distributed task scheduling method and device, storage medium and electronic equipment
CN113741872A (en) * 2021-09-03 2021-12-03 上海新炬网络信息技术股份有限公司 Software application automatic publishing method based on job scheduling
CN113741872B (en) * 2021-09-03 2024-04-23 上海新炬网络信息技术股份有限公司 Automatic software application publishing method based on job scheduling
CN113760513A (en) * 2021-09-16 2021-12-07 康键信息技术（深圳）有限公司 Distributed task scheduling method, device, equipment and medium
CN114567625A (en) * 2022-03-01 2022-05-31 上海创远仪器技术股份有限公司 Android Http service-based radio monitoring equipment control processing system, method, device, processor and storage medium thereof
CN115185673B (en) * 2022-05-17 2023-10-31 贝壳找房(北京)科技有限公司 Distributed timing task scheduling method, system, storage medium and program product
CN115185673A (en) * 2022-05-17 2022-10-14 贝壳找房(北京)科技有限公司 Distributed timed task scheduling method, system, storage medium and program product
CN116680064B (en) * 2023-08-03 2023-10-10 中航信移动科技有限公司 Task node management method, electronic equipment and storage medium
CN116680064A (en) * 2023-08-03 2023-09-01 中航信移动科技有限公司 Task node management method, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN111290854B (en) 2024-03-15

Similar Documents

Publication Publication Date Title
CN111290854B (en) Task management method, device, system, computer storage medium and electronic equipment
US10768988B2 (en) Real-time partitioned processing streaming
CN107590001B (en) Load balancing method and device, storage medium and electronic equipment
CN105447088B (en) A kind of multi-tenant profession cloud crawler system based on volunteer computing mode
Dai et al. Cloud service reliability: Modeling and analysis
CN106919445B (en) Method and device for scheduling containers in cluster in parallel
Bhattacharjee et al. IBM deep learning service
WO2020258290A1 (en) Log data collection method, log data collection apparatus, storage medium and log data collection system
CN109117252B (en) Method and system for task processing based on container and container cluster management system
CN112035213B (en) Multi-tenant network car booking system and dynamic isolation method
CN101146127B (en) A client buffer update method and device in distributed system
CN111694888A (en) Distributed ETL data exchange system and method based on micro-service architecture
Sun et al. Building a fault tolerant framework with deadline guarantee in big data stream computing environments
CN115280325A (en) Parameter sharing in federated learning
CN110874272A (en) Resource allocation method and device, computer readable storage medium and electronic device
Kijsipongse et al. A hybrid GPU cluster and volunteer computing platform for scalable deep learning
Kanwal et al. Multiphase fault tolerance genetic algorithm for vm and task scheduling in datacenter
Nguyen et al. A low-cost two-tier fog computing testbed for streaming IoT-based applications
Shah et al. Dynamic multilevel hybrid scheduling algorithms for grid computing
US20150067097A1 (en) Managing data distribution to networked client computing devices
Skałkowski et al. QoS-based storage resources provisioning for grid applications
Pop et al. Dynamic meta-scheduling architecture based on monitoring in distributed systems
CN115361382A (en) Data processing method, device, equipment and storage medium based on data group
Chard et al. Experiences in the design and implementation of a social cloud for volunteer computing
CN112231093A (en) Data acquisition method and system based on code template and coroutine pool and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40024407

Country of ref document: HK

TA01 Transfer of patent application right

Effective date of registration: 20230922

Address after: 101, 4th Floor, Building 9, West District, No. 10 Courtyard, Northwest Wangdong Road, Haidian District, Beijing, 100080

Applicant after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd.

Address before: 518000 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 Floors

Applicant before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant