CN110955508B - Asynchronous task scheduling method and device based on distributed architecture and computer equipment - Google Patents
Asynchronous task scheduling method and device based on distributed architecture and computer equipment Download PDFInfo
- Publication number
- CN110955508B CN110955508B CN201911192841.6A CN201911192841A CN110955508B CN 110955508 B CN110955508 B CN 110955508B CN 201911192841 A CN201911192841 A CN 201911192841A CN 110955508 B CN110955508 B CN 110955508B
- Authority
- CN
- China
- Prior art keywords
- task
- state
- target
- scheduling
- linked list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to an asynchronous task scheduling method and device based on a distributed architecture and computer equipment. The method comprises the following steps: after the distributed lock is obtained, a first target task is obtained from the distributed key value storage system; the distributed key value storage system comprises at least one task; judging whether the first target task is finished according to the task state of the first target task; if the first target task is not completed, judging whether the first target task exists in the task chain table or not; and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list. By adopting the method, when the server carries out asynchronous task scheduling downtime, other servers seize the distributed locks and continue to process the asynchronous tasks according to the task state of the tasks.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to an asynchronous task scheduling method and apparatus based on a distributed architecture, and a computer device.
Background
With the development of internet technology, distributed computing environments are often used in larger computer systems.
In a distributed computing environment, some tasks that are processed for longer periods of time typically employ asynchronous processing methods, such as migration of large amounts of data. However, the server goes down during the asynchronous task processing, which results in abnormal completion of task processing, resulting in failure of task execution and affecting the efficiency of the server in processing tasks.
Disclosure of Invention
In view of the foregoing, it is desirable to provide an asynchronous task scheduling method, an asynchronous task scheduling apparatus, and a computer device based on a distributed architecture.
In a first aspect, the present application provides an asynchronous task scheduling method, including:
after the distributed lock is obtained, a first target task is obtained from the distributed key value storage system; the distributed key value storage system comprises at least one task;
judging whether the first target task is finished according to the task state of the first target task;
if the first target task is not completed, judging whether the first target task exists in a task chain table or not;
and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and executing scheduling on each task according to the task state in the task linked list.
In one embodiment, the task state includes a new state, a start state, a process state, and a complete state.
In one embodiment, if the task state is the new state, the performing scheduling on each task according to the task state in the task linked list includes:
acquiring a task type of a second target task of which the task state is a new state in the task chain table;
respectively creating task threads for each task according to the task type;
and calling the task thread to execute scheduling on the corresponding second target task.
In one embodiment, if the task type is data migration, the invoking the task thread to perform scheduling on the corresponding second target task includes:
calling the task thread to circularly execute data migration and acquiring a migration state of the task;
and if the migration state is complete, modifying the task state of the second target task into a complete state in the task linked list.
In one embodiment, the method further comprises:
modifying the task state of the second target task in the distributed key value storage system into a processing state in the process of circularly executing data migration;
and when the migration state is the completion state, modifying the task state of the second target task in the distributed key value storage system into the completion state.
In one embodiment, the method further comprises:
and modifying the task state of the second target task in the task linked list into a starting state when task threads are respectively established for each task.
In one embodiment, the performing scheduling on each task according to the task state in the task chain table includes:
and deleting the task with the task state being the completion state from the task linked list.
In a second aspect, the present application provides an asynchronous task scheduling device, including:
the acquisition module is used for acquiring a first target task from the distributed key value storage system after the distributed lock is acquired; the distributed key value storage system comprises at least one task;
the first judgment module is used for judging whether the first target task is finished according to the task state of the first target task;
the second judgment module is used for judging whether the first target task exists in a task chain table or not if the first target task is not finished;
and the scheduling module is used for adding the first target task into the task chain table and scheduling each task according to the task state in the task chain table if the first target task does not exist in the task chain table.
In a third aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of any one of the embodiments of the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of any of the embodiments of the first aspect.
The asynchronous task scheduling method, the asynchronous task scheduling device and the computer equipment based on the distributed architecture have the following beneficial effects that:
after the server acquires the distributed lock, acquiring a first target task from a distributed key value storage system; the distributed key value storage system comprises at least one task; judging whether the first target task is finished according to the task state of the first target task; if the first target task is not completed, judging whether the first target task exists in a task chain table or not; and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list. The method determines the processing condition of the task according to the task state of the target task, adds the unprocessed task or the task failed in processing into the task chain table for processing, and can know the processing state of the target task according to the task state. When a server executing task scheduling crashes, the distributed locks are idle, other servers can seize the distributed locks to acquire unfinished tasks from the distributed key value storage system to perform task scheduling, the situations that the tasks are abnormally finished and the task scheduling fails due to crash of the server are avoided, and the efficiency of processing the tasks by the server is improved.
Drawings
FIG. 1 is a diagram of an application environment of an asynchronous task scheduling method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an asynchronous task scheduling method provided in an embodiment of the present application;
FIG. 3 is a flowchart illustrating another asynchronous task scheduling method provided in an embodiment of the present application;
FIG. 4 is a flowchart illustrating another asynchronous task scheduling method provided in an embodiment of the present application;
FIG. 5 is a flowchart illustrating another asynchronous task scheduling method provided in an embodiment of the present application;
fig. 6 is a block diagram of an asynchronous task scheduling device provided in an embodiment of the present application;
FIG. 7 is a block diagram of another asynchronous task scheduler provided in an embodiment of the present application;
FIG. 8 is a block diagram of another asynchronous task scheduler provided in an embodiment of the present application;
fig. 9 is a block diagram of another asynchronous task scheduling device provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The asynchronous task scheduling method provided by the application can be applied to the application environment shown in fig. 1. Fig. 1 provides a computer device, which may be a server, and its internal structure diagram may be as shown in fig. 1. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is for storing asynchronous task scheduling data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an asynchronous task scheduling method.
The following describes in detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems by embodiments and with reference to the drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, in the asynchronous task scheduling method provided in the embodiments of fig. 2 to fig. 5 of the present application, an execution main body of the asynchronous task scheduling method may be a server, or may be an asynchronous task scheduling device, and the asynchronous task scheduling device may become part or all of the server by software, hardware, or a combination of software and hardware. In the following method embodiments, the following method embodiments are all described by taking the example where the execution subject is a server.
In one embodiment, as shown in fig. 2, there is provided an asynchronous task scheduling method, comprising the steps of:
s201, after a distributed lock is obtained, a first target task is obtained from a distributed key value storage system; the distributed key value storage system includes at least one task.
The distributed lock is used for the server to acquire the shared data resource. For example, 100 servers share one database, and in order to avoid mutual interference in the process of accessing the database by the servers, the servers need to preempt the distributed lock when accessing the database, and the servers with successful preemption may access the database.
In this embodiment, when the server is initialized, a task scheduling process is started, and after the distributed lock is successfully preempted, the server can normally operate, and the server acquires the first target task from the distributed key value storage system. The first target task is a task for writing a task to be processed into the distributed key value storage system through the terminal. Optionally, the task state of the task may be set as a new state, and the distributed key value storage system may further store the processed task, the task to be processed, and the task being processed.
S202, judging whether the first target task is finished according to the task state of the first target task.
The task state represents the current state of the task in the processing process, the task state can label the task in the forms of labels, symbols, characters and the like, and the current processing condition of the task can be known at any time according to the task state. Optionally, the task state includes a new state, a start state, a processing state, and a completion state. The new state shows that the task is a newly established task; the start state indicates that the task has started processing; the processing state indicates that the task is processing; the completion status indicates that the task has been processed.
In this embodiment, after acquiring the first target task from the distributed key value storage system, the server determines a state of the first target task in a processing process according to a task state corresponding to the first target task, and determines whether the first target task is completed. Optionally, if the task state of the first target task is a completion state, it indicates that the task is completed; and if the task state of the first target task is a new state, a starting state or a processing state, indicating that the task is not finished. Wherein, a task corresponds to a task state within a certain time.
S203, if the first target task is not completed, judging whether the first target task exists in the task chain table.
The task chain table may include a plurality of first target tasks to be processed.
In this embodiment, after the server determines that the first target task is not completed, it determines whether the task has been added to the task chain table. For example, according to the task identifier of the first target task, querying in a task chain table, and if the same task identifier is queried, indicating that the first target task exists in the task chain table; and if the same task identifier is not inquired, indicating that the first target task does not exist in the task chain table. The task identity may be numeric, alphabetical, or a mixture of numeric and alphabetical. Optionally, if the first target task is completed, the task is skipped, and whether the next task is completed is determined.
And S204, if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list.
In this embodiment, after determining that the first target task does not exist in the task chain table, the server adds the relevant information of the first target task to the task chain table, and sets the task state of the first target task to be a new state. And when the server processes the tasks in the task chain table, processing the tasks according to the task states of the tasks in the task chain table. Optionally, if the first target task exists in the task chain table, the execution step 202 determines whether the next task is completed according to the task state of the next first target task.
According to the asynchronous task scheduling method, after a server acquires a distributed lock, a first target task is acquired from a distributed key value storage system; the distributed key value storage system comprises at least one task; judging whether the first target task is finished according to the task state of the first target task; if the first target task is not completed, judging whether the first target task exists in the task chain table or not; and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list. The method determines the processing condition of the task according to the task state of the target task, adds the unprocessed task or the task failed in processing into the task chain table for processing, and can know the processing state of the target task according to the task state. Because the tasks can be stored in the distributed key value storage system, when the server executing the task scheduling is down, the distributed locks are idle, and other servers can seize the distributed locks to acquire unfinished tasks from the distributed key value storage system to perform task scheduling, the situations that the tasks are abnormally finished and the task scheduling fails due to the down of the server are avoided, and the efficiency of processing the tasks by the server is improved.
On the basis of the embodiment shown in fig. 2, when a task in the task list is scheduled, if the task state is the new state, the server will acquire a task type creation thread of the task and process the task. As shown in fig. 3, the specific process of executing task scheduling on each task in the task chain table includes:
s301, acquiring the task type of the second target task with the task state being the new state in the task chain table.
The task type may include data migration, data download, data analysis, and the like.
In this embodiment, the server traverses the second target task in the task chain table, and if the task state in the task chain table is the new state, it indicates that the task is a task to be processed, and obtains the task type of the second target task in the new state. In the traversal process, if the task state of the second target task is the starting state and indicates that the task is being processed, the task is skipped and the task state of the next second target task is determined. Optionally, the task whose task state is the completion state is deleted from the task linked list. And if the task state of the second target task is the completion state in the traversal process, indicating that the task is processed and completed, and deleting the task from the task linked list.
And S302, respectively creating task threads for each task according to the task type.
In this embodiment, according to the task type corresponding to each task, a corresponding task thread is created for each character, and the tasks are processed respectively. For example, if the task type of task 1 is data migration and the task type of task 2 is data downloading, a data migration thread and a data downloading dominance are respectively created according to the task types of task 1 and task 2, the data migration thread is used for processing task 1, and the data downloading thread is used for processing task 2, which is not limited in this embodiment of the present application. Optionally, when task threads are respectively created for each task, the task state of the second target task in the task chain table is modified to the starting state. For example, when a data migration thread is created for task 1, the task state of task 1 in the task chain table is modified to the start state.
And S303, scheduling the corresponding second target task by calling the task thread.
In this embodiment, the server calls the task thread that has been created to process the corresponding second target task. For example, the server calls the data migration thread to perform task scheduling on task 1, and calls the data download thread to perform task scheduling on task 2. Optionally, the server traverses the tasks in the task chain table once every other period, and performs scheduling on a second target task in the task chain table.
According to the asynchronous task scheduling method, the server traverses the tasks in the task chain table, the tasks with the task states in the task chain table as the new states are executed and scheduled, the task thread is established according to the task type of the second target task, and the second target task is processed through the task thread. Tasks needing to be processed can be distinguished according to task states, a thread is created according to task types to process target tasks, the completed tasks are deleted from the database in time, bearing pressure of the server during operation is reduced, the operation speed of the server is improved, and then the efficiency of the server for processing the tasks is improved.
On the basis of the embodiment shown in fig. 3, optionally, if the task type is data migration, in the process of scheduling the second target task by calling the task thread, the server may further obtain a migration state of the task, and modify the corresponding task state according to the migration state. As shown in fig. 4, a specific procedure of the data migration task is explained.
S401, calling a task thread to circularly execute data migration, and acquiring a migration state of a task.
The migration status represents the progress of data migration, and the migration status may be in the form of a progress bar, a percentage, a migrated number, or the like. Alternatively, the data migration may be migration of data between servers, or migration of data between the inside of servers.
In this embodiment, the server invokes a data migration thread to perform circular migration on data, and obtains a migration state of a task at the same time. For example, a data migration thread is called to migrate a 100G file, the size of data migrated by the data migration thread every time is 1M, the cyclic migration is continued until the 100G file is completely migrated, meanwhile, the percentage of migrated data in the data migration process is obtained, and when the migrated data reaches 100%, the task data migration is completed. Optionally, in the process of circularly executing data migration, the task state of the second target task in the distributed key value storage system is modified into a processing state. When the calling thread migrates the data, the task state of the corresponding task in the distributed key value storage system is modified into the processing state, so that the current processing condition of the task can be clearly understood.
S402, if the migration state is complete, modifying the task state of the second target task into a complete state in the task chain table.
In this embodiment, if the migration status is complete, the server modifies the task status of the second target task in the task chain table to be the complete status. For example, when the progress of the data migration task reaches 100% or the progress bar is full, which indicates that the data is completely migrated, the task state of the second target task in the task chain table is modified from the start state to the completion state. Optionally, when the migration state is complete, modifying the task state of the second target task in the distributed key value storage system to be a complete state. And when the data migration task is completed, the server modifies the task state of the corresponding task in the distributed key value storage system from the processing state to the completion state.
According to the asynchronous task scheduling method, if the task type is data migration, the server calls the corresponding task thread to execute the data migration task, obtains the real-time data migration state, and modifies the task linked list and the task state of the corresponding task in the distributed key value storage system according to the migration state in the data migration process. The data migration task execution progress can be clearly known by acquiring the migration state, the task state can be modified in time, the server can judge the current processing condition of the task accurately, and the task processing speed is accelerated.
Based on all the above embodiments, as shown in fig. 5, a flowchart of an asynchronous task scheduling method is provided, where the flowchart includes:
s501, acquiring a distributed lock; the server seizes a distributed lock for accessing the public data resource, and starts a task scheduling process after the successful seizing;
s502, reading a task; the server reads a first target task from the distributed key value storage system;
s503, judging whether the task is finished or not; the server judges whether the task stone is finished according to the task state of the first target task; if not, go to step S504; if yes, executing step S502, skipping the first target task and continuing to read the next task;
s504, judging whether the task chain table has the task; the server judges whether the first target task exists in the task chain table according to the task identifier of the first target task; if not, go to step S505; if yes, executing step S502, skipping the first target task and continuing to read the next task;
s505, adding the task to the task linked list and setting a task state; the server adds the first target task into a task chain table, and sets the task state of the task as a new state;
s506, the server sleeps; the server continuously executes step S502 through a preset period;
s507, acquiring a task from the task chain table; the server acquires a second target task from the task chain table;
s508, executing task scheduling according to the task state of the task;
s509, if the task state is the completion state, executing the step S5010;
s5010, deleting a task; the server deletes the task from the task linked list, and executes the step S507 after the task is deleted;
s5011, if the task state is the starting state, skipping the task and continuing to execute the step S507;
s5012, if the task state is the new state, executing the step S5013 and the step S5014;
s5013, modifying the task state; the server modifies the task state of the second target task in the task chain table from the new state to the starting state, and continues to execute the step S507 after the state modification is completed.
S5014, acquiring a task type and creating a task thread; the server acquires the task type of the second target task and creates a task thread according to the task type;
s5015, data migration tasks; if the task type is data migration, the server calls a task thread to execute scheduling on the task;
s5016, scheduling the tasks, and acquiring a migration state; the server calls a data migration thread, circularly executes data migration, acquires a migration state of the data migration in real time, and modifies the state of the task in the distributed key value storage system into a processing state in the process of circularly executing the data migration;
and S5017, modifying the task state of the task in the distributed key value storage system and the task linked list to be a finished state when the migration state is finished.
The specific implementation of fig. 5 may refer to the implementation process of the embodiment corresponding to the modeling simulation method, and is not described herein again.
According to the asynchronous task processing method, when the server is initialized, the distributed locks for accessing the public data resources are preempted, and after the preemption is successful, a task scheduling process is started. The server reads a first target task from the distributed key value storage system, judges whether the task is finished according to the task state of the first target task, judges whether the task exists in a task chain table if the task is not finished, adds the task into the task chain table if the task does not exist, and modifies the task state of the task into a new state. And the server reads the second target task from the task chain table, executes scheduling according to the task state of the second target task, deletes the task from the task chain table if the second target task is completed, creates a task thread according to the task type of the task if the task state is a new state, and modifies the task state of the task in the task chain table into a starting state. If the task type is a data migration task, circularly migrating data by calling a task thread, acquiring a migration state in the data migration process, judging whether the data migration is finished according to the migration state, and modifying the task state of the task in the distributed key value storage system and the task linked list. By adopting the method, even if the server is down to finish the task abnormally in the data migration process, after other servers successfully seize the distributed lock, the tasks can be continuously read from the distributed key value storage system to execute task scheduling, the condition that the task execution fails due to the server being down is prevented, and the task processing efficiency of the server is improved.
It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in FIG. 6, there is provided an asynchronous task scheduler 600 comprising: an obtaining module 601, a first determining module 602, a second determining module 603, and a scheduling module 604, wherein:
an obtaining module 601, configured to obtain a first target task from a distributed key value storage system after obtaining a distributed lock; the distributed key value storage system comprises at least one task;
a first determining module 602, configured to determine whether the first target task is completed according to a task state of the first target task;
a second judging module 603, configured to judge whether the first target task exists in the task chain table if the first target task is not completed;
the scheduling module 604 is configured to add the first target task to the task chain table if the first target task does not exist in the task chain table, and perform scheduling on each task according to a task state in the task chain table.
In one embodiment, the task state includes a new state, a start state, a process state, and a complete state.
In one embodiment, as shown in fig. 7, if the task status is new, the scheduling module 504 includes:
an obtaining unit 6041, configured to obtain a task type of a second target task whose task state is a new state in a task chain table;
a creating unit 6042 configured to create task threads for each task according to the task type;
and a calling unit 6043, configured to call the task thread to perform scheduling on the corresponding second target task.
In one embodiment, as shown in fig. 8, if the task type is data migration, the invoking unit 6043 includes:
a calling subunit 6043a, configured to call a task thread to perform data migration circularly, and obtain a migration state of the task;
and a modifying subunit 6043b, configured to modify the task state of the second target task to be the complete state in the task chain table if the migration state is complete.
In one embodiment, as shown in fig. 9, the apparatus 600 further comprises:
a modifying module 605, configured to modify a task state of a second target task in the distributed key value storage system to a processing state in a process of circularly executing data migration;
the modification module 605 is further configured to modify the task state of the second target task in the distributed key value storage system to a completion state when the migration state is complete.
In one embodiment, the modification module 605 is further configured to modify the task state of the second target task in the task chain table to the start state when the task thread is created for each task.
In one embodiment, the scheduling module 604 is further configured to delete tasks whose task state is the completion state from the task list.
The implementation principle and technical effect of all the embodiments of the asynchronous task scheduling device are similar to those of the embodiments corresponding to the asynchronous task scheduling method, and are not described herein again.
For specific limitations of the asynchronous task scheduling device, reference may be made to the above limitations of the asynchronous task scheduling method, which are not described herein again. The modules in the asynchronous task scheduling device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, FIG. 1 above provides a computer device, the internal structure of which may be as shown in FIG. 1. Those skilled in the art will appreciate that the architecture shown in fig. 1 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
after the distributed lock is obtained, a first target task is obtained from the distributed key value storage system; the distributed key value storage system comprises at least one task;
judging whether the first target task is finished according to the task state of the first target task;
if the first target task is not completed, judging whether the first target task exists in the task chain table or not;
and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list.
In one embodiment, the processor, when executing the computer program, further performs the steps of: the task state comprises a new state, a starting state, a processing state and a finishing state.
In one embodiment, the processor, when executing the computer program, further performs the steps of: if the task state is the new state, scheduling each task according to the task state in the task linked list, including:
acquiring a task type of a second target task of which the task state is a new state in a task chain table;
respectively creating task threads for each task according to the task type;
and calling the task thread to perform scheduling on the corresponding second target task.
In one embodiment, the processor, when executing the computer program, further performs the steps of: if the task type is data migration, calling the task thread to execute scheduling on the corresponding second target task, wherein the scheduling comprises the following steps:
calling a task thread to circularly execute data migration and acquiring a migration state of a task;
and if the migration state is the completion state, modifying the task state of the second target task into the completion state in the task linked list.
In one embodiment, the processor, when executing the computer program, further performs the steps of: modifying the task state of a second target task in the distributed key value storage system into a processing state in the process of circularly executing data migration;
and when the migration state is the completion state, modifying the task state of the second target task in the distributed key value storage system into the completion state.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and modifying the task state of a second target task in the task chain table into a starting state when the task thread is respectively established for each task.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and executing scheduling on each task according to the task state in the task chain table, wherein the scheduling comprises the following steps: and deleting the task with the task state as the completion state from the task chain table.
The implementation principle and technical effect of the computer device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
after the distributed lock is obtained, a first target task is obtained from the distributed key value storage system; the distributed key value storage system comprises at least one task;
judging whether the first target task is finished according to the task state of the first target task;
if the first target task is not completed, judging whether the first target task exists in the task chain table or not;
and if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and scheduling each task according to the task state in the task linked list.
In one embodiment, the computer program when executed by the processor further performs the steps of: the task state comprises a new state, a starting state, a processing state and a finishing state.
In one embodiment, the computer program when executed by the processor further performs the steps of: if the task state is the new state, scheduling each task according to the task state in the task linked list, including:
acquiring a task type of a second target task of which the task state is a new state in a task chain table;
respectively creating task threads for each task according to the task type;
and calling the task thread to perform scheduling on the corresponding second target task.
In one embodiment, the computer program when executed by the processor further performs the steps of: if the task type is data migration, calling the task thread to execute scheduling on the corresponding second target task, wherein the scheduling comprises the following steps:
calling a task thread to circularly execute data migration and acquiring a migration state of a task;
and if the migration state is the completion state, modifying the task state of the second target task into the completion state in the task linked list.
In one embodiment, the computer program when executed by the processor further performs the steps of: modifying the task state of a second target task in the distributed key value storage system into a processing state in the process of circularly executing data migration;
and when the migration state is the completion state, modifying the task state of the second target task in the distributed key value storage system into the completion state.
In one embodiment, the computer program when executed by the processor further performs the steps of: and modifying the task state of a second target task in the task chain table into a starting state when the task thread is respectively established for each task.
In one embodiment, the computer program when executed by the processor further performs the steps of: and executing scheduling on each task according to the task state in the task chain table, wherein the scheduling comprises the following steps:
and deleting the task with the task state as the completion state from the task chain table.
The implementation principle and technical effect of the computer-readable storage medium provided by the above embodiments are similar to those of the above method embodiments, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. An asynchronous task scheduling method, the method comprising:
under the condition that a server executing task scheduling is down, the distributed locks are in an idle state, and after the server acquires the distributed locks, the server acquires a first target task from the distributed key value storage system; the distributed key value storage system comprises at least one task; the server is different from any one of the servers in the downtime state and other servers in the normal state;
judging whether the first target task is finished according to the task state of the first target task; the task state comprises a new state, a starting state, a processing state and a finishing state;
if the first target task is not completed, judging whether the first target task exists in a task chain table or not;
if the first target task does not exist in the task linked list, adding the first target task into the task linked list, and executing scheduling on each task according to the task state in the task linked list, wherein if the task state in the task linked list is a new state, the task type of the task is obtained; if the task state in the task chain table is the starting state, skipping the task and judging the task state of the next task; and if the task state in the task linked list is the completion state, deleting the task from the task linked list.
2. The method of claim 1, wherein the determining whether the first target task exists in a task chain table comprises:
inquiring in the task linked list according to the task identifier of the first target task;
if the task identifier is inquired, the first target task exists in the task linked list;
and if the task identifier is not inquired, indicating that the first target task does not exist in the task chain table.
3. The method of claim 1, wherein if the task state is a new state, the performing scheduling on each task according to the task state in the task list comprises:
acquiring a task type of a second target task of which the task state is a new state in the task chain table;
respectively creating task threads for each task according to the task type;
and calling the task thread to execute scheduling on the corresponding second target task.
4. The method of claim 3, wherein if the task type is data migration, said invoking the task thread to perform scheduling on the corresponding second target task comprises:
calling the task thread to circularly execute data migration and acquiring a migration state of the task;
and if the migration state is complete, modifying the task state of the second target task into a complete state in the task linked list.
5. The method of claim 4, further comprising:
modifying the task state of the second target task in the distributed key value storage system into a processing state in the process of circularly executing data migration;
and when the migration state is the completion state, modifying the task state of the second target task in the distributed key value storage system into the completion state.
6. The method of claim 3, further comprising:
and modifying the task state of the second target task in the task linked list into a starting state when task threads are respectively established for each task.
7. The method of claim 1, wherein the performing scheduling for each task according to the task state in the task list comprises:
and deleting the task with the task state being the completion state from the task linked list.
8. An asynchronous task scheduler, the apparatus comprising:
the distributed lock scheduling system comprises an acquisition module, a distributed key value storage system and a task scheduling module, wherein the acquisition module is used for enabling a distributed lock to be in an idle state under the condition that a server for executing task scheduling is down, and acquiring a first target task from the distributed key value storage system after the server acquires the distributed lock; the distributed key value storage system comprises at least one task; the server is different from any one of the servers in the downtime state and other servers in the normal state;
the first judgment module is used for judging whether the first target task is finished according to the task state of the first target task; the task state comprises a new state, a starting state, a processing state and a finishing state;
the second judgment module is used for judging whether the first target task exists in a task chain table or not if the first target task is not finished;
the scheduling module is used for adding the first target task into the task chain table and scheduling each task according to the task state in the task chain table if the first target task does not exist in the task chain table, wherein if the task state in the task chain table is a new state, the task type of the task is acquired; if the task state in the task chain table is the starting state, skipping the task and judging the task state of the next task; and if the task state in the task linked list is the completion state, deleting the task from the task linked list.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program implements the steps of the asynchronous task scheduling method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the asynchronous task scheduling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911192841.6A CN110955508B (en) | 2019-11-28 | 2019-11-28 | Asynchronous task scheduling method and device based on distributed architecture and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911192841.6A CN110955508B (en) | 2019-11-28 | 2019-11-28 | Asynchronous task scheduling method and device based on distributed architecture and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110955508A CN110955508A (en) | 2020-04-03 |
| CN110955508B true CN110955508B (en) | 2021-04-13 |
Family
ID=69978795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911192841.6A Active CN110955508B (en) | 2019-11-28 | 2019-11-28 | Asynchronous task scheduling method and device based on distributed architecture and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110955508B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111782360B (en) * | 2020-06-28 | 2023-08-11 | 中国工商银行股份有限公司 | Distributed task scheduling method and device |
| CN112328392B (en) * | 2020-10-27 | 2024-05-28 | 金蝶软件（中国）有限公司 | Data processing method and related equipment |
| CN112527489B (en) * | 2020-12-22 | 2024-07-19 | 税友软件集团股份有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN113010326A (en) * | 2021-03-22 | 2021-06-22 | 平安科技（深圳）有限公司 | Distributed lock processing method and device, electronic equipment and storage medium |
| CN115061807A (en) * | 2022-07-27 | 2022-09-16 | 中建电子商务有限责任公司 | Method for generating and scheduling execution of tasks as required |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092698A (en) * | 2012-12-24 | 2013-05-08 | 中国科学院深圳先进技术研究院 | System and method of cloud computing application automatic deployment |
| CN109408212A (en) * | 2018-09-28 | 2019-03-01 | 平安科技（深圳）有限公司 | Task schedule component building method, device and storage medium, server |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7546486B2 (en) * | 2006-08-28 | 2009-06-09 | Bycast Inc. | Scalable distributed object management in a distributed fixed content storage system |
| CN103780655B (en) * | 2012-10-24 | 2018-03-06 | 阿里巴巴集团控股有限公司 | A kind of message passing interface task and resource scheduling system and method |
| CN103647834B (en) * | 2013-12-16 | 2017-03-22 | 上海证券交易所 | System and method used for processing multi-phase distributed task scheduling |
| CN104735138B (en) * | 2015-03-09 | 2018-01-09 | 中国科学院计算技术研究所 | A kind of distributed acquisition method and system of user oriented generation content |
| CN106681824A (en) * | 2015-11-10 | 2017-05-17 | 阿里巴巴集团控股有限公司 | Distributed system scheduling method and device and electronic equipment |
| CN106874094A (en) * | 2017-02-17 | 2017-06-20 | 广州爱九游信息技术有限公司 | timed task processing method, device and computing device |
| JP2019121240A (en) * | 2018-01-09 | 2019-07-22 | 東芝メモリ株式会社 | Workflow scheduling system, workflow scheduling method and electronic apparatus |
| CN110245011A (en) * | 2018-03-08 | 2019-09-17 | 北京京东尚科信息技术有限公司 | A kind of method for scheduling task and device |
-
2019
- 2019-11-28 CN CN201911192841.6A patent/CN110955508B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092698A (en) * | 2012-12-24 | 2013-05-08 | 中国科学院深圳先进技术研究院 | System and method of cloud computing application automatic deployment |
| CN109408212A (en) * | 2018-09-28 | 2019-03-01 | 平安科技（深圳）有限公司 | Task schedule component building method, device and storage medium, server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110955508A (en) | 2020-04-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110955508B (en) | Asynchronous task scheduling method and device based on distributed architecture and computer equipment | |
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN110928653B (en) | Cross-cluster task execution method and device, computer equipment and storage medium | |
| CN110286998B (en) | Virtual machine graphical user interface validation and execution process recording method and device | |
| CN111190753B (en) | Distributed task processing method and device, storage medium and computer equipment | |
| CN111008188A (en) | Data migration method and device, computer equipment and storage medium | |
| CN109542719B (en) | Thread state monitoring method and device, computer equipment and storage medium | |
| CN110162344B (en) | Isolation current limiting method and device, computer equipment and readable storage medium | |
| CN111290919A (en) | Log file generation method and device, computer equipment and storage medium | |
| CN110851159A (en) | Business rule updating method and device, computer equipment and storage medium | |
| CN110990132A (en) | Asynchronous task processing method and device, computer equipment and storage medium | |
| CN114531477B (en) | Method and device for configuring functional components, computer equipment and storage medium | |
| CN111984520A (en) | Buried point testing method, computer device and computer-readable storage medium | |
| CN110727509A (en) | Task scheduling method and device, computer equipment and storage medium | |
| US20110078657A1 (en) | Operation management server, rollback execution method, and rollback execution program | |
| CN108389124B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN114006891A (en) | Information reporting method, device, equipment and storage medium | |
| CN109901998B (en) | Resource recovery method, device, system, computer equipment and storage medium | |
| CN112132652A (en) | Order information acquisition method and device, computer equipment and storage medium | |
| CN111078370B (en) | Task execution method and device of fabric alliance chain and computer equipment | |
| CN114567664A (en) | Message processing result monitoring method and device, computer equipment and storage medium | |
| CN112214336B (en) | Data queue pushing method and device, computer equipment and storage medium | |
| CN113268348B (en) | Switching method and device of distributed lock server, computer equipment and medium | |
| CN113868127A (en) | Online debugging method and device, computer equipment and storage medium | |
| CN114416689A (en) | Data migration method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |