CN115509716A

CN115509716A - Task scheduling method, system and storage medium

Info

Publication number: CN115509716A
Application number: CN202211256126.6A
Authority: CN
Inventors: 庄建超; 刘俊
Original assignee: Guangdong 3vjia Information Technology Co Ltd
Current assignee: Guangdong 3vjia Information Technology Co Ltd
Priority date: 2022-10-13
Filing date: 2022-10-13
Publication date: 2022-12-23

Abstract

The application provides a task scheduling method, a system and a storage medium, wherein the task scheduling method is applied to a task scheduling system; the task scheduling system comprises a region scheduling module which is provided with a core interaction submodule, a message generation submodule and a region scheduling submodule. The task scheduling method comprises the following steps: acquiring a target task by a core interaction submodule; the message generation sub-module acquires a target task from the core interaction sub-module and generates a message queue; the regional scheduling submodule acquires a target message queue from the message queue; acquiring a target machine from the core interaction submodule by the regional scheduling submodule; and generating a regional scheduling strategy by the regional scheduling submodule according to the target message queue and the target machine. The average turnover time of the operation is reduced by a two-stage scheduling mode, and the throughput capacity of the system is improved. And the priorities exist between the message queues and in the message queues, so that the task with high priority is guaranteed to be executed preferentially.

Description

Task scheduling method, system and storage medium

Technical Field

The present application relates to the field of computers, and in particular, to a method, a system, and a storage medium for task scheduling.

Background

A rendering platform consists of hundreds or even thousands of rendering nodes. And the rendering platform acquires the rendering task and outputs a high-quality image which is close to reality. The whole rendering process mainly comprises scene file loading and picture rendering, and a large amount of network IO, storage IO, memory, CPU, GPU and other resources are needed in the process.

At present, a traditional rendering platform generally comprises a machine management module, a task management module, a queue module, a scheduling module and the like, the scheduling module is mostly a centralized scheduler in realization, and once the scheduler is down, the rendering task cannot be issued, so that the availability of the system is influenced. And the queue module is generally a single queue or a plurality of queues; often resulting in high priority tasks in the queue not being executed in a timely manner.

Disclosure of Invention

The embodiment of the application aims to provide a task scheduling method, a task scheduling system and a storage medium, wherein a region scheduling module is only responsible for task scheduling at a region level, the average turnover time of operation is reduced through a secondary scheduling mode, and the throughput capacity of the system is improved. And the priorities exist between the message queues and in the message queues, so that the task with high priority is guaranteed to be executed preferentially.

In a first aspect, an embodiment of the present application provides a task scheduling method, where the task scheduling method is applied to a task scheduling system; the task scheduling system comprises a region scheduling module with a core interaction submodule, a message generation submodule, a region scheduling submodule and a data storage submodule; the method comprises the following steps: and the core interaction submodule acquires a target task and stores the target task to the data storage submodule. The message generation submodule acquires a target task from the data storage submodule and generates a message queue; the regional scheduling submodule acquires a target message queue from the message queue; acquiring a target machine from the core interaction submodule by the regional scheduling submodule; the target machine is used for executing a target task, and the target task comprises a rendering task; and generating a regional scheduling strategy by the regional scheduling submodule according to the target message queue and the target machine.

In the implementation process, according to the task scheduling method provided by the embodiment of the application, the regional scheduling module generates the target message queue by acquiring the target task, and the messages in the target message queue have priorities, so that when the tasks are executed, the tasks with high priorities are dequeued firstly, and the tasks with low priorities are dequeued secondly; and ensuring that the tasks with high priority can be executed in time. Further, the correct available area can be selected through the target message queue and the target machine, rather than scheduling individual machines directly; but the target machine is continuously scheduled in the available area, and the state of the task, the state of the available area and the state of the machine are shared, so that the method has extremely high availability.

Optionally, in this embodiment of the present application, the obtaining, by the regional scheduling sub-module, the target message queue from the message queue includes: acquiring an ID of a message queue; acquiring the priority of the message queue according to the ID; and generating a target message queue according to the priority.

In the implementation process, the task scheduling submodule can generate a target message queue with priority according to the ID of the message queue; when executing the task, the target message queue dequeues the task according to the priority of the message queue and the priority in the message queue; the priority exists between the queues, and the priority also exists inside the queues, so that the timely execution of high-priority tasks is guaranteed.

Optionally, in this embodiment of the application, the obtaining, by the region scheduling sub-module, the target machine from the core interaction sub-module includes: the regional scheduling submodule acquires the machine state from the core interaction submodule; wherein the machine state comprises a state of a node lock; the node lock is a universal unique identification code of the machine; the machine state comprises the IP of the node, the running state of the node and the state of the node lock; the node IP is a universal unique identifier of the machine.

In the implementation process, the idle machine, namely the target machine, is found out by judging whether the node lock state in the machine state is empty or not; and determining a final region scheduling strategy by combining the number of the target machines of the available regions and the rendering tasks, wherein the region scheduling strategy comprises the selected available regions. According to the method and the device, the available area is selected first, then the specific machine is selected, and the scheduling strategy with the maximum machine resource utilization rate is made under the condition of reasonably evaluating the machines in all the available areas.

Optionally, in this embodiment of the present application, the task scheduling system further includes an application module; the region scheduling module also comprises a data storage submodule. After the area scheduling submodule generates an area scheduling strategy according to the target message queue, the machine state and the state of the node lock, the task scheduling method further comprises the following steps: and the regional scheduling submodule sends the regional scheduling strategy to the core interaction submodule. And the core interaction submodule sends the region scheduling strategy to the data storage submodule. And the application module acquires the target task and the regional scheduling strategy from the data storage submodule.

In the implementation process, by arranging the MySQL database (data storage module) and the application module, the module can be used for taking data from the MySQL database, and real-time synchronous display of the data is realized; and the existence of the application module can reduce the consumption of CPU resources, memory resources and network bandwidth resources.

Optionally, in this embodiment of the present application, the region scheduling module of the task scheduling system further includes a machine control sub-module; the task scheduling method further comprises the following steps: and the machine control submodule acquires the target message queue and the idle machine from the core interaction submodule. Judging whether the number of target machines corresponding to the target message queue is smaller than the number of idle machines or not; if the number of the target machines corresponding to the target message queue is smaller than the number of the idle machines, closing the idle machines with the designated number; wherein the designated number is a difference between the number of idle machines and the number of target machines.

In the implementation process, machines in the system can be reasonably controlled through the setting of the machine management submodule, a reasonable number of machines are enabled to be on line, and if excessive idle machines exist, the machine management submodule can close the redundant idle machines so as to achieve reasonable utilization of resources.

Optionally, in this embodiment of the present application, the task scheduling system further includes a node scheduling module having a node scheduling submodule and a task execution submodule; the method further comprises the following steps: judging whether the data can be pulled from the core interaction submodule or not by the node scheduling submodule; and if the data can be pulled from the core interaction submodule, acquiring the regional scheduling strategy from the core interaction submodule. The node scheduling submodule issues a scheduling instruction to a target machine according to a regional scheduling strategy; and executing the target task by the task execution submodule according to the scheduling instruction.

In the implementation process, the node scheduling submodule acquires a region scheduling strategy from the region scheduling submodule and further schedules a target machine so as to execute a rendering task; the task scheduling system provided by the embodiment of the application overcomes the defect of centralized scheduling in a two-level scheduling mode, the regional scheduling submodule is responsible for the scheduling tasks at the regional level, and the node scheduling submodule is responsible for the scheduling tasks at the node level; the two-stage scheduling mode can reduce the average turnover time of the operation and improve the throughput capacity of the system.

Optionally, in this embodiment of the present application, the task scheduling method further includes: and if the data can not be pulled from the core interaction submodule, the node scheduling submodule acquires a target message queue from the message generation submodule. And acquiring a target machine, and generating a regional scheduling strategy according to the target message queue and the target machine.

In the implementation process, the node scheduling submodule can directly take the task from the message queue for scheduling under the condition that the regional scheduling submodule completely fails, and system paralysis and machine resource loss are avoided.

Optionally, in this embodiment of the application, after the task execution sub-module executes the target task according to the scheduling instruction, the task scheduling method further includes: and the task execution submodule acquires the machine state and sends the machine state to the node scheduling submodule. And the node scheduling submodule transmits the machine state to the core interaction submodule.

In the implementation process, after the target task is executed, the task execution submodule can reacquire the state of the machine; sending the machine state to a node scheduling submodule; further, the node scheduling submodule returns the machine state to a core interaction submodule of the region scheduling module. Therefore, the core scheduling submodule updates the state of the machine, and the state can be used as basic information for scheduling subsequent tasks, so that the task scheduling system can be guaranteed to operate efficiently.

In a second aspect, an embodiment of the present application provides a task scheduling system, which is applied to a task scheduling system; the task scheduling system comprises a region scheduling module with a core interaction submodule, a message generation submodule and a region scheduling submodule; the core interaction submodule is used for acquiring a target task; and the message generation submodule is used for acquiring the target task from the core interaction submodule and generating a message queue. The regional scheduling submodule is used for acquiring a target message queue from the message queue; the region scheduling submodule is also used for acquiring a target machine from the core interaction submodule; wherein the target machine is used for executing a target task; the target task includes a rendering task. And the regional scheduling sub-module is also used for generating a regional scheduling strategy according to the target message queue and the target machine.

In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores program instructions, and when the processor reads and executes the program instructions, the electronic device executes steps in any one of the foregoing implementation manners.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are read and executed by a processor, the steps in any of the above implementation manners are performed.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a flowchart of regional scheduling provided in an embodiment of the present application;

fig. 2 is a flowchart of generating a target message queue according to an embodiment of the present application;

fig. 3 is a flowchart for acquiring a target machine according to an embodiment of the present disclosure;

FIG. 4 is a flow chart of machine control provided by an embodiment of the present application;

fig. 5 is a flowchart of node scheduling provided in the embodiment of the present application;

FIG. 6 is a schematic diagram of task scheduling provided in an embodiment of the present application;

FIG. 7 is a block diagram of a task scheduling system according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. For example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

In the research process, the applicant finds that the conventional rendering platform generally comprises a machine management module, a task management module, a queue module, a scheduling module and the like, the scheduling module is mainly a centralized scheduler in realization, and once the scheduler is down, the rendering task cannot be issued, so that the availability of the system is influenced. The queue module is generally a single queue or a plurality of queues, and the priority in the same queue is not realized; often resulting in high priority tasks in the queue not being executed in time.

Based on the task scheduling method, the task scheduling system and the storage medium, the regional scheduling module is only responsible for regional-level task scheduling, the average turnover time of operation is reduced through a secondary scheduling mode, and the throughput capacity of the system is improved. And the priorities exist between the message queues and in the message queues, so that the task with high priority is guaranteed to be executed preferentially, and the availability of the system is high.

Before the specific solutions of the embodiments of the present application are introduced, services executed by each module in the task scheduling system in the present application are introduced in a unified manner.

A core interaction submodule, which is used for executing core-service (core service), wherein the core-service is mainly used for providing a global state; the global state mainly includes a machine state, a task state, and the like.

The message generation submodule can be used for executing api-service, the api-service is a service gateway of the task scheduling system in the embodiment of the application, and the functions of protecting internal services and exposing interfaces to the outside can be realized; the exposed interfaces include, but are not limited to, submission of tasks, query of task status, query of real-time milestone log (query of health status of control plane components, query of the number of piled up task queues, and query of memory and CPU usage of all machines, prohibition or permission of machine scheduling), and the like.

The data storage submodule, which may be a MySQL database, is used to store data of the area scheduling module and the node scheduling module, such as machine state, task state, message queue, etc.

And the region scheduling submodule comprises a scheduler service for scheduling the rendering task, wherein the scheduler service in the region scheduling submodule is used for realizing the scheduling task and is focused on realizing the scheduling at the level of the available region.

The application module comprises a UI service and a third-party service, supports the butt joint of an Operation Support System (OSS), can directly upload the scene file to the OSS during operation, and downloads the scene file during the execution of the rendering task.

The machine control submodule can execute a node-scaler service, and the node-scaler service is used for controlling the elastic startup and shutdown of a machine and is mainly used for improving the utilization rate of the machine and unnecessarily wasting resources.

And the node scheduling submodule is mainly used for carrying out scheduler service on the node scheduling module and is used for realizing task scheduling at a node level.

The task execution submodule is used for executing the rendering service.

Referring to fig. 1, fig. 1 is a flowchart of a regional scheduling provided in an embodiment of the present application; the method is applied to a task scheduling system, and the task scheduling system comprises a region scheduling module with a core interaction submodule, a message generation submodule and a region scheduling submodule. The method comprises the following steps:

step S100: and the core interaction submodule acquires a target task and stores the target task to the data storage submodule.

In the step S100, the core-service of the core interaction submodule acquires a target task; and the core-service synchronously updates the target task to the MySQL database. And each time the target task is updated, the core-service synchronizes the updated target task to the MySQL database, so that the latest state in the database can be ensured. The task scheduling method provided by the embodiment of the application is mainly used for a pollution scheduling system capable of executing any rendering task, and the target task mainly comprises the rendering task.

It should be noted that, in the task scheduling system provided in the embodiment of the present application, the core interaction submodule may be configured to execute a core-service (core service), where the core-service is mainly used to provide a global state; the global state mainly includes a machine state, a state of a machine node lock, a task state and the like.

Step S101: and the message generation submodule acquires the target task from the data storage submodule and generates a message queue.

In the step S101, the message generation submodule acquires the target task from the data storage submodule, and generates a message queue according to the target task. and the api-service acquires the target task from the MySQL database and generates a message queue according to the target task (the api-service serves as a producer of kafka and creates a new message in the queue of kafka).

It should be noted that the message generation submodule in the task scheduling system provided in the embodiment of the present application may be configured to execute api-service, where api-service is a service gateway of the task scheduling system in the embodiment of the present application, and may implement a function of protecting an internal service and exposing an interface to the outside; the exposed interfaces include, but are not limited to, submission of tasks, query of task status, query of real-time milestone logs (query of health status of control plane components, query of number of piles of task queues, and query of memory and CPU usage of all machines, prohibition or permission of machine scheduling), and the like.

Optionally, the api-service pushes the task data to a message queue after generating the message queue, which may be a kafka message queue. Wherein, tasks with different priorities can be submitted to different priorities by the api-service, and each priority is guaranteed to only comprise one queue.

As will be understood by those skilled in the art, kafka is a distributed, partition-enabled, multi-copy (replenisher) based zookeeper coordinated, fractional messaging system, which has the greatest characteristic of being able to process large amounts of data in real time to meet various demand scenarios.

In the implementation process, after the message queue is generated, the api-service pushes the task data to the message queue, so that the task with high priority can be preferentially executed, and the important and urgent rendering task is not covered by non-urgent unimportant (the task with high priority is covered by the task with low priority) to cause delayed execution of the important task.

Step S102: and the regional scheduling submodule acquires a target message queue from the message queue.

In step S102, the regional scheduling sub-module obtains the target message queue from the message queue. Illustratively, the scheduler service pulls the ID of the task in the target message queue from the kafka queue and stores the ID in a memory queue of the scheduler service; it can be understood that the message queues in the scheduler service correspond to the message queues in kafka one-to-one.

Step S103: and the regional scheduling submodule acquires the target machine from the core interaction submodule.

In the step S103, the regional scheduling sub-module scheduler service obtains the target machine from the core interaction sub-module; it should be noted that the target machine is a machine for executing a target task, and the target task in the embodiment of the present application includes a rendering task.

Step S104: and generating a regional scheduling strategy by the regional scheduling submodule according to the target message queue and the target machine.

In the step S104, the regional scheduling sub-module generates a regional scheduling policy according to the target message queue and the target machine; for example, when the number of machines required for the target message queue is greater than the number of free machines in the machine availability zone a, the scheduler service ignores the machine availability zone a, i.e., does not schedule the machines in the area a. When the number of machines required by the target message queue is less than or equal to the number of idle machines in the machine available area B, the scheduler service selects the machine available area B; after the available area is selected, setting the area name of a machine available area B in a scheduledZone field (the field represents the address of the machine available area where the target message queue is to be executed) of the rendering task in the target message queue; meanwhile, the region scheduling strategy is transmitted back to the core-service, so that the region scheduling strategy exists in the core-service, and therefore the scheduling of the usable region level can be completed.

As can be seen from fig. 1, in the task scheduling method provided in the embodiment of the present application, a target message queue is generated by obtaining a target task at a regional scheduling module, where messages in the target message queue have priorities, so that it can be ensured that a task with a high priority is dequeued first and then dequeued after the task with a low priority is executed; and ensuring that the tasks with high priority can be executed in time. Further, the correct available area can be selected by the target message queue and the target machine, rather than scheduling individual machines directly; but the target machine is continuously scheduled in the available area, and the state of the task, the state of the available area and the state of the machine are shared, so that the method has extremely high availability.

Referring to fig. 2, fig. 2 is a flowchart of generating a target message queue according to an embodiment of the present application; the method comprises the following steps:

step S200: and acquiring the ID of the message queue.

In the step S200, the scheduler service pulls the ID of the task in the target message queue from the kafka queue and stores the ID in the memory of the scheduler service; it should be noted that the message queues in the scheduler service correspond to the message queues in the kafka one-to-one.

Step S201: and acquiring the priority of the message queue according to the ID.

In the step S201, the scheduler service obtains the priority of the message queue according to the ID.

Step S202: and generating a target message queue according to the priority.

In the above steps S201 to S202, a target message queue can be generated according to the ID of the message queue, where the target message queue has a priority, and for example, the messages in the target message queue are dequeued according to the priority, the elements with high priority are dequeued first, and the elements with low priority are dequeued later. For example, 3 message queues a, B, and C exist in the target message queue, and the corresponding priorities are 3, 2, and 1; when dequeuing, the tasks in the queue A are preferentially dequeued, and the tasks in the queue C are finally dequeued; the tasks in each queue also have priority, and dequeuing also divides the sequence.

As can be seen from fig. 2, the task scheduling sub-module can generate a target message queue with priority according to the ID of the message queue; when executing the task, the target message queue dequeues the task according to the priority of the message queue and the priority in the message queue; the priority exists between the queues, and the priority also exists in the queues, so that the high-priority tasks are guaranteed to be executed in time.

Please refer to fig. 3, fig. 3 is a flowchart for acquiring a target machine according to an embodiment of the present disclosure; the method comprises the following steps:

step S300: and the regional scheduling submodule acquires the machine state from the core interaction submodule.

In the step S300, the scheduler service of the regional scheduling submodule acquires the state of the machine from the core-service of the core interaction submodule; wherein the state of the machine may be available, unavailable; the unavailable condition includes a malfunctioning machine and a machine that is performing a task.

It should be noted that the machine state includes a node lock state, and the node lock may be understood as a field characterizing the machine state.

Step S301: and judging whether the state of the node lock of the machine state is null or not.

In step S301, the state of the node lock may be null or not null; when the state of the node lock is not empty, the machine corresponding to the node which is not empty is determined to be running a task, and the scheduler service ignores the node; when the value of the node lock is empty and the state of the machine is normal, and the number of idle machines in the available area is greater than or equal to the number of machines required in the rendering task, the available area is selected, and the final area scheduling result is written back to the core-service.

Step S302: and if the state of the node lock is empty, taking the machine corresponding to the node lock as a target machine.

In the step S302, the scheduler service determines that the node lock is empty, and takes the machine corresponding to the node lock as the target machine; if the target machine number of the available area is larger than or equal to the machine number required in the rendering task, the available area is selected.

As can be seen from fig. 3, in the embodiment of the present application, an idle machine, that is, a target machine, is found by determining whether a node lock state in a machine state is empty; and determining a final region scheduling strategy by combining the number of the target machines of the available regions and the rendering tasks, wherein the region scheduling strategy comprises the selected available regions. According to the method and the device, the available area is selected first, then the specific machine is selected, and the scheduling strategy with the maximum machine resource utilization rate is made under the condition of reasonably evaluating the machines in all the available areas.

In an optional embodiment, the task scheduling system provided in the embodiment of the present application further includes an application module, where the application module includes a UI service and a third-party service, and supports docking with an Operation Support System (OSS), and when running, the application module may directly upload a scene file to the OSS and download the scene file when executing a rendering task; therefore, resources and cost of the server, such as CPU resources, memory resources, network bandwidth resources and the like, can be saved to the greatest extent.

After the area scheduling sub-module generates the area scheduling policy according to the target message queue, the machine state and the state of the node lock, the method further includes: the regional scheduling submodule sends the regional scheduling strategy to the core interaction submodule; and the core interaction submodule sends the region scheduling strategy to the data storage submodule. Further, the application module acquires the target task and the regional scheduling policy from the data storage submodule. Exemplarily, the MySQL database in the embodiment of the present application synchronizes data to the MySQL database after the core interaction sub-module obtains the target task, obtains the machine state, and generates the regional scheduling policy. The application module can directly acquire data from the MySQL database and can transmit the data to a third-party service or a UI (user interface); for example, the task progress, the machine state and the like can be directly acquired from the UI interface, and the task state can be globally grasped in real time.

Therefore, by arranging the MySQL database (data storage module) and the application module, the module can be used for taking data from the MySQL database, and real-time synchronous display of the data is realized; and the existence of the application module can reduce the consumption of CPU resources, memory resources and network bandwidth resources.

Referring to fig. 4, fig. 4 is a flowchart illustrating a machine control according to an embodiment of the present disclosure; the regional scheduling module of the task scheduling system provided by the embodiment of the application further comprises a machine control sub-module; the method comprises the following steps:

step S400: and the machine control submodule acquires the target message queue and the idle machine from the core interaction submodule.

In the step S400, the node-scaler service of the machine control sub-module obtains the target message queue and the idle machine from the core-service of the core interaction sub-module; it is mentioned that the core-service can obtain the machine status and the task status, including the idle machine.

Step S401: and judging whether the number of the target machines corresponding to the target message queue is less than the number of the idle machines or not.

Step S402: and if the number of the target machines corresponding to the target message queue is less than the number of the idle machines, closing the idle machines with the specified number.

In the above steps S401 to S402, when the rendering task is executed, when the number of target machines required in the generated target message queue is less than the number of free machines in the region, the node-scaler service turns off a specified number of free machines, and only leaves the corresponding number of free machines to execute the target message queue. Wherein the specified number of idle machines is the difference of the number of idle machines minus the number of target machines.

Illustratively, when a plurality of machines are idle, the interface of the node-scaler service calling core-service actively takes off excessive idle machines, namely, the idle machines are not scheduled tasks; the cloud manufacturer API can be enabled to close the machine, so that waste of machine resources is avoided. On the other hand, when a large number of tasks are waiting to be executed, the cloud manufacturer API is called to start the closed machines, so that more machines execute rendering tasks, and the consumption of queues is accelerated.

As can be seen from fig. 4, the machines in the system can be reasonably managed and controlled by the setting of the machine management submodule, so that a reasonable number of machines are online, and if there are too many idle machines, the machine management submodule closes the redundant idle machines, so as to achieve reasonable utilization of resources.

Referring to fig. 5, fig. 5 is a flowchart of node scheduling provided in the embodiment of the present application; the task scheduling system provided by the embodiment of the application also comprises a node scheduling module with a node scheduling submodule and a task execution submodule; the method comprises the following steps:

step S500: and the node scheduling submodule judges whether the data can be pulled from the core interaction submodule or not.

In the step S500, the scheduler service of the node scheduling sub-module determines whether the scheduler service can pull data from the core-service of the core interaction sub-module, mainly a regional scheduling policy.

Step S501: and if the data can be pulled from the core interaction submodule, acquiring a regional task scheduling strategy from the core interaction submodule.

In the step S501, if the scheduler service of the node scheduling submodule can pull data from the core-service of the core interaction submodule, an area scheduling policy is obtained from the core-service, and it can be clearly known based on the foregoing description that the area scheduling policy includes an available area corresponding to the target task.

Step S502: and the node scheduling submodule issues a scheduling instruction to the target machine according to the regional scheduling strategy.

In the step S502, the scheduler service of the node scheduling sub-module issues a scheduling instruction to the target machine according to the regional scheduling policy and according to the regional scheduling policy, so that the target machine is scheduled.

Step S503: and the task execution sub-module executes the target task according to the scheduling instruction.

In step S503, the task execution sub-module causes the machine to execute the target task according to the scheduling command.

Illustratively, a scheduler service of the node scheduling submodule acquires all machine information in the local available area in the area scheduling policy from the core-service; selecting machines in the area with the normal machine state, the empty node lock state and the idle machine number larger than the target machine number; the IP of the selected idle machines is transmitted back to the core-service to indicate that the scheduling task is completed. Further, after the core-service receives the IP of the selected idle machine, the node locks of the machines are locked, and the value of the node lock is set as the task ID. And the task execution submodule issues a rendering task to the machine according to the task scheduling instruction.

As can be seen from fig. 5, the scheduler service in the node scheduling sub-module obtains the region scheduling policy from the core-service, and further schedules the target machine, so as to execute the rendering task; the task scheduling system provided by the embodiment of the application overcomes the defect of centralized scheduling by a two-level scheduling mode, the regional scheduling submodule is responsible for scheduling tasks at a regional level, and the node scheduling submodule is responsible for scheduling tasks at a node level; the two-stage scheduling mode can reduce the average turnover time of the operation and improve the throughput capacity of the system.

In an optional embodiment, if the data cannot be pulled from the core interaction submodule, the node scheduling submodule acquires the target message queue from the message generation submodule. And acquiring a target machine, and generating a regional scheduling strategy according to the target message queue and the target machine. Illustratively, if the node scheduler finds that the offline of the regional scheduler exceeds the preset time through the service registration center, the node scheduling sub-module acquires the target message queue from the message generation sub-module. And acquiring a target machine, and generating a regional scheduling strategy according to the target message queue and the target machine. It should be noted that, in the embodiment of the present application, the preset time may be 1 minute, and in practical applications, the preset time may be set according to actual requirements, and the preset time provided in the embodiment of the present application is 1 minute, which cannot be a limitation of the preset time.

Illustratively, if the scheduler service in the node scheduling sub-module cannot obtain the area scheduling policy from the core-service, the node scheduling sub-module may obtain the area scheduling policy from the core-service. Then the scheduler service in the node scheduling submodule acquires a target message queue from the api-service of the message generation submodule; exemplarily, if the scheduler service in the node scheduling sub-module is offline for more than 1 minute from the service registry discovery zone scheduler; the scheduler service in the node scheduling submodule acquires the message in the kafka and generates a target queue in the memory. Further, generating a regional scheduling strategy according to the target message queue and the target machine; that is, when the area scheduling submodule fails, the node scheduling submodule can also independently complete the scheduling of the task. If the node scheduler finds from the service registry that the regional scheduler has recovered, the logic of regional scheduling is no longer performed directly, and responsibility for regional scheduling is performed by the regional scheduler that has just recovered.

Therefore, the node scheduling submodule can directly take the tasks from the message queue for scheduling under the condition that the regional scheduling submodule completely fails, and system paralysis and machine resource loss are avoided.

In an optional embodiment, after the task execution sub-module executes the target task according to the scheduling instruction, the task scheduling method further includes: the task execution submodule acquires the machine state and sends the machine state to the node scheduling submodule; and the node scheduling submodule transmits the machine state to the core interaction submodule.

Therefore, after the target task is executed, the task execution submodule can acquire the state of the machine again; sending the machine state to a node scheduling submodule; further, the node scheduling submodule returns the machine state to a core interaction submodule of the area scheduling module. Therefore, the core scheduling submodule updates the machine state, and the core scheduling submodule can be used as basic information of subsequent task scheduling to ensure that a task scheduling system operates efficiently.

Referring to fig. 6, fig. 6 is a schematic diagram of task scheduling according to an embodiment of the present application. The task scheduling method of the present application is described in a module-by-module manner. Before introduction, it should be noted that the same registry instance and configuration center instance are used among the modules provided in the embodiments of the present application, and the same registry and configuration center are shared, which is convenient for an operation and maintenance engineer to manage.

The application module mainly comprises a UI service and a third-party service. UI services may interface directly with the OSS system; the scene file can be directly uploaded to an OSS system and downloaded when a task is executed; the waste of CPU resources, memory resources and network bandwidth resources can be reduced to the maximum extent.

The region scheduling module mainly realizes region-level scheduling. The system mainly comprises a region scheduling submodule, a message generating submodule, a data storage submodule, a machine management submodule and a core interaction submodule. The rendering service is pushed to the core interaction submodule in a log mode, the core interaction submodule places the rendering service in a database to the data storage submodule and informs the message generation submodule of the time for adding the log; further, the message generation submodule reads the log from the data storage submodule and forwards the log to the application module. The data of the rendering task is persisted in a data storage module and then is queried (mainly querying the rendering task and the finished historical task), the task is pushed to a message queue of a message generation submodule, and the message queue can be a kafka queue; and the regional scheduling submodule takes the message queue and finally generates a regional scheduling strategy. Meanwhile, when more idle machines than the target machines needed by the target task exist in the system, the machine management submodule closes redundant idle machines; and when a large amount of accumulation of the task queue occurs in the system, the closed machine is restarted.

The node scheduling module mainly realizes task scheduling at a node level. The node scheduling module comprises a node scheduling submodule and a task execution submodule; the node scheduling sub-module firstly acquires a regional scheduling strategy from the regional scheduling sub-module, and the acquired task execution sub-module executes the target task according to the task scheduling instruction, so that the rendering task is successfully issued. If the rendering task fails to be issued, the node scheduling submodule directly pulls the task from the message queue, and the kafka queue is prevented from accumulating the task due to the fault factor of the region scheduling submodule.

The task execution submodule is used for executing the rendering task and integrates various rendering engines and OSS storage to externally provide rendering capability. The method comprises the steps that a scheduler service of a node scheduling module issues a task, a rendering engine is called to render after the rendering service acquires the task, an output file is transmitted to an OSS after rendering is completed, an important milestone log from the rendering engine is written back to a core-service in real time in the whole rendering process, the final task state is returned to the core-service after the task is completed, an interface of the core-service for releasing a node lock is called to release the lock, and therefore a regional scheduling submodule can know that a machine is in an idle state. Meanwhile, the rendering service periodically sends the heartbeat (the local CPU memory usage rate, the number of CPU cores, the memory capacity, etc.) to the core-service.

As can be seen from fig. 6, the task scheduling method provided in the embodiment of the present application implements zone-level scheduling by a two-level scheduling manner, that is, the zone scheduling module implements zone-level scheduling, and the node scheduling module implements node-level scheduling; when the downtime of the regional scheduling module cannot realize regional-level scheduling, the node scheduling module senses through the registration center, and automatically acquires a task from the message queue for scheduling, at the moment, the system still operates normally, and the utilization rate of machine resources is not lost.

Referring to fig. 7, fig. 7 is a schematic block diagram of a task scheduling system according to an embodiment of the present invention; the task scheduling system 100 includes: a region scheduling module 110 having a core interaction sub-module 111, a message generation sub-module 112, and a region scheduling sub-module 113.

The core interaction submodule 111 is used for acquiring a target task;

the message generation sub-module 112 is used for acquiring the target task from the core interaction sub-module 111 and generating a message queue;

the region scheduling submodule 113 is configured to obtain a target message queue from the message queue;

the region scheduling submodule 113 is further configured to obtain a target machine from the core interaction submodule 111; wherein the target machine is used for executing a target task; the target task includes a rendering task.

And the area scheduling submodule 113 is further configured to generate an area scheduling policy according to the target message queue and the target machine.

In an optional embodiment, the obtaining, by the regional scheduling sub-module 113, the target message queue from the message queue includes: the region scheduling submodule 113 obtains the ID of the message queue; acquiring the priority of the message queue according to the ID; the regional scheduling submodule 113 generates a target message queue according to the priority.

In an optional embodiment, the obtaining, by the area scheduling submodule 113, the target machine from the core interaction submodule 111 includes: the regional scheduling submodule 113 acquires the machine state from the core interaction submodule 111; wherein the machine state comprises a state of a node lock; the node lock is a universal unique identification code of the machine. The area scheduling submodule 113 determines whether the node lock state of the machine state is empty; if the node lock status is not null, the area scheduling sub-module 113 sets the device corresponding to the node lock as the target device.

In an optional embodiment, the task scheduling system 100 further includes an application module 120, and the region scheduling module 110 further includes a data storage submodule 114; after the regional scheduling sub-module 113 generates the regional scheduling policy according to the target message queue, the machine state, and the state of the node lock, the method further includes: the regional scheduling submodule 113 sends the regional scheduling strategy to the core interaction submodule 111; the core interaction submodule 111 sends the region scheduling policy to the data storage submodule 114. The target task and regional scheduling policy are retrieved from the data storage sub-module 114 by the application module 120.

In an optional embodiment, the region scheduling module 110 of the task scheduling system 100 further includes a machine control sub-module 150; the machine control sub-module 150 obtains the target message queue and the idle machine from the core interaction sub-module 111. The machine control sub-module 150 determines whether the number of target machines corresponding to the target message queue is less than the number of idle machines; if the number of the target machines corresponding to the target message queue is smaller than the number of the idle machines, the machine control submodule 150 closes the idle machines with the designated number; wherein the designated number is a difference between the number of idle machines and the number of target machines.

In an optional embodiment, the task scheduling system 100 further includes a node scheduling module 130 having a node scheduling submodule 131 and a task execution submodule 132; the node scheduling submodule 131 judges whether data can be pulled from the core interaction submodule; if the data can be pulled from the core interaction submodule, the node scheduling submodule 131 acquires the area scheduling policy from the core interaction submodule. And the node scheduling submodule 131 issues a scheduling instruction to the target machine according to the regional scheduling policy. The task execution sub-module 132 executes the target task according to the scheduling instruction.

In an optional embodiment, if the node scheduling sub-module 131 cannot pull data from the core interaction sub-module 111, the node scheduling sub-module 131 obtains a target message queue from the message generating sub-module 112 to obtain a target machine, and the node scheduling sub-module 131 generates a regional scheduling policy according to the target message queue and the target machine.

In an optional embodiment, after the task execution submodule 132 executes the target task according to the scheduling instruction, the task execution submodule 132 acquires the machine state and sends the machine state to the node scheduling submodule 131; the node scheduling submodule 131 sends the machine state to the core interaction submodule 111.

Please refer to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. An embodiment of the present application provides an electronic device 300, including: a processor 301 and a memory 302, the memory 302 storing machine readable instructions executable by the processor 301, the machine readable instructions when executed by the processor 301 performing the method as above.

Based on the same inventive concept, embodiments of the present application further provide a computer-readable storage medium, where computer program instructions are stored, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the steps in any of the above implementation manners.

The computer-readable storage medium may be a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and other various media capable of storing program codes. The method executed by the electronic terminal defined by the process disclosed by any embodiment of the invention can be applied to the processor or realized by the processor.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some communication interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical, mechanical or other form.

In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

Alternatively, all or part of the implementation may be in software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.).

In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A task scheduling method is characterized in that the method is applied to a task scheduling system; the task scheduling system comprises a region scheduling module which is provided with a core interaction submodule, a message generation submodule, a region scheduling submodule and a data storage submodule; the method comprises the following steps:

acquiring a target task by the core interaction submodule and storing the target task to the data storage submodule;

the message generation submodule acquires the target task from the data storage submodule and generates a message queue;

the regional scheduling submodule acquires a target message queue from the message queue;

the regional scheduling submodule acquires a target machine from the core interaction submodule; wherein the target machine is configured to execute the target task, the target task including a rendering task;

and generating a regional scheduling strategy by the regional scheduling submodule according to the target message queue and the target machine.

2. The method of claim 1, wherein said obtaining, by the regional scheduling submodule, a target message queue from the message queues comprises:

acquiring the ID of the message queue;

acquiring the priority of the message queue according to the ID;

and generating the target message queue according to the priority.

3. The method of claim 1, wherein the obtaining, by the regional dispatch submodule, a target machine from the core interaction submodule comprises:

the regional scheduling submodule acquires a machine state from the core interaction submodule; the machine state comprises a node IP, a running state of a node and a state of a node lock; the node IP is a universal unique identification code of the machine;

judging whether the state of the node lock of the machine state is empty or not;

and if the node running state is normal and the node lock state is empty, taking the machine corresponding to the node lock as the target machine.

4. The method of claim 1, wherein the task scheduling system further comprises an application module; the region scheduling module also comprises a data storage submodule;

after the generating, by the regional scheduling sub-module, a regional scheduling policy according to the target message queue and the target machine, the method further includes:

the regional scheduling sub-module sends the regional scheduling strategy to the core interaction sub-module;

the core interaction submodule sends the region scheduling strategy to the data storage submodule;

and the application module acquires the target task and the regional scheduling strategy from the data storage submodule.

5. The method of claim 1, wherein the regional scheduling module of the task scheduling system further comprises a machine control sub-module; the method further comprises the following steps:

the machine control sub-module acquires the target message queue and the idle machine from the core interaction sub-module;

judging whether the number of target machines corresponding to the target message queue is smaller than the number of idle machines or not;

if the number of the target machines corresponding to the target message queue is smaller than the number of the idle machines, closing the idle machines with the specified number; wherein the specified number is a difference between the number of idle machines and the target number of machines.

6. The method of claim 1, wherein the task scheduling system further comprises a node scheduling module having a node scheduling submodule and a task execution submodule; the method further comprises the following steps:

the node scheduling submodule judges whether data can be pulled from the core interaction submodule or not;

if the data can be pulled from the core interaction submodule, acquiring a regional scheduling strategy from the core interaction submodule;

the node scheduling submodule issues a scheduling instruction to the target machine according to the regional scheduling strategy;

and executing the target task by the task execution submodule according to the scheduling instruction.

7. The method of claim 6, further comprising:

if the data can not be pulled from the core interaction submodule, the node scheduling submodule acquires the target message queue from the message generation submodule;

and acquiring the target machine, and generating a regional scheduling strategy according to the target message queue and the target machine.

8. The method of claim 6, wherein after said executing, by said task execution submodule, said target task in accordance with said scheduling instruction, said method further comprises:

the task execution submodule acquires a machine state and sends the machine state to the node scheduling submodule;

and the node scheduling submodule transmits the machine state to the core interaction submodule.

9. A task scheduling system is characterized by being applied to the task scheduling system; the task scheduling system comprises a region scheduling module with a core interaction submodule, a message generation submodule and a region scheduling submodule;

the core interaction submodule is used for acquiring a target task;

the message generation submodule is used for acquiring the target task from the core interaction submodule and generating a message queue;

the region scheduling submodule is used for acquiring a target message queue from the message queue;

the region scheduling submodule is also used for acquiring a target machine from the core interaction submodule; wherein the target machine is to perform the target task; the target task comprises a rendering task;

and the region scheduling submodule is also used for generating a region scheduling strategy according to the target message queue and the target machine.

10. A computer-readable storage medium having computer program instructions stored thereon for execution by a processor to perform the steps of the method of any one of claims 1-8.