CN115658248A

CN115658248A - Task scheduling method and device, electronic equipment and storage medium

Info

Publication number: CN115658248A
Application number: CN202211107236.6A
Authority: CN
Inventors: 刘桐仁
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-09-09
Filing date: 2022-09-09
Publication date: 2023-01-31

Abstract

The disclosure provides a task scheduling method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, in particular to the fields of intelligent search, webpage search and information retrieval. The specific implementation scheme is as follows: in response to receiving a selection instruction, selecting a target template from a plurality of predetermined templates according to the selection instruction; in response to receiving the task information, creating a target task according to the task information and the target template; and scheduling the target task.

Description

Task scheduling method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to the fields of intelligent search, web search, and information retrieval, and more particularly, to a method and an apparatus for task scheduling, an electronic device, a storage medium, and a computer program product.

Background

At present, a series of architecture systems such as a search system, a recommendation system and the like can be provided with a timing task, and the timing task is used for undertaking the work of data access, data processing, model training, monitoring and maintenance and the like.

Disclosure of Invention

The disclosure provides a task scheduling method, a task scheduling device, an electronic device, a storage medium and a computer program product.

According to an aspect of the present disclosure, there is provided a task scheduling method, including: in response to receiving a selection instruction, selecting a target template from a plurality of predetermined templates according to the selection instruction; in response to receiving the task information, creating a target task according to the task information and the target template; and scheduling the target task.

According to another aspect of the present disclosure, a task scheduling apparatus is provided, which includes a selection module, a creation module, and a scheduling module. The selection module is used for responding to the received selection instruction and selecting a target template from a plurality of preset templates according to the selection instruction; the creating module is used for creating a target task according to the task information and the target template in response to the received task information; the scheduling module is used for scheduling the target task.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods provided by the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method provided by the present disclosure.

According to another aspect of the disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method provided by the disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of an application scenario of a task scheduling method and apparatus according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart diagram of a task scheduling method according to an embodiment of the present disclosure;

FIG. 3 is a schematic flow chart diagram of a task scheduling method in accordance with another embodiment of the present disclosure;

FIG. 4 is a schematic flow chart diagram of a task scheduling method in accordance with another embodiment of the present disclosure;

FIG. 5 is a schematic system architecture diagram of a task scheduling method according to an embodiment of the present disclosure;

fig. 6 is a schematic block diagram of a task scheduling apparatus according to an embodiment of the present disclosure; and

fig. 7 is a block diagram of an electronic device for implementing a task scheduling method according to an embodiment of the disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic application scenario diagram of a task scheduling method and apparatus according to an embodiment of the present disclosure.

It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, a system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104, and a server 105. Network 104 is the medium used to provide communication links between

terminal devices

101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server can analyze and process the received data such as the user request and feed back the processing result to the terminal equipment.

It should be noted that the task scheduling method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the task scheduling device provided by the embodiment of the present disclosure may be generally disposed in the server 105. The task scheduling method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Correspondingly, the task scheduling apparatus provided in the embodiment of the present disclosure may also be disposed in a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 2 is a schematic flow chart diagram of a task scheduling method according to an embodiment of the present disclosure.

As shown in fig. 2, the task scheduling method 200 may include operations S210 to S230.

In operation S210, in response to receiving a selection instruction, a target template is selected from a plurality of predetermined templates according to the selection instruction.

For example, a plurality of predefined templates may be created in advance according to actual requirements, and each predefined template may correspond to a template identifier.

For example, the business platform tier can provide a human-machine interaction interface, and the business platform tier can provide a plurality of predefined templates. A user can perform selection operation on a human-computer interaction interface according to actual requirements, the selection operation can trigger a selection instruction, and a target template can be selected according to a template identifier included in the selection instruction.

In operation S220, in response to receiving the task information, a target task is created according to the task information and the target template.

For example, the user may perform a configuration operation at the service platform layer according to actual requirements to configure information related to the task, such as parameters and codes of the task, where the information may indicate task content, an execution period, a time when execution starts, and the like of the task. The configuration operation triggers a configuration instruction, and task information configured by a user can be acquired from the configuration instruction.

For example, a task template and task information may be assembled into a template task. The task may be, for example, data export, data import, issuing a coupon for a specific commodity on a certain e-commerce platform, and the like, and the task may be, for example, a timing task, and the content of the task is not limited in the embodiment of the present disclosure.

Taking the example that the target task is to export data, the task may include multiple steps, and the steps of connecting a database, searching for data, performing protocol conversion, performing data verification, and the like may be encapsulated as a task template, and other steps required for executing the task may be used as configured task information, or a data source address, a target storage address of exported data, and the like may be used as configured task information, so that the task template and the task information are assembled to obtain the target task.

In operation S230, a target task is scheduled.

For example, after the target task is created, the task may be executed at a predetermined time and a predetermined number of times, thereby achieving scheduling of the task.

In the related art, a plurality of tasks can be developed on a plurality of machines, and as the timed task scripts are more and more, the exposed problem becomes more and more obvious. On the other hand, a plurality of scripts are distributed over a plurality of devices, and omission is likely to occur at the time of service delivery. On the other hand, the stability requirement of the task is difficult to be ensured, and especially when a single machine fails, the tasks running on the failed machine are all abnormal. Therefore, the tasks on the multiple machines can be migrated to the business task system, and the multiple tasks are managed by the business task system. However, in the process of actually performing task migration, a plurality of tasks need to be developed separately, which results in higher cost for migrating the tasks to the business task system.

According to the technical scheme provided by the embodiment of the disclosure, in the process of migrating the tasks on other machines to the business task system, the tasks can be migrated by using the preset template and the configuration information, and the preset template can be reused, so that the timed tasks on other machines can be more simply migrated to the business task system, the resource cost and the labor cost of task migration are reduced, and the task migration efficiency is improved.

In practical application, the method and the device are suitable for timing tasks such as data access migration, data processing and training aiming at large-scale complex scenes such as searching and recommending scenes.

It should be noted that, in an actual application, the predetermined template and the configuration task information may be created according to actual requirements, and the predetermined template and the task information are not limited in the embodiment of the present disclosure.

In one example, the target template includes a task template that includes code related to a task. The task template is suitable for creating common tasks, the execution flow of the tasks is fixed and generalized, and the parameters of the tasks can be customized according to the actual requirements of the tasks. Thus, a task template can be created from code related to the task, and when the task template is used, code for the task does not need to be written.

In practical application, a designer can design a general operator based on a plurality of similar tasks in a unified abstract mode, the operator can be packaged into a task template, the operator can be loaded by a general framework in a unified mode and is loaded and executed by an executor in a unified mode, and therefore high-speed flow tasks can be configured at low cost and can be executed stably.

Accordingly, when the task template includes task code, the task information may include code parameters. It should be understood that the task template may pre-encapsulate some code that performs this type of task, but parameters in the code are in an empty state, and these empty parameters may be configured as task information.

For example, a task is to export a certain type of data from MySQL or other databases and then perform a library process on the exported data. The task can comprise a plurality of steps, which relate to hundreds of lines of codes, and in practical application, if a task template is not used, all codes of the task need to be written, and the writing process is complex. When the task template is used, some steps of the task can be packaged in advance, for example, steps of connecting a database, searching data, then performing protocol conversion, performing data verification and the like are packaged, and the steps subjected to packaging processing do not need to write codes. Therefore, when a task needs to be executed, a user can select a pre-created task template, provide task information such as an SQL statement of MySQL (relational database management system), UGI parameters (user account, password, address), a target storage address to which data needs to be exported, and the like according to the actual requirement of the task, and assemble the code of the task with the task information to create the task. It should be understood that when multiple tasks are multiplexed with the same task template, the task information, and therefore the tasks, of the multiple tasks are different.

In another example, the target template includes a business environment template, which is suitable for a scenario where a business needs to be developed flexibly, for example, a task of a service type with a high business flexibility may be a task of a distributed script type.

Accordingly, when the target template comprises a business environment template, the task information may include code related to the task, i.e., in which case the user may configure the code of the task, thereby supporting business-customized code development.

It should be noted that the business needs are diverse, and various needs need to be mapped to different basic environments, and although the differences of different businesses may be relatively large, the basic environments of multiple tasks included in the same business line are substantially the same and have high similarity. The service environment template can be created according to actual service requirements, the created service environment template can be reused, and scripts of a plurality of target tasks can use the same service environment template. In practical application, the codes of the target tasks and the selected business environment template can be assembled in an overlapping mode.

For example, the target task of the custom code class can be created according to the task code of the service environment template, the architecture execution framework and the configuration, so that the service can be migrated from the original physical machine to the service task system at lower cost.

For example, a service may customize a customized service environment template, the same service environment template may be multiplexed among a plurality of tasks, and the service environment template may include, for example, an environment with a Nginx Redis, a basic environment with a Php environment, an environment with a product search basic component, and the like. The service environment template can be selected according to actual needs, and the basic environment supports self-definition.

The architecture execution framework is a framework provided for services that can be multiplexed by different services without modification. The business can execute target tasks in parallel under the framework.

The task code is a script needing to be executed, can be executed in a certain environment, and supports common script languages such as Python, shell, php and the like in languages. Code that multiple tasks can correspond to respectively; multiple tasks may also correspond to the same code, and the code parameters of different tasks may be different.

FIG. 3 is a schematic flow chart diagram of a task scheduling method according to another embodiment of the present disclosure.

As shown in fig. 3, the task scheduling method 300 may include operations S310 to S320, and may further include operations S331 to S333.

In some embodiments, the method 300 may be performed where the target template includes a task template and the task template includes code related to a task. It should be understood that the method 300 may also be performed in other situations, such as where the target template includes a business environment template and the task information may include code related to the task, for example, performing the method 300.

For example, the operations S210 and S220 may be referred to in the operations S310 and S320, which are not described in detail herein.

In operation S331, in response to detecting that the current time is the trigger time of the target task, the target task is added to the message queue.

In operation S332, a target task is acquired from the message queue using at least one of the plurality of work nodes.

In operation S333, the target task is executed using at least one work node.

This may be accomplished, for example, by a configuration database, task manager, task trigger, message queue, work node.

The configuration database may store information related to the task, such as the content of the task, the execution period of the task, the time the task was last executed, the current running state of the task, which may include triggered, in progress, completed, etc.

The task manager may retrieve data from the configuration database and then determine whether the current time is the trigger time for the target task based on the retrieved data. Upon determining that a trigger is needed at the current time, the task manager may send a trigger instruction to the task trigger.

After receiving the trigger, the task trigger performs initial data loading, and the loading process may include sending the target task to a message queue, which may be a kafka queue. The task trigger may also split the target task into multiple subtasks.

The message queue can receive a target task issued by an upstream task trigger and then distribute the target task to at least one downstream work node. In addition, when the target task is split into multiple subtasks, the message queue may issue the full amount of information for each subtask to the work node.

The working nodes can receive the target tasks issued by the message queues, and the plurality of working nodes can independently process the received target tasks or subtasks, so that the target tasks are executed.

In addition, in practical applications, a downstream work node executes a target task by consuming messages in an upstream message queue, and if the work node fails, the message queue may distribute the target task to other work nodes through a Rebalance (Rebalance) mechanism.

In some embodiments, the method 300 may further include an operation S334 of updating and recording the execution state of the target task in real time.

For example, the target task may be added to the message queue using a task trigger and also registered in the task manager using the task trigger, and the registering may include updating the execution state of the target task to be triggered. For another example, when the work node acquires the target task or a subtask split from the target task, the work node may perform state synchronization with the task manager, so as to update the state information of the target task to be in process. For example, when the work node has completed the target task, the work node may update the state information of the target task to completion by performing state synchronization with the task manager.

For example, the task manager may maintain state information for task triggers and target tasks submitted by the worker nodes. The task manager may also record the state information in a configuration database.

In addition, an observation interface can be provided, and the state information of the target task can be acquired through the observation interface. In practical applications, the cause of the fault can be traced through the state information of the target task, for example, if a certain target task is executed every 5 minutes in an execution period, but is not executed due to timeout, it can be determined through the state information whether the cause of the fault is not successfully triggered, or the fault is caused and stopped in the execution process, and the like.

In some embodiments, the adding the target task to the message queue may include: in response to detecting that the target task meets a predetermined splitting condition, splitting the target task into a plurality of subtasks, and then adding the plurality of subtasks to the message queue. Accordingly, the above-mentioned operation of performing the target task by using at least one work node may include the operations of: a plurality of subtasks are executed in parallel using at least two working nodes.

For example, the predetermined splitting condition includes at least one of: the data quantity of the target task is larger than or equal to a first threshold value, and the calculation quantity of the target task is larger than or equal to a second threshold value.

For example, taking a data export task as an example, for a large batch of data export tasks, data may be partitioned (partition), and data of each partition corresponds to one subtask. The target task can be divided into 10 subtasks, and 5 or other working nodes can execute a plurality of subtasks in parallel at the downstream, so that the processing efficiency of the target task is improved.

FIG. 4 is a schematic flow chart diagram of a task scheduling method according to another embodiment of the present disclosure.

As shown in fig. 4, the task scheduling method 400 may include operations S410 to S420, and may further include operations S431 to S437.

In some embodiments, the method 400 may be performed where the target template includes a business environment template and the task information may include code related to the task. It should be understood that the method 400 may also be performed in other situations, such as where the target template includes a task template and the task template includes code related to a task, for example, performing the method 400.

For example, the operations S210 and S220 may be referred to in the operations S410 and S420, which are not described in detail herein.

In operation S431, it is determined whether the container satisfies a predetermined pre-scheduling condition according to a scheduling interval duration of the target task and an expected duration required for a scheduling process for the container. In case it is determined that the container satisfies the predetermined pre-scheduling condition, operation S432 may be performed. In case it is determined that the container does not satisfy the predetermined pre-scheduling condition, operation S435 may be performed.

For example, performing the scheduling process on the container may indicate starting the container, initializing the container, or the like.

For example, the configuration server may store information about the tasks for the scheduler to read and trigger scheduling. The information stored by the configuration server may include information configured by the user at the service platform layer, and may also include information calculated according to actual situations of the task, for example, the information may include information such as an execution command of the task, an execution period of the task, a current execution state of the task, an actual time required for container scheduling, a time when the container is scheduled next time, and a time when the target task is executed next time.

For example, the container may be a docker container.

For example, the expected duration may be determined by: at least one actual time length required by the scheduling container is obtained from historical data of a preset time period, and then the expected time length is determined according to the at least one actual time length and a preset quantile value.

For example, the predetermined period of time may be the past 1 day, 7 days, 30 days, etc., the container has been scheduled multiple times in the past predetermined period of time, and each time it is scheduled, the actual length of time required to schedule the container may be recorded. The predetermined quantile value may be a 75% quantile value, a 90% quantile value, etc.

Taking the case where the container was scheduled 100 times in the last 7 days, and the actual duration for the 90% quantile value is 5 minutes, this can be expressed as: in the last 7 days, the actual time length required for scheduling containers 90 times is less than or equal to 5 minutes, and it can be seen that the probability that the time length required for scheduling containers is less than or equal to 5 minutes is 90%. Thus, by predetermining the quantile value, it can be ensured that the container completes scheduling within an expected duration with a high probability.

For example, the scheduler may read its stored information from the configuration server, then compare the expected duration of time required to schedule the container with the execution interval of the task, and determine whether pre-scheduling is required based on the comparison.

For example, prescheduling may mean starting the container before the target task is executed, so that the target task may be executed directly after the container is started without consuming time by waiting for the container to be started.

For example, the predetermined pre-scheduling conditions include: the duration of the scheduling interval is greater than the product of the expected duration and a predetermined factor, the predetermined factor being greater than or equal to 1, preferably greater than or equal to 1.5.

Taking the expected time period required for the scheduling process for the container as 10 minutes and the predetermined coefficient equal to 1 as an example, if the scheduling interval time period is 1 hour, then pre-scheduling may be performed. If the scheduling interval duration is 1 minute, no pre-scheduling may be performed.

Taking the expected duration as 10 minutes and the scheduling interval duration as 13 minutes as an example, if the predetermined coefficient is equal to 1, pre-scheduling can be performed; if the predetermined factor is equal to 1.5, no pre-scheduling may be performed. It can be seen that when the predetermined factor is adjusted to be greater than 1, for example greater than 1.5 or even greater than 2, the additional consumption of resources by frequent activation of the container can be avoided.

In operation S432, a container is pre-scheduled before a target execution time according to the expected duration and the target execution time of the target task.

For example, the container may be prescheduled at a predetermined time, the time difference between the predetermined time and the target execution time of the target task being equal to the expected duration. By adopting the method, some time is needed for scheduling the container, and through the pre-scheduling method, after the container is scheduled, the container can directly execute the target task without waiting for the completion of the container scheduling when the target task needs to be executed, so that the execution efficiency of the target task is improved. In addition, the containers are scheduled at the preset time, so that the target task can be prevented from being executed after the containers are started for a long time, and resource waste caused by the fact that the containers are in a starting state and do not execute the tasks is avoided.

For example, a scheduler may be used to generate scheduling instructions by which to implement prescheduling of containers.

In operation S433, the target task is executed using the prescheduled container.

For example, a PaaS (Platform as a Service) management layer may be utilized to receive a pre-scheduling instruction of a scheduler, generate a container according to the pre-scheduling instruction, initialize a Service environment, run a timed task framework, and pull up a Service code by the framework, thereby executing a target task. In practical applications, the PaaS management layer may use K8S. The container provides the characteristics of resource isolation, space isolation independent deployment and the like, and the same container can simultaneously execute a plurality of target tasks.

In operation S434, in case it is determined that the container performs the completion target task, the container is released.

For example, after the container completes the target task, it needs to wait for a period of time before executing the next target task again, and the waiting time is longer than the expected time required for scheduling the container because the predetermined pre-scheduling condition is satisfied. Therefore, the container can be released, so that the container is prevented from being in a starting state and additionally consuming resources without executing the target task, and the effect of saving the resources is achieved.

In operation S435, a container is scheduled according to a target execution time of a target task.

For example, the container may be scheduled at a target execution time of a target task. For another example, referring to operation S432, the container may be pre-scheduled during the first container scheduling process.

In operation S436, the target task is executed using the container subjected to the scheduling process.

In operation S437, in the case where it is determined that the container performs the completion target task, the container of the scheduled process is reserved.

For example, the expected duration of time required for the container to perform the scheduling process is 10 minutes, the predetermined factor is equal to 1 for example, and the scheduling interval duration is 1 minute. It can be seen that, at this time, if the container is released after the container is scheduled, when the target task is executed next time, it is necessary to wait for the container to complete scheduling, which results in a timeout of the target task. Therefore, the container subjected to the scheduling processing can be retained, so that the execution efficiency of the target task can be improved, and the stability can also be improved.

In some embodiments, the method may further include the operations of: and migrating the abnormal container.

For example, a new container is scheduled in response to detecting that the container is in a failed state, and the execution state of the target task is obtained, and then the target task continues to be executed based on the execution state using the new container.

For example, the container detects that the container has a failure through its own operation logic, and may report the failure to the scheduler, and the scheduler may release the failed container and schedule a new container, and continue to execute the target task executed by the failed container using the new container.

In addition, in practical applications, the execution state of the target task may be synchronized by using the remote distributed file system, for example, during the execution of the target task, the execution state of the target task is updated in the local file in real time, and simultaneously, data synchronization is performed between the local file and the remote distributed file system. When a container fails, the target task may continue to run based on its execution state through the synchronized data.

In the related art, a single machine and single instance mode can be adopted to execute a target task, when a machine is down, the target task can be continuously executed only after the machine needs to be restarted, and the problem of low stability exists. Compared with the related art, the embodiment of the disclosure can migrate the target task to other containers for execution after the container fails, thereby improving the stability.

Fig. 5 is a schematic system architecture diagram of a task scheduling method according to an embodiment of the present disclosure.

As shown in FIG. 5, the business task system includes a business platform layer 510, a business logic layer 520, and a base services layer 530.

The business platform layer 510 can provide a human-machine interaction interface, and the business platform layer 510 can provide a plurality of predefined templates, such as a plurality of task templates and a plurality of business environment templates. The user can select the required task template or business environment template at the business platform layer 510 according to the actual requirement. The user may also configure task information, such as configuration code parameters or task code, at the business platform layer 510.

The business logic layer 520 may include generic template class tasks and custom code class tasks. For the service task with fixed flow, the processing flow of the universal template task can be adopted. For other tasks which cannot be covered by the general template tasks, the processing flow of the custom code tasks can be used.

The base services layer 530 is used to provide base services for the target task. For example, for data import, data export tasks, the base services layer 530 may include a variety of databases, which may include Mongo (a distributed file storage based database), HDFS (Hadoop distributed file system), mySQL, ES (ElasticSearch), and so on.

In practical application, when a user selects a task template at a task platform layer and configures code parameters, the target task belongs to a general template task. The task template and code parameters may be assembled into a target task. Processing generic template class tasks may be implemented using a configuration database, task manager, task triggers, message queues, work nodes.

For example, a configuration database may store information related to tasks, and a task manager may retrieve data from the configuration database and then determine whether the current time is the trigger time of the target task based on the retrieved data. Upon determining that a trigger is needed at the current time, the task manager may send a trigger instruction to the task trigger. And the task trigger sends the target task to the message queue, and the working node acquires the target task from the message queue and executes the target task.

In practical application, when a user selects a business environment template at a task platform layer and configures codes related to tasks, the target tasks belong to custom code tasks. The business environment templates, the architectural execution framework, and the configured task code may be assembled into a target task. The task of processing the custom code class can be realized by utilizing a configuration server, a scheduler, a PaaS management layer and a container.

For example, the configuration server may store information about the task for the scheduler to read and trigger scheduling. The trigger reads information from the configuration server and compares the scheduling interval duration of the target task with the expected duration required to schedule the container to determine whether the container meets the predetermined pre-scheduling condition.

In the case that it is determined that the predetermined pre-scheduling condition is satisfied, the scheduler pre-schedules the container at a predetermined time, for example, the scheduler may send a pre-scheduling instruction to the PaaS management layer, the PaaS management layer generates the container according to the pre-scheduling instruction, and the container executes the target task. And releasing the container after the container performs the target task. The next time the target task is executed, the container is pre-scheduled again.

In the case that it is determined that the predetermined pre-scheduling condition is not satisfied, the scheduler schedules the container, for example, the scheduler may send a pre-scheduling instruction to the PaaS management layer, the PaaS management layer generates the container according to the pre-scheduling instruction, and the container executes the target task. And reserving the container scheduled for processing after the container executes the target task.

By adopting the service task system, efficient creation and efficient migration of services can be ensured. In practical application, aiming at a service cloud native timing task system under a scene of a service complex task, the task can be divided into a general template task and a custom code task according to different service use scenes. In the migration process, the universal template tasks can reuse the task templates, and the custom code tasks can reuse the service environment templates, so that the multiplexing and custom requirements of different levels are realized, and the effect of batch migration of low service cost is realized.

In practical application, the service task system can realize the user-defined code tasks of migrating hundreds of original physical machines at a single service day level. In addition, the average access period of the universal template tasks is less than 1 hour, and the stability and the access efficiency of the service tasks are improved.

Fig. 6 is a schematic block diagram of a task scheduling device according to an embodiment of the present disclosure.

As shown in FIG. 6, the task scheduler 600 may include a selection module 610, a creation module 620, and a scheduling module 630.

The selection module 610 is configured to select a target template from a plurality of predetermined templates in response to receiving a selection instruction according to the selection instruction.

The creation module 620 is configured to, in response to receiving the task information, create a target task according to the task information and the target template.

The scheduling module 630 is used for scheduling the target task.

According to another embodiment of the present disclosure, the scheduling module includes a first determining submodule, a first scheduling submodule, a first executing submodule, and a releasing submodule. The first determining submodule is used for determining whether the container meets a preset pre-scheduling condition according to the scheduling interval duration of the target task and the expected duration required by scheduling the container; the first scheduling submodule is used for performing pre-scheduling processing on the container before the target execution time according to the expected time length and the target execution time of the target task under the condition that the container is determined to meet the preset pre-scheduling condition; the first execution submodule is used for executing the target task by utilizing the container subjected to the prescheduling processing; and the releasing submodule is used for releasing the container under the condition that the container is determined to execute the completed target task.

According to another embodiment of the present disclosure, the predetermined pre-scheduling condition includes: the scheduling interval duration is greater than the product of the expected duration and a predetermined coefficient, the predetermined coefficient being greater than or equal to 1.

According to another embodiment of the present disclosure, the sub-module according to the first scheduling comprises a pre-scheduling unit for pre-scheduling the container at a predetermined time instant; wherein the time difference between the predetermined time and the target execution time of the target task is equal to the expected time.

According to another embodiment of the present disclosure, the apparatus further includes a second scheduling submodule, a second executing submodule, and a reserving submodule. The second scheduling submodule is used for scheduling the container according to the target execution time of the target task under the condition that the container is determined not to meet the preset pre-scheduling condition; the second execution submodule is used for executing the target task by using the container subjected to the scheduling processing; and the reservation submodule is used for reserving the containers subjected to the scheduling processing under the condition that the containers are determined to execute the completed target tasks.

According to another embodiment of the present disclosure, the apparatus further includes a first obtaining sub-module and a second determining sub-module. The first obtaining submodule is used for obtaining at least one actual time length required by the scheduling container from historical data of a preset time period; the second determining submodule is used for determining the expected duration according to the at least one actual duration and the preset quantile value.

According to another embodiment of the present disclosure, the apparatus further includes a third scheduling submodule, a second obtaining submodule, and a third executing submodule. The third scheduling submodule is used for responding to the detection that the container is in a fault state and scheduling a new container; the second obtaining submodule is used for obtaining the execution state of the target task; and the third execution submodule is used for utilizing the new container to continuously execute the target task based on the execution state.

According to another embodiment of the present disclosure, the target template includes a business environment template, and the task information includes code related to the task.

According to another embodiment of the present disclosure, the scheduling module includes an adding submodule, a third obtaining submodule, and a fourth executing submodule. The adding submodule is used for responding to the detected current moment which is the trigger moment of the target task and adding the target task into the message queue; the third obtaining sub-module is used for obtaining the target task from the message queue by using at least one working node in the plurality of working nodes; and the fourth execution submodule is used for executing the target task by utilizing at least one working node.

According to another embodiment of the present disclosure, the apparatus further includes a first status determining sub-module and a second status determining sub-module. The first state determining submodule is used for determining the state information of the target task as triggered in response to the fact that the target task is added to the message queue; the second state determining submodule is used for responding to the fact that at least one working node acquires the target task from the message queue, and determining the state information of the target task to be processed; the third state determining submodule is used for responding to the fact that the target task is executed by at least one working node, and determining the state information of the target task to be completed; the recording submodule is used for recording the state information of the target task.

According to another embodiment of the present disclosure, the adding submodule includes a splitting unit and an adding unit. The splitting unit is used for splitting the target task into a plurality of subtasks in response to detecting that the target task meets a preset splitting condition; the adding unit is used for adding a plurality of subtasks to the message queue; wherein the predetermined splitting condition comprises at least one of: the data volume of the target task is greater than or equal to a first threshold value, and the calculated volume of the target task is greater than or equal to a second threshold value; the fourth execution submodule is further used for executing a plurality of subtasks in parallel by utilizing at least two working nodes.

According to another embodiment of the present disclosure, the target template includes a task template, the task template includes code related to a task, and the task information includes code parameters.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

In the technical scheme of the disclosure, before the personal information of the user is acquired or collected, the authorization or the consent of the user is acquired.

According to an embodiment of the present disclosure, there is also provided an electronic device, comprising at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the task scheduling method described above.

According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the above task scheduling method.

According to an embodiment of the present disclosure, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the above task scheduling method.

FIG. 7 illustrates a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the device 700 comprises a computing unit 701, which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Computing unit 701 may be a variety of general purpose and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 701 performs the respective methods and processes described above, such as the task scheduling method. For example, in some embodiments, the task scheduling method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto device 700 via ROM 702 and/or communications unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the task scheduling method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A task scheduling method comprises the following steps:

in response to receiving a selection instruction, selecting a target template from a plurality of predetermined templates according to the selection instruction;

in response to receiving task information, creating a target task according to the task information and the target template; and

and scheduling the target task.

2. The method of claim 1, wherein the scheduling the target task comprises:

determining whether the container meets a preset pre-scheduling condition according to the scheduling interval duration of the target task and the expected duration required by the scheduling processing of the container;

under the condition that the container is determined to meet the preset pre-scheduling condition, performing pre-scheduling processing on the container before the target execution time according to the expected duration and the target execution time of the target task;

executing the target task by using the container subjected to the prescheduling processing; and

in an instance in which it is determined that a container performs the target task, the container is released.

3. The method of claim 2, wherein the predetermined pre-scheduling condition comprises:

the scheduling interval duration is greater than the product of the expected duration and a predetermined coefficient, and the predetermined coefficient is greater than or equal to 1.

4. The method of claim 2, wherein pre-scheduling a container prior to a target execution time for the target task based on the expected duration and the target execution time comprises:

pre-scheduling the container at a predetermined time;

wherein a time difference between the predetermined time and a target execution time of the target task is equal to the expected duration.

5. The method of claim 2, further comprising:

under the condition that the container is determined not to meet the preset pre-scheduling condition, scheduling the container according to the target execution time of the target task;

executing the target task using the container subjected to the scheduling processing; and

in an instance in which it is determined that the container execution completed the target task, retaining the scheduled processed container.

6. The method of claim 2, further comprising:

acquiring at least one actual time length required for scheduling the container from historical data of a preset time period; and

and determining the expected duration according to the at least one actual duration and a preset quantile value.

7. The method of claim 2, further comprising:

in response to detecting that the container is in a failed state, scheduling a new container;

acquiring the execution state of the target task; and

continuing to execute the target task based on the execution state using the new container.

8. The method of any of claims 1 to 7, wherein the target template comprises a business environment template and the task information comprises code related to a task.

9. The method of claim 1, wherein the scheduling the target task comprises:

in response to detecting that the current time is the trigger time of the target task, adding the target task to a message queue;

acquiring the target task from the message queue by using at least one working node in a plurality of working nodes; and

executing the target task with the at least one worker node.

10. The method of claim 9, further comprising:

in response to detecting that the target task is added to the message queue, determining status information of the target task as triggered;

in response to detecting that the at least one worker node has acquired the target task from the message queue, determining status information of the target task as in-process;

in response to detecting that the at least one worker node has completed executing the target task, determining status information of the target task as completed; and

and recording the state information of the target task.

11. The method of claim 9, wherein adding the target task to a message queue comprises:

splitting the target task into a plurality of subtasks in response to detecting that the target task meets a predetermined splitting condition; and

adding the plurality of subtasks to the message queue;

wherein the predetermined splitting condition comprises at least one of: the data volume of the target task is greater than or equal to a first threshold value, and the calculated volume of the target task is greater than or equal to a second threshold value;

wherein the performing the target task with the at least one worker node comprises:

executing the plurality of subtasks in parallel with at least two worker nodes.

12. The method of any of claims 9 to 11, wherein the target template comprises a task template comprising code related to a task, the task information comprising code parameters.

13. A task scheduling apparatus, comprising:

the selection module is used for responding to the received selection instruction and selecting a target template from a plurality of preset templates according to the selection instruction;

the creating module is used for creating a target task according to the task information and the target template in response to receiving the task information; and

and the scheduling module is used for scheduling the target task.

14. The apparatus of claim 13, wherein the scheduling module comprises:

the first determining submodule is used for determining whether the container meets a preset pre-scheduling condition according to the scheduling interval duration of the target task and the expected duration required by scheduling the container;

the first scheduling submodule is used for performing pre-scheduling processing on the container before the target execution time according to the expected duration and the target execution time of the target task under the condition that the container is determined to meet the preset pre-scheduling condition;

a first execution submodule, configured to execute the target task using the container subjected to the pre-scheduling processing; and

and the releasing submodule is used for releasing the container under the condition that the container is determined to execute the target task.

15. The apparatus of claim 14, wherein the predetermined pre-scheduling condition comprises:

16. The apparatus of claim 14, wherein the first scheduling sub-module comprises:

a pre-scheduling unit for pre-scheduling the container at a predetermined time;

17. The apparatus of claim 14, further comprising:

the second scheduling submodule is used for scheduling the container according to the target execution time of the target task under the condition that the container is determined not to meet the preset pre-scheduling condition;

a second execution submodule, configured to execute the target task by using the container subjected to the scheduling processing; and

and the reservation submodule is used for reserving the containers subjected to scheduling processing under the condition that the containers are determined to execute the target tasks.

18. The apparatus of claim 14, further comprising:

the first obtaining submodule is used for obtaining at least one actual time length required by the scheduling container from historical data of a preset time period; and

and the second determining submodule is used for determining the expected time length according to the at least one actual time length and a preset quantile value.

19. The apparatus of claim 14, further comprising:

a third scheduling sub-module for scheduling a new container in response to detecting that the container is in a failed state;

the second obtaining submodule is used for obtaining the execution state of the target task; and

and the third execution submodule is used for continuously executing the target task based on the execution state by utilizing the new container.

20. The apparatus of any of claims 13 to 19, wherein the target template comprises a business environment template and the task information comprises code related to a task.

21. The apparatus of claim 13, wherein the scheduling module comprises:

the adding submodule is used for adding the target task into a message queue in response to the fact that the current moment is detected to be the trigger moment of the target task;

a third obtaining submodule, configured to obtain the target task from the message queue by using at least one of the plurality of working nodes; and

and the fourth execution submodule is used for executing the target task by utilizing the at least one working node.

22. The apparatus of claim 21, further comprising:

a first state determination submodule, configured to determine, in response to detecting that the target task is added to the message queue, state information of the target task as triggered;

a second state determination submodule, configured to determine, in response to detecting that the at least one work node has acquired the target task from the message queue, state information of the target task as being in processing;

a third state determination submodule, configured to determine, in response to detecting that the target task has been executed by the at least one work node, state information of the target task as completed; and

and the recording submodule is used for recording the state information of the target task.

23. The apparatus of claim 21, wherein adding sub-modules comprises:

the splitting unit is used for splitting the target task into a plurality of subtasks in response to detecting that the target task meets a preset splitting condition; and

an adding unit, configured to add the plurality of subtasks to the message queue;

wherein the fourth execution sub-module is further configured to execute the plurality of subtasks in parallel using at least two work nodes.

24. The apparatus of any of claims 21 to 23, wherein the target template comprises a task template comprising code related to a task, the task information comprising code parameters.

25. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

26. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1 to 12.

27. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.