CN113641484A - Task timing scheduling method, device and equipment and computer readable storage medium - Google Patents
Task timing scheduling method, device and equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113641484A CN113641484A CN202111191917.0A CN202111191917A CN113641484A CN 113641484 A CN113641484 A CN 113641484A CN 202111191917 A CN202111191917 A CN 202111191917A CN 113641484 A CN113641484 A CN 113641484A
- Authority
- CN
- China
- Prior art keywords
- task
- storage space
- time
- migration
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
- G06F9/4887—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues involving deadlines, e.g. rate based, periodic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the disclosure discloses a task timing scheduling method, a device, an electronic device and a computer readable storage medium, wherein the task timing scheduling method comprises the following steps: in response to the detection of the timed scheduling task registration operation, acquiring task scheduling time and task data volume from the timed scheduling task registration information, and storing the task data into a first storage space; calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount; in response to reaching the migration time, the task data is migrated from the first storage space to the second storage space. According to the technical scheme, the data are prepared in advance based on data migration among different storage spaces, so that high throughput and timeliness of task timing batch processing can be effectively guaranteed even when a large number of tasks are processed, such as a large number of tasks to be executed.
Description
Technical Field
The present disclosure relates to the field of task scheduling technologies, and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for task timing scheduling.
Background
With the development of data processing requirements and data processing technologies, batch processing of data tasks is widely applied, wherein batch processing of data tasks refers to simultaneous input and automatic execution of a plurality of data tasks, and execution results are sent to a requester after execution. The timed batch processing of the data tasks is based on the batch processing, so that the tasks are processed at a specified time point, such as timed batch payroll, timed batch log clearing in a log system and the like, wherein certain application scenarios have high requirements on timeliness of the timed batch processing of the data tasks, namely that the batch task processing is expected to be completed at the specified time point on time, such as stock timed trading and the like in a financial scenario, and under the scenario, the execution delay of the tasks may bring immeasurable loss. However, if the number of tasks to be executed is huge, such as massive task data, how to simultaneously ensure high throughput and timeliness of task timing batch processing is a subject of great attention.
Disclosure of Invention
The embodiment of the disclosure provides a task timing scheduling method and device, electronic equipment and a computer readable storage medium.
In a first aspect, an embodiment of the present disclosure provides a task timing scheduling method.
Specifically, the task timing scheduling method includes:
in response to the detection of the timed scheduling task registration operation, acquiring task scheduling time and task data volume from the timed scheduling task registration information, and storing the task data into a first storage space;
calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
in response to reaching the migration time, the task data is migrated from the first storage space to the second storage space.
With reference to the first aspect, in a first implementation manner of the first aspect, the calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount includes:
estimating and obtaining estimated migration time required for migrating the task data from a first storage space to a second storage space based on the task data volume;
and calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time.
With reference to the first aspect and the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the estimating, based on the task data amount, an estimated migration time required for migrating the task data from the first storage space to the second storage space includes:
Acquiring a reference data migration speed;
and dividing the task data volume by the reference data migration speed to obtain the estimated migration time required for migrating the task data from the first storage space to the second storage space.
With reference to the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration duration includes:
and subtracting the estimated migration duration from the task scheduling time to obtain the migration time for migrating the task data from the first storage space to the second storage space.
With reference to the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, and the third implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the estimated migration time is longer than an actual migration time required for migrating the task data from the first storage space to the second storage space.
With reference to the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, and the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect, an embodiment of the present disclosure further includes:
Reading the task data from the second storage space to a scheduling queue;
and executing the task scheduling operation in response to the arrival of the task scheduling time.
With reference to the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, the fourth implementation manner of the first aspect, and the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the migration is a multi-thread or distributed migration, and the reading is a multi-thread or distributed reading.
With reference to the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, the fourth implementation manner of the first aspect, the fifth implementation manner of the first aspect, and the sixth implementation manner of the first aspect, in a seventh implementation manner of the first aspect, the first storage space is an external memory, and the second storage space is an internal memory.
In a second aspect, an embodiment of the present disclosure provides a task timing scheduling apparatus.
Specifically, the task timing scheduling device includes:
The storage module is configured to respond to the detection of the timed scheduling task registration operation, acquire task scheduling time and task data volume from the timed scheduling task registration information and store the task data into a first storage space;
the computing module is configured to compute migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
a migration module configured to migrate the task data from the first storage space to the second storage space in response to reaching a migration time.
With reference to the second aspect, in a first implementation manner of the second aspect, the computing module is configured to:
estimating and obtaining estimated migration time required for migrating the task data from a first storage space to a second storage space based on the task data volume;
and calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time.
With reference to the second aspect and the first implementation manner of the second aspect, in a second implementation manner of the second aspect, the estimating, based on the task data amount, a portion of an estimated migration time period required for migrating the task data from the first storage space to the second storage space is configured to:
Acquiring a reference data migration speed;
and dividing the task data volume by the reference data migration speed to obtain the estimated migration time required for migrating the task data from the first storage space to the second storage space.
With reference to the second aspect, the first implementation manner of the second aspect, and the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the calculating, based on the task scheduling time and the estimated migration time, a portion of a migration time for migrating the task data from the first storage space to the second storage space is configured to:
and subtracting the estimated migration duration from the task scheduling time to obtain the migration time for migrating the task data from the first storage space to the second storage space.
With reference to the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, and the third implementation manner of the second aspect, in a fourth implementation manner of the second aspect, the estimated migration time is longer than an actual migration time required for migrating the task data from the first storage space to the second storage space.
With reference to the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, and the fourth implementation manner of the second aspect, in a fifth implementation manner of the second aspect, the embodiment of the present disclosure further includes:
the reading module is configured to read the task data from the second storage space to a scheduling queue;
an execution module configured to execute the task scheduling operation in response to reaching the task scheduling time.
With reference to the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, the fourth implementation manner of the second aspect, and the fifth implementation manner of the second aspect, in a sixth implementation manner of the second aspect, the migration is a multi-thread or distributed migration, and the reading is a multi-thread or distributed reading.
With reference to the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, the fourth implementation manner of the second aspect, the fifth implementation manner of the second aspect, and the sixth implementation manner of the second aspect, in a seventh implementation manner of the second aspect, the first storage space is an external memory, and the second storage space is an internal memory.
In a third aspect, the disclosed embodiments provide an electronic device, including a memory and at least one processor, where the memory is configured to store one or more computer instructions, where the one or more computer instructions are executed by the at least one processor to implement the method steps of the task timing scheduling method described above.
In a fourth aspect, the disclosed embodiments provide a computer-readable storage medium for storing computer instructions for a task timing scheduling apparatus, which includes computer instructions for executing the task timing scheduling method described above to the task timing scheduling apparatus.
In a fifth aspect, the disclosed embodiments provide a computer program product comprising a computer program/instructions, wherein the computer program/instructions, when executed by a processor, implement the method steps of the task timing scheduling method described above.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
according to the technical scheme, the data to be processed is stored in the first storage space, and then the data to be processed is migrated to the second storage space before the task execution time comes, so that data preparation is made for the data processing queue to call the data. According to the technical scheme, the data are prepared in advance based on data migration among different storage spaces, so that high throughput and timeliness of task timing batch processing can be effectively guaranteed even when a large number of tasks are processed, such as a large number of tasks to be executed.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments when taken in conjunction with the accompanying drawings. The following is a description of the drawings.
Fig. 1 shows a flowchart of a task timing scheduling method according to an embodiment of the present disclosure.
Fig. 2 illustrates an overall flowchart of a task timing scheduling method according to an embodiment of the present disclosure.
Fig. 3 is a block diagram illustrating a task timing scheduling apparatus according to an embodiment of the present disclosure.
Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
FIG. 5 is a schematic structural diagram of a computer system suitable for implementing a task timing scheduling method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.
In the present disclosure, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.
It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
According to the technical scheme provided by the embodiment of the disclosure, the data to be processed is stored in the first storage space, and then the data to be processed is migrated to the second storage space before the task execution time comes, so that data preparation is made for the data processing queue to call the data. According to the technical scheme, the data are prepared in advance based on data migration among different storage spaces, so that high throughput and timeliness of task timing batch processing can be effectively guaranteed even when a large number of tasks are processed, such as a large number of tasks to be executed.
Fig. 1 shows a flowchart of a task timing scheduling method according to an embodiment of the present disclosure, and as shown in fig. 1, the task timing scheduling method includes the following steps S101 to S103:
in step S101, in response to detecting a timed scheduling task registration operation, acquiring task scheduling time and task data amount from the timed scheduling task registration information, and storing the task data in a first storage space;
In step S102, calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
in step S103, in response to the migration time being reached, the task data is migrated from the first storage space to the second storage space.
As mentioned above, with the development of data processing requirements and data processing technologies, batch processing of data tasks is widely used, where batch processing of data tasks refers to that multiple data tasks are input at the same time, automatically executed, and after execution, an execution result is sent to a requester. The timed batch processing of the data tasks is based on the batch processing, so that the tasks are processed at a specified time point, such as timed batch payroll, timed batch log clearing in a log system and the like, wherein certain application scenarios have high requirements on timeliness of the timed batch processing of the data tasks, namely that the batch task processing is expected to be completed at the specified time point on time, such as stock timed trading and the like in a financial scenario, and under the scenario, the execution delay of the tasks may bring immeasurable loss. However, if the number of tasks to be executed is huge, such as massive task data, how to simultaneously ensure high throughput and timeliness of task timing batch processing is a subject of great attention.
In view of the above drawbacks, in this embodiment, a task timing scheduling method is proposed, which stores data to be processed in a first storage space, and then migrates the data to be processed to a second storage space before the task execution time comes, so as to prepare the data for the data processing queue to fetch the data. According to the technical scheme, the data are prepared in advance based on data migration among different storage spaces, so that high throughput and timeliness of task timing batch processing can be effectively guaranteed even when a large number of tasks are processed, such as a large number of tasks to be executed.
In an embodiment of the present disclosure, the task timing scheduling method may be applied to a task scheduler such as a computer, a computing device, an electronic device, and a server that performs timing scheduling on task data.
In an embodiment of the present disclosure, the timed scheduling task registration operation is used to refer to a request operation of a timed scheduling task requester for a timed scheduling task, and when the task requester performs the timed scheduling task registration operation, the task requester needs to provide task data amount of the timed scheduling task, such as the number of tasks, the data amount related to each task, and the like, and also needs to provide task scheduling time corresponding to each task. That is to say, the task scheduler can make sure the request of the task requester, and at the same time, also the amount of the task that needs to be scheduled subsequently and the time that needs to execute the task scheduling by the task scheduler through the timed scheduling task registration operation. The task requester can execute the timed scheduling task registration operation through an external interface with preset limitation.
In an embodiment of the present disclosure, the first storage space is an external storage space with a relatively large storage space, that is, an external memory, and the second storage space is an internal storage space with a relatively small storage space, that is, an internal memory.
In an embodiment of the present disclosure, the migration time refers to a time point when task data to be scheduled needs to be migrated from the first storage space to the second storage space in advance in order to perform scheduling of a task in time when the task scheduling time arrives, that is, the migration time is determined based on a principle that the task can be migrated to the second storage space and then scheduled in time when the task scheduling time arrives, as seen from the above, migration of the task data is prior and scheduled later, so the migration time is earlier than the task scheduling time. In addition, the specific value of the migration time is related to the task scheduling time of the task to be scheduled and the data volume of the task to be scheduled, and the larger the data volume of the task to be scheduled is, the larger the time difference between the migration time and the task scheduling time is, that is, the earlier the migration time is, that is, the more the migration time needs to be spent; the smaller the data amount of the task to be scheduled, the smaller the time difference between the migration time and the task scheduling time, i.e. the later the migration time, i.e. the less migration time that needs to be spent.
The existing timing task scheduling method is based on either internal memory processing or external memory processing. The processing method based on the memory is that after a requester registers a timing scheduling task, the task to be scheduled is stored in the memory, and the task is executed after the scheduled time of the task is reached, and the processing method can generally ensure the on-time execution of the task. The timed task scheduling method based on the external memory stores tasks to be scheduled based on the external memory, then polls external memory records, and reads the tasks into the internal memory for processing if the to-be-executed time of a certain task is found to be earlier than or equal to the current time. According to the method, the external memory storage space is large, so that the throughput of timing task scheduling can be guaranteed, when the magnitude of the task to be scheduled is large, the time consumed for fetching the task from the external memory and the time consumed for processing the task per se can possibly prolong the time consumed for task execution, the timeliness of the timing task execution cannot be guaranteed, and even an avalanche effect can be generated under the condition of limited resources, so that the execution of subsequent tasks is influenced.
In the above embodiment, after detecting that the task requester has performed the timed scheduling task registration operation, first obtaining task scheduling time and task data amount of a plurality of timed scheduling tasks from corresponding timed scheduling task registration information, and storing task data to be scheduled in a first storage space such as an external memory; then calculating migration time for migrating the task data from the first storage space to the second storage space based on the acquired task scheduling time and the task data amount; if the migration time is detected to be reached, data migration operation can be executed, namely the task data are migrated from a first storage space such as an external memory to a second storage space such as an internal memory, so that the time consumed by reading a mass task into the internal memory is effectively offset, and the buffer time is provided for the timing scheduling of a subsequent task.
In an embodiment of the present disclosure, the step S102 of calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount may include the following steps:
estimating and obtaining estimated migration time required for migrating the task data from a first storage space to a second storage space based on the task data volume;
And calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time.
In order to store the task data to be scheduled into the second storage space when the task scheduling time arrives, prepare the data for timely scheduling of the execution task, and avoid resource waste caused by unnecessary resource occupation due to too early time for storing the task data to be scheduled into the second storage space, in this embodiment, the migration time for migrating the task data from the first storage space to the second storage space needs to be accurately calculated. Specifically, firstly, a migration time length required for migrating the task data from the first storage space to the second storage space, that is, an estimated migration time length, is estimated based on the task data amount, and then, based on the task scheduling time and the estimated migration time length, migration time point information for starting to migrate the task data from the first storage space to the second storage space can be calculated.
In an embodiment of the present disclosure, the step of estimating, based on the task data amount, an estimated migration time required to migrate the task data from the first storage space to the second storage space may include the following steps:
Acquiring a reference data migration speed;
and dividing the task data volume by the reference data migration speed to obtain the estimated migration time required for migrating the task data from the first storage space to the second storage space.
In this embodiment, in order to obtain an estimated migration time length required for migrating the task data from the first storage space to the second storage space, a reference data migration speed may be first obtained, where the reference data migration speed may be set according to a requirement of an actual application, for example, the reference data migration speed may be set as a current data migration speed, may be set as an average migration speed within a preset historical time period, and may also be set as a highest or lowest migration speed within the preset historical time period, where the preset historical time period refers to a historical time period with a preset time length before a current time, and the time length of the preset historical time period may also be set according to a requirement of the actual application, which is not specifically limited by the present disclosure; then, the migration time required for migrating the task data from the first storage space to the second storage space can be estimated by dividing the task data amount by the reference data migration speed, that is, the estimated migration time is estimated.
In addition, it should be noted that, in order to ensure that all task data are migrated to the second storage space when the task scheduling time is reached, after the estimated migration duration is obtained through calculation, upward fine tuning may be performed on the estimated migration duration so as to make the estimated migration duration greater than an actual migration duration required for migrating the task data from the first storage space to the second storage space. When the estimated migration time length is trimmed upwards or downwards, fine trimming can be performed based on a preset fine trimming coefficient, for example, the estimated migration time length is multiplied by a preset fine trimming coefficient larger than 1 or added with a preset fine trimming coefficient to obtain an estimated migration time length after upward fine trimming, and the estimated migration time length is multiplied by a preset fine trimming coefficient smaller than 1 or subtracted with a preset fine trimming coefficient to obtain an estimated migration time length after downward fine trimming, wherein the preset fine trimming coefficient can be determined according to historical task migration conditions.
In an embodiment of the present disclosure, the step of calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time may include the following steps:
And subtracting the estimated migration duration from the task scheduling time to obtain the migration time for migrating the task data from the first storage space to the second storage space.
As mentioned above, the migration time is earlier than the task scheduling time, and therefore, after the migration time required for migrating the task data from the first storage space to the second storage space is estimated, the estimated migration time may be pushed forward with the task scheduling time as a reference time, so as to obtain a migration time point at which the task data starts to migrate from the first storage space to the second storage space, that is, in this embodiment, the estimated migration time is subtracted from the task scheduling time, and the interface obtains the migration time for migrating the task data from the first storage space to the second storage space.
In an embodiment of the present disclosure, the method may further include the steps of:
reading the task data from the second storage space to a scheduling queue;
and executing the task scheduling operation in response to the arrival of the task scheduling time.
After the task data is migrated from the first storage space to the second storage space, the task data can be read from the second storage space to the scheduling queue, and after the task scheduling time is reached, the task scheduling operation is automatically executed. The scheduling queue may be an automatically running scheduling queue, such as a Quartz queue, where Quartz serves as an open-source job scheduling framework and supports automatic timing execution of jobs. Assuming that the migration time is tc, the estimated migration time is Te, the actual migration time is Tr, and the task scheduling time is t, the time for adding the task data into the scheduling queue can be represented as tc + Tr, and since t = tc + Te and Te > Tr exists, tc + Tr < t, namely the time for adding the task data into the scheduling queue is earlier than the task scheduling time, the capability of automatically and timely executing the task by means of the Quartz scheduling framework can be used for ensuring that the task is executed on time at the task scheduling time t, namely the timeliness of task timing scheduling is ensured.
In one embodiment of the present disclosure, to improve the throughput of batch task scheduling, data migration and reading are performed using a multi-threaded or distributed data processing method. That is, the migration is a multi-threaded or distributed migration and the reading is a multi-threaded or distributed reading.
Fig. 2 is a flowchart illustrating a task timing scheduling method according to an embodiment of the present disclosure, as shown in fig. 2, a timing scheduling task requester registers a timing scheduling task with a database; then, based on the registration information, obtaining task scheduling time and task data volume, and calculating migration time for migrating the task data from the first storage space to the second storage space; when migration time is reached, the task data is migrated from the first storage space to the second storage space by using a multithreading or distributed data processing method, the process can also be called the early entry of the task data, and the migration of the task data can also be called the fishing of the task; after the task data is migrated to the second storage space, the task data can be read into the Quartz queue by using a multithread or distributed data processing method, so that the task scheduling operation is automatically executed when the task scheduling time is reached, and the process can also be called as the timed consumption of the task data.
The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods.
Fig. 3 shows a block diagram of a task timing scheduling apparatus according to an embodiment of the present disclosure, which may be implemented as part or all of an electronic device by software, hardware, or a combination of the two. As shown in fig. 3, the task timing scheduling apparatus includes:
the storage module 301 is configured to, in response to detecting a timed scheduling task registration operation, obtain task scheduling time and task data amount from the timed scheduling task registration information, and store the task data in the first storage space;
a calculating module 302 configured to calculate a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
a migration module 303 configured to migrate the task data from the first storage space to the second storage space in response to reaching the migration time.
As mentioned above, with the development of data processing requirements and data processing technologies, batch processing of data tasks is widely used, where batch processing of data tasks refers to that multiple data tasks are input at the same time, automatically executed, and after execution, an execution result is sent to a requester. The timed batch processing of the data tasks is based on the batch processing, so that the tasks are processed at a specified time point, such as timed batch payroll, timed batch log clearing in a log system and the like, wherein certain application scenarios have high requirements on timeliness of the timed batch processing of the data tasks, namely that the batch task processing is expected to be completed at the specified time point on time, such as stock timed trading and the like in a financial scenario, and under the scenario, the execution delay of the tasks may bring immeasurable loss. However, if the number of tasks to be executed is huge, such as massive task data, how to simultaneously ensure high throughput and timeliness of task timing batch processing is a subject of great attention.
In view of the above drawbacks, in this embodiment, a task timing scheduling apparatus is proposed, which stores data to be processed in a first storage space, and then migrates the data to be processed to a second storage space before the task execution time comes, so as to prepare the data for the data processing queue to fetch the data. According to the technical scheme, the data are prepared in advance based on data migration among different storage spaces, so that high throughput and timeliness of task timing batch processing can be effectively guaranteed even when a large number of tasks are processed, such as a large number of tasks to be executed.
In an embodiment of the present disclosure, the task timing scheduling apparatus may be implemented as a task scheduler, such as a computer, a computing device, an electronic device, and a server, that performs timing scheduling on task data.
In an embodiment of the present disclosure, the timed scheduling task registration operation is used to refer to a request operation of a timed scheduling task requester for a timed scheduling task, and when the task requester performs the timed scheduling task registration operation, the task requester needs to provide task data amount of the timed scheduling task, such as the number of tasks, the data amount related to each task, and the like, and also needs to provide task scheduling time corresponding to each task. That is to say, the task scheduler can make sure the request of the task requester, and at the same time, also the amount of the task that needs to be scheduled subsequently and the time that needs to execute the task scheduling by the task scheduler through the timed scheduling task registration operation. The task requester can execute the timed scheduling task registration operation through an external interface with preset limitation.
In an embodiment of the present disclosure, the first storage space is an external storage space with a relatively large storage space, that is, an external memory, and the second storage space is an internal storage space with a relatively small storage space, that is, an internal memory.
In an embodiment of the present disclosure, the migration time refers to a time point when task data to be scheduled needs to be migrated from the first storage space to the second storage space in advance in order to perform scheduling of a task in time when the task scheduling time arrives, that is, the migration time is determined based on a principle that the task can be migrated to the second storage space and then scheduled in time when the task scheduling time arrives, as seen from the above, migration of the task data is prior and scheduled later, so the migration time is earlier than the task scheduling time. In addition, the specific value of the migration time is related to the task scheduling time of the task to be scheduled and the data volume of the task to be scheduled, and the larger the data volume of the task to be scheduled is, the larger the time difference between the migration time and the task scheduling time is, that is, the earlier the migration time is, that is, the more the migration time needs to be spent; the smaller the data amount of the task to be scheduled, the smaller the time difference between the migration time and the task scheduling time, i.e. the later the migration time, i.e. the less migration time that needs to be spent.
The existing timing task scheduling method is based on either internal memory processing or external memory processing. The processing method based on the memory is that after a requester registers a timing scheduling task, the task to be scheduled is stored in the memory, and the task is executed after the scheduled time of the task is reached, and the processing method can generally ensure the on-time execution of the task. The timed task scheduling method based on the external memory stores tasks to be scheduled based on the external memory, then polls external memory records, and reads the tasks into the internal memory for processing if the to-be-executed time of a certain task is found to be earlier than or equal to the current time. According to the method, the external memory storage space is large, so that the throughput of timing task scheduling can be guaranteed, when the magnitude of the task to be scheduled is large, the time consumed for fetching the task from the external memory and the time consumed for processing the task per se can possibly prolong the time consumed for task execution, the timeliness of the timing task execution cannot be guaranteed, and even an avalanche effect can be generated under the condition of limited resources, so that the execution of subsequent tasks is influenced.
In the above embodiment, after detecting that the task requester has performed the timed scheduling task registration operation, first obtaining task scheduling time and task data amount of a plurality of timed scheduling tasks from corresponding timed scheduling task registration information, and storing task data to be scheduled in a first storage space such as an external memory; then calculating migration time for migrating the task data from the first storage space to the second storage space based on the acquired task scheduling time and the task data amount; if the migration time is detected to be reached, data migration operation can be executed, namely the task data are migrated from a first storage space such as an external memory to a second storage space such as an internal memory, so that the time consumed by reading a mass task into the internal memory is effectively offset, and the buffer time is provided for the timing scheduling of a subsequent task.
In an embodiment of the present disclosure, the calculation module 302 may be configured to:
estimating and obtaining estimated migration time required for migrating the task data from a first storage space to a second storage space based on the task data volume;
and calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time.
In order to store the task data to be scheduled into the second storage space when the task scheduling time arrives, prepare the data for timely scheduling of the execution task, and avoid resource waste caused by unnecessary resource occupation due to too early time for storing the task data to be scheduled into the second storage space, in this embodiment, the migration time for migrating the task data from the first storage space to the second storage space needs to be accurately calculated. Specifically, firstly, a migration time length required for migrating the task data from the first storage space to the second storage space, that is, an estimated migration time length, is estimated based on the task data amount, and then, based on the task scheduling time and the estimated migration time length, migration time point information for starting to migrate the task data from the first storage space to the second storage space can be calculated.
In an embodiment of the present disclosure, the portion of estimating, based on the task data amount, an estimated migration time required to migrate the task data from the first storage space to the second storage space may be configured to:
acquiring a reference data migration speed;
and dividing the task data volume by the reference data migration speed to obtain the estimated migration time required for migrating the task data from the first storage space to the second storage space.
In this embodiment, in order to obtain an estimated migration time length required for migrating the task data from the first storage space to the second storage space, a reference data migration speed may be first obtained, where the reference data migration speed may be set according to a requirement of an actual application, for example, the reference data migration speed may be set as a current data migration speed, may be set as an average migration speed within a preset historical time period, and may also be set as a highest or lowest migration speed within the preset historical time period, where the preset historical time period refers to a historical time period with a preset time length before a current time, and the time length of the preset historical time period may also be set according to a requirement of the actual application, which is not specifically limited by the present disclosure; then, the migration time required for migrating the task data from the first storage space to the second storage space can be estimated by dividing the task data amount by the reference data migration speed, that is, the estimated migration time is estimated.
In addition, it should be noted that, in order to ensure that all task data are migrated to the second storage space when the task scheduling time is reached, after the estimated migration duration is obtained through calculation, upward fine tuning may be performed on the estimated migration duration so as to make the estimated migration duration greater than an actual migration duration required for migrating the task data from the first storage space to the second storage space. When the estimated migration time length is trimmed upwards or downwards, fine trimming can be performed based on a preset fine trimming coefficient, for example, the estimated migration time length is multiplied by a preset fine trimming coefficient larger than 1 or added with a preset fine trimming coefficient to obtain an estimated migration time length after upward fine trimming, and the estimated migration time length is multiplied by a preset fine trimming coefficient smaller than 1 or subtracted with a preset fine trimming coefficient to obtain an estimated migration time length after downward fine trimming, wherein the preset fine trimming coefficient can be determined according to historical task migration conditions.
In an embodiment of the present disclosure, the calculating a portion of the migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time duration may be configured to:
and subtracting the estimated migration duration from the task scheduling time to obtain the migration time for migrating the task data from the first storage space to the second storage space.
As mentioned above, the migration time is earlier than the task scheduling time, and therefore, after the migration time required for migrating the task data from the first storage space to the second storage space is estimated, the estimated migration time may be pushed forward with the task scheduling time as a reference time, so as to obtain a migration time point at which the task data starts to migrate from the first storage space to the second storage space, that is, in this embodiment, the estimated migration time is subtracted from the task scheduling time, and the interface obtains the migration time for migrating the task data from the first storage space to the second storage space.
In an embodiment of the present disclosure, the apparatus may further include:
the reading module is configured to read the task data from the second storage space to a scheduling queue;
An execution module configured to execute the task scheduling operation in response to reaching the task scheduling time.
After the task data is migrated from the first storage space to the second storage space, the task data can be read from the second storage space to the scheduling queue, and after the task scheduling time is reached, the task scheduling operation is automatically executed. The scheduling queue may be an automatically running scheduling queue, such as a Quartz queue, where Quartz serves as an open-source job scheduling framework and supports automatic timing execution of jobs. Assuming that the migration time is tc, the estimated migration time is Te, the actual migration time is Tr, and the task scheduling time is t, the time for adding the task data into the scheduling queue can be represented as tc + Tr, and since t = tc + Te and Te > Tr exists, tc + Tr < t, namely the time for adding the task data into the scheduling queue is earlier than the task scheduling time, the capability of automatically and timely executing the task by means of the Quartz scheduling framework can be used for ensuring that the task is executed on time at the task scheduling time t, namely the timeliness of task timing scheduling is ensured.
In one embodiment of the present disclosure, to improve the throughput of batch task scheduling, data migration and reading are performed using a multi-threaded or distributed data processing method. That is, the migration is a multi-threaded or distributed migration and the reading is a multi-threaded or distributed reading.
The present disclosure also discloses an electronic device, fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure, and as shown in fig. 4, the electronic device 400 includes a memory 401 and a processor 402; wherein the content of the first and second substances,
the memory 401 is used to store one or more computer instructions that are executed by the processor 402 to implement the above-described method steps.
FIG. 5 is a schematic structural diagram of a computer system suitable for implementing a task timing scheduling method according to an embodiment of the present disclosure.
As shown in fig. 5, the computer system 500 includes a processing unit 501 that can execute various processes in the above-described embodiments according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the system 500 are also stored. The processing unit 501, the ROM502, and the RAM503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary. The processing unit 501 may be implemented as a CPU, a GPU, a TPU, an FPGA, an NPU, or other processing units.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present disclosure may be implemented by software or hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.
As another aspect, the present disclosure also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the apparatus in the above-described embodiment; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is possible without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Claims (11)
1. A task timing scheduling method comprises the following steps:
in response to the detection of the timed scheduling task registration operation, acquiring task scheduling time and task data volume from the timed scheduling task registration information, and storing the task data into a first storage space;
Calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
in response to reaching the migration time, the task data is migrated from the first storage space to the second storage space.
2. The method of claim 1, wherein calculating a migration time for migrating the task data from a first storage space to a second storage space based on the task scheduling time and the amount of task data comprises:
estimating and obtaining estimated migration time required for migrating the task data from a first storage space to a second storage space based on the task data volume;
and calculating migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration time.
3. The method of claim 2, wherein estimating the estimated migration duration required to migrate the task data from the first storage space to the second storage space based on the amount of task data comprises:
acquiring a reference data migration speed;
and dividing the task data volume by the reference data migration speed to obtain the estimated migration time required for migrating the task data from the first storage space to the second storage space.
4. The method of claim 2 or 3, wherein calculating a migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the estimated migration duration comprises:
and subtracting the estimated migration duration from the task scheduling time to obtain the migration time for migrating the task data from the first storage space to the second storage space.
5. The method of claim 2 or 3, wherein the estimated migration duration is greater than an actual migration duration required to migrate the task data from a first storage space to a second storage space.
6. The method of any of claims 1-3, further comprising:
reading the task data from the second storage space to a scheduling queue;
and executing the task scheduling operation in response to the arrival of the task scheduling time.
7. The method of claim 6, wherein the migration is multi-threaded or distributed and the reading is multi-threaded or distributed.
8. The method according to any one of claims 1-3, wherein the first storage space is an external memory and the second storage space is an internal memory.
9. A task timing scheduler comprising:
The storage module is configured to respond to the detection of the timed scheduling task registration operation, acquire task scheduling time and task data volume from the timed scheduling task registration information and store the task data into a first storage space;
the computing module is configured to compute migration time for migrating the task data from the first storage space to the second storage space based on the task scheduling time and the task data amount;
a migration module configured to migrate the task data from the first storage space to the second storage space in response to reaching a migration time.
10. An electronic device comprising a memory and at least one processor; wherein the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the at least one processor to implement the method steps of any one of claims 1-8.
11. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, carry out the method steps of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111191917.0A CN113641484A (en) | 2021-10-13 | 2021-10-13 | Task timing scheduling method, device and equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111191917.0A CN113641484A (en) | 2021-10-13 | 2021-10-13 | Task timing scheduling method, device and equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113641484A true CN113641484A (en) | 2021-11-12 |
Family
ID=78426674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111191917.0A Pending CN113641484A (en) | 2021-10-13 | 2021-10-13 | Task timing scheduling method, device and equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641484A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106951311A (en) * | 2017-03-21 | 2017-07-14 | 联想(北京)有限公司 | A kind of data processing method and server cluster |
| CN107391031A (en) * | 2017-06-27 | 2017-11-24 | 北京邮电大学 | Data migration method and device in a kind of computing system based on mixing storage |
| CN108762931A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method for scheduling task, server based on distributed scheduling system and storage medium |
| WO2021189857A1 (en) * | 2020-09-23 | 2021-09-30 | 平安科技（深圳）有限公司 | Task state detection method and apparatus, and computer device and storage medium |
-
2021
- 2021-10-13 CN CN202111191917.0A patent/CN113641484A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106951311A (en) * | 2017-03-21 | 2017-07-14 | 联想(北京)有限公司 | A kind of data processing method and server cluster |
| CN107391031A (en) * | 2017-06-27 | 2017-11-24 | 北京邮电大学 | Data migration method and device in a kind of computing system based on mixing storage |
| CN108762931A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method for scheduling task, server based on distributed scheduling system and storage medium |
| WO2021189857A1 (en) * | 2020-09-23 | 2021-09-30 | 平安科技（深圳）有限公司 | Task state detection method and apparatus, and computer device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107992359B (en) | Task scheduling method for cost perception in cloud environment | |
| US8812639B2 (en) | Job managing device, job managing method and job managing program | |
| US8914805B2 (en) | Rescheduling workload in a hybrid computing environment | |
| US9235440B2 (en) | Managing job execution | |
| US8739171B2 (en) | High-throughput-computing in a hybrid computing environment | |
| US8839243B2 (en) | Remediating resource overload | |
| KR101686010B1 (en) | Apparatus for fair scheduling of synchronization in realtime multi-core systems and method of the same | |
| US20040015973A1 (en) | Resource reservation for large-scale job scheduling | |
| CN111338804B (en) | Resolving method and device for laser radar point cloud data and multithreading processing system | |
| EP2581829B1 (en) | Dynamic scheduling for frames representing views of a geographic information environment | |
| US8612991B2 (en) | Dynamic critical-path recalculation facility | |
| CN107430526B (en) | Method and node for scheduling data processing | |
| CN113391911B (en) | Dynamic scheduling method, device and equipment for big data resources | |
| CN113434303A (en) | Batch-processed remote sensing image intelligent processing model prediction performance optimization system and method | |
| EP2840513B1 (en) | Dynamic task prioritization for in-memory databases | |
| CN113641484A (en) | Task timing scheduling method, device and equipment and computer readable storage medium | |
| Ouyang et al. | An approach for modeling and ranking node-level stragglers in cloud datacenters | |
| CN110175078B (en) | Service processing method and device | |
| CN109189581B (en) | Job scheduling method and device | |
| CN110825342A (en) | Memory scheduling device and system, method and apparatus for processing information | |
| CN112114967B (en) | GPU resource reservation method based on service priority | |
| CN112527482A (en) | Task management method and system based on mobile edge cloud platform | |
| CN109669780B (en) | Video analysis method and system | |
| US20120124582A1 (en) | Calculating Processor Load | |
| Lee et al. | Interrupt handler migration and direct interrupt scheduling for rapid scheduling of interrupt-driven tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211112 |