CN114168297A - Method, device, equipment and medium for scheduling collection tasks - Google Patents
Method, device, equipment and medium for scheduling collection tasks Download PDFInfo
- Publication number
- CN114168297A CN114168297A CN202111518958.6A CN202111518958A CN114168297A CN 114168297 A CN114168297 A CN 114168297A CN 202111518958 A CN202111518958 A CN 202111518958A CN 114168297 A CN114168297 A CN 114168297A
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- scheduling
- tasks
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Abstract
The application discloses a method, a device, equipment and a medium for scheduling a collection task, wherein the method comprises the following steps: acquiring a task to be processed generated by a system service flow; determining the task category of a task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set; acquiring a scheduling instruction sent by a distributed task scheduling platform, then extracting a task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from a plurality of preset task sets; and sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executing a task processing flow corresponding to the received tasks to be processed through the external interface. The application stores the task to be processed to the cache component so as to reduce the pressure of the external interface in subsequent processing.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for aggregated task scheduling.
Background
With the development of system scale, the number of tasks to be processed in the system is increasing, and the tasks become more and more complex, especially in a distributed environment, a plurality of service systems exist, each service system has a demand for a large number of tasks to be processed, and therefore a task platform is required to perform unified management and scheduling on the dispersed tasks to be processed.
In the prior art, generally, an open source scheduling platform schedules tasks to be processed at regular time, for example, an XXL-JOB (distributed task scheduling platform) acquires the tasks to be processed and calls an interface corresponding to the tasks to process the tasks, and since XXL-JOB can process the tasks to be processed in parallel, when a large number of tasks to be processed call one external interface at the same time, external interface breakdown, untimely response of the external interface, loss of the tasks to be processed, and the like can be caused.
In summary, how to call an external interface to process a large number of tasks in a distributed task scheduling platform, the situations of external interface breakdown and task loss are avoided, and the problem that the task to be processed is smoothly responded by the external interface is solved.
Disclosure of Invention
In view of this, an object of the present invention is to provide a method, an apparatus, a device, and a medium for scheduling a collection task, which can avoid situations of external interface breakdown and task loss when a distributed task scheduling platform calls an external interface to process a large number of tasks, and enable a task to be processed to be smoothly responded by the external interface.
The specific scheme is as follows:
in a first aspect, the present application discloses a method for scheduling a collection task, which is applied to a cache component, and includes:
acquiring a task to be processed generated by a system service flow;
determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set;
acquiring a scheduling instruction sent by a distributed task scheduling platform, then extracting a task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from the plurality of preset task sets;
and sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
Optionally, the acquiring the to-be-processed task generated by the system business process includes:
acquiring a task to be processed, which is obtained by assigning task parameters in a preset entity class in a system service flow; the task parameters in the entity class comprise task keywords, interface request parameters, retry times and expiration time;
correspondingly, the determining the task category of the task to be processed includes:
and determining the task category of the task to be processed based on the task keywords in the task to be processed.
Optionally, the screening out a preset task set corresponding to the task category from a plurality of preset task sets created in advance locally, and then storing the to-be-processed task to the process of the screened out preset task set, further includes:
if the plurality of preset task sets created locally in advance do not have the preset task set corresponding to the task type, an initialization task set corresponding to the task type is created, and the task to be processed is stored in the initialization task set.
Optionally, the obtaining of the scheduling instruction sent by the distributed task scheduling platform includes:
the method comprises the steps of obtaining a scheduling instruction which is triggered by a distributed task scheduling platform according to a preset instruction triggering period and carries a task set identifier, wherein the scheduling instruction is generated by the distributed task scheduling platform by taking a task keyword as the task set identifier.
Optionally, the sending the to-be-processed task in the target task set to the distributed task scheduling platform includes:
and according to the preset single-batch task quantity sent by the distributed task scheduling platform, screening a corresponding quantity of tasks to be processed from the target task set, and sending the screened tasks to be processed to the distributed task scheduling platform.
Optionally, before the obtaining of the scheduling instruction sent by the distributed task scheduling platform, the method further includes:
transmitting the statistical information of all the tasks to be processed which are cached locally at present to the distributed task scheduling platform so that the distributed task scheduling platform configures an actuator corresponding to each task to be processed according to the statistical information;
correspondingly, the sending the to-be-processed task in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received to-be-processed task, so as to execute a task processing flow corresponding to the received to-be-processed task through the external interface includes:
and sending the to-be-processed tasks in the target task set to the distributed task scheduling platform, so that the distributed task scheduling platform calls a pre-configured executor corresponding to the to-be-processed tasks, schedules an external interface corresponding to the received to-be-processed tasks through the executor, and executes a task processing flow corresponding to the received to-be-processed tasks through the external interface.
In a second aspect, the present application discloses a method for scheduling a collection task, which is applied to a distributed task scheduling platform, and includes:
sending a scheduling instruction carrying a task set identifier to a cache component so that the cache component can determine a target task set determined by the task set identifier from a plurality of preset task sets created in advance locally;
acquiring the tasks to be processed in the target task set sent by the cache component;
and calling a pre-configured executor corresponding to the task to be processed, and scheduling an external interface corresponding to the received task to be processed through the executor so as to execute a task processing flow corresponding to the received task to be processed through the external interface.
In a third aspect, the present application discloses a rendezvous task scheduling apparatus, including:
the task acquisition module is used for acquiring a task to be processed generated by a system business process;
the task storage module is used for determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets which are created locally in advance, and storing the task to be processed to the screened preset task set;
the target task set determining module is used for acquiring a scheduling instruction sent by the distributed task scheduling platform, then extracting a task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from the plurality of preset task sets;
and the task processing module is used for sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
In a fourth aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
a processor for executing said computer program for implementing the steps of the aforementioned disclosed rendezvous task scheduling method.
In a fifth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program realizes the steps of the aforementioned disclosed rendezvous task scheduling method when executed by a processor.
Therefore, the method comprises the steps of firstly acquiring a task to be processed generated by a system business process; determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set; then, a scheduling instruction sent by a distributed task scheduling platform is obtained, then a task set identifier carried in the scheduling instruction is extracted, and a target task set determined by the task set identifier is determined from the plurality of preset task sets; and finally, sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executing a task processing flow corresponding to the received tasks to be processed through the external interface. Therefore, the tasks to be processed are classified according to the task categories and stored in the corresponding preset task sets, an intermediate caching process is performed on the tasks to be processed by virtue of the caching component, the situation that the tasks to be processed are lost can be avoided, then the caching component sends the corresponding tasks to be processed in the preset task sets to the distributed task scheduling platform through the task set identification in the scheduling instruction sent by the distributed task scheduling platform, so that the distributed task scheduling platform schedules the corresponding external interfaces to perform subsequent data processing on the tasks to be processed, the distributed task scheduling platform obtains the corresponding tasks to be processed in the preset task sets according to the task set identification, in this way, the classified storage of the tasks to be processed can be realized, the request amount of the tasks to be processed using the distributed task scheduling platform is reduced, and the pressure of the corresponding distributed task scheduling platform for calling the external interfaces is also reduced, the situations that the external interface is crashed, the response of the external interface is not timely and the task to be processed is lost are reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a method for aggregated task scheduling as disclosed herein;
FIG. 2 is a flow chart of a particular aggregated task scheduling method disclosed herein;
FIG. 3 is a flowchart of a particular aggregated task scheduling method disclosed herein;
FIG. 4 is a flow diagram illustrating a specific timing scheduling process disclosed herein;
FIG. 5 is a flow chart of another aggregated task scheduling method disclosed herein;
FIG. 6 is a schematic diagram of an aggregated task scheduler according to the present disclosure;
fig. 7 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the prior art, generally, an open source scheduling platform schedules tasks to be processed at regular time, for example, an XXL-JOB (distributed task scheduling platform) acquires the tasks to be processed and calls an interface corresponding to the tasks to process the tasks, and since XXL-JOB can process the tasks to be processed in parallel, when a large number of tasks to be processed call one external interface at the same time, external interface breakdown, untimely response of the external interface, loss of the tasks to be processed, and the like can be caused. Therefore, the embodiment of the application discloses a method, a device, equipment and a medium for scheduling a collection task, which can avoid the situations of external interface breakdown and task loss when a distributed task scheduling platform calls an external interface to process a large number of tasks, and realize the problem that the task to be processed is smoothly responded by the external interface.
Referring to fig. 1, an embodiment of the present application discloses a rendezvous task scheduling method, which is applied to a cache component, and the method includes:
step S11: and acquiring a task to be processed generated by the system business process.
In this embodiment, a to-be-processed task generated according to a system business process needs to be acquired, for example, the to-be-processed task may be a data acquisition task, a data cleaning task, a data analysis task, a log storage task, and the like.
Step S12: determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set.
In this embodiment, the task category of the task to be processed is determined, and a preset task set corresponding to the task category is screened out from a plurality of preset task sets created in advance locally, it may be understood that the plurality of preset task sets may exist in advance in the cache component, and the type of the preset task set may be a queue set created based on a list, or a task set created based on a set. And then storing the tasks to be processed into the corresponding screened preset task sets, thereby obtaining a plurality of preset task sets carrying the tasks to be processed.
Step S13: the method comprises the steps of obtaining a scheduling instruction sent by a distributed task scheduling platform, then extracting task set identification carried in the scheduling instruction, and determining a target task set determined by the task set identification from a plurality of preset task sets.
In this embodiment, before the obtaining of the scheduling instruction sent by the distributed task scheduling platform, the method further includes: and transmitting the statistical information of all the tasks to be processed which are cached locally at present to the distributed task scheduling platform so that the distributed task scheduling platform configures the executors corresponding to the tasks to be processed according to the statistical information. It is understood that the statistical information may specifically include: information such as actual business, task size, environment and the like, for example, on an XXL-JOB platform, an XXL-JOB initialization script needs to be run first, otherwise, the starting of the XXL-JOB service will report an error, and subsequent activities cannot be performed; yml configuration files of application of XXL-JOB-ADMIN service can change information such as ports and ip (Internet Protocol, Protocol for interconnection between networks) according to the environment condition; the actuator configuration of the XXL-JOB platform can be integrated into an actual project, and when there are multiple actuators, the actuators need to be split into multiple modules, so as to avoid the actuator port collision. When an actuator of the distributed task scheduling platform is configured, firstly, a cache component is needed to transmit statistical information of all locally cached tasks to be processed to the distributed task scheduling platform, so that the distributed task scheduling platform configures the actuator corresponding to each task to be processed according to the received statistical information, wherein when the number of the actuators exceeds the size of all the tasks to be processed, an individual actuator can be configured for each task to be processed, so that the task execution efficiency is improved, but meanwhile, the maintenance cost of a server is also improved; when the number of the executors is smaller than the size of all the tasks to be processed, a certain number of executors can be allocated to all the tasks to be processed of different task categories, so that the executor resources are saved.
After the cache component receives a task scheduling instruction sent by the distributed task scheduling platform, a target task set required to be executed at this time is determined according to a task set identifier in the task scheduling instruction.
Step S14: and sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
In this embodiment, the sending the to-be-processed task in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received to-be-processed task, so as to execute a task processing flow corresponding to the received to-be-processed task through the external interface includes: and sending the to-be-processed tasks in the target task set to the distributed task scheduling platform, so that the distributed task scheduling platform calls a pre-configured executor corresponding to the to-be-processed tasks, schedules an external interface corresponding to the received to-be-processed tasks through the executor, and executes a task processing flow corresponding to the received to-be-processed tasks through the external interface. It can be understood that, when executing the task, the distributed task scheduling platform calls the external interface corresponding to the to-be-processed task through the configured executor, and completes the task processing flow, and referring to step S13, after configuring the executor corresponding to each to-be-processed task, the pre-configured executor is directly called to schedule the external interface corresponding to the to-be-processed task, so as to execute the task processing flow corresponding to the to-be-processed task through the external interface.
Therefore, the embodiment of the application firstly acquires the task to be processed generated by the system business process; determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set; then, a scheduling instruction sent by a distributed task scheduling platform is obtained, then a task set identifier carried in the scheduling instruction is extracted, and a target task set determined by the task set identifier is determined from the plurality of preset task sets; and finally, sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executing a task processing flow corresponding to the received tasks to be processed through the external interface. Therefore, the tasks to be processed are classified according to the task categories and stored in the corresponding preset task sets, an intermediate caching process is performed on the tasks to be processed by virtue of the caching component, the situation that the tasks to be processed are lost can be avoided, then the caching component sends the corresponding tasks to be processed in the preset task sets to the distributed task scheduling platform through the task set identification in the scheduling instruction sent by the distributed task scheduling platform, so that the distributed task scheduling platform schedules the corresponding external interfaces to perform subsequent data processing on the tasks to be processed, the distributed task scheduling platform obtains the corresponding tasks to be processed in the preset task sets according to the task set identification, in this way, the classified storage of the tasks to be processed can be realized, the request amount of the tasks to be processed using the distributed task scheduling platform is reduced, and the pressure of the corresponding distributed task scheduling platform for calling the external interfaces is also reduced, the situations that the external interface is crashed, the response of the external interface is not timely and the task to be processed is lost are reduced.
Referring to fig. 2, the embodiment of the present application discloses a specific aggregation task scheduling method, and compared with the previous embodiment, the present embodiment further describes and optimizes the technical solution. The method specifically comprises the following steps:
step S21: acquiring a task to be processed, which is obtained by assigning task parameters in a preset entity class in a system service flow; the task parameters in the entity class comprise task keywords, interface request parameters, retry times and expiration time.
In this embodiment, the task to be processed is obtained by assigning task parameters in a preset entity class, and the task parameters in the entity class include, but are not limited to, task keywords, interface request parameters, retry times, expiration time, and the like. Moreover, the cache component needs to be configured correspondingly according to different service scenarios, for example, in a Redis (key-value storage system), persistent configuration of the to-be-processed task needs to be involved, that is, the to-be-processed task in the Redis is written into a disk space according to a certain time period, so that the to-be-processed task can be reserved and persisted, and the to-be-processed task is prevented from being lost. In addition, the Redis needs to perform certain adjustment on parameters such as expiration time in the task to be processed according to the actual situation and the situation such as the need of the memory size, so as to meet the specific expiration time requirement on the task to be processed in the actual situation.
Step S22: determining the task category of the task to be processed based on the task keywords in the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set.
In this embodiment, when determining the task category of the task to be processed, the task category is determined based on the task keywords in the task to be processed, that is, different task keywords correspond to different task categories. And screening a corresponding preset task set from a plurality of preset task sets which are pre-established locally, and storing the task to be processed to the screened preset task set. It should be noted that if a plurality of preset task sets created in advance locally do not have a preset task set corresponding to the task category, an initialization task set corresponding to the task category is created, and the task to be processed is stored in the initialization task set. It can be understood that, if the task category of the current task to be processed is first present in the cache component, that is, the task keyword in the task to be processed does not appear in the historical task to be processed, a preset task set corresponding to the task keyword is not created locally, and then an initialization task set corresponding to the task category or the task keyword needs to be automatically created, and the task to be processed is stored in the initialization task set. For example, when storing the to-be-processed task in the Redis, a List-based queue is first created in the Redis, and a different queue is already created locally for the key of a different to-be-processed task in advance, but when a task corresponding to a specific key that has not occurred before occurs, that is, when an initialization queue corresponding to the key of the to-be-processed task does not exist locally, a blank queue may be automatically created according to the key of the to-be-processed task; and then, the task to be processed is put into a corresponding queue of Redis according to the key of the task to be processed, wherein different queues are created by calling common set methods such as opsForList (). leftPush () in redisplate. Therefore, when the task to be processed which is the same as the task type of this time appears later, a preset task set corresponding to the task type exists locally.
Step S23: the method comprises the steps of obtaining a scheduling instruction sent by a distributed task scheduling platform, then extracting task set identification carried in the scheduling instruction, and determining a target task set determined by the task set identification from a plurality of preset task sets.
Step S24: and sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
For more specific processing procedures of the steps S23 and S24, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
As can be seen, in the embodiment of the present application, the task obtained by the cache component is obtained by assigning the task parameter in the preset entity class, and when the task class of the task to be processed is determined, the task class is determined based on the task keyword in the task to be processed, that is, different task keywords correspond to different task classes. In addition, when the task to be processed is stored in the preset task set corresponding to the task category, if the preset task set corresponding to the task category does not exist in the plurality of preset task sets created in advance locally, an initialization task set corresponding to the task category needs to be created, and the task to be processed is stored in the initialization task set.
Referring to fig. 3, the embodiment of the present application discloses a specific aggregation task scheduling method, and compared with the previous embodiment, the present embodiment further describes and optimizes the technical solution. The method specifically comprises the following steps:
step S31: and acquiring a task to be processed generated by the system business process.
Step S32: determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set.
Step S33: the method comprises the steps of obtaining a scheduling instruction which is triggered by a distributed task scheduling platform according to a preset instruction triggering period and carries a task set identifier, wherein the scheduling instruction is generated by the distributed task scheduling platform by taking a task keyword as the task set identifier, then extracting the task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from a plurality of preset task sets.
In this embodiment, referring to fig. 4, the scheduling instruction sent by the distributed task scheduling platform and acquired by the cache component is triggered according to a preset instruction triggering cycle, where the preset instruction triggering cycle may be triggered in a timed manner, and the scheduling instruction is generated by the distributed task scheduling platform by using a task keyword as a task set identifier. It can be understood that, the distributed task scheduling platform may send the task scheduling instruction carrying the task set identifier to the cache component at a preset time node or time period, for example, in order not to affect normal operation of the service, the data extraction task is usually executed at early morning (e.g., 0:00), and then the task scheduling instruction is sent to the cache component at zero point every day to execute the data extraction task, where the above process may also be understood as setting the sending time of the initial scheduling instruction to 0:00, and then sending the scheduling instruction to the cache component every 24 hours.
Step S34: and according to the preset single-batch task quantity sent by the distributed task scheduling platform, selecting a corresponding quantity of tasks to be processed from the target task set, and sending the selected tasks to be processed to the distributed task scheduling platform, so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
In this embodiment, a single batch of executable tasks may be configured to obtain a corresponding number of tasks to be processed when executing the tasks, so as to limit the number of times of scheduling the external interface within a time period, thereby better alleviating the pressure of the external interface. Firstly, the distributed task scheduling platform is required to send the number of preset single batch tasks to the cache component, then the cache component screens out the corresponding number of tasks to be processed from the target task set, so that the distributed task scheduling platform schedules external interfaces corresponding to the tasks to be processed, and executes corresponding task processing flows through the external interfaces. In a specific embodiment, the preset single batch task number may be a number uniformly specified in advance, where the number is specified by the distributed task scheduling platform according to an actual configuration situation of the platform, and the number is sent to the cache component in advance, and if the number is 100, after the distributed task scheduling platform sends the scheduling instruction to the cache component, the cache component sends 100 to-be-processed tasks to the distributed task scheduling platform by default each time. In another specific embodiment, the preset single-batch task number may be sent to the cache component together with the scheduling instruction, that is, when the distributed task scheduling platform sends the task scheduling instruction to the cache component, the scheduling instruction may carry a task set identifier representing a category of a target task to be scheduled this time, or may carry a task number to be scheduled this time, and then a corresponding task number of tasks to be processed is selected from the determined target task set.
For more specific processing procedures of the steps S31, S32, and S33, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
As can be seen, in the embodiment of the present application, the scheduling instruction sent by the distributed task scheduling platform and acquired by the cache component is triggered according to a preset instruction triggering cycle, and the scheduling instruction is a scheduling instruction generated by the distributed task scheduling platform by using a task keyword as a task set identifier. And when executing the tasks, a single batch of executable tasks can be configured to obtain a corresponding number of to-be-processed tasks, that is, the cache component is based on the preset single batch of tasks sent by the distributed task scheduling platform, and screens out a corresponding number of to-be-processed tasks from the target task set and sends the to-be-processed tasks to the distributed task scheduling platform for subsequent task execution. Therefore, the number of tasks in a single batch is limited, which is equivalent to correspondingly limiting the number of requests of the external interface in a time period, so that the pressure of the external interface is relieved, and the situation that the external interface is broken down due to overlarge pressure is reduced.
Referring to fig. 5, an embodiment of the present application discloses a method for scheduling a collection task, which is applied to a distributed task scheduling platform, and the method includes:
step S41: and sending a scheduling instruction carrying a task set identifier to a cache component so that the cache component determines a target task set determined by the task set identifier from a plurality of preset task sets created in advance locally.
In this embodiment, the scheduling instruction sent to the cache component and carrying the task set identifier may be a scheduling instruction triggered by a trigger instruction according to a preset period, and the scheduling instruction is a scheduling instruction generated by a distributed task scheduling platform by using a task keyword as a task set identifier, so that the cache component determines a target task set from a plurality of preset task sets created locally in advance according to task categories.
Step S42: and acquiring the tasks to be processed in the target task set sent by the cache component.
In this embodiment, when obtaining the to-be-processed tasks sent by the cache component, the cache component may collectively screen a corresponding number of to-be-processed tasks from the target task according to a preset single-batch task number sent in advance, and send the screened to-be-processed tasks to the distributed task scheduling platform.
It should be noted that task management and executor management need to be performed on the distributed task scheduling platform, after the task to be processed is obtained, a new task needs to be added on the task management interface, and a corresponding executor, a running mode, a JobHandler (configuration file of a scheduling center), retry times, an alarm mail and the like need to be configured; adding an actuator on an actuator management interface, and configuring information such as a registration method, a name and the like; and then executing the task through the @ XxlJob annotation, wherein the specific steps comprise: what needs to be configured in @ XxlJob is JobHandler configured by the task to be processed; then inquiring a task queue corresponding to a key (keyword) by a redisplate. keys () (fuzzy matching) method; then determining the task queue and acquiring the task to be detected from the task queue; and finally, calling respective interfaces of the acquired tasks to acquire external interfaces for subsequent processing. And traversing all external interfaces by the executor according to interface request parameter information in the task to be processed, and determining the external interface corresponding to the task to be processed so that the external interface executes task processing and data requests corresponding to the received task to be processed.
Step S43: and calling a pre-configured executor corresponding to the task to be processed, and scheduling an external interface corresponding to the received task to be processed through the executor so as to execute a task processing flow corresponding to the received task to be processed through the external interface.
In this embodiment, before sending the scheduling instruction carrying the task set identifier to the cache component, the statistical information of all the to-be-processed tasks of the current local cache sent by the cache component needs to be obtained, so as to configure the executor corresponding to each to-be-processed task according to the statistical information. Therefore, after the to-be-processed task is acquired, a pre-configured executor needs to be called, and an external interface corresponding to the received to-be-processed task is scheduled through the executor, so that a related task processing flow is executed through the external interface.
Therefore, the tasks to be processed are classified according to the task categories and stored in the corresponding preset task sets, an intermediate caching process is performed on the tasks to be processed by virtue of the caching component, the situation that the tasks to be processed are lost can be avoided, then the caching component sends the corresponding tasks to be processed in the preset task sets to the distributed task scheduling platform through the task set identification in the scheduling instruction sent by the distributed task scheduling platform, so that the distributed task scheduling platform schedules the corresponding external interfaces to perform subsequent data processing on the tasks to be processed, the distributed task scheduling platform obtains the corresponding tasks to be processed in the preset task sets according to the task set identification, in this way, the classified storage of the tasks to be processed can be realized, the request amount of the tasks to be processed using the distributed task scheduling platform is reduced, and the pressure of the corresponding distributed task scheduling platform for calling the external interfaces is also reduced, the situations that the external interface is crashed, the response of the external interface is not timely and the task to be processed is lost are reduced.
Referring to fig. 6, an embodiment of the present application discloses a method for scheduling a rendezvous task, which is applied to a cache component, and the apparatus includes:
the task obtaining module 11 is configured to obtain a to-be-processed task generated by a system business process;
the task storage module 12 is configured to determine a task category of the task to be processed, screen a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and store the task to be processed to the screened preset task set;
a target task set determining module 13, configured to obtain a scheduling instruction sent by a distributed task scheduling platform, extract a task set identifier carried in the scheduling instruction, and determine a target task set determined by the task set identifier from the plurality of preset task sets;
and the task processing module 14 is configured to send the to-be-processed task in the target task set to the distributed task scheduling platform, so that the distributed task scheduling platform schedules an external interface corresponding to the received to-be-processed task, and executes a task processing flow corresponding to the received to-be-processed task through the external interface.
Therefore, the method comprises the steps of firstly acquiring a task to be processed generated by a system business process; determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set; then, a scheduling instruction sent by a distributed task scheduling platform is obtained, then a task set identifier carried in the scheduling instruction is extracted, and a target task set determined by the task set identifier is determined from the plurality of preset task sets; and finally, sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executing a task processing flow corresponding to the received tasks to be processed through the external interface. Therefore, the tasks to be processed are classified according to the task categories and stored in the corresponding preset task sets, an intermediate caching process is performed on the tasks to be processed by virtue of the caching component, the situation that the tasks to be processed are lost can be avoided, then the caching component sends the corresponding tasks to be processed in the preset task sets to the distributed task scheduling platform through the task set identification in the scheduling instruction sent by the distributed task scheduling platform, so that the distributed task scheduling platform schedules the corresponding external interfaces to perform subsequent data processing on the tasks to be processed, the distributed task scheduling platform obtains the corresponding tasks to be processed in the preset task sets according to the task set identification, in this way, the classified storage of the tasks to be processed can be realized, the request amount of the tasks to be processed using the distributed task scheduling platform is reduced, and the pressure of the corresponding distributed task scheduling platform for calling the external interfaces is also reduced, the situations that the external interface is crashed, the response of the external interface is not timely and the task to be processed is lost are reduced.
Fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present application. The method specifically comprises the following steps: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein the memory 22 is used for storing a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the aggregate task scheduling method executed by the computer device disclosed in any of the foregoing embodiments.
In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the computer device 20; the communication interface 24 can create a data transmission channel between the computer device 20 and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to specific application requirements, which is not specifically limited herein.
The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 21 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may further include an AI (Artificial Intelligence) processor for processing a calculation operation related to machine learning.
In addition, the storage 22 is used as a carrier for storing resources, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., the resources stored thereon include an operating system 221, a computer program 222, data 223, etc., and the storage may be a transient storage or a permanent storage.
The operating system 221 is used for managing and controlling each hardware device and the computer program 222 on the computer device 20, so as to realize the operation and processing of the mass data 223 in the memory 22 by the processor 21, which may be Windows, Unix, Linux, or the like. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the aggregated task scheduling method performed by the computer device 20 disclosed in any of the foregoing embodiments. The data 223 may include data received by the computer device and transmitted from an external device, data collected by the input/output interface 25, and the like.
Further, an embodiment of the present application further discloses a storage medium, where a computer program is stored, and when the computer program is loaded and executed by a processor, the method steps executed in the process of scheduling the collection task disclosed in any of the foregoing embodiments are implemented.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The method, the apparatus, the device and the storage medium for scheduling the collection tasks provided by the present invention are described in detail above, and a specific example is applied in the present document to illustrate the principle and the implementation manner of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for scheduling a collection task is applied to a cache component and comprises the following steps:
acquiring a task to be processed generated by a system service flow;
determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance, and storing the task to be processed to the screened preset task set;
acquiring a scheduling instruction sent by a distributed task scheduling platform, then extracting a task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from the plurality of preset task sets;
and sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
2. The collection task scheduling method according to claim 1, wherein the acquiring the to-be-processed task generated by the system business process includes:
acquiring a task to be processed, which is obtained by assigning task parameters in a preset entity class in a system service flow; the task parameters in the entity class comprise task keywords, interface request parameters, retry times and expiration time;
correspondingly, the determining the task category of the task to be processed includes:
and determining the task category of the task to be processed based on the task keywords in the task to be processed.
3. The collection task scheduling method according to claim 1, wherein the process of screening a preset task set corresponding to the task category from a plurality of preset task sets created locally in advance and storing the task to be processed in the screened preset task set further includes:
if the plurality of preset task sets created locally in advance do not have the preset task set corresponding to the task type, an initialization task set corresponding to the task type is created, and the task to be processed is stored in the initialization task set.
4. The method of claim 1, wherein the obtaining of the scheduling instruction sent by the distributed task scheduling platform comprises:
the method comprises the steps of obtaining a scheduling instruction which is triggered by a distributed task scheduling platform according to a preset instruction triggering period and carries a task set identifier, wherein the scheduling instruction is generated by the distributed task scheduling platform by taking a task keyword as the task set identifier.
5. The method of claim 1, wherein sending the to-be-processed task in the target task set to the distributed task scheduling platform comprises:
and according to the preset single-batch task quantity sent by the distributed task scheduling platform, screening a corresponding quantity of tasks to be processed from the target task set, and sending the screened tasks to be processed to the distributed task scheduling platform.
6. The rendezvous task scheduling method according to any one of claims 1 to 5, wherein before the obtaining of the scheduling instruction sent by the distributed task scheduling platform, the method further comprises:
transmitting the statistical information of all the tasks to be processed which are cached locally at present to the distributed task scheduling platform so that the distributed task scheduling platform configures an actuator corresponding to each task to be processed according to the statistical information;
correspondingly, the sending the to-be-processed task in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received to-be-processed task, so as to execute a task processing flow corresponding to the received to-be-processed task through the external interface includes:
and sending the to-be-processed tasks in the target task set to the distributed task scheduling platform, so that the distributed task scheduling platform calls a pre-configured executor corresponding to the to-be-processed tasks, schedules an external interface corresponding to the received to-be-processed tasks through the executor, and executes a task processing flow corresponding to the received to-be-processed tasks through the external interface.
7. A method for scheduling a collection task is applied to a distributed task scheduling platform and comprises the following steps:
sending a scheduling instruction carrying a task set identifier to a cache component so that the cache component can determine a target task set determined by the task set identifier from a plurality of preset task sets created in advance locally;
acquiring the tasks to be processed in the target task set sent by the cache component;
and calling a pre-configured executor corresponding to the task to be processed, and scheduling an external interface corresponding to the received task to be processed through the executor so as to execute a task processing flow corresponding to the received task to be processed through the external interface.
8. An apparatus for scheduling aggregated tasks, applied to a cache component, includes:
the task acquisition module is used for acquiring a task to be processed generated by a system business process;
the task storage module is used for determining the task category of the task to be processed, screening a preset task set corresponding to the task category from a plurality of preset task sets which are created locally in advance, and storing the task to be processed to the screened preset task set;
the target task set determining module is used for acquiring a scheduling instruction sent by the distributed task scheduling platform, then extracting a task set identifier carried in the scheduling instruction, and determining a target task set determined by the task set identifier from the plurality of preset task sets;
and the task processing module is used for sending the tasks to be processed in the target task set to the distributed task scheduling platform so that the distributed task scheduling platform schedules an external interface corresponding to the received tasks to be processed, and executes a task processing flow corresponding to the received tasks to be processed through the external interface.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing said computer program for carrying out the steps of the rendezvous task scheduling method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program; wherein the computer program realizes the steps of the rendezvous task scheduling method according to any one of claims 1 to 7 when being executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111518958.6A CN114168297A (en) | 2021-12-13 | 2021-12-13 | Method, device, equipment and medium for scheduling collection tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111518958.6A CN114168297A (en) | 2021-12-13 | 2021-12-13 | Method, device, equipment and medium for scheduling collection tasks |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114168297A true CN114168297A (en) | 2022-03-11 |
Family
ID=80485962
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111518958.6A Pending CN114168297A (en) | 2021-12-13 | 2021-12-13 | Method, device, equipment and medium for scheduling collection tasks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114168297A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041353A (en) * | 2023-10-08 | 2023-11-10 | 北京小米移动软件有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN117105382A (en) * | 2023-09-23 | 2023-11-24 | 广东水清环保科技有限公司 | Industrial garden heavy metal wastewater treatment method, device, equipment and storage medium |
-
2021
- 2021-12-13 CN CN202111518958.6A patent/CN114168297A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117105382A (en) * | 2023-09-23 | 2023-11-24 | 广东水清环保科技有限公司 | Industrial garden heavy metal wastewater treatment method, device, equipment and storage medium |
| CN117105382B (en) * | 2023-09-23 | 2024-03-19 | 广东水清环保科技有限公司 | Industrial garden heavy metal wastewater treatment method, device, equipment and storage medium |
| CN117041353A (en) * | 2023-10-08 | 2023-11-10 | 北京小米移动软件有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN117041353B (en) * | 2023-10-08 | 2024-01-16 | 北京小米移动软件有限公司 | Task processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11005969B2 (en) | Problem solving in a message queuing system in a computer network | |
| CN109582466B (en) | Timed task execution method, distributed server cluster and electronic equipment | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| US8185624B2 (en) | Efficient on-demand provisioning of servers for specific software sets | |
| CN114168297A (en) | Method, device, equipment and medium for scheduling collection tasks | |
| CN113742031B (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| US20220201024A1 (en) | Modeling Topic-Based Message-Oriented Middleware within a Security System | |
| CN112738060B (en) | Method and device for processing micro-service data, micro-service processing platform and medium | |
| CN113886069A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN111984505A (en) | Operation and maintenance data acquisition engine and acquisition method | |
| CN107729213A (en) | A kind of background task monitoring method and device | |
| EP4024761A1 (en) | Communication method and apparatus for multiple management domains | |
| CN113760677A (en) | Abnormal link analysis method, device, equipment and storage medium | |
| CN109324892B (en) | Distributed management method, distributed management system and device | |
| CN114564249B (en) | Recommendation scheduling engine, recommendation scheduling method and computer readable storage medium | |
| CN112231073A (en) | Distributed task scheduling method and device | |
| CN114090268B (en) | Container management method and container management system | |
| CN112685157B (en) | Task processing method, device, computer equipment and storage medium | |
| CN114675954A (en) | Task scheduling method and device | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN114217867A (en) | Automatic operation and maintenance agent device, equipment and storage medium | |
| CN115686813A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN112416980A (en) | Data service processing method, device and equipment | |
| CN111400003A (en) | Task processing method and device | |
| CN111741097B (en) | Method for tenant to monopolize node, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |