CN108628677A - Distributed task scheduling processing system, method and apparatus - Google Patents
Distributed task scheduling processing system, method and apparatus Download PDFInfo
- Publication number
- CN108628677A CN108628677A CN201710166286.4A CN201710166286A CN108628677A CN 108628677 A CN108628677 A CN 108628677A CN 201710166286 A CN201710166286 A CN 201710166286A CN 108628677 A CN108628677 A CN 108628677A
- Authority
- CN
- China
- Prior art keywords
- task
- processing end
- monitoring information
- waiting
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
- Debugging And Monitoring (AREA)
Abstract
This application discloses distributed task scheduling processing system, method and apparatus.One specific implementation mode of the system includes：Task access end, for extracting waiting task from inactive queue of task, and according to the loading condition of at least one task processing end connected, selection target task processing end sends extracted waiting task from least one task processing end；Task processing end, waiting task for receiving task access end transmission, start the task execution process of received waiting task, and start the monitoring process being monitored to the task execution process, and the monitoring information that the monitoring process is generated is sent to the task access end.This embodiment improves task treatment effeciencies.
Description
Technical field
This application involves field of computer technology, and in particular at Internet technical field more particularly to distributed task scheduling
Reason system, method and apparatus.
Background technology
Nowadays system data increasingly increases, and business complexity is also higher and higher, thus for the processing capacity of system with
And the monitoring of data handling procedure also made higher requirement.In order to handle the efficient stable of data, from initial list
Point is serial to execute task, to distributed multithreading high concurrent processing.Doing queue scheduling from Java memories realizes distribution to dependence
Middleware implementation colony dispatching, being all will be efficiently to big data processing.
Rely on the message-oriented middleware structure producer, the communication module that consumer's message is route.Due to introducing among message
Part, the reliability of message-oriented middleware, fault-tolerance, robustness often become the deciding factor of this kind of framework；Meanwhile use is multi-thread
It is also possible to there is thread-safe in Cheng Zhihang.
Invention content
The purpose of the application is to propose a kind of distributed task scheduling processing system, method and apparatus, to solve background above
The technical issues of technology segment is mentioned.
In a first aspect, this application provides a kind of distributed task scheduling processing system, above system includes task access end, is used
In extracting waiting task from inactive queue of task, and according to the load feelings of at least one task processing end connected
Condition, selection target task processing end sends extracted waiting task from above-mentioned at least one task processing end；At task
End is managed, the waiting task sent for receiving above-mentioned task access end starts the task execution of received waiting task
Process, and start the monitoring process being monitored to above-mentioned task execution process, and above-mentioned monitoring process is generated
Monitoring information is sent to above-mentioned task access end.
In some embodiments, above-mentioned task access end is additionally operable to, according to above-mentioned monitoring information, determine above-mentioned pending
Whether business is completed, and is determining that above-mentioned waiting task is completed, is generated at task according to above-mentioned monitoring information
Manage result.
In some embodiments, above-mentioned task processing end is additionally operable to the abnormal end in response to above-mentioned task execution process,
The monitoring information that above-mentioned monitoring process is generated is sent to above-mentioned task access end, wherein above-mentioned monitoring information includes above-mentioned
Loading condition, the task execution progress information of business processing end；Above-mentioned task access end is additionally operable to receive above-mentioned monitoring information, and protects
Deposit above-mentioned task execution progress information；Above-mentioned task processing end is additionally operable in above-mentioned for receiving above-mentioned task access end and sending
After executive process information of being engaged in, using the above-mentioned task execution progress information received, task execution process is restarted.
In some embodiments, above-mentioned loading condition includes at least one of following：The memory usage of task processing end, net
The history processing speed of network rate and task.
Second aspect, this application provides a kind of distributed task scheduling processing method, the above method includes：From waiting task
Waiting task is extracted in queue；According to the loading condition of at least one task processing end connected, from above-mentioned at least one
Selection target task processing end sends extracted waiting task in task processing end, so that above-mentioned goal task processing end exists
After receiving waiting task, start the task execution process of received waiting task, and start and held to above-mentioned task
The monitoring process that traveling journey is monitored, and return to the monitoring information that above-mentioned monitoring process is generated；Believed according to above-mentioned monitoring
Breath generates the handling result of above extracted waiting task.
In some embodiments, the above-mentioned processing knot that above extracted waiting task is generated according to above-mentioned monitoring information
Fruit, including：It according to above-mentioned monitoring information, determines whether above-mentioned waiting task is completed, and is determining above-mentioned waiting task
In the case of completed, task handling result is generated according to above-mentioned monitoring information.
In some embodiments, monitoring information is received, wherein above-mentioned monitoring information is above-mentioned task processing end in response to upper
The abnormal end for stating task execution process sends the monitoring information that above-mentioned monitoring process is generated, and above-mentioned monitoring information includes upper
State loading condition, the task execution progress information of task processing end；In response to receiving above-mentioned monitoring information, preserves above-mentioned task and hold
Row progress information；Above-mentioned task execution progress information is sent to above-mentioned task processing end, so that the use of above-mentioned task processing end connects
The above-mentioned task execution progress information received restarts task execution process.
In some embodiments, above-mentioned loading condition includes at least one of following：The memory usage of task processing end, net
The history processing speed of network rate and task.
The third aspect, this application provides a kind of distributed task scheduling processing unit, above-mentioned apparatus includes：Extraction module is used
In extracting waiting task from inactive queue of task；Distribution module is selected, for according at least one task connected
The loading condition of processing end, from above-mentioned at least one task processing end selection target task processing end send extracted wait for from
Reason task starts holding for received waiting task so that above-mentioned goal task processing end is after receiving waiting task
Row task process, and start the monitoring process being monitored to above-mentioned task execution process, and return to above-mentioned monitoring process
The monitoring information generated；Generation module, the place for generating above extracted waiting task according to above-mentioned monitoring information
Manage result.
In some embodiments, above-mentioned apparatus further includes：Receiving module, for receiving monitoring information, wherein above-mentioned monitoring
Information is abnormal end of the above-mentioned task processing end in response to above-mentioned task execution process, sends the monitoring letter of above-mentioned monitoring process
Breath, above-mentioned monitoring information includes the loading condition of above-mentioned task processing end, task execution progress information；Memory module, for ringing
Above-mentioned task execution progress information should be preserved in receiving above-mentioned monitoring information；Sending module, for being sent out to above-mentioned task processing end
The above-mentioned task execution progress information preserved is sent, so that above-mentioned task processing end uses the above-mentioned task execution process received
Information restarts task execution process.
Distributed task scheduling system provided by the embodiments of the present application, method and apparatus, according to task processing end loading condition into
Different task amount is distributed in row selected objective target task processing end to different task processing end, and task processing end is in task processes
In, it applies multi-process and ensures that processing task is effectively run, improve processing task efficiency.
Description of the drawings
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon：
Fig. 1 is the exemplary architecture figure according to the distributed task scheduling processing system of the application；
Fig. 2 is the sequence diagram according to an embodiment of the distributed task scheduling processing system of the application；
Fig. 3 is the sequence diagram according to the another embodiment of the distributed task scheduling processing system of the application；
Fig. 4 is the flow chart according to an embodiment of the distributed task scheduling processing method of the application；
Fig. 5 is the structural schematic diagram according to an embodiment of the distributed task scheduling processing unit of the application；
Fig. 6 is the structural schematic diagram according to another embodiment of the distributed task scheduling processing unit of the application；
Fig. 7 is adapted for the structural schematic diagram of the computer system of the server for realizing the embodiment of the present application.
Specific implementation mode
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 shows the exemplary architecture 100 of the distributed task scheduling processing system according to the application.
As shown in Figure 1, system architecture 100 may include server 101,102,103, network 104 and server 105, clothes
Business device 106.As an example, server 101,102,103 can be task access end, server 105, server 106 can be
Task processing end.Network 104 between server 101,102,103 and server 105, server 106 providing communication chain
The medium on road.Network 104 may include various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be interacted server 101,102,103 by network 104 with server 105, server 106, to connect
Receive or send message etc..
Server 101,102,103 can be various types, provide the server of various services, such as database service
Device, high performance computing service device etc. can also be to be allocated task to server 105, server 106, send mission bit stream
Server.
Server 105, server 106 can be to provide the server of various services, for example, receive server 101,102,
103 sending for tasks, and feed back to server 101, server to the server that task is handled, and by task processing information
102, server 103.
It should be understood that the number of the network and server in Fig. 1 is only schematical.According to needs are realized, can have
There are any number of network and server.
With continued reference to Fig. 2, the sequence diagram of an embodiment of the distributed task scheduling processing system according to the application is shown.
The distributed task scheduling processing system of the present embodiment includes：Task access end and task processing end.Wherein, task is distributed
End, for extracting waiting task from inactive queue of task, and according to the negative of at least one task processing end connected
Situation is carried, selection target task processing end sends extracted waiting task from above-mentioned at least one task processing end；Appoint
Business processing end, the waiting task sent for receiving above-mentioned task access end, starts the task of received waiting task
Executive process, and start the monitoring process being monitored to above-mentioned task execution process, and above-mentioned monitoring process is given birth to
At monitoring information be sent to above-mentioned task access end.
Distributed task scheduling processing system task access end provided in this embodiment is according to the load feelings of above-mentioned task processing end
Condition distributes waiting task, so that above-mentioned task processing end carries out task processing, realizes effective resource utilization, and then improve
The efficiency of task processing.
As shown in Fig. 2, in step 201, task access end from inactive queue of task for extracting waiting task.
In the present embodiment, task access end extracts waiting task from inactive queue of task, at task
Reason end handles waiting task.
It should be noted that task access end can be there is task to obtain, the server of distribution function.
In the present embodiment, waiting task be stored in the form of queue lock in database etc. pending appoint
Business.In storing process, there are correspondences with task queue for task, can different types of task be stored in different queue
In.Such as：Type A tasks are stored in a queues, and type B task is stored in b queues.In practice, task queue is an elder generation
Enter the chained list task queue first gone out, the header elements of the task queue are i.e. earliest into the task of enqueue, the tail portion element of queue
The task of i.e. laggard enqueue.The waiting task of task access end extraction is the task on head in task queue.
In the present embodiment, above-mentioned database can be Redis databases, wherein Redis databases are one and increase income
Use ANSI C (standard about C language released by American National Standards Association ANSI and International Organization for standardization ISO) language
Speech write, support network, can based on memory also can persistence log type, distributed memory system database.Above-mentioned database
Can also be arbitrary database, as long as ensureing that the data stored in database are all unique in the database.
In the present embodiment, extraction task detailed process is：It is locked in database since waiting task is stored in, thus
Acquisition lock request operation is first carried out in task access end extraction task, is locked successfully if obtaining, obtains extraction waiting task permission,
Waiting task can be extracted；If obtaining lock failure, acquisition lock request operation is continued to execute, is locked successfully until obtaining.In this way may be used
To ensure that only there are one requests to run succeeded, it is equivalent to and ensures that a lock distributes to unique applicant, can ensure so not
Duplicate allocation task.
In step 202, task access end is according to the loading condition of at least one task processing end connected, from above-mentioned
Selection target task processing end at least one task processing end.
In the present embodiment, task access end is at least one according to what is connected after extracting pending task
The loading condition at pending end, the selection target task processing end from above-mentioned at least one task processing end, at the goal task
Reason end be used to receive waiting task, and handle task.
Task processing end can be the server for having certain processing function.
In some optional realization methods of the present embodiment, above-mentioned loading condition includes at least one of following：At task
Manage memory usage, the history processing speed of network rate and task at end.
Wherein, the history processing speed of task refers to the processing within a period before receiving waiting task
The speed degree of task.
In the present embodiment, select to load low task processing end from above-mentioned at least one task processing end as target
Task processing end.For example, task access end can select the task that memory usage is low in task processing end according to loading condition
Processing end is as goal task processing end；The task processing end that the history processing speed of task can also be selected relatively high is as mesh
Mark task processing end.
In step 203, the waiting task extracted is sent to goal task processing end by task access end.
In the present embodiment, task access end extracts waiting task, and the selection target task from task processing end
After processing end, waiting task is sent to goal task processing end, so that goal task processing end carries out task processing.
In step 204, goal task processing end receives the waiting task that above-mentioned task access end is sent, and startup is connect
The task execution process of the waiting task of receipts, and start the monitoring process being monitored to above-mentioned task execution process, on
It states monitoring process and generates monitoring information.
In the present embodiment, above-mentioned goal task processing end receives the waiting task that above-mentioned task access end is sent
Afterwards, according to the difference of task, different task execution processes is created.Above-mentioned task execution process be for handle task into
Journey, number, the loading condition comprehensive analysis of goal task processing end that goal task processing end can also be according to task amount, also may be used
Task is handled to create multiple tasks executive process simultaneously.After creating task execution process, each task execution process is opened
Move a monitoring process.Above-mentioned monitoring process is the process for monitoring task execution process situation in real time.Above-mentioned monitoring process can be with
Low memory is distributed, ensures normal operation.Monitoring process can generate monitoring information, and monitoring information includes the negative of task processing end
Carry situation and task execution progress information.Also, the monitoring information that monitoring process generates is sent to by task processing end in real time appoints
Business access end.
In some optional realization methods of the present embodiment, the task execution process of above-mentioned goal task processing end occurs
Abnormal conditions, task execution process terminate.The monitoring information that the monitoring process of above-mentioned goal task processing end generates, which is sent to, appoints
Business access end, after task access end receives above-mentioned monitoring information, and preserves the task execution progress information in monitoring information.When
The goal task processing end handle again task when, task access end by the monitoring information of preservation task execution process believe
Breath is sent to above-mentioned task processing end, which reruns task execution process.
In the task execution process normal operation of above-mentioned task processing end, need to modify to the Logic of Tasks, therefore,
Task execution process is terminated.The monitoring information that monitoring process generates is sent to task distribution by above-mentioned goal task processing end
End.After task access end receives above-mentioned monitoring information, preserved.After the Logic of Tasks is changed in the goal task processing end,
Task execution progress information in the monitoring information of preservation can be sent to above-mentioned task processing end, above-mentioned mesh by task access end
Mark task processing end can re-create a new task execution process according to task execution progress information and start the task
Executive process.
In step 205, task processing end sends the monitoring information that monitoring process generates.
In the present embodiment, the monitoring information that monitoring process generates is sent to task access end by above-mentioned task processing end,
Monitor the operating condition of task execution process and the machine loading situation of task processing end in real time in order to task access end.Example
Such as：When the task execution process exception of task processing end terminates, transmission monitoring information gives task access end, task access end real-time
Information is received, in order to carry out subsequent operation.
In step 206, task access end receives task processing end and sends monitoring information, and generates task handling result.
In the present embodiment, task access end receives the monitoring information of task processing end transmission, according to monitoring information, comes
It determines task performance, and generates task handling result.
In some optional realization methods of the present embodiment, task access end comes true according to the monitoring information received
Determine whether waiting task is completed, in the case where determining that waiting task has been completed, a task handling result can be generated.
Such as：The return value 1 for including task execution process inside the monitoring information received, passes through the return value 1, it can be determined that task
It has been completed that, information is completed according to this in task access end, generates a task handling result.
In the present embodiment, task access end receives the monitoring information of task processing end transmission, even if in task processing
End found the abnormal situation, and task execution process terminates suddenly, it is also ensured that not losing for task execution information also helps in this way
In the stability of processing task.
With continued reference to Fig. 3, the sequence diagram of an embodiment of the distributed task scheduling processing system according to the application is shown.
The distributed task scheduling processing system of the present embodiment includes：Task access end and task processing end；Wherein, task is distributed
End, for extracting waiting task from inactive queue of task, and according to the negative of at least one task processing end connected
Situation is carried, selection target task processing end sends extracted waiting task from above-mentioned at least one task processing end；Appoint
Business processing end, the waiting task sent for receiving above-mentioned task access end, starts the task of received waiting task
Executive process, and start the monitoring process being monitored to above-mentioned task execution process, and above-mentioned monitoring process is given birth to
At monitoring information be sent to above-mentioned task access end.
Above-mentioned goal task processing end is additionally operable to the abnormal end in response to above-mentioned task execution process, to above-mentioned task point
Originator sends the monitoring information that above-mentioned monitoring process is generated, wherein above-mentioned monitoring information includes the negative of above-mentioned task processing end
Carry situation, task execution progress information；
Above-mentioned task access end is additionally operable to receive above-mentioned monitoring information, and preserves above-mentioned task execution progress information；
Above-mentioned goal task processing end is additionally operable in the above-mentioned task execution process for receiving above-mentioned task access end transmission
After information, using the above-mentioned task execution progress information received, task execution process is restarted.
As shown in figure 3, in step 301, goal task processing end in response to above-mentioned task execution process abnormal end,
Monitoring information is sent to task access end.
In the present embodiment, there are abnormal conditions in the task execution process of goal task processing end, at above-mentioned goal task
End is managed to terminate in response to task execution process.The goal task end sends the monitoring letter of the monitoring process of above-mentioned task execution process
It ceases and gives task access end, in order to subsequently restart task execution process.Monitoring information is the information that monitoring process generates, prison
Control information includes the loading condition and task execution progress information of task processing end.Also, above-mentioned goal task processing end is real-time
The monitoring information that monitoring process generates is sent to task access end.As an example, can also be in the goal task processing end
Task execution process normal operation when, need to modify to the Logic of Tasks, therefore, task execution process is terminated.The mesh
The monitoring information that monitoring process generates is sent to task access end by mark task processing end.
In step 302, task access end receives monitoring information, and preserves monitoring information.
In the present embodiment, task access end receives the task execution for the above-mentioned termination that above-mentioned goal task processing end is sent
The monitoring information of the monitoring process of process, and monitoring information is preserved.
In step 303, task access end sends monitoring information and gives goal task processing end.
In the present embodiment, the monitoring information of preservation is sent to goal task processing end by task access end.In practice,
The goal task processing end re-execute task process when, task access end by the task execution in the monitoring information of preservation into
Journey information is sent to above-mentioned goal task processing end, in order to which task processing end executes subsequent operation.As an example, can also
It is in goal task processing end after changing the Logic of Tasks, task access end can be by the task execution in the monitoring information of preservation
Progress information is sent to goal task processing end.
In step 304, goal task processing end receives monitoring information and executes task treatment progress.
In the present embodiment, goal task processing end receives the task execution in the monitoring information of task access end transmission
After progress information, and task treatment progress is executed according to actual conditions.Such as：If it is the abnormal end of task process, the mesh
Mark task processing end can utilize the monitoring information received to re-execute task process；It is repaiied if it is due to the Logic of Tasks needs
Change, task execution process is terminated, then the goal task processing end can utilize the monitoring information received, re-creates one
A task execution process is handled for task.
With continued reference to Fig. 4, it illustrates the flows according to one embodiment of the distributed task scheduling processing method of the application
400.Server 101, server 102, the server of distributed task scheduling processing method shown in the present embodiment generally as shown in Figure 1
Any server executes in 103.
In step 401, waiting task is extracted from inactive queue of task.
In the present embodiment, electronic equipment (such as the service shown in FIG. 1 of distributed task scheduling processing method operation thereon
Any server is as task access end in device 101, server 102, server 103), above-mentioned electronic equipment is from pending
Waiting task is extracted in task queue.Above-mentioned waiting task be stored in the form of queue lock database it is medium pending
Task.In storing process, different types of task is stored in different queue.Such as：Type A tasks are stored in a queues
In, type B task is stored in b queues.In practice, task queue is the chained list task queue of a first in, first out, the task
The header elements of queue are i.e. earliest into the task of enqueue, the task of tail portion element, that is, laggard enqueue of queue.Task is distributed
End extraction task from locking database is the extraction task since queue head.
Above-mentioned database can be Redis databases, can also be arbitrary database, as long as being stored in above-mentioned database
Data be all unique in the database.It is locked in database since waiting task is stored in, thus task access end
Extraction task, which first has to execute, obtains lock request operation, lock successfully if obtaining, waiting task permission is extracted in acquisition, and then can be with
Extract waiting task；If obtaining lock failure, acquisition lock request operation is continued to execute, is locked successfully until obtaining.It can protect in this way
Only there are one requests to run succeeded for card, is equivalent to and ensures that a lock distributes to unique applicant, helps to ensure that do not weigh in this way
Subdivision matches task.
In step 402, according to the loading condition of at least one task processing end connected, from above-mentioned at least one
Selection target task processing end sends extracted waiting task in business processing end.
In the present embodiment, task access end is at least one according to what is connected after extracting pending task
The loading condition at pending end, the selection target task processing end from above-mentioned at least one task processing end, at the goal task
Reason end be used to receive waiting task, and handle task.
Goal task processing end can be have certain processing function server (such as server shown in FIG. 1 105,
A certain server is as goal task access end in server 106).
In some optional realization methods of the present embodiment, above-mentioned loading condition includes at least one of following：At task
Manage memory usage, the history processing speed of network rate and task at end.
In the present embodiment, select to load low task processing end from above-mentioned at least one task processing end as target
Task processing end.For example, task access end can select the task that memory usage is low in task processing end according to loading condition
Processing end is as goal task processing end；The task processing end that the history processing speed of task can also be selected relatively high is as mesh
Mark task processing end.
In step 403, monitoring information is received.
In the present embodiment, monitoring information is the information generated by monitoring process, and monitoring information includes task processing end
Loading condition and task execution progress information；Monitoring process is goal task processing end after receiving waiting task, according to
The difference of task creates different task execution processes, after creating task execution process, starts monitoring process and appoints to monitor
Business executive process situation.
In some optional realization methods of the present embodiment, there is exception in the task execution process of goal task processing end
Situation, task execution process terminate.The monitoring information that the monitoring process of the goal task processing end generates is sent to above-mentioned task
Access end after the task access end receives above-mentioned monitoring information, and preserves monitoring information.When the goal task processing end again
Processing task when, the task execution progress information in the monitoring information of preservation is sent at the goal task by task access end
End is managed, which reruns task execution process.
In the task execution process normal operation of above-mentioned goal task processing end, need to make an amendment the Logic of Tasks,
Above-mentioned task execution process is terminated.The monitoring information that monitoring process generates is sent to above-mentioned by above-mentioned goal task processing end
Business access end, after which receives above-mentioned monitoring information, and is preserved.When above-mentioned goal task processing end is changed
After the Logic of Tasks, the task execution progress information in the monitoring information of preservation can be sent to the target by above-mentioned task access end
Task processing end, the goal task processing end can re-create a task according to received task executive process information and hold
Traveling journey and the task execution process for starting the new establishment.
In step 404, the handling result of waiting task is generated according to monitoring information.
Task access end receives the monitoring information that goal task processing end is sent, when the task of the goal task processing end is held
There are abnormal conditions in traveling journey, which terminates suddenly, it can be ensured that and the task execution information is not lost, in this way,
Also contribute to the stability of processing task.
In some optional realization methods of the present embodiment, task access end comes true according to the monitoring information received
Determine whether waiting task is completed, in the case where determining that the waiting task has been completed, a task processing knot can be generated
Fruit.Such as：The return value 1 for including task execution process in the monitoring information received, passes through the return value 1, it can be determined that appoints
Business has been completed, and information is completed according to this in task access end, generates a task handling result.
With further reference to Fig. 5, as the realization to above-mentioned shown method, this application provides distributed task scheduling processing units
One embodiment, device embodiment embodiment of the method as shown in fig. 4 is corresponding, which specifically can be applied to various
In electronic equipment.
As shown in figure 5, the above-mentioned distributed task scheduling processing unit 500 of the present embodiment includes：Extraction module, selection distribution mould
Block and generation module；Wherein, extraction module 501 are configured to extract waiting task from inactive queue of task；Selection point
Module 502 is sent out, the loading condition according at least one task processing end connected is configured to, from above-mentioned at least one task
Selection target task processing end sends extracted waiting task in processing end, so that above-mentioned goal task processing end is receiving
To after waiting task, start received waiting task execution task process, and start to above-mentioned task execution into
The monitoring process that journey is monitored, and return to the monitoring information that above-mentioned monitoring process is generated；Generation module 503, configuration are used
In the handling result for generating extracted waiting task according to above-mentioned monitoring information.
In the present embodiment, the extraction module 501 in distributed task scheduling processing unit 500 can be from inactive queue of task
Middle extraction waiting task, which, which is stored in, locks in database, and extraction task, which first has to obtain, locks successfully, is locked
The waiting task in inactive queue of task can be extracted after success.Wherein, inactive queue of task is a kind of first to enter elder generation
Dequeue.The waiting task that extraction module 501 extracts is the waiting task being first stored in task queue.
In the present embodiment, above-mentioned locking database can be any type of database, as long as being stored in the database
Data be unique in the database.For example, Redis databases can be stored in, which is there are one each
Request can run succeeded, and be equivalent to and ensure that a lock distributes to unique applicant, also help to ensure that distribution task does not weigh in this way
It is multiple.
In the present embodiment, selection distribution module 502 is after extraction module 501 extracts waiting task, according to being connected
At least one task processing end loading condition, selected from above-mentioned at least one task processing end load it is low as target
Task processing end, and the waiting task extracted is sent to the goal task processing end.
In some optional realization methods of the present embodiment, distributed task scheduling processing unit further include receiving module 601,
Memory module 602 and sending module 603.Wherein, receiving module 601 are configured to receive monitoring information.Monitoring information is task
The information that processing end is sent.Monitoring information includes：The loading condition and task execution progress information of task processing end.Task processing
In task executive process there are abnormal conditions in end, which terminates, above-mentioned task processing end by the task execution into
The monitoring information that the monitoring process of journey generates is sent to task access end.Receiving module 601 receives above-mentioned monitoring in above-mentioned apparatus
Information, memory module 602 preserve the above-mentioned monitoring information received；Sending module 603 believes the task execution process preserved
Breath is sent to above-mentioned task processing end and sends the above-mentioned task execution progress information preserved, so that task processing end is using again
Start task execution process.
Can also be normal in task processing end task execution process in some optional realization methods of the present embodiment
It when operation, needs to do the Logic of Tasks and changes, above-mentioned task execution process is terminated, the monitoring that task processing end monitoring process generates
Information is sent to task access end, and task access end receiving module 601 receives above-mentioned monitoring information, and task execution process is believed
Breath preserve, when task processing end change the Logic of Tasks after, task access end sending module 603 by the monitoring information of preservation appoint
Business executive process information is sent to task processing end, and task processing end can newly create one according to task execution progress information and appoint
Business executive process simultaneously starts.
Below with reference to Fig. 7, it illustrates the computer systems 700 suitable for the server for realizing the embodiment of the present application
Structural schematic diagram.Terminal device/server shown in Fig. 7 is only an example, should not to the function of the embodiment of the present application and
Use scope brings any restrictions.
As shown in fig. 7, computer system 700 includes central processing unit (CPU) 701, it can be read-only according to being stored in
Program in memory (ROM) 702 or be loaded into the program in random access storage device (RAM) 703 from storage section 708 and
Execute various actions appropriate and processing.In RAM 703, also it is stored with system 700 and operates required various programs and data.
CPU 701, ROM702 and RAM 703 are connected with each other by bus 704.Input/output (I/O) interface 705 is also connected to always
Line 704.
It is connected to I/O interfaces 705 with lower component：Importation 706 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 707 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 708 including hard disk etc.；
And the communications portion 709 of the network interface card including LAN card, modem etc..Communications portion 709 via such as because
The network of spy's net executes communication process.Driver 710 is also according to needing to be connected to I/O interfaces 705.Detachable media 711, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 710, as needed in order to be read from thereon
Computer program be mounted into storage section 708 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed by communications portion 709 from network, and/or from detachable media
711 are mounted.When the computer program is executed by central processing unit (CPU) 701, limited in execution the present processes
Above-mentioned function.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for a part for one module, program segment, or code of table, the module, program segment, or code includes one or more uses
The executable instruction of the logic function as defined in realization.It should also be noted that in some implementations as replacements, being marked in box
The function of note can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are actually
It can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it to note
Meaning, the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart can be with holding
The dedicated hardware based system of functions or operations as defined in row is realized, or can use specialized hardware and computer instruction
Combination realize.
Being described in module involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be arranged in the processor, for example, can be described as：A kind of processor packet
Include extraction module, selection distribution module and generation module.Wherein, the title of these modules is not constituted to this under certain conditions
The restriction of module itself, for example, extraction module is also described as " module of extraction waiting task ".
As on the other hand, present invention also provides a kind of computer-readable medium, which can be
Included in device described in above-described embodiment；Can also be individualism, and without be incorporated the device in.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the device so that should
Device：Waiting task is extracted from inactive queue of task；According to the load feelings of at least one task processing end connected
Condition, selection target task processing end sends extracted waiting task from above-mentioned at least one task processing end, for upper
Goal task processing end is stated after receiving waiting task, starts the task execution process of received waiting task, and
And start the monitoring process being monitored to above-mentioned task execution process, and return to the monitoring letter that above-mentioned monitoring process is generated
Breath；The handling result of above extracted waiting task is generated according to above-mentioned monitoring information.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art
Member should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent feature
Other technical solutions of arbitrary combination and formation.Such as features described above has similar work(with (but not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (12)
1. a kind of distributed task scheduling processing system, which is characterized in that the system comprises：
Task access end, for extracting waiting task from inactive queue of task, and according at least one connected
The loading condition of business processing end, what the transmission of selection target task processing end was extracted from least one task processing end waits for
Processing task；
Task processing end, the waiting task sent for receiving the task access end, starts received waiting task
Task execution process, and start the monitoring process that is monitored to the task execution process, and by it is described monitor into
The monitoring information that journey is generated is sent to the task access end.
2. distributed task scheduling processing system according to claim 1, which is characterized in that
The task access end is additionally operable to, according to the monitoring information, determine whether the waiting task is completed, and true
In the case of the fixed waiting task is completed, task handling result is generated according to the monitoring information.
3. distributed task scheduling processing system according to claim 1, which is characterized in that
The task processing end is additionally operable to the abnormal end in response to the task execution process, is sent to the task access end
The monitoring information that the monitoring process is generated, wherein the monitoring information includes the loading condition of the task processing end, appoints
Business executive process information；
The task access end is additionally operable to receive the monitoring information, and preserves the task execution progress information；
The task processing end is additionally operable to after receiving the task execution progress information that the task access end is sent,
Using the task execution progress information received, task execution process is restarted.
4. the distributed task scheduling processing system according to one of claim 1-3, which is characterized in that
The loading condition includes at least one of following：The memory usage of task processing end, the history of network rate and task
Processing speed.
5. a kind of distributed task scheduling processing method, which is characterized in that the method includes：
Waiting task is extracted from inactive queue of task；
According to the loading condition of at least one task processing end connected, mesh is selected from least one task processing end
Mark task processing end sends extracted waiting task, so that the goal task processing end is receiving waiting task
Afterwards, start the task execution process of received waiting task, and start and the task execution process is monitored
Monitoring process, and return to the monitoring information that the monitoring process is generated；
The handling result of the extracted waiting task is generated according to the monitoring information.
6. distributed task scheduling processing method according to claim 5, which is characterized in that described to be given birth to according to the monitoring information
At the handling result of the extracted waiting task, including：
It according to the monitoring information, determines whether the waiting task is completed, and is determining that the waiting task is complete
In the case of, task handling result is generated according to the monitoring information.
7. distributed task scheduling processing method according to claim 5, which is characterized in that the method further includes：
Receive monitoring information, wherein the monitoring information is the goal task processing end in response to the task execution process
Abnormal end and monitoring information that the monitoring process that sends generates, the monitoring information includes the goal task processing
Loading condition, the task execution progress information at end；
In response to receiving the monitoring information, the task execution progress information is preserved；
The task execution progress information is sent to the goal task processing end, so that goal task processing end use connects
The task execution progress information received restarts task execution process.
8. distributed task scheduling processing method according to claim 5, which is characterized in that
The loading condition includes at least one of following：The memory usage of task processing end, the history of network rate and task
Processing speed.
9. a kind of distributed task scheduling processing unit, which is characterized in that described device includes：
Extraction module, for extracting waiting task from inactive queue of task；
Distribution module is selected, for the loading condition according at least one task processing end connected, from described at least one
Selection target task processing end sends extracted waiting task in task processing end, so that the goal task processing end exists
After receiving waiting task, start the execution task process of received waiting task, and start and held to the task
The monitoring process that traveling journey is monitored, and return to the monitoring information that the monitoring process is generated；
Generation module, the handling result for generating the extracted waiting task according to the monitoring information.
10. distributed task scheduling processing unit according to claim 9, which is characterized in that described device further includes：
Receiving module, for receiving monitoring information, wherein the monitoring information is the task processing end in response to the task
The abnormal end of executive process, sends the monitoring information of the monitoring process, and monitoring information includes the goal task processing end
Loading condition, task execution progress information；
Memory module, in response to the monitoring information, preserving the task execution progress information；
Sending module, for sending the task execution progress information preserved to the goal task processing end, for institute
Goal task processing end is stated using the task execution progress information received, restarts task execution process.
11. a kind of server, which is characterized in that the server includes：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors so that one or more of processors are real
The now method as described in any in claim 5-8.
12. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The method as described in any in claim 5-8 is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710166286.4A CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710166286.4A CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108628677A true CN108628677A (en) | 2018-10-09 |
| CN108628677B CN108628677B (en) | 2022-12-02 |
Family
ID=63687178
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710166286.4A Active CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108628677B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109656694A (en) * | 2018-11-02 | 2019-04-19 | 国网青海省电力公司 | A kind of distributed approach and system of energy storage monitoring data |
| CN109886327A (en) * | 2019-02-12 | 2019-06-14 | 北京奇艺世纪科技有限公司 | The processing system and method for Java data in a kind of distributed system |
| CN110187971A (en) * | 2019-05-30 | 2019-08-30 | 口碑(上海)信息技术有限公司 | Service request processing method and device |
| CN113127158A (en) * | 2019-12-30 | 2021-07-16 | 百度在线网络技术（北京）有限公司 | Method and device for executing data processing task |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941805B2 (en) * | 2006-08-15 | 2011-05-10 | International Business Machines Corporation | Affinity dispatching load balancer with precise CPU consumption data |
| CN103246592A (en) * | 2013-05-13 | 2013-08-14 | 北京搜狐新媒体信息技术有限公司 | Monitoring acquisition system and method |
| CN104184823A (en) * | 2014-09-05 | 2014-12-03 | 北京京东尚科信息技术有限公司 | Automatic task scheduling method and system |
| CN104360923A (en) * | 2014-11-03 | 2015-02-18 | 中国银行股份有限公司 | Monitoring method and monitoring system for batch application process |
| CN105893147A (en) * | 2016-03-29 | 2016-08-24 | 乐视控股（北京）有限公司 | Multi-task queue management method, equipment and system |
| CN106227596A (en) * | 2016-07-13 | 2016-12-14 | 百度在线网络技术（北京）有限公司 | Mission Monitor method and apparatus for task scheduling server |
| CN106406983A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Task scheduling method and device in cluster |
-
2017
- 2017-03-20 CN CN201710166286.4A patent/CN108628677B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941805B2 (en) * | 2006-08-15 | 2011-05-10 | International Business Machines Corporation | Affinity dispatching load balancer with precise CPU consumption data |
| CN103246592A (en) * | 2013-05-13 | 2013-08-14 | 北京搜狐新媒体信息技术有限公司 | Monitoring acquisition system and method |
| CN104184823A (en) * | 2014-09-05 | 2014-12-03 | 北京京东尚科信息技术有限公司 | Automatic task scheduling method and system |
| CN104360923A (en) * | 2014-11-03 | 2015-02-18 | 中国银行股份有限公司 | Monitoring method and monitoring system for batch application process |
| CN106406983A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Task scheduling method and device in cluster |
| CN105893147A (en) * | 2016-03-29 | 2016-08-24 | 乐视控股（北京）有限公司 | Multi-task queue management method, equipment and system |
| CN106227596A (en) * | 2016-07-13 | 2016-12-14 | 百度在线网络技术（北京）有限公司 | Mission Monitor method and apparatus for task scheduling server |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109656694A (en) * | 2018-11-02 | 2019-04-19 | 国网青海省电力公司 | A kind of distributed approach and system of energy storage monitoring data |
| CN109886327A (en) * | 2019-02-12 | 2019-06-14 | 北京奇艺世纪科技有限公司 | The processing system and method for Java data in a kind of distributed system |
| CN110187971A (en) * | 2019-05-30 | 2019-08-30 | 口碑(上海)信息技术有限公司 | Service request processing method and device |
| CN110187971B (en) * | 2019-05-30 | 2020-08-04 | 口碑(上海)信息技术有限公司 | Service request processing method and device |
| CN113127158A (en) * | 2019-12-30 | 2021-07-16 | 百度在线网络技术（北京）有限公司 | Method and device for executing data processing task |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108628677B (en) | 2022-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113641457B (en) | Container creation method, device, apparatus, medium, and program product | |
| KR100322724B1 (en) | Apparatus and method for scheduling and dispatching queued client requests within a server in a client/server computer system | |
| EP0849921B1 (en) | Apparatus and method for dispatching client method calls within a server computer system | |
| CN108628677A (en) | Distributed task scheduling processing system, method and apparatus | |
| CN112667414A (en) | Message queue-based message consumption method and device, computer equipment and medium | |
| CN106802826A (en) | A kind of method for processing business and device based on thread pool | |
| US8984530B2 (en) | Queued message dispatch | |
| US10693816B2 (en) | Communication methods and systems, electronic devices, and computer clusters | |
| CN109726004B (en) | Data processing method and device | |
| CN108228349A (en) | For handling the method for task, system and storage medium | |
| US9448861B2 (en) | Concurrent processing of multiple received messages while releasing such messages in an original message order with abort policy roll back | |
| CN111126895A (en) | Management warehouse and scheduling method for scheduling intelligent analysis algorithm in complex scene | |
| US9104486B2 (en) | Apparatuses, systems, and methods for distributed workload serialization | |
| US20180329814A1 (en) | Method and system for automated storage provisioning | |
| US9860307B2 (en) | System and method for virtual desktop infrastructure user level virtual channel | |
| CN109582445A (en) | Message treatment method, device, electronic equipment and computer readable storage medium | |
| CN111984505A (en) | Operation and maintenance data acquisition engine and acquisition method | |
| CN113626173B (en) | Scheduling method, scheduling device and storage medium | |
| CN113051049B (en) | Task scheduling system, method, electronic device and readable storage medium | |
| Sai et al. | Producer-Consumer problem using Thread pool | |
| CN116932147A (en) | Streaming job processing method and device, electronic equipment and medium | |
| CN111209311A (en) | Method and apparatus for processing data | |
| CN109829005A (en) | A kind of big data processing method and processing device | |
| CN111538491B (en) | Data event processing method, device, equipment and storage medium | |
| CN114385351A (en) | Cloud management platform load balancing performance optimization method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |