CN108628677B - Distributed task processing system, method and device - Google Patents
Distributed task processing system, method and device Download PDFInfo
- Publication number
- CN108628677B CN108628677B CN201710166286.4A CN201710166286A CN108628677B CN 108628677 B CN108628677 B CN 108628677B CN 201710166286 A CN201710166286 A CN 201710166286A CN 108628677 B CN108628677 B CN 108628677B
- Authority
- CN
- China
- Prior art keywords
- task
- monitoring
- processed
- execution process
- task processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Multi Processors (AREA)
Abstract
The application discloses a distributed task processing system, a method and a device. One embodiment of the system includes: the task distributing terminal is used for extracting the tasks to be processed from the task queue to be processed, and selecting a target task processing terminal from at least one task processing terminal to send the extracted tasks to be processed according to the load condition of the at least one task processing terminal; the task processing terminal is used for receiving the to-be-processed task sent by the task sending terminal, starting a task execution process of the received to-be-processed task, starting a monitoring process for monitoring the task execution process, and sending monitoring information generated by the monitoring process to the task sending terminal. The embodiment improves the task processing efficiency.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to the field of internet technologies, and in particular, to a distributed task processing system, method, and apparatus.
Background
Nowadays, the system data is increasingly larger and the service complexity is higher, so that higher requirements are made on the processing capacity of the system and the monitoring of the data processing process. For efficient stability of processing data, tasks are executed serially from an initial single point to distributed multi-threaded high-concurrency processing. The cluster scheduling is realized from the Java memory queue scheduling realization of distributed type to the dependent middleware realization, and the efficient processing of the big data is required.
And a communication module for constructing the routing of the producer and consumer messages by relying on the message middleware. Due to the introduction of the message middleware, the reliability, fault tolerance and robustness of the message middleware often become the decisive factors of the architecture; also, there may be thread safety issues with multi-threaded execution.
Disclosure of Invention
The present application is directed to a distributed task processing system, method and device, which solve the technical problems mentioned in the background section above.
In a first aspect, the present application provides a distributed task processing system, where the system includes a task distributing end, configured to extract a task to be processed from a task queue to be processed, and select a target task processing end from at least one task processing end to send the extracted task to be processed according to a load condition of the at least one task processing end connected to the target task distributing end; and the task processing terminal is used for receiving the to-be-processed task sent by the task distribution terminal, starting a task execution process of the received to-be-processed task, starting a monitoring process for monitoring the task execution process, and sending monitoring information generated by the monitoring process to the task distribution terminal.
In some embodiments, the task distributing end is further configured to determine whether the task to be processed is completed according to the monitoring information, and generate a task processing result according to the monitoring information when it is determined that the task to be processed is completed.
In some embodiments, the task processing end is further configured to send, to the task distribution end, monitoring information generated by the monitoring process in response to an abnormal termination of the task execution process, where the monitoring information includes a load condition of the task processing end and task execution process information; the task distributing end is also used for receiving the monitoring information and storing the task execution progress information; the task processing terminal is further configured to restart the task execution process by using the received task execution process information after receiving the task execution process information sent by the task distribution terminal.
In some embodiments, the load condition comprises at least one of: the memory occupancy rate of the task processing terminal, the network rate and the historical processing rate of the task.
In a second aspect, the present application provides a distributed task processing method, where the method includes: extracting a task to be processed from a task queue to be processed; according to the load condition of at least one connected task processing end, selecting a target task processing end from the at least one task processing end to send the extracted task to be processed, so that the target task processing end starts a task execution process of the received task to be processed after receiving the task to be processed, starts a monitoring process for monitoring the task execution process and returns monitoring information generated by the monitoring process; and generating a processing result of the extracted task to be processed according to the monitoring information.
In some embodiments, the generating a processing result of the extracted task to be processed according to the monitoring information includes: and determining whether the task to be processed is completed or not according to the monitoring information, and generating a task processing result according to the monitoring information under the condition that the task to be processed is determined to be completed.
In some embodiments, monitoring information is received, where the monitoring information is generated by the monitoring process in response to an abnormal termination of the task execution process by the task processing end, and the monitoring information includes a load condition of the task processing end and task execution process information; in response to receiving the monitoring information, saving the task execution process information; and sending the task execution process information to the task processing end so that the task processing end can restart the task execution process by using the received task execution process information.
In some embodiments, the load condition comprises at least one of: the memory occupancy rate of the task processing terminal, the network rate and the historical processing rate of the task.
In a third aspect, the present application provides a distributed task processing apparatus, where the apparatus includes: the extraction module is used for extracting the tasks to be processed from the task queue to be processed; the selection distribution module is used for selecting a target task processing end from the at least one task processing end to send the extracted task to be processed according to the load condition of the connected at least one task processing end, so that the target task processing end starts a task execution process of the received task to be processed after receiving the task to be processed, starts a monitoring process for monitoring the task execution process and returns monitoring information generated by the monitoring process; and the generating module is used for generating the processing result of the extracted task to be processed according to the monitoring information.
In some embodiments, the above apparatus further comprises: a receiving module, configured to receive monitoring information, where the monitoring information is monitoring information sent by the task processing end in response to an abnormal termination of the task execution process, and the monitoring information includes a load condition of the task processing end and task execution process information; the storage module is used for responding to the received monitoring information and storing the task execution progress information; and the sending module is used for sending the saved task execution process information to the task processing end so that the task processing end can restart the task execution process by using the received task execution process information.
According to the distributed task system, the distributed task method and the distributed task device, the target task processing end is optimized according to the load condition of the task processing end, different task quantities are distributed to different task processing ends, the task processing end ensures effective operation of processing tasks by applying multiple processes in the task processing process, and the task processing efficiency is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary architecture diagram of a distributed task processing system according to the present application;
FIG. 2 is a timing diagram of an embodiment of a distributed task processing system according to the present application;
FIG. 3 is a timing diagram of yet another embodiment of a distributed task processing system according to the present application;
FIG. 4 is a flow diagram of one embodiment of a distributed task processing method according to the present application;
FIG. 5 is a schematic block diagram of an embodiment of a distributed task processing apparatus according to the present application;
FIG. 6 is a schematic block diagram of another embodiment of a distributed task processing apparatus according to the present application;
FIG. 7 is a block diagram of a computer system suitable for use in implementing a server according to embodiments of the present application.
Detailed Description
It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
FIG. 1 illustrates an exemplary architecture 100 of a distributed task processing system according to the present application.
As shown in fig. 1, system architecture 100 may include servers 101, 102, 103, network 104 and servers 105, 106. By way of example, the servers 101, 102, 103 may be task distribution terminals, and the servers 105, 106 may be task processing terminals. Network 104 is the medium used to provide communication links between servers 101, 102, 103 and servers 105, 106. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may interact with servers 101, 102, 103, via network 104, with server 105, server 106, to receive or send messages, etc.
The servers 101, 102, and 103 may be various types of servers that provide various services, such as a database server, a high-performance computing server, and the like, or may be servers that distribute tasks and transmit task information to the server 105 and the server 106.
The servers 105 and 106 may be servers that provide various services, for example, servers that receive and process tasks transmitted by the servers 101, 102, and 103, and feed back task processing information to the servers 101, 102, and 103.
It should be understood that the number of networks and servers in fig. 1 is merely illustrative. There may be any number of networks and servers, as desired for implementation.
With continued reference to FIG. 2, a timing diagram of an embodiment of a distributed task processing system according to the present application is shown.
The distributed task processing system of the embodiment includes: a task distributing end and a task processing end. The task distributing terminal is used for extracting tasks to be processed from the task queue to be processed, and selecting a target task processing terminal from at least one task processing terminal to send the extracted tasks to be processed according to the load condition of the at least one task processing terminal connected with the task distributing terminal; and the task processing terminal is used for receiving the to-be-processed task sent by the task distribution terminal, starting a task execution process of the received to-be-processed task, starting a monitoring process for monitoring the task execution process, and sending monitoring information generated by the monitoring process to the task distribution terminal.
The task distributing end of the distributed task processing system provided by this embodiment distributes the task to be processed according to the load condition of the task processing end, so that the task processing end performs task processing, thereby achieving effective resource utilization rate and further improving the efficiency of task processing.
As shown in fig. 2, in step 201, the task distributor is configured to extract the to-be-processed task from the to-be-processed task queue.
In this embodiment, the task distribution end extracts the to-be-processed task from the to-be-processed task queue, so that the task processing end processes the to-be-processed task.
It should be noted that the task distributing end may be a server with task acquiring and distributing functions.
In this embodiment, the pending task is a task waiting for processing that is stored in the locking database in a queue. In the storage process, the corresponding relation exists between the tasks and the task queues, and different types of tasks can be stored in different queues. For example: the type A tasks are stored in the queue a, and the type B tasks are stored in the queue B. In practice, the task queue is a first-in first-out linked list task queue, the head element of the task queue is the task entering the queue at the earliest time, and the tail element of the task queue is the task entering the queue at the later time. The task to be processed extracted by the task distributing terminal is the task at the head of the task queue.
In this embodiment, the database may be a Redis database, wherein the Redis database is an open-source database written in ANSI C (a standard for C language proposed by american national standards institute ANSI and international organization for standardization ISO) language, supporting network, and may be a memory-based or persistent database of a log-type, distributed storage system. The database may be any database as long as it is ensured that the data stored in the database is unique in the database.
In this embodiment, the specific process of extracting the task is as follows: the task to be processed is stored in the locking database, so that the task distributing end extracts the task and firstly executes the operation of acquiring the lock request, if the lock is successfully acquired, the permission of extracting the task to be processed is obtained, and the task to be processed can be extracted; and if the lock acquisition fails, continuing to execute the lock acquisition request operation until the lock acquisition is successful. This ensures that only one request can be successfully executed, which is equivalent to ensuring that the lock is assigned to a unique applicant, which ensures that the assignment task is not repeated.
In step 202, the task distributing terminal selects a target task processing terminal from the at least one task processing terminal according to a load condition of the at least one task processing terminal connected thereto.
In this embodiment, after the task distribution end extracts the task to be processed, a target task processing end is selected from the at least one task processing end according to a load condition of the connected at least one task processing end, and the target task processing end is used for receiving the task to be processed and processing the task.
The task processing end can be a server with certain processing functions.
In some optional implementations of this embodiment, the load condition includes at least one of: the memory occupancy rate of the task processing end, the network rate and the historical processing rate of the task.
The historical processing rate of the task refers to the speed of processing the task in a time period before the task to be processed is received.
In this embodiment, a task processing side with a low load is selected from the at least one task processing side as a target task processing side. For example, the task distributing end may select a task processing end with a low memory occupancy rate from the task processing ends as a target task processing end according to a load condition; or selecting a task processing end with a higher historical processing rate of the task as the target task processing end.
In step 203, the task distributing terminal sends the extracted task to be processed to the target task processing terminal.
In this embodiment, after the task distributing end extracts the task to be processed and selects the target task processing end from the task processing ends, the task to be processed is sent to the target task processing end, so that the target task processing end performs task processing.
In step 204, the target task processing end receives the to-be-processed task sent by the task distributing end, starts a task execution process of the received to-be-processed task, and starts a monitoring process for monitoring the task execution process, where the monitoring process generates monitoring information.
In this embodiment, after receiving the to-be-processed task sent by the task distributing terminal, the target task processing terminal creates different task execution processes according to different tasks. The task execution process is a process for processing tasks, and the target task processing end can also comprehensively analyze the load condition of the target task processing end according to the quantity of the tasks and the load condition of the target task processing end, and can also create a plurality of task execution process processing tasks at the same time. After the task execution processes are created, a monitor process is started for each task execution process. The monitoring process is a process for monitoring the process status of the task execution in real time. The monitoring process can allocate low memory and ensure normal operation. The monitoring process generates monitoring information, and the monitoring information comprises the load condition of the task processing terminal and the task execution process information. And the task processing terminal sends the monitoring information generated by the monitoring process to the task distribution terminal in real time.
In some optional implementation manners of this embodiment, when an exception occurs in the task execution process at the target task processing end, the task execution process is terminated. And the monitoring information generated by the monitoring process of the target task processing end is sent to the task distributing end, and the task distributing end stores the task execution process information in the monitoring information after receiving the monitoring information. When the target task processing terminal reprocesses the task, the task distributing terminal sends the task execution progress information in the stored monitoring information to the task processing terminal, and the target task processing terminal reruns the task execution progress.
When the task execution process at the task processing end normally runs, the task logic needs to be modified, and therefore, the task execution process is terminated. And the target task processing terminal sends the monitoring information generated by the monitoring process to the task distribution terminal. And the task distributing terminal receives the monitoring information and stores the monitoring information. After the target task processing end modifies the task logic, the task distribution end can send the task execution progress information in the stored monitoring information to the task processing end, and the target task processing end can recreate a new task execution progress according to the task execution progress information and start the task execution progress.
In step 205, the task processing side sends monitoring information generated by the monitoring process.
In this embodiment, the task processing end sends the monitoring information generated by the monitoring process to the task distribution end, so that the task distribution end monitors the running condition of the task execution process and the machine load condition of the task processing end in real time. For example: when the task execution process of the task processing end is abnormally terminated, the monitoring information is sent to the task distributing end, and the task distributing end receives the information in real time so as to facilitate subsequent operation.
In step 206, the task distributing end receives the monitoring information sent by the task processing end and generates a task processing result.
In this embodiment, the task distributing end receives the monitoring information sent by the task processing end, determines the task completion condition according to the monitoring information, and generates a task processing result.
In some optional implementation manners of this embodiment, the task distribution end determines whether the task to be processed is completed according to the received monitoring information, and generates a task processing result when it is determined that the task to be processed is completed. For example: the received monitoring information contains a return value 1 of the task execution process, the task can be judged to be completed through the return value 1, and the task distribution end generates a task processing result according to the completed information.
In this embodiment, the task distribution end receives the monitoring information sent by the task processing end, and even if the task processing end finds an abnormal condition and the task execution process is suddenly terminated, the task distribution end can ensure that the task execution information is not lost, which is also beneficial to the stability of processing tasks.
With continued reference to FIG. 3, a timing diagram of an embodiment of a distributed task processing system according to the present application is shown.
The distributed task processing system of the embodiment includes: a task distributing end and a task processing end; the task distributing terminal is used for extracting tasks to be processed from the task queue to be processed, and selecting a target task processing terminal from at least one task processing terminal to send the extracted tasks to be processed according to the load condition of the at least one task processing terminal connected with the task distributing terminal; and the task processing terminal is used for receiving the to-be-processed task sent by the task sending terminal, starting a task execution process of the received to-be-processed task, starting a monitoring process for monitoring the task execution process, and sending monitoring information generated by the monitoring process to the task sending terminal.
The target task processing terminal is further configured to send monitoring information generated by the monitoring process to the task distributing terminal in response to the abnormal termination of the task execution process, where the monitoring information includes a load condition of the task processing terminal and task execution process information;
the task distributing terminal is also used for receiving the monitoring information and storing the task execution progress information;
and the target task processing terminal is also used for restarting the task execution process by using the received task execution process information after receiving the task execution process information sent by the task distributing terminal.
As shown in fig. 3, in step 301, the target task processing end sends monitoring information to the task distributing end in response to the abnormal termination of the task execution process.
In this embodiment, when an exception occurs in the task execution process of the target task processing end, the target task processing end terminates in response to the task execution process. And the target task terminal sends the monitoring information of the monitoring process of the task execution process to the task distribution terminal so as to restart the task execution process subsequently. The monitoring information is information generated by a monitoring process, and the monitoring information comprises the load condition of the task processing terminal and the task execution process information. And the target task processing terminal sends the monitoring information generated by the monitoring process to the task distributing terminal in real time. As an example, when the task execution process at the target task processing end runs normally, the task logic may need to be modified, and therefore, the task execution process is terminated. And the target task processing terminal sends the monitoring information generated by the monitoring process to the task distributing terminal.
In step 302, the task distributing terminal receives the monitoring information and stores the monitoring information.
In this embodiment, the task distribution end receives the monitoring information of the monitoring process of the terminated task execution process sent by the target task processing end, and stores the monitoring information.
In step 303, the task distributing terminal sends the monitoring information to the target task processing terminal.
In this embodiment, the task distributing end sends the stored monitoring information to the target task processing end. In practice, when the target task processing end re-executes the task process, the task distributing end sends the task execution process information in the stored monitoring information to the target task processing end, so that the task processing end can execute subsequent operations conveniently. As an example, after the target task processing end modifies the task logic, the task distribution end may send the task execution progress information in the stored monitoring information to the target task processing end.
In step 304, the target task processing terminal receives the monitoring information and executes the task processing process.
In this embodiment, after receiving the task execution process information in the monitoring information sent by the task distributing terminal, the target task processing terminal executes the task processing process according to the actual situation. For example: if the task process is abnormally terminated, the target task processing end can utilize the received monitoring information to execute the task process again; if the task execution process is terminated because the task logic needs to be modified, the target task processing terminal can recreate a task execution process for task processing by using the received monitoring information.
With continued reference to FIG. 4, a flow 400 of one embodiment of a distributed task processing method according to the present application is shown. The distributed task processing method shown in the present embodiment is generally executed by any one of the server 101, the server 102, and the server 103 shown in fig. 1.
In step 401, a to-be-processed task is extracted from a to-be-processed task queue.
In this embodiment, an electronic device (for example, any one of the server 101, the server 102, and the server 103 shown in fig. 1 as a task distribution terminal) on which the distributed task processing method operates extracts a task to be processed from a task queue to be processed. The pending tasks are pending tasks stored in the locked database in a queue. And in the storage process, different types of tasks are stored in different queues. For example: the type A task is stored in the queue a, and the type B task is stored in the queue B. In practice, the task queue is a linked list task queue with first-in first-out, the head element of the task queue is the task entering the queue at the earliest time, and the tail element of the queue is the task entering the queue at the later time. The task dispatching end extracts the task from the locking database by starting to extract the task from the head of the queue.
The database may be a Redis database, or may be any database as long as the data stored in the database is unique in the database. The task to be processed is stored in the locking database, so that the task distribution end extracts the task and firstly executes the operation of acquiring the lock request, if the lock is successfully acquired, the permission of extracting the task to be processed is obtained, and the task to be processed can be extracted; and if the lock acquisition fails, continuing to execute the lock acquisition request operation until the lock acquisition is successful. This ensures that only one request can be successfully executed, which is equivalent to ensuring that the lock is assigned to a unique applicant, which helps to ensure that the assignment task is not repeated.
In step 402, according to the load condition of the at least one task processing end connected to the task processing end, a target task processing end is selected from the at least one task processing end to send the extracted task to be processed.
In this embodiment, after extracting the task to be processed, the task distribution end selects a target task processing end from the at least one task processing end according to a load condition of the connected at least one task processing end, where the target task processing end is used to receive the task to be processed and process the task.
The target task processing side may be a server with certain processing functions (for example, one of the server 105 and the server 106 shown in fig. 1 is used as the target task distributing side).
In some optional implementations of the embodiment, the load condition includes at least one of: the memory occupancy rate of the task processing terminal, the network rate and the historical processing rate of the task.
In this embodiment, a task processing side with a low load is selected from the at least one task processing side as a target task processing side. For example, the task distributing end may select a task processing end with a low memory occupancy rate from the task processing ends as a target task processing end according to a load condition; a task processing end with a relatively high historical processing rate of the task may also be selected as the target task processing end.
In step 403, monitoring information is received.
In this embodiment, the monitoring information is information generated by a monitoring process, and the monitoring information includes a load condition of a task processing end and task execution process information; the monitoring process is that after the target task processing terminal receives the task to be processed, different task execution processes are created according to different tasks, and after the task execution processes are created, the monitoring process is started to monitor the status of the task execution processes.
In some optional implementation manners of this embodiment, when an abnormal condition occurs in the task execution process at the target task processing end, the task execution process is terminated. And the task distribution end receives the monitoring information and stores the monitoring information. When the target task processing end processes the task again, the task distributing end sends the task execution progress information in the stored monitoring information to the target task processing end, and the target task processing end operates the task execution progress again.
When the task execution process of the target task processing terminal normally runs, the task logic needs to be modified, and the task execution process is terminated. And the target task processing terminal sends the monitoring information generated by the monitoring process to the task distribution terminal, and the task distribution terminal receives and stores the monitoring information. After the target task processing end modifies the task logic, the task distributing end may send the task execution process information in the stored monitoring information to the target task processing end, and the target task processing end may re-create a task execution process according to the received task execution process information and start the newly created task execution process.
In step 404, a processing result of the task to be processed is generated according to the monitoring information.
The task distributing end receives the monitoring information sent by the target task processing end, and when the task execution progress of the target task processing end is abnormal, the task execution progress is suddenly stopped, so that the task execution information can be ensured not to be lost, and the stability of processing the task is also facilitated.
In some optional implementation manners of this embodiment, the task distribution terminal determines whether the to-be-processed task is completed according to the received monitoring information, and generates a task processing result when it is determined that the to-be-processed task is completed. For example: the received monitoring information contains a return value 1 of the task execution process, the task can be judged to be completed through the return value 1, and the task distributing end generates a task processing result according to the completed information.
With further reference to fig. 5, as an implementation of the method shown above, the present application provides an embodiment of a distributed task processing apparatus, which corresponds to the method embodiment shown in fig. 4, and which can be applied to various electronic devices.
As shown in fig. 5, the distributed task processing apparatus 500 of the present embodiment includes: the device comprises an extraction module, a selection distribution module and a generation module; the extracting module 501 is configured to extract a to-be-processed task from a to-be-processed task queue; a selection distribution module 502 configured to select a target task processing end from the at least one task processing end to send the extracted to-be-processed task according to a load condition of the connected at least one task processing end, so that the target task processing end starts a task execution process of the received to-be-processed task after receiving the to-be-processed task, starts a monitoring process for monitoring the task execution process, and returns monitoring information generated by the monitoring process; the generating module 503 is configured to generate a processing result of the extracted task to be processed according to the monitoring information.
In this embodiment, the extracting module 501 in the distributed task processing apparatus 500 may extract the to-be-processed task from the to-be-processed task queue, where the task queue is stored in the locking database, the to-be-processed task in the to-be-processed task queue may be extracted after the task queue is successfully locked. The pending task queue is a first-in first-out queue. The to-be-processed task extracted by the extraction module 501 is a to-be-processed task stored in a task queue first.
In this embodiment, the locking database may be any database as long as the data stored in the database is unique in the database. For example, it may be stored in a Redis database where only one request at a time can be successfully executed, which is equivalent to ensuring that a lock is assigned to a unique applicant, which also helps to ensure that the assignment task is not duplicated.
In this embodiment, after the to-be-processed task is extracted by the extraction module 501, the selection distribution module 502 selects, according to the load condition of at least one task processing end connected thereto, a low-load task from the at least one task processing end as a target task processing end, and sends the extracted to-be-processed task to the target task processing end.
In some optional implementations of this embodiment, the distributed task processing apparatus further includes a receiving module 601, a storing module 602, and a sending module 603. The receiving module 601 is configured to receive monitoring information. The monitoring information is information sent by the task processing terminal. The monitoring information includes: load condition of the task processing terminal and task execution progress information. When the task execution progress is abnormal, the task execution progress is terminated, and the task processing end sends monitoring information generated by the monitoring progress of the task execution progress to the task distribution end. The receiving module 601 in the device receives the monitoring information, and the storage module 602 stores the received monitoring information; the sending module 603 sends the saved task execution process information to the task processing end to send the saved task execution process information, so that the task processing end restarts the task execution process.
In some optional implementation manners of this embodiment, it may also be that, when the task processing end task execution process runs normally, a task logic needs to be changed, the task execution process is terminated, monitoring information generated by the task processing end monitoring process is sent to the task distribution end, the task distribution end receiving module 601 receives the monitoring information and stores task execution process information, after the task processing end modifies the task logic, the task distribution end sending module 603 sends task execution process information in the stored monitoring information to the task processing end, and the task processing end may newly create a task execution process according to the task execution process information and start the task execution process.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use in implementing a server according to embodiments of the present application. The terminal device/server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that the computer program read out therefrom is mounted in the storage section 708 as necessary.
In particular, the processes described above with reference to the flow diagrams may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the method of the present application when executed by the Central Processing Unit (CPU) 701.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an extraction module, a selection distribution module, and a generation module. The names of the modules do not constitute a limitation to the module itself in some cases, and for example, the extraction module may also be described as a "module that extracts a task to be processed".
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: extracting a task to be processed from a task queue to be processed; according to the load condition of at least one connected task processing end, selecting a target task processing end from the at least one task processing end to send the extracted task to be processed, so that the target task processing end starts a task execution process of the received task to be processed after receiving the task to be processed, starts a monitoring process for monitoring the task execution process and returns monitoring information generated by the monitoring process; and generating a processing result of the extracted task to be processed according to the monitoring information.
The foregoing description is only exemplary of the preferred embodiments of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (9)
1. A distributed task processing system, the system comprising:
the task distributing terminal is used for extracting the tasks to be processed from the task queue to be processed stored in the locking database based on the lock corresponding to each task to be processed, and selecting a target task processing terminal from the at least one task processing terminal to send the extracted tasks to be processed according to the load condition of the at least one task processing terminal, wherein the load condition comprises the historical processing speed of the tasks, and the target task processing terminal is the task processing terminal with the high historical processing speed of the tasks in the at least one task processing terminal;
the task distribution terminal is used for receiving a task to be processed sent by the task distribution terminal, starting a task execution process of the received task to be processed, starting a monitoring process for monitoring the task execution process, and sending monitoring information generated by the monitoring process to the task distribution terminal, wherein the monitoring process is a process for monitoring the status of the task execution process in real time, and the monitoring information comprises the real-time load condition of the task processing terminal and the task execution process information;
the task processing end is also used for responding to the abnormal termination of the task execution process and sending the monitoring information generated by the monitoring process to the task distributing end;
the task distributing end is also used for receiving the monitoring information and storing the task execution progress information;
and the task processing terminal is also used for restarting the task execution process according to the actual condition corresponding to the task execution process information by using the received task execution process information after receiving the task execution process information sent by the task distribution terminal.
2. The distributed task processing system of claim 1,
the task distributing terminal is further used for determining whether the task to be processed is completed or not according to the monitoring information, and generating a task processing result according to the monitoring information under the condition that the task to be processed is determined to be completed.
3. The distributed task processing system of one of claims 1-2,
the load condition includes at least one of: memory occupancy rate and network rate of the task processing terminal.
4. A distributed task processing method, the method comprising:
extracting the tasks to be processed from the task queues to be processed stored in the locking database based on the corresponding lock of each task to be processed;
according to the load condition of at least one connected task processing end, selecting a target task processing end from the at least one task processing end to send the extracted task to be processed, so that the target task processing end starts a task execution process of the received task to be processed after receiving the task to be processed, starts a monitoring process for monitoring the task execution process and returns monitoring information generated by the monitoring process, wherein the load condition comprises the historical processing rate of the task, the target task processing end is a task processing end with a high historical processing rate of the task in the at least one task processing end, the monitoring process is a process for monitoring the status of the task execution process in real time, and the monitoring information comprises the real-time load condition of the task processing end and the task execution process information;
generating a processing result of the extracted task to be processed according to the monitoring information;
receiving monitoring information, wherein the monitoring information is monitoring information generated by the monitoring process and sent by the target task processing end in response to the abnormal termination of the task execution process;
in response to receiving the monitoring information, saving the task execution process information;
and sending the task execution process information to the target task processing terminal so that the target task processing terminal can use the received task execution process information to restart the task execution process according to the actual situation corresponding to the task execution process information.
5. The distributed task processing method according to claim 4, wherein the generating a processing result of the extracted task to be processed according to the monitoring information includes:
and determining whether the task to be processed is completed or not according to the monitoring information, and generating a task processing result according to the monitoring information under the condition that the task to be processed is determined to be completed.
6. The distributed task processing method of claim 4,
the load condition includes at least one of: memory occupancy rate and network rate of the task processing terminal.
7. A distributed task processing apparatus, the apparatus comprising:
the extraction module is used for extracting the tasks to be processed from the task queue to be processed stored in the locking database based on the lock corresponding to each task to be processed;
the system comprises a selection distribution module, a task execution module and a monitoring module, wherein the selection distribution module is used for selecting a target task processing end from at least one task processing end to send an extracted task to be processed according to the load condition of the connected task processing end, so that the target task processing end starts a task execution process of the received task to be processed after receiving the task to be processed, starts a monitoring process for monitoring the task execution process and returns monitoring information generated by the monitoring process, the load condition comprises the historical processing rate of the task, the target task processing end is a task processing end with a high historical processing rate of the task in the task processing end, the monitoring process is a process for monitoring the status of the task execution process in real time, and the monitoring information comprises the real-time load condition of the task processing end and task execution process information;
the generating module is used for generating the processing result of the extracted task to be processed according to the monitoring information;
a receiving module, configured to receive monitoring information, where the monitoring information is monitoring information sent by the task processing end in response to an abnormal termination of the task execution process, and the monitoring information includes a load condition of the target task processing end and task execution process information;
the storage module is used for responding to the monitoring information and storing the task execution progress information;
and the sending module is used for sending the saved task execution process information to the target task processing end so that the target task processing end can use the received task execution process information to restart the task execution process according to the actual situation corresponding to the task execution process information.
8. A server, characterized in that the server comprises:
one or more processors;
a storage device to store one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 4-6.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 4-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710166286.4A CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710166286.4A CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108628677A CN108628677A (en) | 2018-10-09 |
| CN108628677B true CN108628677B (en) | 2022-12-02 |
Family
ID=63687178
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710166286.4A Active CN108628677B (en) | 2017-03-20 | 2017-03-20 | Distributed task processing system, method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108628677B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109656694A (en) * | 2018-11-02 | 2019-04-19 | 国网青海省电力公司 | A kind of distributed approach and system of energy storage monitoring data |
| CN109886327B (en) * | 2019-02-12 | 2021-11-19 | 北京奇艺世纪科技有限公司 | System and method for processing Java data in distributed system |
| CN110187971B (en) * | 2019-05-30 | 2020-08-04 | 口碑(上海)信息技术有限公司 | Service request processing method and device |
| CN113127158B (en) * | 2019-12-30 | 2023-07-14 | 百度在线网络技术（北京）有限公司 | Method and device for executing data processing task |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104360923A (en) * | 2014-11-03 | 2015-02-18 | 中国银行股份有限公司 | Monitoring method and monitoring system for batch application process |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941805B2 (en) * | 2006-08-15 | 2011-05-10 | International Business Machines Corporation | Affinity dispatching load balancer with precise CPU consumption data |
| CN103246592B (en) * | 2013-05-13 | 2016-12-28 | 北京搜狐新媒体信息技术有限公司 | A kind of monitoring acquisition system and method |
| CN104184823B (en) * | 2014-09-05 | 2017-11-14 | 北京京东尚科信息技术有限公司 | A kind of method and system for automating task scheduling |
| CN106406983B (en) * | 2015-07-27 | 2021-01-08 | 创新先进技术有限公司 | Task scheduling method and device in cluster |
| CN105893147A (en) * | 2016-03-29 | 2016-08-24 | 乐视控股（北京）有限公司 | Multi-task queue management method, equipment and system |
| CN106227596A (en) * | 2016-07-13 | 2016-12-14 | 百度在线网络技术（北京）有限公司 | Mission Monitor method and apparatus for task scheduling server |
-
2017
- 2017-03-20 CN CN201710166286.4A patent/CN108628677B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104360923A (en) * | 2014-11-03 | 2015-02-18 | 中国银行股份有限公司 | Monitoring method and monitoring system for batch application process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108628677A (en) | 2018-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108628677B (en) | Distributed task processing system, method and device | |
| CN105677469B (en) | Timed task execution method and device | |
| CN108733461B (en) | Distributed task scheduling method and device | |
| CN111078396B (en) | Distributed data access method and system based on multitasking examples | |
| CN108829512B (en) | Cloud center hardware accelerated computing power distribution method and system and cloud center | |
| CN110611707B (en) | Task scheduling method and device | |
| CN111182061B (en) | Task distribution processing method, system, computer device and storage medium | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN111897633A (en) | Task processing method and device | |
| CN111209110A (en) | Task scheduling management method, system and storage medium for realizing load balance | |
| CN110659131A (en) | Task processing method, electronic device, computer device, and storage medium | |
| CN111258741B (en) | Warehouse task execution method, distributed server cluster and computer equipment | |
| CN113703997A (en) | Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method | |
| CN109257396B (en) | Distributed lock scheduling method and device | |
| CN115544044A (en) | Data consistency keeping method, device, equipment and storage medium | |
| CN109388501B (en) | Communication matching method, device, equipment and medium based on face recognition request | |
| CN111538585A (en) | Js-based server process scheduling method, system and device | |
| CN113760522A (en) | Task processing method and device | |
| CN110888739B (en) | Distributed processing method and device for delayed tasks | |
| CN112860401A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN112600842A (en) | Cluster shell method and device, electronic equipment and computer readable storage medium | |
| CN111541646A (en) | Method for enhancing security service access capability of cipher machine | |
| CN110807058B (en) | Method and system for exporting data | |
| CN111767122A (en) | Distributed task scheduling management method and device | |
| CN112486502A (en) | Distributed task deployment method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |