WO2017148268A1 - 基于分布式系统的任务处理方法、装置和系统 - Google Patents

基于分布式系统的任务处理方法、装置和系统 Download PDF

Info

Publication number
WO2017148268A1
WO2017148268A1 PCT/CN2017/073721 CN2017073721W WO2017148268A1 WO 2017148268 A1 WO2017148268 A1 WO 2017148268A1 CN 2017073721 W CN2017073721 W CN 2017073721W WO 2017148268 A1 WO2017148268 A1 WO 2017148268A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
information
processing device
sub
task information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/073721
Other languages
English (en)
French (fr)
Inventor
方正
刘少伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to US16/080,854 priority Critical patent/US11256539B2/en
Publication of WO2017148268A1 publication Critical patent/WO2017148268A1/zh
Anticipated expiration legal-status Critical
Priority to US17/677,773 priority patent/US12346723B2/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G06F9/4856—Task life-cycle, e.g. stopping, restarting, resuming execution resumption being on a different machine, e.g. task migration, virtual machine migration
    • G06F9/4862—Task life-cycle, e.g. stopping, restarting, resuming execution resumption being on a different machine, e.g. task migration, virtual machine migration the task being a mobile agent, i.e. specifically designed to migrate
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/48—Indexing scheme relating to G06F9/48
    • G06F2209/486—Scheduler internals

Definitions

  • the present invention relates to the field of distributed systems, and in particular to a task processing method, apparatus and system based on a distributed system.
  • the distributed execution system in the prior art is a distributed execution system designed in a specific application environment, and its architecture is as shown in FIG. 1.
  • the system architecture is divided into layers: resource management layer and task execution layer, where resource management layer
  • resource management layer As the central control device of the whole system, it can be called driver, which is responsible for the assignment of tasks, the allocation of resources, the api interface of the client, etc.
  • the task processing device of the task execution layer can implement specific task execution logic, called node. Each node is deployed on a machine, and the node is horizontally expandable.
  • the execution flow chart of the task is shown in Figure 1:
  • the client submits the execution task through the HTTP interface provided by the driver; the Driver distributes the task to the specified node based on the internal resource management model; the Node reads the task assigned to the node; after the Node acquires the task, the task information is registered.
  • start the task execution process submit to the calculation engine to execute; save the task execution log and the process pid to the local; the state of the task execution (execution, execution is completed), feedback the task information to the driver; the driver is calling back the task status To the client.
  • the task processing device is in a special application environment, for example, during the process of upgrading the task processing device node, or entering the interrupted state, the task information is not caused during the restart process because the device needs to be restarted. Law enforcement and other issues;
  • the node has the problem of repeatedly performing the same task information, resulting in waste of resources.
  • the embodiment of the present invention provides a task processing method, apparatus, and system based on a distributed system, so as to at least solve the problem that the distributed processing system in the prior art is established in a specific application environment, and the task processing device cannot successfully perform task information, resulting in Technical problem with lost tasks.
  • a task processing method based on a distributed system including: a task processing device acquires task information from a master device by initiating a task request, wherein the task request is saved in the master device Corresponding task information; after receiving the task information, the main process of the task processing device starts the corresponding task sub-process by triggering the agent process to execute the task information; the task processing device saves the task execution information to the target location through the agent process, wherein the task
  • the execution information includes any one or more of the following: a task identifier of the task information, a process information of the task child process, log information generated by the task child process executing the task information, and an exit code.
  • a task processing apparatus based on a distributed system
  • a first acquiring module configured to: obtain, by a task processing device, a task information from a master device by initiating a task request, where The task information corresponding to the task request is saved in the main control device; the first execution module, after the main process of the task processing device receives the task information, starts the corresponding task sub-process by triggering the proxy process to execute the task information; a saving module, configured to save the task execution information to the target location by the agent process, where the task execution information includes any one or more of the following: a task identifier of the task information, a process information of the task subprocess, and a task subprocess Execute the log information and exit code generated by the task information.
  • a task processing system based on a distributed system, including: a master control device, configured to store task information; and a task processing device, communicating with the master control device, for passing Initiating a task request to obtain task information from the master device, wherein the task information corresponding to the task request is saved in the master device; the task processing device is further configured to: after the master process of the task processing device receives the task information, start by triggering the proxy process The task sub-process is used to execute the task information; the task processing device is further configured to save the task execution information to the target location by using the proxy process, where the task execution information includes any one or more of the following: task identifier of the task information, task sub-task The process information of the process, the log information generated by the task sub-process executing the task information, and Exit code.
  • the task processing device obtains the task information saved in the master control device from the master control device by initiating the task request, and after receiving the task information, the main process of the task processing device starts the corresponding task by triggering the proxy process.
  • the process executes the task information, and the task processing device saves the task execution information to the target location through the agent process.
  • the main process and the task sub-process in the task processing device are isolated by the agent process, the main process is stopped or restarted, and the agent process and the task sub-process are not perceived, and the task information can be executed normally, thereby realizing the execution.
  • the process is not affected during the startup of the main process.
  • the task processing device actively obtains the task information from the master control device, that is, the task processing device accesses the one-way master device. Therefore, by changing the http mode and using the database to synchronously store the data, the task processing device and the master control device are not in the same network. Deployment of distributed systems in a segment environment. It can support a million-level task execution, and can deploy nodes at any time through HTTP, which greatly improves the system's environment adaptability and scalability.
  • the embodiment provided by the present invention solves the technical problem that the distributed execution system in the prior art is established in a specific application environment, and the task processing device cannot successfully execute the task information, resulting in the loss of the task.
  • 1 is a flow chart of task processing of a distributed execution system according to the prior art
  • FIG. 2 is a block diagram showing a hardware structure of a computer terminal based on a task processing method of a distributed system according to an embodiment of the present application;
  • FIG. 3 is a flowchart of a task processing method based on a distributed system according to an embodiment of the present application
  • FIG. 4 is a schematic structural diagram of a distributed execution system according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an optional task processing device according to an embodiment of the present application.
  • FIG. 6 is a flowchart of an optional task processing device performing tasks according to an embodiment of the present application.
  • FIG. 7 is a flowchart of an optional task processing device recovering task information after restarting according to an embodiment of the present application
  • FIG. 8 is an interaction diagram of a task processing method based on a distributed system according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a task processing apparatus based on a distributed system according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • FIG. 11 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • FIG. 12 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • FIG. 13 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • FIG. 14 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • 15 is a schematic diagram of an optional distributed system based task processing apparatus in accordance with an embodiment of the present application.
  • 16 is a schematic diagram of a distributed system based task processing system in accordance with an embodiment of the present application.
  • FIG. 17 is a structural block diagram of a computer terminal according to an embodiment of the present application.
  • a distributed system which is a software system built on the network, with high cohesion and transparency.
  • Cohesion means that each database distribution node is highly autonomous and has a local database management system.
  • Transparency means that each database distribution node is transparent to the user's application and does not see whether it is local or remote.
  • the main control device The main control device, the service layer (central control) in the distributed execution system, responsible for resource management.
  • the task processing device the cluster node of the distributed execution system, also known as the gateway, is responsible for the execution of the task.
  • Pid The task process number, which is saved when the agent process is started.
  • an embodiment of a task processing method based on a distributed system is also provided.
  • the steps shown in the flowchart of the drawing may be in a computer system such as a set of computer executable instructions.
  • the steps shown and described may be performed in a different order than the ones described herein, although the logical order is shown in the flowchart.
  • FIG. 2 is a hardware structural block diagram of a computer terminal based on a task processing method of a distributed system according to an embodiment of the present application.
  • computer terminal 20 may include one or more (only one shown) processor 202 (processor 202 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA).
  • FIG. 2 is merely illustrative and does not limit the structure of the above electronic device.
  • computer terminal 20 may also include more or fewer components than those shown in FIG. 2, or have a different configuration than that shown in FIG. 2.
  • the memory 204 can be used to store software programs and modules of the application software, such as the program instructions/modules corresponding to the distributed system-based task processing method in the embodiment of the present application, and the processor 202 runs the software programs and modules stored in the memory 204. Thus, various functional applications and data processing are performed, that is, the above-described distributed system-based task processing method is implemented.
  • Memory 204 can include high speed random access memory and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
  • memory 204 can further include memory remotely located relative to processor 202, which can be connected to computer terminal 20 over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • Transmission device 206 is for receiving or transmitting data via a network.
  • the network specific examples described above may include a wireless network provided by a communication provider of the computer terminal 20.
  • transmission device 206 includes a Network Interface Controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet.
  • NIC Network Interface Controller
  • the transmission device 206 can be a Radio Frequency (RF) module for communicating with the Internet wirelessly.
  • RF Radio Frequency
  • FIG. 3 is a flowchart of a method for processing a task based on a distributed system according to an embodiment of the present application.
  • the method shown in FIG. 3 may include the following steps S301 to S305:
  • Step S301 The task processing device acquires task information from the master device by initiating a task request, where the task information corresponding to the task request is saved in the master device.
  • the task processing device may be a machine node node of a cluster deployed in a distributed system, where the task request may be an http request, the master device may be a driver, and the task processing device is saved in a database of the master device. Required task information.
  • the tasks in the distributed system are distributed at the resource layer of the resource management layer, and the driver manages the nodes and tasks.
  • the list of tasks being executed on each node is stored in the DB of the driver, and the storage period is the task life cycle.
  • a task processing device actively sends an http request to a master control device, and the master control device receives After the HTTP request is received, the task information corresponding to the http request is extracted, and the task information is sent to the task processing device through the interface.
  • Step S303 After receiving the task information, the main process of the task processing device starts the task task by triggering the agent process to start the task task.
  • the proxy process may be a proxy script, and the proxy script may serve as a channel for the primary process and the task subprocess.
  • FIG. 5 is a schematic diagram of an optional task processing device according to an embodiment of the present application.
  • the main process (AlisaNode) in the task processing device node is isolated from the task subprocess, and the proxy process (Python process) is associated.
  • Task child process and read and write logs.
  • FIG. 6 is a flowchart of an optional task processing device performing tasks according to an embodiment of the present application. As shown in FIG. 5 and FIG. 6, in an optional solution, after a node acquires a task from a driver, the node starts.
  • the agent process which passes the parameters required for the task execution, and the agent process starts the specific task child process to execute the task information.
  • Step S305 The task processing device saves the task execution information to the target location by using the proxy process, where the task execution information includes any one or more of the following: the task identifier of the task information, the process information of the task subprocess, and the task subprocess execution task.
  • the log information and exit code generated by the information are included in the proxy process.
  • the target location may be a local file of the proxy process
  • the path may be a task execution path
  • the task identifier of the task information may be a task id
  • the process information of the task subprocess may be a process number pid of the task subprocess.
  • log information generated by the task sub-process execution task information is complete log information, that is, the log action is irrelevant to the task processing device restart, and the log can be read and written normally as long as the agent process exists.
  • the task sub-process returns all the log information to the proxy server through a streaming manner, and the proxy process receives the log information, the process ID of the sub-process, the task information label, and drop out The code is saved to a local file.
  • the task processing device obtains the task information saved in the master device from the master device by initiating the task request, and the master process of the task processing device starts the task by triggering the proxy process after receiving the task information.
  • the corresponding task sub-process executes the task information, and the task processing device saves the task execution information to the target location through the proxy process.
  • the main process and the task sub-process in the task processing device are isolated by the agent process, the main process is stopped or restarted, and the agent process and the task sub-process are not perceived, and the task information can be executed normally, thereby realizing the execution.
  • the process is not affected during the startup of the main process.
  • the task processing device actively obtains the task information from the master control device, that is, the task processing device accesses the one-way master device. Therefore, by changing the http mode and using the database to synchronously store the data, the task processing device and the master control device are not in the same network. Deployment of distributed systems in a segment environment. It can support a million-level task execution, and can deploy nodes at any time through HTTP, which greatly improves the system's environment adaptability and scalability.
  • Embodiment 1 solves the technical problem that the distributed execution system in the prior art is established in a specific application environment, and the task processing device cannot successfully execute the task information, resulting in the loss of the task.
  • step S307 After the main process receives the task information in step S303, the method further includes the following step S307:
  • Step S307 if the main process is restarted, the task sub-process is started offline by the proxy process to execute the task information, and the task execution information is saved offline.
  • the agent process offline starts the task sub-process to execute the task information, and the task execution information sent by the task sub-process is stored in the local file through the offline storage mode.
  • step S307 the main process is restarted, and the agent process protects the task state by storing the task execution information of the task sub-process offline, so that the hot restart of the task processing device does not affect the task execution and does not lose the log information.
  • the method further includes the following steps S311 to S313:
  • Step S311 The task processing device acquires at least one task information currently being executed by accessing the master control device, where the task information includes: a task identifier and a task execution path.
  • the task execution path may be a globally unique critical path, and the path may maintain a life cycle of the entire task, and the task execution path is generated by: client name+date+random number+task number.
  • FIG. 7 is a flowchart of an optional task processing device recovering task information after restarting according to an embodiment of the present application;
  • the task information retained on the node is in memory, and the task information is lost after the node is restarted.
  • the task child process is separated from the node main process, the node is restarted, and the task is executed normally.
  • the recovery interface is called to obtain a list of tasks currently being executed in the driver (ie, at least one task information currently being executed), and each task includes a task number and a task execution path.
  • step S313 the task processing device determines a task that has been executed or not executed in at least one task information when the main process is restarted by detecting the task execution information recorded in the target location.
  • the log information interface can be obtained by querying the log file in the task execution path.
  • the task information can be obtained by the task sub-process;
  • the exit code file in the task execution path can be used to determine whether the task information has been executed by the task sub-process.
  • the task processing device performs, for each task, a process ID of the corresponding child process based on the task execution path length training, and if there is a process number of the corresponding child process, The task has been executed; if there is no process number of the corresponding child process, the task is executed.
  • the at least one task information currently being executed is acquired from the master device, and the task execution information recorded in the target location is detected, and at least one task information is determined. Has been executed or not executed, so that after the main process restarts, the task sub-process performs the consistency of the task information.
  • step S313 the task processing device determines the task that has been executed or not executed in the at least one task information by detecting the task execution information recorded in the target location, including the following steps S3133 to S3135:
  • step S3131 the task processing device performs a local query at the target location to determine whether process information matching any one of the at least one task information already exists.
  • the process information may be a process ID of the sub-process, and each task sub-process stores the process ID corresponding to the task information in the task information when the task information is started.
  • the task processing is performed based on the task execution path, and determining, according to the task identifier, whether the task ID of the corresponding child process exists in any one of the at least one task information.
  • Step S3133 if yes, it is determined that when the main process is restarted, the task information has been started and executed by the corresponding task sub-process, and the task processing device registers the task information only in the memory.
  • the task processing device resumes the task information (ie, the above-mentioned RAM Register the task information), and read the task process information of the task (ie, the task execution information described above).
  • step S3135 if it does not exist, it is determined that when the main process is restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to execute the task information.
  • the task processing device saves the task information in the memory, and Start the agent process to start the corresponding task child process to execute the task information.
  • the task processing device performs a local query from the target location to determine whether there is a process number of the corresponding child process, and if so, re-records the task information; if not, starts the task child process through the proxy process.
  • the task sub-process performs the consistency of the task information after the main process is restarted.
  • the method before the task processing device registers the task information in the memory only in step S3133, the method further includes the following steps S317 to S319:
  • step S315 the task processing device performs a local query at the target location to determine whether there is already an exit code that matches the task information that has been executed by the corresponding task sub-process.
  • the above return code is used to indicate that the task information has been executed.
  • Step S317 if yes, it is determined that when the main process is restarted, the task information has been executed by the corresponding task sub-process, and the task processing device terminates the registration of the task information in the memory, and directly feeds the corresponding task log to the main control device. .
  • Step S319 if it does not exist, it is determined that when the main process is restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to re-execute the task information. .
  • the task processing device determines that the process ID of the corresponding sub-process exists, it determines whether there is an exit code corresponding to the task information, and if yes, it indicates that the task information has been executed, then You need to register the task information in the memory and directly report the corresponding task log to the master device. If it does not exist, the task is not executed. You need to start the task sub-process through the agent process to re-execute the task information.
  • the task processing device performs a local query at the target location to determine whether a corresponding exit code already exists. If yes, the task log is fed back. If not, the corresponding task sub-process is started to re-execute the task information. Therefore, after the main process is restarted, the task information being executed is restored, and the operation is guaranteed. The status of the task information.
  • the method further includes the following steps S321 to S323:
  • Step S321 extracting a path parameter from the task information, and generating a task execution path according to the path parameter.
  • the task processing device when the task processing device obtains the task information, the task processing device obtains a path parameter from the task information, and the task processing device uses the parameter as the current task execution path, the task execution log, and the process ID.
  • the exit code is saved in the path.
  • step S323 the agent process saves the task identifier to the target location according to the task execution path.
  • the task child process saves the process number of the current child process to a local file when the startup is performed, and the saved path is the task execution path.
  • the task processing device extracts the path parameter from the task information, generates a task execution path, and the agent process saves the task identifier in the target location according to the task execution path, thereby ensuring the integrity of the task log before and after the main process restarts. Checkability.
  • the foregoing method further includes the following step S325:
  • Step S325 The main process of the task processing device monitors whether the task information is started according to the task identifier, and monitors whether the task information ends according to the exit code.
  • the task processing device can monitor whether the task information is executed by the task identification and the exit code, and whether the execution ends, thereby ensuring that the task information is all executed and all the execution is completed.
  • the task processing device includes a main process, an agent process, and a task sub-process as an application scenario, and an optional distributed system-based task processing method is provided.
  • the method may include the following steps S81 to S87. :
  • the client terminal 81 submits a task.
  • the client can issue task information through the client terminal 81.
  • the main process of the task processing device 163 acquires task information saved in the master device 161.
  • the main process of the task processing device 163 obtains task information from the main control device 161 through the http request, and the task information is saved in the main control device.
  • the implementation manner of the foregoing step S802 is the same as the implementation manner of the step S301 in the foregoing embodiment of the present application, and details are not described herein again.
  • the task processing device 163 starts the task sub-process to execute the task information through the proxy process.
  • the main process of the task processing device 163 starts the task sub-process to execute the task information by triggering the proxy process.
  • the main process AlisaNode
  • the agent process Panethon process
  • the implementation manner of the foregoing step S803 is the same as the implementation manner of the step S303 in the foregoing embodiment of the present application, and details are not described herein again.
  • the task processing device 163 saves the task execution information through the proxy process.
  • the task processing device 163 when acquiring the task information, obtains the path parameter from the task information, generates a task execution path, and the agent process saves the task identifier of the task information, the process information of the task subprocess, the log information, and the exit code.
  • step S804 is the same as the implementation of the step S305 in the foregoing embodiment of the present application, and details are not described herein again.
  • the agent process starts the task sub-process offline to execute the task information, and saves the task execution information generated by the task sub-process execution task information in the offline storage manner at the target location.
  • the task processing device 163 acquires the task information being executed from the master device 161, and queries the task execution information from the target location to determine whether the task information has been executed or not executed when the main process is restarted. .
  • the task processing device 163 calls the recovery interface to obtain a task list saved in the main control device 161, and detects whether a corresponding task sub-process exists in the target location.
  • the process ID if it does not exist, indicating that the task information is not executed by the corresponding task sub-process, the task processing device 163 saves the task information in the memory, and starts the corresponding task sub-process to execute the task information by starting the proxy process; If the task information has been activated by the corresponding task sub-process, the task processing device 163 detects whether there is a corresponding exit code in the target location. If not, the corresponding task sub-process is restarted by the proxy process to re-execute the task. The information, if any, returns a task log to the master device 161.
  • the task processing device 163 returns the task status to the master device 161.
  • the main process monitors whether the task information is started according to the task identifier, and whether the execution of the task information is terminated according to the exit code. If the main process monitors that the task has been executed, the task execution information is fed back to the main control device 161, if the monitoring is performed. After the task information has been executed, the task log is fed back to the master device 161.
  • step S806 is the same as the implementation of the step S325 in the foregoing embodiment of the present application, and details are not described herein again.
  • the main control device 161 feeds back the task log to the client terminal 81.
  • the master device 161 feeds back the task log of the executed task information to the client, and performs the task execution information to the client terminal 81 via the task information being executed.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present application which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present application.
  • a distributed system-based task processing apparatus for implementing the above-described distributed system-based task processing method is further provided.
  • the apparatus includes: a first obtaining module 91, An execution module 93 and a first save module 95.
  • the first obtaining module 91 is configured to acquire task information from the master device by initiating a task request, where the task information corresponding to the task request is saved in the master device.
  • the first execution module 93 is configured to execute the task information by triggering the proxy process to start the corresponding task sub-process after the main process of the task processing device receives the task information.
  • the first saving module 95 is configured to save the task execution information to the target location by the agent processing device, where the task execution information includes any one or more of the following: task identifier of the task information, process information of the task subprocess, and task
  • the child process executes the log information and the exit code generated by the task information.
  • the task processing device may be a machine node node of a cluster deployed in a distributed system, where the task request may be an http request, the master device may be a driver, and the task processing device is saved in a database of the master device.
  • the above proxy process can be a proxy script, and the proxy script can serve as a channel for the main process and the task subprocess.
  • the target location may be a local file of the proxy process, and the path may be a task execution path.
  • the task identifier of the task information may be a task id, and the process information of the task subprocess may be a process number pid of the task subprocess.
  • the tasks in the distributed system are distributed at the resource layer of the resource management layer, and the driver manages the nodes and tasks.
  • the list of tasks being executed on each node is stored in the DB of the driver, and the storage period is the task life cycle.
  • the log information generated by the task sub-process execution task information is complete log information, which means that the log action is irrelevant to the task processing device restart, and the log can be read and written normally as long as the agent process exists.
  • the first obtaining module 91, the first executing module 93, and the first saving module 95 correspond to steps S301 to S305 in the first embodiment, and the three modules and the corresponding steps are implemented by the corresponding steps.
  • the application scenario is the same, but is not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the task processing device obtains the task information saved in the master control device from the master device by initiating the task request, and the master process of the task processing device starts the task by triggering the proxy process after receiving the task information.
  • the corresponding task sub-process executes the task information, and the task processing device saves the task execution information to the target location through the proxy process.
  • the main process and the task sub-process in the task processing device are isolated by the agent process, the main process is stopped or restarted, and the agent process and the task sub-process are not perceived, and the task information can be executed normally, thereby realizing the execution.
  • the process is not affected during the startup of the main process.
  • the task processing device actively obtains the task information from the master control device, that is, the task processing device accesses the one-way master device. Therefore, by changing the http mode and using the database to synchronously store the data, the task processing device and the master control device are not in the same network. Deployment of distributed systems in a segment environment. It can support a million-level task execution, and can deploy nodes at any time through HTTP, which greatly improves the system's environment adaptability and scalability.
  • the solution of the foregoing embodiment 2 provided by the present application solves the technical problem that the distributed execution system in the prior art is established in a specific application environment, and the task processing device cannot successfully execute the task information, resulting in the loss of the task.
  • the foregoing apparatus further includes: a second execution module 101.
  • the second execution module 101 is configured to: after the main process receives the task information, if the main process is restarted, the task sub-process is started offline by the proxy process to execute the task information, and the task execution information is saved offline.
  • the foregoing second execution module 101 corresponds to step S307 in the first embodiment, and the foregoing modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the foregoing apparatus further includes: a second obtaining module 111 and a first determining module 113.
  • the second obtaining module 111 is configured to acquire, by the task processing device, at least one task information that is currently being executed by accessing the master control device, where the task information includes: a task identifier and a task execution path.
  • the first determining module 113 is configured to determine, by the task processing device, the task that has been executed or not executed in the at least one task information when the main process is restarted by detecting the task execution information recorded in the target location.
  • the task execution path may be a globally unique critical path, and the path may maintain a life cycle of the entire task, and the task execution path is generated by: client name+date+random number+task number.
  • the exit code file can be used to determine whether the task information has been executed by the task sub-process.
  • the foregoing second obtaining module 111 and the first determining module 113 correspond to steps S311 to S313 in the first embodiment, and the two modules are the same as the examples and application scenarios implemented by the corresponding steps, but not It is limited to the content disclosed in the above embodiment 1. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the first determining module 113 includes: a second determining module 121, a first registration module 123, and a second registration module 125.
  • the second determining module 121 is configured to perform a local query by the task processing device at the target location, and determine whether there is already process information that matches any one of the at least one task information.
  • the first registration module 123 is configured to determine, if present, that the task information has been started by the corresponding task sub-process when the main process is restarted, and the task processing device registers the task information only in the memory.
  • the second registration module 125 is configured to: if it does not exist, determine that when the main process is restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process Execute task information.
  • the process information may be a process ID of the sub-process, and each task sub-process stores the process ID corresponding to the task information in the task information when the task information is started.
  • the foregoing second determining module 121, the first registration module 123 and the second registration module 125 correspond to the steps S3133 to S3135 in the first embodiment, and the three modules and the corresponding steps are implemented by the corresponding steps.
  • the application scenario is the same, but is not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the foregoing apparatus further includes: a third determining module 131, a feedback module 133, and a third registration module 135.
  • the third determining module 131 is configured to perform a local query by the task processing device at the target location to determine whether The exit code is matched by the task information that has been executed by the corresponding task sub-process.
  • the feedback module 133 is configured to: if yes, determine that when the main process is restarted, the task information has been executed by the corresponding task sub-process, and the task processing device terminates registering the task information in the memory, and directly returns the corresponding task log to the main task. Control equipment.
  • the third registration module 135 is configured to: if it does not exist, determine that when the main process is restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process. Re-execute the task information.
  • the above return code is used to indicate that the task information has been executed.
  • the foregoing third determining module 131, the feedback module 133, and the third registration module 135 correspond to the steps S317 to S319 in the first embodiment, and the examples and application scenarios implemented by the three modules and corresponding steps. The same, but not limited to, the content disclosed in the above embodiment 1. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the foregoing apparatus further includes: a generating module 141 and a second saving module 143.
  • the generating module 141 is configured to extract a path parameter from the task information, and generate a task execution path according to the path parameter.
  • the second saving module 143 is configured to save the task identifier to the target location according to the task execution path.
  • the foregoing generating module 141 and the second saving module 143 correspond to steps S321 to S323 in the first embodiment, and the two modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the above.
  • the content disclosed in the first embodiment It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • the foregoing apparatus further includes: a listening module 151.
  • the monitoring module 151 is used by the main process of the task processing device to monitor whether the task information is started according to the task identifier, and whether the execution of the task information is terminated according to the exit code.
  • the foregoing monitoring module 151 corresponds to the step S325 in the first embodiment.
  • the foregoing modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 20 provided in the first embodiment.
  • a distributed system-based task processing system for implementing the above-described distributed system-based task processing method is further provided.
  • the system includes: a master control device 161 and task processing.
  • the main control device 161 is configured to store task information.
  • the task processing device 163 communicates with the main control device 161 to obtain task information from the main control device by initiating a task request, wherein the task information corresponding to the task request is saved in the main control device.
  • the task processing device 163 may be a machine node node of a cluster deployed in a distributed system, where the task request may be an http request, the master device may be a driver, and a task processing is saved in a database of the master device. The task information required by the device.
  • the tasks in the distributed system are distributed at the resource layer of the resource management layer, and the driver manages the nodes and tasks.
  • the list of tasks being executed on each node is stored in the DB of the driver, and the storage period is the task life cycle.
  • the task processing device actively sends an http request to the main control device, and after receiving the http request, the main control device extracts the task information corresponding to the http request, and the task information is obtained through the interface. Send to the task processing device.
  • the task processing device 163 is further configured to: after the main process of the task processing device receives the task information, start the task task by triggering the agent process to start the task task.
  • the proxy process may be a proxy script, and the proxy script may serve as a channel for the primary process and the task subprocess.
  • the main process (AlisaNode) in the task processing device node is isolated from the task subprocess, the proxy process (Python process) is associated with the task subprocess, and the log is read and written.
  • the agent process is started, and the parameters required for the task execution are transmitted, and the agent process starts the specific task child process to execute the task information.
  • the task processing device 163 is further configured to save the task execution information to the target location by using a proxy process, where the task execution information includes any one or more of the following: a task identifier of the task information, a process information of the task subprocess, and a task subprocess execution. Log information and exit code generated by the task information.
  • the target location may be a local file of the proxy process
  • the path may be a task execution path
  • the task identifier of the task information may be a task id
  • the process information of the task subprocess may be a process number pid of the task subprocess.
  • log information generated by the task sub-process execution task information is complete log information, that is, the log action is irrelevant to the task processing device restart, and the log can be read and written normally as long as the agent process exists.
  • the task sub-process returns all the log information to the proxy server through a streaming manner, and the proxy process receives the log information, the process ID of the sub-process, the task information label, and drop out The code is saved to a local file.
  • the task processing device obtains the task information saved in the master device from the master device by initiating the task request, and the master process of the task processing device starts the task by triggering the proxy process after receiving the task information.
  • the corresponding task sub-process executes the task information, and the task processing device saves the task execution information to the target location through the proxy process.
  • the main process and the task sub-process in the task processing device are isolated by the agent process, the main process is stopped or restarted, and the agent process and the task sub-process are not perceived, and the task information can be executed normally, thereby realizing the execution.
  • the process is not affected during the startup of the main process.
  • the task processing device actively obtains the task information from the master control device, that is, the task processing device accesses the one-way master device. Therefore, by changing the http mode and using the database to synchronously store the data, the task processing device and the master control device are not in the same network. Deployment of distributed systems in a segment environment. It can support a million-level task execution, and can deploy nodes at any time through HTTP, which greatly improves the system's environment adaptability and scalability.
  • the solution of the foregoing embodiment 3 provided by the present application solves the technical problem that the distributed execution system in the prior art is established in a specific application environment, and the task processing device cannot successfully execute the task information, resulting in the loss of the task.
  • the task processing device 163 is further configured to: if the main process is restarted, start the task sub-process offline by the proxy process to execute the task information, and Save task execution information offline.
  • the task processing device 163 is further configured to obtain at least one task information currently being executed by accessing the main control device, where the task information includes: a task identifier. And a task execution path; determining a task that has been executed or not executed in at least one task information when the main process is restarted by detecting task execution information recorded in the target location.
  • the task processing device 163 is further configured to perform a local query at the target location to determine whether there is already process information that matches any one of the at least one task information; if yes, It is determined that when the main process is restarted, the task information is started by the corresponding task sub-process, and the task processing device registers the task information only in the memory; if not, it determines that the task information is not when the main process is restarted.
  • the corresponding task sub-process executes, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to execute the task information.
  • the task processing device 163 is further configured to perform a local query at the target location to determine whether an exit code that matches the task information that has been started and executed by the corresponding task sub-process already exists; If it exists, it is determined that when the main process is restarted, the task information has been executed by the corresponding task sub-process.
  • the task processing device terminates the registration of the task information in the memory, and directly feeds the corresponding task log to the master device; if not, determines that the task information is not executed by the corresponding task child process when the main process is restarted, the task is completed.
  • the processing device registers the task information in the memory, and starts the corresponding task sub-process to re-execute the task information.
  • the task processing device 163 is further configured to extract a path parameter from the task information, and generate a task execution path according to the path parameter; wherein the agent process saves the task identifier according to the task execution path to target location.
  • the task processing device 163 is further configured to: the main process monitors whether the task information is started according to the task identifier, and monitors whether the task information ends according to the exit code.
  • Embodiments of the present application may provide a computer terminal, which may be any one of computer terminal groups.
  • the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.
  • the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.
  • the computer terminal may execute the program code of the following steps in the task processing method of the distributed system: the task processing device acquires task information from the master device by initiating a task request, where the file is saved in the master device The task information corresponding to the task request; after receiving the task information, the main process of the task processing device starts the corresponding task sub-process by triggering the agent process to execute the task information; the task processing device saves the task execution information to the target location through the proxy process, wherein
  • the task execution information includes any one or more of the following: a task identifier of the task information, a process information of the task sub-process, a log information generated by the task sub-process executing the task information, and an exit code.
  • FIG. 17 is a structural block diagram of a computer terminal according to an embodiment of the present application.
  • the computer terminal A may include one or more (only one shown in the figure) processor 171, memory 173, and transmission device 175.
  • the memory 173 can be used to store a software program and a module, such as a distributed system-based task processing method and a program instruction/module corresponding to the device in the embodiment of the present application, and the processor 171 runs the software program and the module stored in the memory. Thus, various functional applications and data processing are performed, that is, the above-described distributed system-based task processing method is implemented.
  • Memory 173 can include high speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
  • memory 173 can further include memory remotely located relative to the processor, the remote memory can pass through the network Connect to terminal A. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the processor 171 can call the information and the application stored in the memory by the transmission device to perform the following steps: the task processing device acquires the task information from the master device by initiating a task request, where the task request corresponding to the task is saved in the master device Task information; after receiving the task information, the main process of the task processing device starts the corresponding task sub-process by triggering the agent process to execute the task information; the task processing device saves the task execution information to the target location through the agent process, wherein the task execution information
  • the information includes any one or more of the following: a task identifier of the task information, a process information of the task sub-process, a log information generated by the task sub-process executing the task information, and an exit code.
  • the processor 171 may further execute the following program code: after the main process receives the task information, if the main process is restarted, the task sub-process is started offline by the proxy process to execute the task information, and the task is saved offline. Execution information.
  • the processor 171 may further execute the following program code: the task processing device acquires at least one task information currently being executed by accessing the master control device, where the task information includes: a task identifier and a task execution path; and task processing The device determines a task that has been executed or not executed in at least one of the task information when the main process is restarted by detecting task execution information recorded in the target location.
  • the processor 171 may further execute the following step: the task processing device performs a local query at the target location, and determines whether there is already process information that matches any one of the at least one task information; if yes, It is determined that when the main process is restarted, the task information is started by the corresponding task sub-process, and the task processing device registers the task information only in the memory; if not, it determines that the task information is not when the main process is restarted.
  • the corresponding task sub-process executes, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to execute the task information.
  • the processor 171 may further execute the following program code: the task processing device performs a local query at the target location, and determines whether an exit code that matches the task information that has been executed by the corresponding task sub-process has existed; If yes, it is determined that when the main process is restarted, the task information has been executed by the corresponding task sub-process, and the task processing device terminates the registration of the task information in the memory, and directly returns the corresponding task log to the main control device; If yes, it is determined that when the main process is restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to re-execute the task information.
  • the processor 171 may further execute the following program code: extract a path parameter from the task information, and generate a task execution path according to the path parameter; wherein the agent process marks the task according to the task execution path. Save to the target location.
  • the processor 171 may further execute the following program code: the main process of the task processing device monitors whether the task information is started according to the task identifier, and monitors whether the task information ends according to the exit code.
  • a task processing method based on a distributed system obtains the task information saved in the master device from the master device by initiating the task request, and after receiving the task information, the main process of the task processing device starts the corresponding task child process by triggering the proxy process to execute the task information.
  • the task processing device saves the task execution information to the target location through the proxy process.
  • FIG. 17 is merely illustrative, and the computer terminal can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, MID). ), PAD and other terminal devices.
  • Fig. 17 does not limit the structure of the above electronic device.
  • computer terminal A may also include more or fewer components (such as a network interface, display device, etc.) than shown in FIG. 17, or have a different configuration than that shown in FIG.
  • Embodiments of the present application also provide a storage medium.
  • the foregoing storage medium may be used to save the program code executed by the distributed system-based task processing method provided in Embodiment 1 above.
  • the foregoing storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.
  • the storage medium is configured to store program code for performing the following steps: the task processing device acquires task information from the master device by initiating a task request, wherein the task is saved in the master device Requesting the corresponding task information; after receiving the task information, the main process of the task processing device starts the task task by triggering the agent process to execute the task information; the task processing device saves the task execution information to the target location through the agent process, where
  • the task execution information includes any one or more of the following: a task identifier of the task information, a process information of the task subprocess, log information generated by the task subprocess executing the task information, and an exit code.
  • the storage medium is arranged to store program code for performing the following steps: After the main process receives the task information, if the main process is restarted, the task sub-process is started offline by the agent process to execute the task information, and the task execution information is saved offline.
  • the storage medium is configured to store program code for performing the following steps: the task processing device acquires at least one task information currently being executed by accessing the master control device, and the task information includes: the task The identification and task execution path; the task processing device determines a task that has been executed or not executed in the at least one task information when the main process is restarted by detecting the task execution information recorded in the target location.
  • the storage medium is configured to store program code for performing the following steps: the task processing device performs a local query at the target location, determining whether there is already any task identifier associated with the at least one task information Matching process information; if yes, it is determined that when the main process is restarted, the task information has been started by the corresponding task sub-process, and the task processing device registers the task information only in the memory; if not, it determines the main process When restarted, the task information is not executed by the corresponding task sub-process, the task processing device registers the task information in the memory, and starts the corresponding task sub-process to execute the task information.
  • the storage medium is configured to store program code for performing the following steps: the task processing device performs a local query at the target location, and determines whether there is already a start to be executed by the corresponding task sub-process.
  • the exit code matching the task information; if yes, it is determined that when the main process is restarted, the task information has been executed by the corresponding task sub-process, and the task processing device terminates the registration of the task information in the memory, directly corresponding to the task log.
  • Feedback to the master device if not, it is determined that when the master process is restarted, the task information is not executed by the corresponding task child process, the task processing device registers the task information in the memory, and starts the corresponding task child process to restart Execute task information.
  • the storage medium is configured to store program code for performing the following steps: extracting a path parameter from the task information, and generating a task execution path according to the path parameter; wherein the agent process executes the path according to the task Save the task ID to the target location.
  • the storage medium is configured to store program code for performing the following steps: the main process of the task processing device monitors whether the task information is started according to the task identifier, and monitors whether the task information is detected according to the exit code. Execution ends.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
  • the foregoing storage medium includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Computer And Data Communications (AREA)

Abstract

一种基于分布式系统的任务处理方法、装置和系统。其中，该方法包括：任务处理设备通过发起任务请求从主控设备中获取任务信息（S301），其中，在主控设备中保存任务请求对应的任务信息；任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息（S303）；任务处理设备通过代理进程将任务执行信息保存至目标位置（S305），其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。所述方法、装置和系统解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。

Description

基于分布式系统的任务处理方法、装置和系统
本申请要求2016年02月29日递交的申请号为201610112497.5、发明名称为“基于分布式系统的任务处理方法、装置和系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及分布式系统领域，具体而言，涉及一种基于分布式系统的任务处理方法、装置和系统。
背景技术
随着微处理机技术的发展，分布式系统可以潜在地得到比单个大型集中式系统更好的性价比，可以通过较低廉的价格实现相似的功能，因此，分布式系统成为越来越普遍使用的系统架构。
但是，现有技术中的分布式执行系统是在特定应用环境中设计的分布式执行系统，其架构如图1所示，系统架构分为层：资源管理层和任务执行层，其中资源管理层作为整个系统的中控设备，可以称之为driver，负责对任务的分配、资源的分配、客户端的api接口等功能；任务执行层的任务处理设备可以实现具体的任务执行逻辑，称之为node，每个node部署在一台机器上(gateway)，node是可水平扩展。任务的执行流程图如图1所示：
客户端通过driver提供的HTTP接口提交执行任务；Driver基于内部的资源管理模型，将任务分发到指定的node上；Node读取到分配到node上的任务；Node获取到任务后，将任务信息注册到内存中；启动任务执行进程，提交到计算引擎执行；保存任务执行日志以及进程pid到本地；任务执行的状态(执行中、执行完毕)，将任务信息反馈到driver；Driver在将任务状态回调给客户端。
通过上述流程可以发现，上述系统中至少存在如下三个问题：
1、如果driver与其中某些node网络不通，则driver发起的任务信息将无法成功发给node，也无法获知哪些任务信息没有发送成功；
2、如果任务处理设备处于特殊的应用环境，例如在任务处理设备node进行升级的过程中，或者进入中断状态下，由于需要重启设备，因此重启过程中会导致任务信息无 法执行等问题；
3、若存在driver重复发了多次任务到一台node上，node存在重复执行相同的任务信息的问题，导致资源浪费。
针对现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题，目前尚未提出有效的解决方案。
发明内容
本发明实施例提供了一种基于分布式系统的任务处理方法、装置和系统，以至少解决现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
根据本发明实施例的一个方面，提供了一种基于分布式系统的任务处理方法，包括：任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
根据本发明实施例的另一方面，还提供了一种基于分布式系统的任务处理装置，包括：第一获取模块，用于任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；第一执行模块，用于任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；第一保存模块，用于任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
根据本发明实施例的另一方面，还提供了一种基于分布式系统的任务处理系统，包括：主控设备，用于存储任务信息；任务处理设备，与主控设备进行通讯，用于通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；任务处理设备还用于任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；任务处理设备还用于通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和 退出码。
在本发明实施例中，任务处理设备通过发起任务请求从主控设备中获取主控设备中保存的任务信息，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息，任务处理设备通过代理进程将任务执行信息保存至目标位置。
容易注意到，由于任务处理设备中主进程和任务子进程通过代理进程进行隔离，使得主进程停止或者被重启，代理进程及任务子进程都不感知，可以正常执行任务信息，从而实现正在执行的进程在主进程启动过程中不受影响。并且由于任务处理设备主动向主控设备获取任务信息，即任务处理设备单向主控设备访问，因此通过改变http方式，利用数据库同步存储数据，从而实现任务处理设备、主控设备不在同一个网段环境下分布式系统的部署问题。能够支持百万级的任务执行，并且通过HTTP的方式，可以随时部署node，大大提升了系统的环境适应能力和可扩展能力。
由此，本发明提供的实施例解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
附图说明
此处所说明的附图用来提供对本申请的进一步理解，构成本申请的一部分，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的不当限定。在附图中：
图1是根据现有技术的一种分布式执行系统的任务处理的流程图；
图2是根据本申请实施例的一种基于分布式系统的任务处理方法的计算机终端的硬件结构框图；
图3是根据本申请实施例的一种基于分布式系统的任务处理方法的流程图；
图4是根据本申请实施例的一种分布式执行系统的结构示意图；
图5是根据本申请实施例的一种可选的任务处理设备的示意图；
图6是根据本申请实施例的一种可选的任务处理设备执行任务的流程图；
图7是根据本申请实施例的一种可选的任务处理设备重启后恢复任务信息的流程图；
图8是根据本申请实施例的基于分布式系统的任务处理方法的交互图；
图9是根据本申请实施例的基于分布式系统的任务处理装置的示意图；
图10是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图11是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图12是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图13是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图14是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图15是根据本申请实施例的可选的基于分布式系统的任务处理装置的示意图；
图16是根据本申请实施例的基于分布式系统的任务处理系统的示意图；以及
图17是根据本申请实施例的一种计算机终端的结构框图。
具体实施方式
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分的实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本申请保护的范围。
需要说明的是，本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
首先，在对本申请实施例进行描述的过程中出现的部分名词或术语适用于如下解释：
分布式系统：distributed system，是建立在网络上的软件系统，具有高度的内聚性和透明性。内聚性是指每一个数据库分布节点高度自治，有本地的数据库管理系统。透明性是指每一个数据库分布节点对用户的应用来说都是透明的，看不出是本地还是远程。
driver：主控设备，分布式执行系统中的服务层(中控)，负责资源管理。
node：任务处理设备，分布式执行系统中的集群的机器节点，也称之为gateway，具体负责任务的执行。
pid：任务进程编号，启动代理进程时进行保存。
实施例1
根据本申请实施例，还提供了一种基于分布式系统的任务处理方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
本申请实施例一所提供的方法实施例可以在移动终端、计算机终端或者类似的运算装置中执行。以运行在计算机终端上为例，图2是根据本申请实施例的一种基于分布式系统的任务处理方法的计算机终端的硬件结构框图。如图2所示，计算机终端20可以包括一个或多个(图中仅示出一个)处理器202(处理器202可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)、用于存储数据的存储器204、以及用于通信功能的传输模块206。本领域普通技术人员可以理解，图2所示的结构仅为示意，其并不对上述电子装置的结构造成限定。例如，计算机终端20还可包括比图2中所示更多或者更少的组件，或者具有与图2所示不同的配置。
存储器204可用于存储应用软件的软件程序以及模块，如本申请实施例中的基于分布式系统的任务处理方法对应的程序指令/模块，处理器202通过运行存储在存储器204内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的基于分布式系统的任务处理方法。存储器204可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器204可进一步包括相对于处理器202远程设置的存储器，这些远程存储器可以通过网络连接至计算机终端20。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输装置206用于经由一个网络接收或者发送数据。上述的网络具体实例可包括计算机终端20的通信供应商提供的无线网络。在一个实例中，传输装置206包括一个网络适配器(Network Interface Controller，NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输装置206可以为射频(Radio Frequency，RF)模块，其用于通过无线方式与互联网进行通讯。
在上述运行环境下，本申请提供了如图3所示的基于分布式系统的任务处理方法。图3是根据本申请实施例的一种基于分布式系统的任务处理方法的流程图，如图3所示的方法可以包括如下步骤S301至步骤S305：
步骤S301，任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息。
可选地，上述任务处理设备可以是部署在分布式系统中的集群的机器节点node，上述任务请求可以是http请求，上述主控设备可以是driver，主控设备的数据库中保存有任务处理设备需要的任务信息。
需要说明的是，分布式系统中任务都在资源管理层Driver层做分发，driver管理node、任务。每台node上正在执行的任务列表都存储在driver的DB中，存储周期为任务生命周期。
图4是根据本申请实施例的一种分布式执行系统的结构示意图，如图4所示，在一种可选的方案中，任务处理设备主动发送http请求给主控设备，主控设备接收到http请求之后，提取http请求对应的任务信息，通过接口将该任务信息发送给任务处理设备。
步骤S303，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息。
可选地，上述代理进程可以是代理脚本，代理脚本可以作为主进程与任务子进程的通道。
图5是根据本申请实施例的一种可选的任务处理设备的示意图，如图5所示，任务处理设备node中的主进程(AlisaNode)与任务子进程隔离，代理进程(Python进程)关联任务子进程，并且读写日志。图6是根据本申请实施例的一种可选的任务处理设备执行任务的流程图，结合图5和图6可知，在一种可选的方案中，node从driver中获取到任务之后，启动代理进程，传入任务执行需要的参数，代理进程启动具体的任务子进程执行任务信息。
步骤S305，任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选地，上述目标位置可以是代理进程的本地文件，路径可以是任务执行路径，上述任务信息的任务标识可以是任务id，上述任务子进程的进程信息可以是任务子进程的进程号pid。
需要说明的是，上述任务子进程执行任务信息所生成的日志信息为完整的日志信息，是指日志动作与任务处理设备重启不相关，并且只要代理进程存在，日志就可以正常读写。
如图6所示，在一种可选的方案中，任务子进程将所有的日志信息通过流方式返回给代理服务器，代理进程将接收到的日志信息、子进程的进程号、任务信息标号和退出 码保存到本地文件中。
本申请上述实施例一公开的方案中，任务处理设备通过发起任务请求从主控设备中获取主控设备中保存的任务信息，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息，任务处理设备通过代理进程将任务执行信息保存至目标位置。
容易注意到，由于任务处理设备中主进程和任务子进程通过代理进程进行隔离，使得主进程停止或者被重启，代理进程及任务子进程都不感知，可以正常执行任务信息，从而实现正在执行的进程在主进程启动过程中不受影响。并且由于任务处理设备主动向主控设备获取任务信息，即任务处理设备单向主控设备访问，因此通过改变http方式，利用数据库同步存储数据，从而实现任务处理设备、主控设备不在同一个网段环境下分布式系统的部署问题。能够支持百万级的任务执行，并且通过HTTP的方式，可以随时部署node，大大提升了系统的环境适应能力和可扩展能力。
由此，本申请提供的上述实施例一的方案解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
在本申请上述实施例中，在步骤S303主进程接收到任务信息之后，上述方法还包括如下步骤S307：
步骤S307，如果主进程被重启，则通过代理进程离线启动任务子进程来执行任务信息，并离线保存任务执行信息。
在一种可选的方案中，如果node主进程被重启，代理进程离线启动任务子进程执行任务信息，并将任务子进程发送的任务执行信息通过离线存储方式储存在本地文件中。
通过上述步骤S307，主进程被重启，代理进程通过离线存储任务子进程的任务执行信息的方式保护任务状态，从而实现任务处理设备热重启不影响任务执行、不丢失日志信息的目的。
在本申请上述实施例中，在步骤S307主进程被重启之后，上述方法还包括如下步骤S311至步骤S313：
步骤S311，任务处理设备通过访问主控设备，来获取当前正在执行的至少一个任务信息，任务信息包括：任务标识和任务执行路径。
可选地，上述任务执行路径可以是全局唯一的关键路径，这个路径可以维持整个任务的生命周期，任务执行路径的生成方式为：客户端名称+日期+随机数+任务编号。
图7是根据本申请实施例的一种可选的任务处理设备重启后恢复任务信息的流程 图，如图7所示，在一种可选的方案中，node上保留任务信息都在内存中，node重启后任务信息丢失。但是任务子进程与node主进程分离，node重启，任务正常执行。待node恢复后，调用恢复(recover)接口，获取driver中找到当前正在执行的任务列表(即上述的当前正在执行的至少一个任务信息)，每个任务包含有任务编号和任务执行路径。
步骤S313，任务处理设备通过检测在目标位置中记录的任务执行信息，来确定在主进程被重启时至少一个任务信息中已经被执行或未被执行的任务。
可选地，通过查询该任务执行路径下的日志文件，可以得到日志信息接口；通过查询该任务执行路径下的任务子进程文件，可以得到该任务信息是否已经被任务子进程执行；通过查询该任务执行路径下的退出码文件，可以得到该任务信息是否已经被任务子进程执行结束。
如图7所示，在一种可选的方案中，任务处理设备对每个任务基于任务执行路径长训是否存在对应的子进程的进程号，如果存在对应的子进程的进程号，则说明该任务已经被执行；如果不存在对应的子进程的进程号，则说明该任务为被执行。
通过上述步骤S311至步骤S313，任务处理设备主进程被重启后，从主控设备中获取当前正在执行的至少一个任务信息，并检测在目标位置中记录的任务执行信息，确定至少一个任务信息是否已经被执行或未被执行，从而实现主进程重启后，任务子进程执行任务信息的一致性。
在本申请上述实施例中，步骤S313任务处理设备通过检测在目标位置中记录的任务执行信息，来确定至少一个任务信息中已经被执行或未被执行的任务，包括如下步骤S3133至步骤S3135：
步骤S3131，任务处理设备在目标位置进行本地查询，确定是否已经存在与至少一个任务信息中任意一个任务标识相匹配的进程信息。
可选地，上述进程信息可以是子进程的进程号，每一个任务子进程在启动执行任务信息时，都会将自己的进程号对应的存放在该任务信息中。
在一种可选的方案中，任务处理设基于任务执行路径进行查询，根据任务标识确定至少一个任务信息中任意一个任务标识是否存在对应的子进程的进程号。
步骤S3133，如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程启动执行，任务处理设备仅在内存中注册该任务信息。
如图7所示，在一种可选的方案中，如果存在子进程的进程号，说明该任务信息已经被相应的任务子进程启动执行，则任务处理设备重新恢复任务信息(即上述的在内存 中注册该任务信息)，并读取该任务的任务进程信息(即上述的任务执行信息)。
步骤S3135，如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程所执行，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程执行任务信息。
如图7所示，在一种可选的方案中，如果不存在子进程的进程号，说明该任务信息未被相应的任务子进程执行，则任务处理设备在内存中保存该任务信息，通过启动代理进程来启动相应的任务子进程执行任务信息。
通过上述步骤S3133至步骤S3135，任务处理设备从在目标位置进行本地查询，确定是否存在相应的子进程的进程号，如果存在则重新记录任务信息；如果不存在，则通过代理进程启动任务子进程执行任务信息，从而实现主进程重启后，任务子进程执行任务信息的一致性。
在本申请上述实施例中，在步骤S3133任务处理设备仅在内存中注册该任务信息之前，上述方法还包括如下步骤S317至步骤S319：
步骤S315，任务处理设备在目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码。
可选地，上述退出码(returncode)用于表征该任务信息已经执行结束。
步骤S317，如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程执行结束，任务处理设备终止在内存中注册该任务信息，直接将对应的任务日志反馈给主控设备。
步骤S319，如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程执行完毕，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程重新执行任务信息。
在一种可选的方案中，如果任务处理设备确定存在对应的子进程的进程号，则判断是否存在任务信息对应的退出码，如果存在，则说明该任务信息已经被执行完毕，那么就不需要在内存中注册该任务信息，直接将相应的任务日志反馈给主控设备；如果不存在，则说明该任务未被执行完毕，需要通过代理进程启动相应的任务子进程重新执行该任务信息。
通过上述步骤S317至步骤S319，任务处理设备在目标位置进行本地查询，确定是否已经存在对应的退出码，如果存在则反馈任务日志，如果不存在则启动相应的任务子进程重新执行该任务信息，从而实现主进程重启后，恢复正在执行的任务信息，保证了 任务信息的状态。
在本申请上述实施例中，在步骤S301任务处理设备通过发起任务请求从主控设备中获取任务信息之后，上述方法还包括如下步骤S321至步骤S323：
步骤S321，从任务信息中提取路径参数，并根据路径参数生成任务执行路径。
在一种可选的方案中，任务处理设备在获取到任务信息时，会从任务信息中获取到一条路径参数，任务处理设备将该参数作为当前的任务执行路径，任务的执行日志、进程号、退出码都保存在该路径下。
步骤S323，代理进程根据任务执行路径将任务标识保存至目标位置。
在一种可选的方案中，任务子进程在执行启动时会将当前子进程的进程号保存到本地文件中，保存的路径为任务执行路径。
通过上述步骤S321至步骤S323，任务处理设备从任务信息中提取路径参数，生成任务执行路径，代理进程根据任务执行路径将任务标识保存在目标位置，从而保证主进程重启前后任务日志的完整性和可查性。
在本申请上述实施例中，上述方法还包括如下步骤S325：
步骤S325，任务处理设备的主进程根据任务标识监听任务信息是否被启动执行，并根据退出码监听任务信息是否执行结束。
通过上述步骤S325，任务处理设备可以通过任务标识和退出码监听任务信息是否被执行，以及是否执行结束，从而确保任务信息全部被执行且全部执行完毕。
下面结合图5、图7和图8详细介绍本申请的一种优选实施例。
如图8所示，以任务处理设备包括主进程、代理进程和任务子进程为应用场景，提供了一种可选的基于分布式系统的任务处理方法，该方法可以包括如下步骤S81至步骤S87：
S801，客户终端81提交任务。
可选地，客户可以通过客户终端81发布任务信息。
S802，任务处理设备163的主进程获取主控设备161中保存的任务信息。
可选地，任务处理设备163的主进程通过http请求从主控设备161中获取任务信息，任务信息保存在主控设备中。
在该实施例中，上述步骤S802的实现方式与本申请上述实施例中的步骤S301的实现方式一致，在此不再赘述。
S803，任务处理设备163通过代理进程启动任务子进程执行任务信息。
可选地，任务处理设备163的主进程收到任务信息之后，通过触发代理进程启动任务子进程执行任务信息，如图5所示，任务处理设备node中的主进程(AlisaNode)与任务子进程隔离，代理进程(Python进程)关联任务子进程，并且读写日志。
在该实施例中，上述步骤S803的实现方式与本申请上述实施例中的步骤S303的实现方式一致，在此不再赘述。
S804，任务处理设备163通过代理进程保存任务执行信息。
可选地，任务处理设备163在获取到任务信息时，从任务信息中获取路径参数，生成任务执行路径，代理进程将任务信息的任务标识、任务子进程的进程信息、日志信息和退出码保存在该任务执行路径对应的目标位置。
在该实施例中，上述步骤S804的实现方式与本申请上述实施例中的步骤S305的实现方式一致，在此不再赘述。
S805，主进程重启，任务处理设备163通过理进程离线启动任务子进程执行任务信息，并离线保存任务执行信息。
可选地，如果主进程被重启，代理进程离线启动任务子进程来执行任务信息，并将任务子进程执行任务信息生成的任务执行信息通过离线存储方式保存在目标位置。主进程被重启后，任务处理设备163从主控设备161中获取正在执行的任务信息，并从目标位置中查询任务执行信息，确定主进程被重启时该任务信息是否已经被执行或未被执行。
如图7所示，在一种可选的方案中，主进程被重启后，任务处理设备163调用恢复接口获取主控设备161中保存的任务列表，检测目标位置中是否存在对应的任务子进程的进程号，如果不存在，说明该任务信息未被相应的任务子进程执行，则任务处理设备163在内存中保存该任务信息，通过启动代理进程来启动相应的任务子进程执行任务信息；如果存在，说明该任务信息已经被相应的任务子进程启动执行，则任务处理设备163检测目标位置中是否存在对应的退出码，如果不存在，则通过代理进程启动对应的任务子进程重新执行该任务信息，如果存在，则返回任务日志给主控设备161。
S806，任务处理设备163返回任务状态给主控设备161。
可选地，主进程根据任务标识监听任务信息是否被启动执行，根据退出码监听任务信息是否执行结束，如果主进程监听该任务已经被执行，则反馈任务执行信息给主控设备161，如果监听该任务信息已经执行结束，则反馈任务日志给主控设备161。
在该实施例中，上述步骤S806的实现方式与本申请上述实施例中的步骤S325的实现方式一致，在此不再赘述。
S807，主控设备161将任务日志反馈给客户终端81。
可选地，主控设备161将执行结束的任务信息的任务日志反馈给客户端，经正在执行的任务信息的任务执行信息给客户终端81。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
实施例2
根据本申请实施例，还提供了一种用于实施上述基于分布式系统的任务处理方法的基于分布式系统的任务处理装置，如图9所示，该装置包括：第一获取模块91，第一执行模块93和第一保存模块95。
其中第一获取模块91用于任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息。第一执行模块93用于任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息。第一保存模块95用于任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选地，上述任务处理设备可以是部署在分布式系统中的集群的机器节点node，上述任务请求可以是http请求，上述主控设备可以是driver，主控设备的数据库中保存有任务处理设备需要的任务信息。上述代理进程可以是代理脚本，代理脚本可以作为主进程与任务子进程的通道。上述目标位置可以是代理进程的本地文件，路径可以是任务执行路径，上述任务信息的任务标识可以是任务id，上述任务子进程的进程信息可以是任务子进程的进程号pid。
需要说明的是，分布式系统中任务都在资源管理层Driver层做分发，driver管理node、任务。每台node上正在执行的任务列表都存储在driver的DB中，存储周期为任务生命周期。上述任务子进程执行任务信息所生成的日志信息为完整的日志信息，是指日志动作与任务处理设备重启不相关，并且只要代理进程存在，日志就可以正常读写。
此处需要说明的是，上述第一获取模块91，第一执行模块93和第一保存模块95对应于实施例一中的步骤S301至步骤S305，三个模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
本申请上述实施例二公开的方案中，任务处理设备通过发起任务请求从主控设备中获取主控设备中保存的任务信息，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息，任务处理设备通过代理进程将任务执行信息保存至目标位置。
容易注意到，由于任务处理设备中主进程和任务子进程通过代理进程进行隔离，使得主进程停止或者被重启，代理进程及任务子进程都不感知，可以正常执行任务信息，从而实现正在执行的进程在主进程启动过程中不受影响。并且由于任务处理设备主动向主控设备获取任务信息，即任务处理设备单向主控设备访问，因此通过改变http方式，利用数据库同步存储数据，从而实现任务处理设备、主控设备不在同一个网段环境下分布式系统的部署问题。能够支持百万级的任务执行，并且通过HTTP的方式，可以随时部署node，大大提升了系统的环境适应能力和可扩展能力。
由此，本申请提供的上述实施例二的方案解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
在本申请上述实施例中，如图10所示，上述装置还包括：第二执行模块101。
其中，第二执行模块101用于在主进程接收到任务信息之后，如果主进程被重启，则通过代理进程离线启动任务子进程来执行任务信息，并离线保存任务执行信息。
此处需要说明的是，上述第二执行模块101对应于实施例一中的步骤S307，上述模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
在本申请上述实施例中，如图11所示，上述装置还包括：第二获取模块111和第一确定模块113。
其中。第二获取模块111用于任务处理设备通过访问主控设备，来获取当前正在执行的至少一个任务信息，任务信息包括：任务标识和任务执行路径。第一确定模块113用于任务处理设备通过检测在目标位置中记录的任务执行信息，来确定在主进程被重启时至少一个任务信息中已经被执行或未被执行的任务。
可选地，上述任务执行路径可以是全局唯一的关键路径，这个路径可以维持整个任务的生命周期，任务执行路径的生成方式为：客户端名称+日期+随机数+任务编号。通过查询该任务执行路径下的日志文件，可以得到日志信息接口；通过查询该任务执行路径下的任务子进程文件，可以得到该任务信息是否已经被任务子进程执行；通过查询该任务执行路径下的退出码文件，可以得到该任务信息是否已经被任务子进程执行结束。
此处需要说明的是，上述第二获取模块111和第一确定模块113对应于实施例一中的步骤S311至步骤S313，两个模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
在本申请上述实施例中，如图12所示，第一确定模块113包括：第二确定模块121，第一注册模块123和第二注册模块125。
其中，第二确定模块121用于任务处理设备在目标位置进行本地查询，确定是否已经存在与至少一个任务信息中任意一个任务标识相匹配的进程信息。第一注册模块123用于如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程启动执行，任务处理设备仅在内存中注册该任务信息。第二注册模块125用于如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程所执行，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程执行任务信息。
可选地，上述进程信息可以是子进程的进程号，每一个任务子进程在启动执行任务信息时，都会将自己的进程号对应的存放在该任务信息中。
此处需要说明的是，上述第二确定模块121，第一注册模块123和第二注册模块125对应于实施例一中的步骤S3133至步骤S3135，三个模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
在本申请上述实施例中，如图13所示，上述装置还包括：第三确定模块131，反馈模块133和第三注册模块135。
其中，第三确定模块131用于任务处理设备在目标位置进行本地查询，确定是否已 经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码。反馈模块133用于如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程执行结束，任务处理设备终止在内存中注册该任务信息，直接将对应的任务日志反馈给主控设备。第三注册模块135用于如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程执行完毕，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程重新执行任务信息。
可选地，上述退出码(returncode)用于表征该任务信息已经执行结束。
此处需要说明的是，上述第三确定模块131，反馈模块133和第三注册模块135对应于实施例一中的步骤S317至步骤S319，三个模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
在本申请上述实施例中，如图14所示，上述装置还包括：生成模块141和第二保存模块143。
其中，生成模块141用于从任务信息中提取路径参数，并根据路径参数生成任务执行路径。第二保存模块143用于代理进程根据任务执行路径将任务标识保存至目标位置。
此处需要说明的是，上述生成模块141和第二保存模块143对应于实施例一中的步骤S321至步骤S323，两个模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
在本申请上述实施例中，如图15所示，上述装置还包括：监听模块151。
其中，监听模块151用于任务处理设备的主进程根据任务标识监听任务信息是否被启动执行，并根据退出码监听任务信息是否执行结束。
此处需要说明的是，上述监听模块151对应于实施例一中的步骤S325，上述模块与对应的步骤所实现的实例和应用场景相同，但不限于上述实施例一所公开的内容。需要说明的是，上述模块作为装置的一部分可以运行在实施例一提供的计算机终端20中。
实施例3
根据本申请实施例，还提供了一种用于实施上述基于分布式系统的任务处理方法的基于分布式系统的任务处理系统，如图16所示，该系统包括：主控设备161和任务处理设备163。
其中，主控设备161用于存储任务信息。
任务处理设备163与主控设备161进行通讯，用于通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息。
可选地，上述任务处理设备163可以是部署在分布式系统中的集群的机器节点node，上述任务请求可以是http请求，上述主控设备可以是driver，主控设备的数据库中保存有任务处理设备需要的任务信息。
需要说明的是，分布式系统中任务都在资源管理层Driver层做分发，driver管理node、任务。每台node上正在执行的任务列表都存储在driver的DB中，存储周期为任务生命周期。
如图3所示，在一种可选的方案中，任务处理设备主动发送http请求给主控设备，主控设备接收到http请求之后，提取http请求对应的任务信息，通过接口将该任务信息发送给任务处理设备。
任务处理设备163还用于任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息。
可选地，上述代理进程可以是代理脚本，代理脚本可以作为主进程与任务子进程的通道。
如图5所示，任务处理设备node中的主进程(AlisaNode)与任务子进程隔离，代理进程(Python进程)关联任务子进程，并且读写日志。结合图5和图6可知，在一种可选的方案中，node从driver中获取到任务之后，启动代理进程，传入任务执行需要的参数，代理进程启动具体的任务子进程执行任务信息。
任务处理设备163还用于通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选地，上述目标位置可以是代理进程的本地文件，路径可以是任务执行路径，上述任务信息的任务标识可以是任务id，上述任务子进程的进程信息可以是任务子进程的进程号pid。
需要说明的是，上述任务子进程执行任务信息所生成的日志信息为完整的日志信息，是指日志动作与任务处理设备重启不相关，并且只要代理进程存在，日志就可以正常读写。
如图6所示，在一种可选的方案中，任务子进程将所有的日志信息通过流方式返回给代理服务器，代理进程将接收到的日志信息、子进程的进程号、任务信息标号和退出 码保存到本地文件中。
本申请上述实施例三公开的方案中，任务处理设备通过发起任务请求从主控设备中获取主控设备中保存的任务信息，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息，任务处理设备通过代理进程将任务执行信息保存至目标位置。
容易注意到，由于任务处理设备中主进程和任务子进程通过代理进程进行隔离，使得主进程停止或者被重启，代理进程及任务子进程都不感知，可以正常执行任务信息，从而实现正在执行的进程在主进程启动过程中不受影响。并且由于任务处理设备主动向主控设备获取任务信息，即任务处理设备单向主控设备访问，因此通过改变http方式，利用数据库同步存储数据，从而实现任务处理设备、主控设备不在同一个网段环境下分布式系统的部署问题。能够支持百万级的任务执行，并且通过HTTP的方式，可以随时部署node，大大提升了系统的环境适应能力和可扩展能力。
由此，本申请提供的上述实施例三的方案解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
在本申请提供的一种可选实施例中，在主进程接收到任务信息之后，任务处理设备163还用于如果主进程被重启，则通过代理进程离线启动任务子进程来执行任务信息，并离线保存任务执行信息。
在本申请提供的一种可选实施例中，在主进程被重启之后，任务处理设备163还用于通过访问主控设备，来获取当前正在执行的至少一个任务信息，任务信息包括：任务标识和任务执行路径；通过检测在目标位置中记录的任务执行信息，来确定在主进程被重启时至少一个任务信息中已经被执行或未被执行的任务。
在本申请提供的一种可选实施例中，任务处理设备163还用于在目标位置进行本地查询，确定是否已经存在与至少一个任务信息中任意一个任务标识相匹配的进程信息；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程启动执行，任务处理设备仅在内存中注册该任务信息；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程所执行，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程执行任务信息。
在本申请提供的一种可选实施例中，任务处理设备163还用于在目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程执行结束， 任务处理设备终止在内存中注册该任务信息，直接将对应的任务日志反馈给主控设备；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程执行完毕，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程重新执行任务信息。
在本申请提供的一种可选实施例中，任务处理设备163还用于从任务信息中提取路径参数，并根据路径参数生成任务执行路径；其中，代理进程根据任务执行路径将任务标识保存至目标位置。
在本申请提供的一种可选实施例中，任务处理设备163还用于主进程根据任务标识监听任务信息是否被启动执行，并根据退出码监听任务信息是否执行结束。
实施例4
本申请的实施例可以提供一种计算机终端，该计算机终端可以是计算机终端群中的任意一个计算机终端设备。可选地，在本实施例中，上述计算机终端也可以替换为移动终端等终端设备。
可选地，在本实施例中，上述计算机终端可以位于计算机网络的多个网络设备中的至少一个网络设备。
在本实施例中，上述计算机终端可以执行基于分布式系统的任务处理方法中以下步骤的程序代码：任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选地，图17是根据本申请实施例的一种计算机终端的结构框图。如图17所示，该计算机终端A可以包括：一个或多个(图中仅示出一个)处理器171、存储器173、以及传输装置175。
其中，存储器173可用于存储软件程序以及模块，如本申请实施例中的基于分布式系统的任务处理方法和装置对应的程序指令/模块，处理器171通过运行存储在存储器内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的基于分布式系统的任务处理方法。存储器173可包括高速随机存储器，还可以包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器173可进一步包括相对于处理器远程设置的存储器，这些远程存储器可以通过网 络连接至终端A。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
处理器171可以通过传输装置调用存储器存储的信息及应用程序，以执行下述步骤：任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选的，上述处理器171还可以执行如下步骤的程序代码：在主进程接收到任务信息之后，如果主进程被重启，则通过代理进程离线启动任务子进程来执行任务信息，并离线保存任务执行信息。
可选的，上述处理器171还可以执行如下步骤的程序代码：任务处理设备通过访问主控设备，来获取当前正在执行的至少一个任务信息，任务信息包括：任务标识和任务执行路径；任务处理设备通过检测在目标位置中记录的任务执行信息，来确定在主进程被重启时至少一个任务信息中已经被执行或未被执行的任务。
可选的，上述处理器171还可以执行如下步骤的程序代码：任务处理设备在目标位置进行本地查询，确定是否已经存在与至少一个任务信息中任意一个任务标识相匹配的进程信息；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程启动执行，任务处理设备仅在内存中注册该任务信息；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程所执行，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程执行任务信息。
可选的，上述处理器171还可以执行如下步骤的程序代码：任务处理设备在目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程执行结束，任务处理设备终止在内存中注册该任务信息，直接将对应的任务日志反馈给主控设备；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程执行完毕，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程重新执行任务信息。
可选的，上述处理器171还可以执行如下步骤的程序代码：从任务信息中提取路径参数，并根据路径参数生成任务执行路径；其中，代理进程根据任务执行路径将任务标 识保存至目标位置。
可选的，上述处理器171还可以执行如下步骤的程序代码：任务处理设备的主进程根据任务标识监听任务信息是否被启动执行，并根据退出码监听任务信息是否执行结束。
采用本申请实施例，提供了一种基于分布式系统的任务处理方法。本任务处理设备通过发起任务请求从主控设备中获取主控设备中保存的任务信息，任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息，任务处理设备通过代理进程将任务执行信息保存至目标位置。解决了现有技术中分布式执行系统建立在特定的应用环境中，任务处理设备无法成功执行任务信息，导致任务丢失的技术问题。
本领域普通技术人员可以理解，图17所示的结构仅为示意，计算机终端也可以是智能手机(如Android手机、iOS手机等)、平板电脑、掌声电脑以及移动互联网设备(Mobile Internet Devices，MID)、PAD等终端设备。图17其并不对上述电子装置的结构造成限定。例如，计算机终端A还可包括比图17中所示更多或者更少的组件(如网络接口、显示装置等)，或者具有与图17所示不同的配置。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令终端设备相关的硬件来完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：闪存盘、只读存储器(Read-Only Memory，ROM)、随机存取器(Random Access Memory，RAM)、磁盘或光盘等。
实施例5
本申请的实施例还提供了一种存储介质。可选地，在本实施例中，上述存储介质可以用于保存上述实施例一所提供的基于分布式系统的任务处理方法所执行的程序代码。
可选地，在本实施例中，上述存储介质可以位于计算机网络中计算机终端群中的任意一个计算机终端中，或者位于移动终端群中的任意一个移动终端中。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在主控设备中保存任务请求对应的任务信息；任务处理设备的主进程接收到任务信息之后，通过触发代理进程启动对应的任务子进程来执行任务信息；任务处理设备通过代理进程将任务执行信息保存至目标位置，其中，任务执行信息包括如下任意一个或多个信息：任务信息的任务标识、任务子进程的进程信息、任务子进程执行任务信息所生成的日志信息和退出码。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：在 主进程接收到任务信息之后，如果主进程被重启，则通过代理进程离线启动任务子进程来执行任务信息，并离线保存任务执行信息。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：任务处理设备通过访问主控设备，来获取当前正在执行的至少一个任务信息，任务信息包括：任务标识和任务执行路径；任务处理设备通过检测在目标位置中记录的任务执行信息，来确定在主进程被重启时至少一个任务信息中已经被执行或未被执行的任务。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：任务处理设备在目标位置进行本地查询，确定是否已经存在与至少一个任务信息中任意一个任务标识相匹配的进程信息；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程启动执行，任务处理设备仅在内存中注册该任务信息；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程所执行，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程执行任务信息。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：任务处理设备在目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码；如果存在，则确定在主进程被重启时，任务信息已经被对应的任务子进程执行结束，任务处理设备终止在内存中注册该任务信息，直接将对应的任务日志反馈给主控设备；如果不存在，则确定在主进程被重启时，任务信息未被对应的任务子进程执行完毕，任务处理设备在内存中注册该任务信息，并启动对应的任务子进程重新执行任务信息。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：从任务信息中提取路径参数，并根据路径参数生成任务执行路径；其中，代理进程根据任务执行路径将任务标识保存至目标位置。
可选地，在本实施例中，存储介质被设置为存储用于执行以下步骤的程序代码：任务处理设备的主进程根据任务标识监听任务信息是否被启动执行，并根据退出码监听任务信息是否执行结束。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
在本申请的上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的技术内容，可通过其它的方式实现。其中，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，仅 仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，单元或模块的间接耦合或通信连接，可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅是本申请的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本申请的保护范围。

Claims (15)

  1. 一种基于分布式系统的任务处理方法，其特征在于，包括：
    任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在所述主控设备中保存所述任务请求对应的所述任务信息；
    所述任务处理设备的主进程接收到所述任务信息之后，通过触发代理进程启动对应的任务子进程来执行所述任务信息；
    所述任务处理设备通过所述代理进程将任务执行信息保存至目标位置，其中，所述任务执行信息包括如下任意一个或多个信息：所述任务信息的任务标识、所述任务子进程的进程信息、所述任务子进程执行所述任务信息所生成的日志信息和退出码。
  2. 根据权利要求1所述的方法，其特征在于，在所述主进程接收到所述任务信息之后，如果所述主进程被重启，则通过所述代理进程离线启动所述任务子进程来执行所述任务信息，并离线保存所述任务执行信息。
  3. 根据权利要求2所述的方法，其特征在于，在所述主进程被重启之后，所述方法还包括：
    所述任务处理设备通过访问所述主控设备，来获取当前正在执行的至少一个任务信息，所述任务信息包括：所述任务标识和任务执行路径；
    所述任务处理设备通过检测在所述目标位置中记录的所述任务执行信息，来确定在所述主进程被重启时所述至少一个任务信息中已经被执行或未被执行的任务。
  4. 根据权利要求3所述的方法，其特征在于，所述任务处理设备通过检测在所述目标位置中记录的所述任务执行信息，来确定所述至少一个任务信息中已经被执行或未被执行的任务，包括：
    所述任务处理设备在所述目标位置进行本地查询，确定是否已经存在与所述至少一个任务信息中任意一个任务标识相匹配的进程信息；
    如果存在，则确定在所述主进程被重启时，任务信息已经被对应的任务子进程启动执行，所述任务处理设备仅在内存中注册该任务信息；
    如果不存在，则确定在所述主进程被重启时，任务信息未被所述对应的任务子进程所执行，所述任务处理设备在所述内存中注册该任务信息，并启动所述对应的任务子进程执行所述任务信息。
  5. 根据权利要求4所述的方法，其特征在于，在所述任务处理设备仅在内存中注册该任务信息之前，所述方法还包括：
    所述任务处理设备在所述目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码；
    如果存在，则确定在所述主进程被重启时，任务信息已经被对应的任务子进程执行结束，所述任务处理设备终止在所述内存中注册该任务信息，直接将对应的任务日志反馈给所述主控设备；
    如果不存在，则确定在所述主进程被重启时，任务信息未被所述对应的任务子进程执行完毕，所述任务处理设备在所述内存中注册该任务信息，并启动所述对应的任务子进程重新执行所述任务信息。
  6. 根据权利要求3所述的方法，其特征在于，在任务处理设备通过发起任务请求从主控设备中获取任务信息之后，所述方法还包括：
    从所述任务信息中提取路径参数，并根据所述路径参数生成所述任务执行路径；
    其中，所述代理进程根据所述任务执行路径将所述任务标识保存至所述目标位置。
  7. 根据权利要求1至6中任意一项所述的方法，其特征在于，所述任务处理设备的所述主进程根据所述任务标识监听所述任务信息是否被启动执行，并根据所述退出码监听所述任务信息是否执行结束。
  8. 一种基于分布式系统的任务处理装置，其特征在于，包括：
    第一获取模块，用于任务处理设备通过发起任务请求从主控设备中获取任务信息，其中，在所述主控设备中保存所述任务请求对应的所述任务信息；
    第一执行模块，用于所述任务处理设备的主进程接收到所述任务信息之后，通过触发代理进程启动对应的任务子进程来执行所述任务信息；
    第一保存模块，用于所述任务处理设备通过所述代理进程将任务执行信息保存至目标位置，其中，所述任务执行信息包括如下任意一个或多个信息：所述任务信息的任务标识、所述任务子进程的进程信息、所述任务子进程执行所述任务信息所生成的日志信息和退出码。
  9. 根据权利要求8所述的装置，其特征在于，所述装置还包括：
    第二执行模块，用于在所述主进程接收到所述任务信息之后，如果所述主进程被重启，则通过所述代理进程离线启动所述任务子进程来执行所述任务信息，并离线保存所述任务执行信息。
  10. 根据权利要求9所述的装置，其特征在于，所述装置还包括：
    第二获取模块，用于所述任务处理设备通过访问所述主控设备，来获取当前正在执 行的至少一个任务信息，所述任务信息包括：所述任务标识和任务执行路径；
    第一确定模块，用于所述任务处理设备通过检测在所述目标位置中记录的所述任务执行信息，来确定在所述主进程被重启时所述至少一个任务信息中已经被执行或未被执行的任务。
  11. 根据权利要求10所述的装置，其特征在于，所述第一确定模块包括：
    第二确定模块，用于所述任务处理设备在所述目标位置进行本地查询，确定是否已经存在与所述至少一个任务信息中任意一个任务标识相匹配的进程信息；
    第一注册模块，用于如果存在，则确定在所述主进程被重启时，任务信息已经被对应的任务子进程启动执行，所述任务处理设备仅在内存中注册该任务信息；
    第二注册模块，用于如果不存在，则确定在所述主进程被重启时，任务信息未被所述对应的任务子进程所执行，所述任务处理设备在所述内存中注册该任务信息，并启动所述对应的任务子进程执行所述任务信息。
  12. 根据权利要求11所述的装置，其特征在于，所述装置还包括：
    第三确定模块，用于所述任务处理设备在所述目标位置进行本地查询，确定是否已经存在与已经被对应的任务子进程启动执行的任务信息相匹配的退出码；
    反馈模块，用于如果存在，则确定在所述主进程被重启时，任务信息已经被对应的任务子进程执行结束，所述任务处理设备终止在所述内存中注册该任务信息，直接将对应的任务日志反馈给所述主控设备；
    第三注册模块，用于如果不存在，则确定在所述主进程被重启时，任务信息未被所述对应的任务子进程执行完毕，所述任务处理设备在所述内存中注册该任务信息，并启动所述对应的任务子进程重新执行所述任务信息。
  13. 根据权利要求10所述的装置，其特征在于，所述装置还包括：
    生成模块，用于从所述任务信息中提取路径参数，并根据所述路径参数生成所述任务执行路径；
    第二保存模块，用于所述代理进程根据所述任务执行路径将所述任务标识保存至所述目标位置。
  14. 根据权利要求8至13中任意一项所述的装置，其特征在于，所述装置还包括：
    监听模块，用于所述任务处理设备的所述主进程根据所述任务标识监听所述任务信息是否被启动执行，并根据所述退出码监听所述任务信息是否执行结束。
  15. 一种基于分布式系统的任务处理系统，其特征在于，包括：
    主控设备，用于存储任务信息；
    任务处理设备，与所述主控设备进行通讯，用于通过发起任务请求从所述主控设备中获取任务信息，其中，在所述主控设备中保存所述任务请求对应的所述任务信息；
    所述任务处理设备还用于所述任务处理设备的主进程接收到所述任务信息之后，通过触发代理进程启动对应的任务子进程来执行所述任务信息；
    所述任务处理设备还用于通过所述代理进程将任务执行信息保存至目标位置，其中，所述任务执行信息包括如下任意一个或多个信息：所述任务信息的任务标识、所述任务子进程的进程信息、所述任务子进程执行所述任务信息所生成的日志信息和退出码。
PCT/CN2017/073721 2016-02-29 2017-02-16 基于分布式系统的任务处理方法、装置和系统 Ceased WO2017148268A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/080,854 US11256539B2 (en) 2016-02-29 2017-02-16 Task processing method, apparatus, and system based on distributed system
US17/677,773 US12346723B2 (en) 2016-02-29 2022-02-22 Task processing method, apparatus, and system based on distributed system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610112497.5A CN107133086B (zh) 2016-02-29 2016-02-29 基于分布式系统的任务处理方法、装置和系统
CN201610112497.5 2016-02-29

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/080,854 A-371-Of-International US11256539B2 (en) 2016-02-29 2017-02-16 Task processing method, apparatus, and system based on distributed system
US17/677,773 Continuation US12346723B2 (en) 2016-02-29 2022-02-22 Task processing method, apparatus, and system based on distributed system

Publications (1)

Publication Number Publication Date
WO2017148268A1 true WO2017148268A1 (zh) 2017-09-08

Family

ID=59720856

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/073721 Ceased WO2017148268A1 (zh) 2016-02-29 2017-02-16 基于分布式系统的任务处理方法、装置和系统

Country Status (4)

Country Link
US (2) US11256539B2 (zh)
CN (1) CN107133086B (zh)
TW (1) TWI753880B (zh)
WO (1) WO2017148268A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109861842A (zh) * 2018-11-23 2019-06-07 阿里巴巴集团控股有限公司 服务器集群的运维方法、装置、电子设备及存储介质

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10649824B1 (en) 2017-09-28 2020-05-12 Massachusetts Mutual Life Insurance Company Event subscription and management system
CN108153654B (zh) * 2017-12-01 2021-01-22 北京奇艺世纪科技有限公司 一种日志收集方法及装置
CN108900482B (zh) * 2018-06-13 2021-04-16 平安科技（深圳）有限公司 脚本的执行方法、服务器管理系统及存储介质
CN110569115B (zh) * 2019-08-15 2023-08-04 贝壳技术有限公司 多点部署的进程管理方法及进程的争夺方法
CN110532099B (zh) * 2019-08-30 2022-05-31 联想(北京)有限公司 资源隔离方法和装置以及电子设备和介质
CN110955460B (zh) * 2019-11-15 2023-10-03 北京金山云网络技术有限公司 一种服务进程启动方法、装置、电子设备和存储介质
CN111338893B (zh) * 2020-02-20 2023-07-18 深圳市腾讯计算机系统有限公司 进程日志处理方法、装置、计算机设备以及存储介质
CN111625326B (zh) * 2020-05-13 2023-06-27 北京字节跳动网络技术有限公司 任务管线执行方法、装置及电子设备
CN114050965B (zh) * 2020-07-22 2024-12-03 中兴通讯股份有限公司 查找中断设备的方法、从设备、主设备及存储介质
CN114154148B (zh) * 2020-09-07 2025-08-08 武汉安天信息技术有限责任公司 一种应用程序行为检测方法及设备
CN112732430B (zh) * 2021-01-20 2025-02-28 北京明略昭辉科技有限公司 一种对Spark任务分片所使用内存的限制方法及系统
CN113067869B (zh) * 2021-03-18 2022-12-27 北京字跳网络技术有限公司 节点控制方法及装置、节点管理方法及装置、分布式系统
CN113377543A (zh) * 2021-06-28 2021-09-10 上海商汤科技开发有限公司 任务处理系统、电子设备和存储介质
CN115604331B (zh) * 2021-06-28 2025-09-12 华为技术有限公司 一种数据的处理系统、方法以及装置
CN113608765A (zh) * 2021-08-05 2021-11-05 北京华云安信息技术有限公司 数据处理方法、装置、设备以及存储介质
CN113783731A (zh) * 2021-09-15 2021-12-10 云茂互联智能科技(厦门)有限公司 边缘服务器的控制方法、装置、存储介质及电子装置
CN113986542B (zh) * 2021-10-27 2025-05-06 北京达佳互联信息技术有限公司 新增任务的处理方法、装置、电子设备及存储介质
CN114237877B (zh) * 2021-12-03 2025-07-25 北京达佳互联信息技术有限公司 执行任务的方法、装置、电子设备及计算机程序产品
US20230401100A1 (en) * 2022-06-09 2023-12-14 Dell Products L.P. Lcs workload in-band resource device management system
CN115242578B (zh) * 2022-07-29 2024-12-03 招商局金融科技有限公司 Vpn连接管理方法、装置、计算机设备及存储介质
CN115905413A (zh) * 2023-01-17 2023-04-04 华东师范大学 一种基于Python协程和DataX的数据同步平台
CN116346838B (zh) * 2023-03-24 2026-02-03 北京星辰天合科技股份有限公司 分布式架构的设计方法、装置、存储介质和处理器
CN119557110B (zh) * 2025-01-24 2025-06-06 全芯智造技术有限公司 用于执行任务的方法、设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102929707A (zh) * 2012-11-06 2013-02-13 无锡江南计算技术研究所 并行任务动态分配方法
CN103677973A (zh) * 2013-09-01 2014-03-26 西安重装渭南光电科技有限公司 一种分布式多任务调度管理系统
CN104331327A (zh) * 2014-12-02 2015-02-04 山东乾云启创信息科技有限公司 大规模虚拟化环境中任务调度的优化方法及优化系统
US20150378406A1 (en) * 2014-06-27 2015-12-31 Fujitsu Limited Method of executing an application on a distributed computer system, a resource manager and a distributed computer system

Family Cites Families (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4633387A (en) * 1983-02-25 1986-12-30 International Business Machines Corporation Load balancing in a multiunit system
DE69220093T2 (de) 1992-06-18 1997-12-04 Ibm Verarbeitungsnetzwerk für verteilte anwendungsprogramme.
IT1278526B1 (it) * 1995-12-12 1997-11-24 Eric Verheecke Procedimento di automatizzazione di procedure informatiche e dispositivo per la sua attuazione
US5956511A (en) * 1996-03-29 1999-09-21 Kabushiki Kaisha Toshiba Program development support apparatus, program development support method, and storage medium therefor
US6446070B1 (en) 1998-02-26 2002-09-03 Sun Microsystems, Inc. Method and apparatus for dynamic distributed computing over a network
US6032172A (en) 1997-05-29 2000-02-29 Sun Microsystems, Inc. Distributed computer process scheduling mechanism
US6112225A (en) 1998-03-30 2000-08-29 International Business Machines Corporation Task distribution processing system and the method for subscribing computers to perform computing tasks during idle time
US7146505B1 (en) 1999-06-01 2006-12-05 America Online, Inc. Secure data exchange between date processing systems
US6775831B1 (en) 2000-02-11 2004-08-10 Overture Services, Inc. System and method for rapid completion of data processing tasks distributed on a network
US7082604B2 (en) 2001-04-20 2006-07-25 Mobile Agent Technologies, Incorporated Method and apparatus for breaking down computing tasks across a network of heterogeneous computer for parallel execution by utilizing autonomous mobile agents
US8024395B1 (en) 2001-09-04 2011-09-20 Gary Odom Distributed processing multiple tier task allocation
US7165256B2 (en) 2001-09-11 2007-01-16 Sun Microsystems, Inc. Task grouping in a distributed processing framework system and methods for implementing the same
JPWO2003083693A1 (ja) 2002-04-03 2005-08-04 富士通株式会社 分散処理システムにおけるタスクスケジューリング装置
US6988139B1 (en) 2002-04-26 2006-01-17 Microsoft Corporation Distributed computing of a job corresponding to a plurality of predefined tasks
US8230426B2 (en) * 2004-10-06 2012-07-24 Digipede Technologies, Llc Multicore distributed processing system using selection of available workunits based on the comparison of concurrency attributes with the parallel processing characteristics
US20070192500A1 (en) * 2006-02-16 2007-08-16 Infoexpress, Inc. Network access control including dynamic policy enforcement point
US20080270589A1 (en) * 2007-03-29 2008-10-30 Begrid, Inc. Multi-Source, Multi-Use Web Processing Through Dynamic Proxy Based Grid Computing Mechanisms
US8060486B2 (en) * 2007-05-07 2011-11-15 Hewlett-Packard Development Company, L.P. Automatic conversion schema for cached web requests
US8635686B2 (en) 2007-05-25 2014-01-21 Apple Inc. Integrated privilege separation and network interception
DE112008002634T5 (de) * 2007-09-27 2011-01-13 Hilton, Ronald N., Holladay Gerät, System und Verfahren zur systemübergreifenden Proxy-basierten Aufgabenentlastung
WO2009134772A2 (en) * 2008-04-29 2009-11-05 Maxiscale, Inc Peer-to-peer redundant file server system and methods
JP4666013B2 (ja) * 2008-06-20 2011-04-06 コニカミノルタビジネステクノロジーズ株式会社 画像処理装置、同装置におけるジョブ実行方法及びジョブ実行プログラム
US8843997B1 (en) 2009-01-02 2014-09-23 Resilient Network Systems, Inc. Resilient trust network services
KR20110097447A (ko) 2010-02-25 2011-08-31 삼성전자주식회사 인터럽트 프록시 기능을 구비한 시스템 온 칩 및 그에 따른 인터럽트 프록시 처리방법
US20120144157A1 (en) * 2010-12-06 2012-06-07 James Reginald Crew Allocation of Mainframe Computing Resources Using Distributed Computing
US9164806B2 (en) 2011-01-28 2015-10-20 Oracle International Corporation Processing pattern framework for dispatching and executing tasks in a distributed computing grid
US20120278120A1 (en) 2011-04-29 2012-11-01 Siemens Product Lifecycle Management Software Inc. Cross-schedule dependencies using proxy tasks
US9244745B2 (en) 2011-06-16 2016-01-26 Kodak Alaris Inc. Allocating tasks by sending task-available messages requesting assistance with an image processing task from a server with a heavy task load to all other servers connected to the computer network
US9424096B2 (en) 2011-06-16 2016-08-23 Kodak Alaris Inc. Task allocation in a computer network
US8769544B2 (en) * 2011-09-01 2014-07-01 Qualcomm Incorporated Method and system for managing parallel resource request in a portable computing device
RU2494453C2 (ru) 2011-11-24 2013-09-27 Закрытое акционерное общество "Лаборатория Касперского" Способ распределенного выполнения задач компьютерной безопасности
US9009564B2 (en) 2011-12-12 2015-04-14 Cleversafe, Inc. Storing data in a distributed storage network
US9430286B2 (en) 2011-12-12 2016-08-30 International Business Machines Corporation Authorizing distributed task processing in a distributed storage network
CN102857363B (zh) 2012-05-04 2016-04-20 运软网络科技(上海)有限公司 一种虚拟网络的自主管理系统和方法
FR2991535B1 (fr) * 2012-05-31 2015-05-01 Streamwide Procedes de delivrance de courriels a la demande, serveurs de courriels et programmes d'ordinateur mettant en oeuvre de tels procedes
US9298455B1 (en) * 2013-03-15 2016-03-29 Instart Logic, Inc. Provisional execution of dynamic content component
US9424027B2 (en) 2013-07-29 2016-08-23 Ralph Moore Message management system for information transfer within a multitasking system
JP2015177491A (ja) * 2014-03-18 2015-10-05 株式会社リコー 情報処理装置、情報処理システム、情報処理方法、及びプログラム
DE102014112478A1 (de) 2014-06-03 2015-12-03 Fujitsu Technology Solutions Intellectual Property Gmbh Verfahren zur Verteilung von Tasks zwischen Computersystemen, Computernetz-Infrastruktur sowie Computerprogramm-Produkt
US9658893B2 (en) 2015-05-06 2017-05-23 Runtime Design Automation Multilayered resource scheduling
US10691817B2 (en) 2016-05-06 2020-06-23 ZeroDB, Inc. Encryption for distributed storage and processing
CN107135268B (zh) 2017-05-22 2020-02-18 南京大学 基于信息中心网络的分布式任务计算方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102929707A (zh) * 2012-11-06 2013-02-13 无锡江南计算技术研究所 并行任务动态分配方法
CN103677973A (zh) * 2013-09-01 2014-03-26 西安重装渭南光电科技有限公司 一种分布式多任务调度管理系统
US20150378406A1 (en) * 2014-06-27 2015-12-31 Fujitsu Limited Method of executing an application on a distributed computer system, a resource manager and a distributed computer system
CN104331327A (zh) * 2014-12-02 2015-02-04 山东乾云启创信息科技有限公司 大规模虚拟化环境中任务调度的优化方法及优化系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109861842A (zh) * 2018-11-23 2019-06-07 阿里巴巴集团控股有限公司 服务器集群的运维方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
US20220179682A1 (en) 2022-06-09
US12346723B2 (en) 2025-07-01
CN107133086A (zh) 2017-09-05
US20190266011A1 (en) 2019-08-29
TW201732629A (zh) 2017-09-16
US11256539B2 (en) 2022-02-22
TWI753880B (zh) 2022-02-01
CN107133086B (zh) 2020-09-04

Similar Documents

Publication Publication Date Title
WO2017148268A1 (zh) 基于分布式系统的任务处理方法、装置和系统
CN110781007B (zh) 任务处理方法、装置、服务器、客户端、系统和存储介质
CN110389815B (zh) 任务处理方法、装置及系统
CN103685491B (zh) 一种应用服务提供方法、系统及相关设备
CN111045893B (zh) 监控任务的执行方法、装置及系统、存储介质、电子装置
CN107391276B (zh) 分布式监听方法、监听控制装置及系统
CN106059825A (zh) 一种分布式系统及配置方法
CN106650427B (zh) 沙箱运行环境的检测方法及检测装置
CN111506283B (zh) 图像显示方法、装置及系统
CN108667639B (zh) 一种私有云环境下的资源管理方法及管理服务器
US20170293537A1 (en) Management system for virtual machine failure detection and recovery
CN111506388B (zh) 容器性能探测方法、容器管理平台及计算机存储介质
WO2015192608A1 (zh) 安卓系统中app的更新方法、终端及存储介质
CN106549789A (zh) 一种实现服务器安装的方法及系统
CN112181677A (zh) 业务的处理方法和装置、存储介质、电子装置
CN105553724A (zh) 处理数据请求及获取服务器信息的方法及装置
WO2017140098A1 (zh) 一种作者关注方法、终端、服务器和系统
CN109408135A (zh) 双系统升级方法、装置、设备及计算机可读存储介质
CN108197958B (zh) 统计线下黄牛的方法、装置及存储介质
CN118862914B (zh) 扫码登录方法、装置、计算机设备及存储介质
CN108270832B (zh) 一种故障重放方法和装置
CN114489956A (zh) 一种基于云平台的实例启动方法及装置
CN112419221B (zh) 图像检测方法及装置、网元、存储介质
CN108111343A (zh) 基于云平台实现终端监控的方法、设备及计算机存储介质
TWI690849B (zh) 應用升級與關閉應用之方法及裝置

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17759119

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17759119

Country of ref document: EP

Kind code of ref document: A1