WO2019019400A1 - 任务分布式处理方法、装置、存储介质和服务器 - Google Patents

任务分布式处理方法、装置、存储介质和服务器 Download PDF

Info

Publication number
WO2019019400A1
WO2019019400A1 PCT/CN2017/104970 CN2017104970W WO2019019400A1 WO 2019019400 A1 WO2019019400 A1 WO 2019019400A1 CN 2017104970 W CN2017104970 W CN 2017104970W WO 2019019400 A1 WO2019019400 A1 WO 2019019400A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
terminal
processing
server
distributed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/104970
Other languages
English (en)
French (fr)
Inventor
严月强
伍琼珂
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OneConnect Financial Technology Co Ltd Shanghai
Original Assignee
OneConnect Financial Technology Co Ltd Shanghai
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OneConnect Financial Technology Co Ltd Shanghai filed Critical OneConnect Financial Technology Co Ltd Shanghai
Publication of WO2019019400A1 publication Critical patent/WO2019019400A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network

Definitions

  • the present application relates to the field of data processing technologies, and in particular, to a task distributed processing method, apparatus, storage medium, and server.
  • the distributed computing system is basically set up on the server cluster, and the tasks to be processed are allocated to the respective nodes on the server for processing according to Zookeeper, so that the processing of the task and the upper allocation are optimized, and the server is improved.
  • Resource utilization efficiency the resources of the server are limited. When the amount of data to be processed is large, the task is more, and even exceeds the processing load of the server, the task efficiency of the traditional method is not high.
  • a task distributed processing method, apparatus, storage medium, and server are provided.
  • a task distributed processing method is applied to a distributed task scheduling system, where the distributed task scheduling system includes a server and a plurality of terminals, wherein the server is a master node in the distributed task scheduling system, and each The terminal is an execution node in the distributed task scheduling system;
  • the method includes: the server receiving a task processing request sent by the terminal; selecting at least one task to be assigned, assigning the selected task to the terminal for processing; and receiving a task processing result fed back by the terminal; Modifying the task status of the corresponding task according to the task processing result.
  • a task distributed processing device for use in a distributed task scheduling system, comprising: a request receiving module, configured to receive a task processing request sent by a terminal, where the terminal is an execution node in the distributed task scheduling system; a distribution module, configured to select at least one task to be assigned, to assign the selected task to the terminal for processing; and a task state modification module, configured to receive a task processing result fed back by the terminal; Modify the task status of the corresponding task.
  • One or more computer readable non-volatile storage media storing computer readable instructions, when executed by one or more processors, causing the one or more processors to perform a task a distributed processing method, the method being applied to a distributed task scheduling system, the distributed task scheduling system comprising a server and a plurality of terminals, wherein the server is a master node in the distributed task scheduling system, each The terminal is an execution node in the distributed task scheduling system; when the computer readable instructions are executed by one or more processors, causing the one or more processors to perform the steps of: receiving the terminal to send The task processing request; selecting at least one task to be assigned, assigning the selected task to the terminal for processing; receiving the task processing result fed back by the terminal; and modifying the task status of the corresponding task according to the task processing result.
  • a server for use in a distributed task scheduling system includes a memory and a processor, wherein the memory stores computer readable instructions, the distributed task scheduling system includes the server and a plurality of terminals, and the server is a master node in the distributed task scheduling system, each of the terminals being an execution node in the distributed task scheduling system; when the instructions are executed by the processor, causing the processor to perform the following steps Receiving a task processing request sent by the terminal; selecting at least one task to be assigned, and assigning the selected task to the terminal for processing;
  • 1 is an application environment diagram of a task distributed processing method in an embodiment
  • FIG. 2 is a flow chart of a task distributed processing method in an embodiment
  • FIG. 3 is an application environment diagram of a distributed processing method for Crash parsing tasks in an embodiment.
  • FIG. 5 is a structural block diagram of a task distributed processing apparatus in an embodiment
  • FIG. 6 is a structural block diagram of a task distributed processing apparatus in another embodiment.
  • Figure 7 is a diagram showing the internal structure of a server in one embodiment.
  • the task distributed processing method provided by the embodiment of the present application can be applied to the application environment shown in FIG. 1 .
  • the server 110 is connected to a plurality of terminals 120.
  • the terminal can be any one of a notebook computer, a desktop computer, a tablet, and the like.
  • Each terminal 120 can send a task processing request to the server.
  • the server 110 may select at least one task to be assigned according to the received task processing request, and allocate the selected task to the terminal 120 that sends the task processing request for processing.
  • the terminal 120 may perform an analysis process on the received task, and send the task processing result to the server 110.
  • the server 110 modifies the task status of the corresponding task according to the received task processing result. Thereby, distributed processing of tasks through the terminal is realized.
  • a task distributed processing method is provided, which can be applied to the application environment shown in FIG. 1, and specifically includes the following steps:
  • Step S202 Receive a task processing request sent by the terminal.
  • the terminal is an employee terminal or other belonging terminal in a company or a group corresponding to the task to be processed.
  • Each terminal may be a task execution node in a DTS system (Distributed Task Scheduler).
  • the DTS system is a general-purpose, plug-in, distributed extended execution framework that supports distributed processing and parsing of tasks. This task refers to a large number of tasks that each task can be distributed, including application compilation tasks, log report statistics tasks, Crash resolution tasks, and so on.
  • Crash parsing task Take the Crash parsing task as an example to illustrate that the Crash can report the Crash stack information reported by an app (application), and can analyze one or more adaptive Crash stack information to be parsed as a Crash. task. The more Crash stack information is collected, the more Crash parsing tasks are pending.
  • the terminal may send a task processing request to the server in the DTS system, where the task processing request carries the terminal identifier.
  • the server is the master node in the DTS system and is responsible for managing tasks and all execution nodes.
  • the management of the task includes receiving, storing, distributing, recording, and monitoring tasks; the management of the node includes receiving the heartbeat packet of the executing node, monitoring the working state of the executing node, and the like.
  • the server can receive the task processing request sent by each terminal in real time, and obtain the terminal identifier in the terminal, so that the corresponding terminal is determined according to the terminal identifier.
  • Step S204 Select at least one task to be assigned, and assign the selected task to the terminal for processing.
  • the server may select a preset number of tasks from the tasks to be allocated in the database, and send the task information of the selected task to the terminal, so that the terminal processes the task.
  • the status of the task includes the status to be assigned, in process, successful in processing, and failed in processing.
  • the preset number may be a preset number or a plurality of suitable numbers, so that the terminal can process multiple tasks at the same time, thereby improving the processing efficiency.
  • the manner of selecting the task to be assigned may include a random selection method and a manner of selecting according to the time sequence generated by the task.
  • the terminal may receive the allocated task, parse the task according to the corresponding task resolution manner, and send the processing result to the server.
  • the method further comprises: modifying the state of the selected task to be processed.
  • Step S206 Receive a task processing result fed back by the terminal, and modify a task state of the corresponding task according to the task processing result.
  • the server may receive the task processing result that is fed back after the terminal processes the task.
  • the task processing results include task processing success and task processing failure.
  • the server may modify the task status of the corresponding assigned task according to the task processing result. For example, when the task is successfully processed, its task status is modified to be processed successfully. If the processing fails, its task status is changed to processing failure.
  • each terminal may be a terminal that is independent of each other in task processing. That is, in the process of task processing, the terminals are processed independently of each other, and there is no connection relationship. In the DTS system, since each terminal is only connected to the server, the independence of the task processing can be maintained to improve the efficiency of the terminal for task processing.
  • the task processing request sent by the terminal is received; at least one task to be assigned is selected, and the selected task is allocated to the terminal for processing; then the task processing result fed back by the terminal is received; and the corresponding task is modified according to the task processing result.
  • Task status the terminal is implemented as a task execution node in the DTS system, and the task is processed.
  • the greater the number of terminals the more distributed processing tasks that can be allocated on the server in a timely manner. Therefore, the server is not required to be added, or the server is expanded, and a plurality of tasks are allocated to each existing terminal for processing by using the existing terminal, thereby improving the efficiency of task processing.
  • the selecting the at least one task to be assigned includes: selecting at least one task identifier to be assigned; reading the task corresponding to each task identifier from the primary database, when reading from the primary database fails When the task corresponding to each task ID is read from the standby database.
  • the task identifier is used to uniquely identify the corresponding task, and the task corresponding to the task
  • the information is associated with a number of characters, letters, and the like of the preset number of digits.
  • the task ID can be the task number.
  • two databases are set in advance, which are a primary database and a standby database, respectively.
  • the task information of each unprocessed task is stored in both the primary database and the standby database, and can be queried from the primary database or the standby database according to the task number.
  • the primary database is the default priority database and the standby database is the standby database.
  • the server may first query the corresponding task from the primary database according to the task identifier, and read the task information. When the reading from the primary database fails, the task information of the corresponding task is read from the standby database, so that the success rate of the extraction of the task can be further improved.
  • the task to be allocated can be read from the standby database when the primary database fails. It can prevent the database from malfunctioning, resulting in a backlog of pending tasks and further impact.
  • the task processing request carries the Internet Protocol address of the terminal; and the selected task is allocated to the terminal for processing, including: assigning the selected task to the terminal for processing according to the Internet Protocol address.
  • the Internet Protocol Address is the local IP address of the terminal. Because the terminal is an employee terminal and needs to handle other work items, there may be operations such as shutting down, restarting, disconnecting the network, and switching the network at any time, so that the IP address or other parameter characteristics of the terminal change. Compared to the traditional zookeeper, you need to configure the IP and port of all the execution nodes before starting.
  • the IP address of the terminal is carried in the task processing request, so that the server can send the assigned task to the IP address, and the IP address of the instant terminal is dynamically changed, and the changed IP address can also be used according to the changed IP address.
  • the task information is successfully sent to the terminal, and the terminal performs task processing.
  • the method includes: receiving a heartbeat packet sent by the terminal, to determine, according to the heartbeat packet, that the terminal is processing the selected task.
  • the terminal may serve the service according to a preset frequency after receiving the assigned task.
  • the heartbeat packet is sent, and the heartbeat packet includes the IP address of the terminal, so that the server can identify the corresponding IP address according to each heartbeat packet received, so that the corresponding terminal can be determined.
  • the server can know that the corresponding terminal is processing the assigned task, thereby implementing monitoring of task processing for each terminal.
  • the method includes: detecting whether the task processing result is received within a preset duration, and if not, reassigning the selected task to another sending task processing request.
  • the terminal is processed.
  • the server further sets a duration threshold, that is, the preset duration described above.
  • the duration threshold is a duration for judging whether the processing of the task is abnormal.
  • the duration threshold can be set according to the historical average time required to complete the processing of the task. It may be a length of time that exceeds the average length. For example, if the average duration of processing a task is 5 minutes, the duration threshold can be set to 8 minutes or 10 minutes.
  • the server can start the processing time of the real-time statistical task, and compare the processing duration and the preset duration. When the processing duration reaches the preset duration, the server does not receive the task processing result sent by the terminal. Then, it is determined that the processing of the task by the terminal fails. And re-set the task status of the assigned task to the status to be assigned, and re-allocate the task according to the received task processing request of other terminals.
  • the success rate of the task processing completion may be further mentioned.
  • the task distributed processing method further includes: calculating a number of allocations of the selected tasks; and when the number of allocations reaches a preset number of times, if the task processing result is not received within the preset duration, or is received If the task processing fails the task processing result, the task status of the selected task is modified to handle the abnormal state.
  • the server may further count the number of allocations of the reassigned tasks, and compare the number of allocations with the preset number of times.
  • the preset number of times may be a suitable number of times, for example, may be 3 times or 5 times.
  • the task When a task that has been assigned a preset number of times occurs, the task has not been successfully processed.
  • the result of the task processing may be caused by problems in the task itself and difficult to be successfully processed. Therefore, if it can be determined that the task has failed to be processed multiple times, the reallocation of the task can be terminated, and the state of the task is set to a state in which the abnormality is handled.
  • the server may summarize task information that is set to handle the abnormal state, send the task information to the corresponding administrator terminal, and cause the corresponding administrator to further detect the task.
  • the aggregation may be performed according to a preset summary frequency, for example, by frequency once every hour.
  • the task can be prevented from being redistributed indefinitely, thereby wasting resources of the terminal and the server.
  • the foregoing method further includes: counting the number of tasks completed by each terminal, and sorting the terminals according to the number of completed tasks.
  • the server can count the number of tasks completed by each terminal within a preset time period. Specifically, the number of task processing results successfully processed by the terminal with the same terminal identifier may be counted according to the terminal identifier carried in the received task processing request.
  • the terminal identifier may be information such as a user name that can uniquely identify the identity of the terminal.
  • the number of tasks completed by each terminal can be counted on a per-weekly or daily basis, and sorted according to the number of completed tasks.
  • the sorting information includes information such as a terminal identifier, a number of completed tasks, and a ranking.
  • the ranking information may be sent to the corresponding terminal, so that the terminal can know the number of tasks and rankings that are completed in the preset time period.
  • the server may further select, from the sorting, the number of tasks that are more than the preset number, or the number of completed tasks exceeds the preset number, and broadcast the reward information to the selected terminal to each terminal to further improve each The enthusiasm of end users to participate in task processing.
  • an application environment diagram of a distributed processing method for a Crash parsing task in an embodiment includes a server 110, a terminal 120, a database 130, and a Crash parsing platform.
  • the server 110 includes a primary server 112 and a backup server 112.
  • the database also includes a primary database 132 and a standby database 134.
  • the Crash parsing platform is a Crash data provider.
  • the Crash parsing task needs to obtain and upload a Crash file.
  • the Crash parsing platform is responsible for providing the Crash source data and receiving the parsed result upload.
  • Each terminal 120 can send a task processing request to the server 110, and the server 110 selects at least one task to be assigned from the database 130 according to the received task processing request, and allocates to the interface defined by the terminal 110 to the task.
  • the terminal 110 can be an interface implemented by http.
  • the terminal 110 receives the task information sent by the server 110 and interacts with the Crash platform to implement parsing processing of the Crash.
  • the terminal 110 sends the task processing result to the server 110, and the server 110 modifies the task status of the corresponding task according to the task processing result, thereby implementing the processing of the Crash parsing task.
  • another task distributed processing method which is a Crash parsing task.
  • the method includes:
  • Step S401 The terminal sends a task processing request to the server, where the task processing request carries the Internet Protocol address of the terminal.
  • the task processing request is a task processing request of the Crash parsing task.
  • the task can be processed for the IOS Crash resolution task
  • the terminal is a terminal that deploys a Mac OSX system, such as a Mac notebook or a Mac desktop computer.
  • the server can generate multiple Crash resolution tasks in advance or in real time and write this into the primary database as well as the standby database.
  • the task identifier is set for the generated Crash parsing task and the task status is set to be assigned.
  • the Crash parsing task contains the Crash file to be parsed.
  • an interface and a web page creation task may be provided, so that the Crash resolution task created by the terminal through the interface and the web page may be received.
  • the server may encapsulate multiple Crash files into one compressed package and encapsulate multiple compressed packages into one Crash parsing task, thereby causing a Crash to be assigned to the terminal.
  • the parsing task contains multiple Crash files to be parsed, which can improve the allocation processing efficiency of Crash files.
  • the terminal can detect if it is in an idle state, and if so, send a task processing request to the server.
  • the idle state includes a state in which the CPU utilization is lower than the preset utilization rate and the memory usage is lower than the preset occupancy rate.
  • the preset utilization rate and the preset occupancy rate may be set to any suitable value, for example, may be 50%.
  • Step S402 the server receives the task processing request, selects at least one task to be assigned, and assigns the selected task to the terminal for processing according to the Internet Protocol address.
  • the server may select a preset number of task identifiers of the tasks to be assigned, and query the tasks corresponding to the task identifiers from the primary database in priority.
  • the queried task is read, and the IP address of the terminal carried in the request processing request is sent to the corresponding terminal, so that the terminal performs task processing.
  • the server may preset a first task table for storing tasks to be assigned, where the task information of the tasks to be assigned may be stored in the order in which the tasks are generated.
  • the task identifier of the task can be stored.
  • the task identifier of the task that has not been assigned is selected from the table in the order of generation.
  • the corresponding task information is read from the primary database according to the task identifier. Wherein, when the read from the primary database fails, the task can be read from the standby database.
  • the server can be a server cluster containing a primary server and a secondary server.
  • the main server can be called by default for task allocation and monitoring.
  • the standby server can be called to perform task allocation and monitoring.
  • the method further includes: modifying the state of the assigned task to be processed. Further, the database logic of the assignable task sets transaction isolation, thereby avoiding the case where the same task is assigned to multiple terminals and repeated execution occurs.
  • the task may be moved from the first task table to the second task table for storing the process, or the task status of the task may be marked as being processed directly in the first task table. Task status.
  • Step S403 the terminal receives the task assigned by the server, processes the task through the Crash analysis platform, and sends the task processing result to the server.
  • the terminal after receiving the task assigned by the server, the terminal can interact with the Crash to implement parsing processing of each Crash file in the task, generate a task processing result, and send the result to the server.
  • the task processing result may include the processing result of each Crash file.
  • Processing results include processing success and processing failure.
  • the terminal may also send a heartbeat packet to the server according to a preset frequency to inform the terminal that the parsing process of the task is in progress.
  • step S404 the server detects whether the task processing result is received within the preset duration. If yes, step S405 is performed; otherwise, step S406 is performed.
  • the server After the server assigns the selected task to the terminal for processing, the server starts to process the processing time of the task, and if the task processing result is received within the preset duration.
  • the task processing result includes a corresponding task identifier, so that the server can identify the task to which the task processing result belongs according to the task.
  • the server may extract a task that has all the states as being processed, and the allocated time is longer than the preset time, and then resets the state of the task to be allocated.
  • the preset duration can be set according to the experience value. Assuming that the normal execution time of a task is 2 minutes and the maximum execution time is 5 minutes, the preset duration can be set to 6 minutes or 8 minutes.
  • Step S405 modifying the task status of the corresponding task according to the task processing result.
  • step S406 the selected task is reassigned to other terminals that send the task processing request for processing.
  • the task is further modified into a state to be allocated, and is again stored in the first task table for reallocation. If the task status is successful, the task is placed in a third task table for storing the processing success. If the task status is a processing exception, the task is stored in a fourth task table for storing processing exceptions.
  • Each task table includes a task identifier of the corresponding task, so that the server can periodically record according to the fourth task table. The service identifier reads the corresponding task information from the primary database or the standby database and sends it to the administrator terminal for the administrator to detect.
  • the terminal may set up multiple threads for task assignment requests and processing. For example, two threads can be set, and each thread is cyclically "task request -> execution -> processing result report -> rest preset duration", thereby improving terminal utilization and task processing efficiency.
  • the above-mentioned task distributed processing method can allocate the task to multiple terminals for processing by using the terminal as an execution node in the DTS system, which can alleviate the pressure of the server on the task processing and improve the efficiency of the task processing.
  • a task distributed processing apparatus which is applied to a distributed task scheduling system, and includes:
  • the request receiving module 502 is configured to receive a task processing request sent by the terminal, where the terminal is an execution node in the distributed task scheduling system.
  • the task assignment module 504 is configured to select at least one task to be assigned, and assign the selected task to the terminal for processing.
  • the task status modification module 506 is configured to receive a task processing result fed back by the terminal, and modify a task status of the corresponding task according to the task processing result.
  • the task assignment module 504 is further configured to select at least one task identifier to be assigned status; read a task corresponding to each task identifier from the primary database, and when the read from the primary database fails, from the standby database. Read the task corresponding to each task ID.
  • the task processing request carries the Internet Protocol address of the terminal; the task assignment module 504 is further configured to allocate the selected task to the terminal for processing according to the Internet Protocol address.
  • the task status modification module 506 is further configured to receive a heartbeat packet sent by the terminal to determine, according to the heartbeat packet, that the terminal is processing the selected task.
  • the task assignment module 504 is further configured to detect whether the task processing result is received within a preset duration, and if not, re-allocate the selected task to another terminal that sends the task processing request for processing.
  • the apparatus further includes:
  • the sorting module 508 is configured to count the number of tasks completed by each terminal, and sort the terminals according to the number of completed tasks.
  • the various modules in the above-described task distributed processing device may be implemented in whole or in part by software, hardware, and combinations thereof.
  • the network interface may be an Ethernet card or a wireless network card.
  • the above modules may be embedded in the hardware in the processor or in the memory in the server, or may be stored in the memory in the server, so that the processor calls the corresponding operations of the above modules.
  • the processor can be a central processing unit (CPU), a microprocessor, a microcontroller, or the like.
  • the task distributed processing apparatus may be implemented in the form of a computer program, which may be run on a server as shown in FIG. 7, and the non-volatile storage medium of the server may be
  • the various program modules that make up the task distributed processing device are stored.
  • the request receiving module 502, the task assigning module 504, and the task state modifying module 506 as shown in FIG. 5 may be included.
  • Each of the program modules includes computer readable instructions for causing the server to perform the steps in the task distributed processing method of various embodiments of the present application described in the present specification, for example, the server may be as shown in FIG.
  • the behavior request receiving module 502 in the illustrated task distributed processing device receives the task processing request sent by the terminal, the terminal is an execution node in the distributed task scheduling system; and at least one task to be assigned is selected by the task assignment module
  • the task is allocated to the terminal for processing; and the task state modification module 506 receives the task processing result fed back by the terminal; and the task state of the corresponding task is modified according to the task processing result.
  • one or more non-transitory computer readable storage media containing computer readable instructions are provided that, when executed by one or more processors, cause the processor to implement a A task distributed processing method, the method is applied to a distributed task scheduling system, the distributed task scheduling system comprising a server and a plurality of terminals, wherein the server is a master node in the distributed task scheduling system, and each terminal For the execution node in the distributed task scheduling system, the processor executes the steps of the task distributed processing method provided by the various embodiments described above when the instruction is executed.
  • the processor when the computer readable instructions are executed by one or more processors, the processor is caused to perform the steps of: receiving a task processing request sent by the terminal; selecting at least one task to be assigned, selecting the selected The task is allocated to the terminal for processing; the task processing result fed back by the terminal is received; and the task state of the corresponding task is modified according to the task processing result.
  • the processor when the computer readable instructions are executed by one or more processors, causing the processor to perform at least one task to be assigned a state, comprising: selecting at least one task identifier to be assigned a status; The task corresponding to each task identifier is read in the primary database. When the read from the primary database fails, the task corresponding to each task identifier is read from the standby database.
  • the task processing request carries an internet protocol address of the terminal; when the computer readable instruction is executed by the one or more processors, causing the processor to perform the task of assigning the selected task to the terminal for processing, including : Assign the selected task to the terminal for processing according to the Internet Protocol address.
  • the computer readable instructions when executed by one or more processors, causing the processor to perform the process of assigning the selected task to the terminal for processing, further comprising the step of: receiving a heartbeat sent by the terminal Packet to determine that the terminal is processing the selected task based on the heartbeat packet.
  • the processor when the computer readable instructions are executed by one or more processors, causing the processor to perform the process of assigning the selected task to the terminal for processing, further comprising the step of: detecting whether the preset is The task processing result is received within the duration, and if not, the selected task is reassigned to other terminals that send the task processing request for processing.
  • the processor when the computer readable instructions are executed by one or more processors, the processor is further caused to perform the steps of counting the number of tasks completed by each terminal and sorting the terminals according to the number of completed tasks.
  • a server for use in a distributed task scheduling system, including a memory and a processor, the memory storing computer readable instructions, the distributed task scheduling system including the server and a plurality of terminals
  • the server is a master node in the distributed task scheduling system, and each terminal is an execution node in the distributed task scheduling system, and the instruction is When the processor executes, the processor is caused to perform the steps of the task distributed processing method provided by the various embodiments described above.
  • the processor when the instruction is executed by the processor, the processor is configured to: receive a task processing request sent by the terminal; select at least one task to be assigned, assign the selected task to the terminal for processing; and receive the terminal Feedback task processing result; modify the task status of the corresponding task according to the task processing result.
  • FIG. 7 is an internal structure diagram of a server in one embodiment.
  • the server includes a processor, memory, and network interface connected through a system bus including connections through a system bus.
  • the processor of the server is used to provide computing and control capabilities to support the operation of the entire terminal.
  • the memory is used to store data, instruction codes, etc., and the network interface is used for network communication with the terminal. For example, the selected task or the like can be sent to the terminal.
  • At least one computer readable instruction is stored on the memory, and the computer readable instructions are executable by the processor to implement the task-distributed processing method for the server provided in the embodiments of the present application.
  • the memory may include a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or the like.
  • the memory includes a non-volatile storage medium and an internal memory.
  • the server's non-volatile storage medium stores operating system and computer readable instructions.
  • the computer readable instructions are executable by a processor for implementing the task distributed processing methods provided by the various embodiments above.
  • the internal memory in the terminal provides a cached operating environment for operating systems, databases, and computer readable instructions in a non-volatile storage medium.
  • the network interface may be an Ethernet card or a wireless network card or the like for communicating with an external terminal or server.
  • the server can be implemented with a stand-alone server or a server cluster consisting of multiple servers. It will be understood by those skilled in the art that the structure shown in FIG. 7 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied.
  • the specific server may include a ratio. More or fewer components are shown in the figures, or some components are combined, or have different component arrangements.
  • the processor when the instruction is executed by the processor, causing the processor to perform the task of selecting at least one state to be allocated, including: selecting at least one task identifier to be assigned; reading each task identifier from the primary database Corresponding task, when reading from the primary database fails, from standby The task corresponding to each task ID is read in the database.
  • the task processing request carries the Internet Protocol address of the terminal; when the instruction is executed by the processor, the processor performs the process of assigning the selected task to the terminal for processing, including: selecting the selected according to the Internet Protocol address. Tasks are assigned to the terminal for processing.
  • the method when the instruction is executed by the processor, after the processor performs the process of assigning the selected task to the terminal for processing, the method further includes the step of: receiving a heartbeat packet sent by the terminal, to determine, according to the heartbeat packet, that the terminal is Process selected tasks.
  • the method when the instruction is executed by the processor, after the processor performs the process of assigning the selected task to the terminal for processing, the method further includes the step of: detecting whether the task processing result is received within the preset duration, If not, the selected task is reassigned to other terminals that send the task processing request for processing.
  • the processor when the instructions are executed by the processor, the processor further performs the steps of: counting the number of tasks completed by each terminal, and ordering the terminals according to the number of completed tasks.
  • the readable storage medium which when executed, may include the flow of an embodiment of the methods as described above.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer And Data Communications (AREA)
  • Hardware Redundancy (AREA)

Abstract

一种任务分布式处理方法，应用于分布式任务调度系统中，所述分布式任务调度系统包括服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点。所述方法包括：服务器接收终端发送的任务处理请求（S202）；选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理（S204）；接收所述终端反馈的任务处理结果；根据所述任务处理结果修改对应任务的任务状态（S206）。

Description

任务分布式处理方法、装置、存储介质和服务器
本申请要求于2017年07月24日提交中国专利局，申请号为201710608537X，发明名称为“任务分布式处理方法、装置、存储介质和服务器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据处理技术领域，特别是涉及一种任务分布式处理方法、装置、存储介质和服务器。
背景技术
分布式计算系统通常都需要处理计算大量的任务，如需要进行统计日志报表、编译应用程序、解析Crash等。而处理这些分布式需要占用大量的服务器资源。
传统方法中，分布式计算系统基本上都是架设在服务器集群上，基于Zookeeper将待处理的任务分配至服务器上的各个节点进行处理，使使得对任务的处理以及上分配进行优化，提高服务器的资源利用效率。然而服务器的资源终究是有限的，当需要处理的数据量较大、任务较多，甚至超过服务器的处理负荷时，传统的方法的任务处理效率并不高。
发明内容
根据本申请公开的各种实施例，提供一种任务分布式处理方法、装置、存储介质和服务器。
一种任务分布式处理方法，应用于分布式任务调度系统中，所述分布式任务调度系统包括服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点； 所述方法包括：所述服务器接收所述终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；接收所述终端反馈的任务处理结果；及根据所述任务处理结果修改对应任务的任务状态。
一种应用于分布式任务调度系统中的任务分布式处理装置，包括：请求接收模块，用于接收终端发送的任务处理请求，所述终端为所述分布式任务调度系统中的执行节点；任务分配模块，用于选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；及任务状态修改模块，用于接收所述终端反馈的任务处理结果；根据所述任务处理结果修改对应任务的任务状态。
一个或多个存储有计算机可读指令的计算机可读非易失性存储介质，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行一种任务分布式处理方法，所述方法应用于分布式任务调度系统中，所述分布式任务调度系统包括服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点；所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：接收所述终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；接收所述终端反馈的任务处理结果；及根据所述任务处理结果修改对应任务的任务状态。
一种应用于分布式任务调度系统中的服务器，包括存储器及处理器，所述存储器中储存有计算机可读指令，所述分布式任务调度系统包括所述服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点；所述指令被所述处理器执行时，使得所述处理器执行以下步骤：接收所述终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；
接收所述终端反馈的任务处理结果；及根据所述任务处理结果修改对应任务的任务状态。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请 的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为一个实施例中任务分布式处理方法的应用环境图；
图2为一个实施例中任务分布式处理方法的流程图；
图3为一个实施例中对Crash解析任务的分布式处理方法的应用环境图
图4为另一个实施例中任务分布式处理方法的流程图；
图5为一个实施例中任务分布式处理装置的结构框图；
图6为另一个实施例中任务分布式处理装置的结构框图；及
图7为一个实施例中服务器的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请实施例所提供的任务分布式处理方法，可应用于如图1所示的应用环境中。参考图1，服务器110与多个终端120相连。其中，该终端可为笔记本电脑、台式电脑、平板等其中的任意一种。每个终端120均可向服务器发送任务处理请求。服务器110可根据所接收到任务处理请求，选取至少一个待分配状态的任务，将选取的任务分配给发送该任务处理请求的终端120进行处理。该终端120可对所接收到的任务进行分析处理，将任务处理结果发送给服务器110。服务器110根据接收到的任务处理结果修改对应任务的任务状态。从而实现了通过终端来对任务进行分布式处理。
在一个实施例中，如图2所示，提供了一种任务分布式处理方法，该方法可应用于如图1所示的应用环境中，具体包括如下步骤：
步骤S202，接收终端发送的任务处理请求。
本实施例中，该终端为对应需要处理该任务的公司或集团内的员工终端或其它所属终端。每个终端可为DTS系统(Distributed Task Scheduler，分布式任务调度系统)中的一个任务执行节点。该DTS系统为一个通用的、插件式的、支持扩展的分布式调度执行框架，用于实现对任务分布式分配和解析等处理。该任务是指数量众多的、每个任务可被分布式处理的任务，包括应用程序编译任务、日志报表统计任务、Crash解析任务等。
以Crash解析任务为例来说明，该Crash可为所收集的某一app(Application，应用)上报的Crash堆栈信息，可将1条或多条适应数量的待解析的Crash堆栈信息作为一个Crash解析任务。当收集Crash堆栈信息越多，则待处理的Crash解析任务也越多。
终端可在处于空闲状态时，可向该DTS系统中的服务器发送任务处理请求，该任务处理请求中携带终端标识。该服务器为DTS系统中的主控节点，负责管理任务和所有执行节点。任务的管理包含了接收、存储、分发、记录、监控任务；节点的管理包含了接收执行节点的心跳包、监控执行节点的工作状态等。服务器可实时接收每个终端所发送的任务处理请求，获取其中的终端标识，使得根据该终端标识确定对应的终端。
步骤S204，选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理。
本实施例中，服务器可从数据库所存储的待分配状态的任务中，选取预设数量的任务，将选取的任务的任务信息发送至终端，使得终端对该任务进行处理。其中，任务的状态包括待分配、处理中、处理成功以及处理失败的状态。预设数量可为预设的1个或几个等合适的数量，以使得终端可同时对多个任务进行处理，提高处理的效率。对待分配状态的任务的选取方式可包括随机选取方式以及按照任务生成的时间顺序来进行选取的方式。
具体地，终端可接收所分配的任务，并按照对应的任务解析方式对该任务进行解析处理，将处理结果发送给服务器。
在一个实施例中，在步骤S204之后，还包括：将选取的任务的状态修改为处理中。
步骤S206，接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。
本实施例中，服务器可接收终端对该任务进行处理后，所反馈的任务处理结果。该任务处理结果包括任务处理成功和任务处理失败。服务器可根据该任务处理结果修改对应已分配的任务的任务状态。比如，当该任务为处理成功，则将其任务状态修改为处理成功，若为处理失败，则将其任务状态修改为处理失败。
在一个实施例中，每个终端之间可为在任务处理上相互独立的终端。即在任务处理的过程中，终端之间进行互相独立的处理，并无连接关系。在DTS系统中，由于每个终端仅与服务器相连，可保持对任务处理的独立性，以提高终端进行任务处理的效率。
本实施例中，通过接收终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理；然后接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。从而实现了将终端作为DTS系统中的任务执行节点，对任务进行处理。当终端数量越多，则服务器上可被及时分配的分布式处理的任务也越多。从而使得无需增加服务器，或者对服务器进行扩容，通过利用已有的终端，将众多的任务分配给每个已有的终端进行处理，提高了任务处理的效率。
在一个实施例中，上述的选取至少一个待分配状态的任务，包括：选取至少一个待分配状态的任务标识；从主数据库中读取每个任务标识对应的任务，当从主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
本实施例中，任务标识用于唯一识别对应的任务，并与对应任务的任务 信息相关联，可由预设位数的数字、字母等字符所构成。任务标识可为任务编号。服务器可在检测到有任务生成时，可为所生成的任务设置相应的任务编号，并将该任务信息分别写入主数据库和备用数据库中。
进一步地，还预先设置了两个数据库，分别为主数据库和备用数据库。每个未处理完成的任务的任务信息在主数据库和备用数据库中均有存储，并可根据该任务编号从主数据库或备用数据库中查询出来。主数据库为默认优先使用的数据库，备用数据库为备用的数据库。服务器可在选取了任务标识后，可根据该任务标识首先从主数据库中查询对应的任务，并读取该任务信息。当从主数据库中读取失败时，则从备用数据库中读取对应任务的任务信息，从而任可进一步提高对任务的提取的成功率。
本实施例中，通过提供主数据库和备用数据库，从而可在主数据库出现故障时，从备用数据库中读取待分配的任务。可防止数据库出现故障，导致待处理的任务积压而造成进一步的影响。
在一个实施例中，任务处理请求中携带终端的互联网协议地址；将选取的任务分配给终端进行处理，包括：根据互联网协议地址，将选取的任务分配给终端进行处理。
本实施例中，互联网协议地址(Internet Protocol Address，IP地址)为终端的本机IP地址。由于终端为员工终端，需要处理其它工作事项，可能存在随时关机、重启、断网、切换网络等操作，使得终端的IP地址或其它参数特征出现变化。相比较于传统的zookeeper需要在启动之前事先配置好所有执行节点的IP和端口。本实施例直接通过在任务处理请求中携带终端的IP地址，使得服务器可向该IP地址发送所分配的任务，实现了即时终端的IP地址出现了动态变化，也可根据变化后的IP地址，成功将任务信息发送至终端，使终端进行任务处理。
在一个实施例中，在将选取的任务分配给终端进行处理之后，包括：接收终端发送的心跳包，以根据心跳包确定终端正在处理选取的任务。
本实施例中，终端可在接收到所分配的任务后，按照预设的频率向服务 器发送心跳包，该心跳包中包含终端的IP地址，使得服务器可根据所接收到的每个心跳包，识别出对应的IP地址，从而可确定对应的终端。服务器通过接收每个终端定时发送的心跳包，可获知对应终端正在处理被分配的任务，从而还实现了对每个终端进行任务处理的监控。
在一个实施例中，在将选取的任务分配给终端进行处理之后，包括：检测是否在预设时长之内接收到任务处理结果，若否，则将选取的任务重新分配给其它发送任务处理请求的终端进行处理。
本实施例中，服务器还设置了时长阈值，即上述的预设时长。该时长阈值为用于评判任务的处理是否异常的时长。时长阈值可根据历史统计出的完成任务的处理所需的平均时长，而设置的时长。可为超过该平均时长一定大小的时长。比如处理一个任务的平均时长为5分钟，则可设置该时长阈值可为8分钟或10分钟等。
服务器可在分配了任务给终端后，开始实时统计任务的处理时长，并比较该处理时长和预设时长的大小，当该处理时长达到预设时长，且还未接收到终端发送的任务处理结果时，则判定终端对该任务的处理失败。并重新将所分配任务的任务状态设置为待分配状态，并根据所接收到的其它终端的任务处理请求，重新对该任务进行分配。
本实施例中，通过设置预设时长，将超过预设时长还未收到任务处理结果的任务进行重新分配，可进一步提到对任务处理完成的成功率。
在一个实施例中，上述的任务分布式处理方法还包括：计算选取的任务的分配次数；当分配次数达到预设次数时，若在预设时长之内未接收到任务处理结果，或者接收到任务处理失败的任务处理结果，则将选取任务的任务状态修改为处理异常的状态。
本实施例中，当产生任务被重新分配时，服务器可进一步统计被重新分配的任务的分配次数，并比较该分配次数和预设次数的大小。其中，预设次数可为设置的合适的次数，比如可为3次或5次等。
当出现被分配次数达到预设次数的任务时，还未接收到任务处理成功的 任务处理结果，产生的原因可能为任务本身出现问题，难以被成功处理。因此，可判定该任务被多次处理失败，则可终止对该任务的重新分配，将该任务的状态设置为处理异常的状态。
在一个实施例中，服务器可汇总每个被设置为处理异常的状态的任务信息，将该任务信息发送至对应的管理员终端，使相应的管理员对该任务进行进一步地检测。具体地，可按照预设的汇总频率进行汇总，比如按照每小时一次的频率进行汇总。
本实施例中，通过设置预设次数，可防止任务被无限地重新分配处理，以浪费终端和服务器的资源。
在一个实施例中，上述的方法还包括：统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
服务器可统计在预设时间段内，每个终端完成的任务数量。具体地，可根据所接收到的任务处理请求中携带的终端标识，统计具有相同终端标识的终端所反馈的任务处理成功的任务处理结果的数量。其中，该终端标识可为用户名等可唯一识别终端身份的信息。
比如可按周或者天为单位，统计每天或每周之内，每个终端完成的任务数量，并根据完成的任务数量进行排序。其中，排序信息中包括终端标识、完成的任务数量以及排名等信息。
本实施例中，通过对每个终端完成的任务数量进行排序，使得每个终端之间形成一种竞赛关系，具有互动性，可提高终端用户参与任务处理的积极性呵和数量，以进一步提高了对任务处理效率。
进一步地，可将排序信息发送至对应的终端，使得终端可获知其在预设时间段内完成的任务数量以及排名。更进一步地，服务器还可从排序中选取中排序超过预设名次的、或者完成的任务数量超过预设数量的终端，并向每个终端广播对所选取的终端的奖励信息，以进一步提高每个终端用户参与任务处理的积极性。
在一个实施例中，上述的任务为Crash解析任务。如图3所示，为一个实施例中对Crash解析任务的分布式处理方法的应用环境图，包括服务器110，终端120，数据库130以及Crash解析平台。其中，服务器110中包含主服务器112以及备用服务器112，数据库同样包含主数据库132以及备用数据库134。Crash解析平台为Crash数据提供者，Crash解析任务需要获取和上传Crash文件，该Crash解析平台负责提供Crash源数据并接收解析后的结果上传。每个终端120均可向服务器110发送任务处理请求，服务器110根据所接收到的任务处理请求，从数据库130中选取至少一个待分配状态的任务，通过与该终端110所定义的接口，分配给该终端110，该接口可为http实现的接口。终端110接收服务器110发送的任务信息，并与Crash平台进行交互，以实现对Crash的解析处理。终端110将任务处理结果发送至服务器110，服务器110根据任务处理结果修改对应任务的任务状态，从而实现了对Crash解析任务的处理。
在一个实施例中，如图4所示，提供了另一种任务分布式处理方法，该任务为Crash解析任务。该方法包括：
步骤S401，终端向服务器发送任务处理请求，任务处理请求中携带终端的互联网协议地址。
本实施例中，该任务处理请求为Crash解析任务的任务处理请求。进一步地，可为IOS Crash解析任务的任务处理请求，终端为部署了Mac OSX系统的终端，比如为Mac笔记本或Mac台式电脑等。
服务器可预先或实时生成了多个Crash解析任务，并将该写入主数据库以及备用数据库中。同时对所生成的Crash解析任务设置任务标识以及设置任务状态为待分配状态。其中，Crash解析任务中包含待解析的Crash文件。具体地，可提供接口和web页面创建任务，从而可接收终端通过该接口和web页面创建的Crash解析任务。
在一个实施例中，服务器可将多个Crash文件封装成一个压缩包，并将多个压缩包封装成一个Crash解析任务，从而使得分配给终端的一个Crash 解析任务中包含多个待解析的Crash文件，从而可提高了Crash文件的分配处理效率。
在一个实施例中，终端可检测自身是否处于空闲状态，若是，则向服务器发送任务处理请求。其中空闲状态是指包括CPU利用率低于预设利用率、内存占用率低于预设占用率的状态。其中，预设利用率和预设占用率可设置为任意合适的数值，比如可均为50％。通过在处于空闲状态时，才发送任务处理请求，可降低处理任务对终端的正常工作造成影响。
步骤S402，服务器接收该任务处理请求，选取至少一个待分配状态的任务，根据互联网协议地址，将选取的任务分配给终端进行处理。
具体地，服务器可任务生成的时间顺序来进行选取预设数量的待分配的任务的任务标识，并优先从主数据库中查询该任务标识所对应的任务。读取所查询出的任务，并按照任务处理请求中携带的终端的IP地址，将其发送至对应的终端，使终端进行任务处理。在一个实施例中，服务器可预设用于存放待分配任务的第一任务表，该表中可按任务的生成顺序来存储待分配任务的任务信息。具体地，可存储任务的任务标识。根据终端的任务处理请求，按照生成顺序从该表中选取还未分配的任务的任务标识。根据该任务标识从主数据库中读取对应的任务信息。其中，当从主数据库中读取失败时，可从备用数据库中读取该任务。
在一个实施例中，该服务器可为一个服务器集群，包含主服务器和备用服务器。可默认调用主服务器进行任务分配和监控等处理，当检测到主服务器处理出现故障时，可调用备用服务器进行任务分配和监控等处理。
在一个实施例中，在将选取的任务分配给终端进行处理之后，还包括：将已分配的任务的状态修改为处理中。进一步地，可分配任务的数据库逻辑设置事务隔离，从而避免将同一个任务分配给了多个终端而重复执行的情况发生。
在一个实施例中，可将该任务从第一任务表中移动到用于存放处理中的第二任务表中，或者直接在第一任务表中将该任务的任务状态标记为处理中 的任务状态。
步骤S403，终端接收服务器分配的任务，通过Crash解析平台对该任务进行处理，将任务处理结果发送至服务器。
本实施例中，终端在接收到服务器分配的任务后，可通过与Crash进行交互，从而实现对任务中的每个Crash文件的解析处理，生成任务处理结果并发送至服务器。其中，该任务处理结果中可包含每个Crash文件的处理结果。处理结果包括处理成功和处理失败。
在一个实施例中，终端还可按照预设的频率向服务器发送心跳包，以告知终端正在进行任务的解析处理。
步骤S404，服务器检测是否在预设时长之内接收到任务处理结果，若是，则执行步骤S405，否则，执行步骤S406。
本实施例中，服务器可在将选取的任务分配给终端进行处理之后，则开始统计任务的处理时长，若在预设时长之内接收到任务处理结果。任务处理结果中包含对应的任务标识，使得服务器可根据该任务标识任务处理结果所属的任务。
服务器可提取将所有状态为处理中，且分配时间距离当前时间已经超出预设时长的任务，然后重置该任务的状态为待分配。预设时长可根据经验值设定，假设一个任务正常执行时间为2分钟，最长执行时间为5分钟，则可设置预设时长为6分钟或者8分钟。
步骤S405，根据任务处理结果修改对应任务的任务状态。
步骤S406，将选取的任务重新分配给其它发送任务处理请求的终端进行处理。
具体地，若任务状态为处理失败，则将该任务再修改为待分配的状态，并再次存放入第一任务表中，以进行重新分配。若任务状态为处理成功，则将该任务存放入用于存储处理成功的第三任务表中。若任务状态为处理异常，则将该任务存放如用于存储处理异常的第四任务表中。其中，每个任务表中均包含对应任务的任务标识，以便于服务器可定时根据第四任务表记录的任 务标识，从主数据库或备用数据库中读取对应的任务信息，发送至管理员终端，使管理员进行检测。
在一个实施例中，终端可设置多个线程来进行任务分配请求与处理。比如可设置2个线程，每个线程均在循环地“任务请求-＞执行-＞处理结果上报-＞休息预设时长”，从而提高终端的利用率和任务处理效率。
上述的任务分布式处理方法，通过将终端作为DTS系统中的执行节点，将任务分配给多个终端进行处理，可缓解服务器对任务处理的压力，提高了对任务处理的效率。
在一个实施例中，如图5所示，提供了一种任务分布式处理装置，该装置应用于分布式任务调度系统中的，包括：
请求接收模块502，用于接收终端发送的任务处理请求，所述终端为所述分布式任务调度系统中的执行节点。
任务分配模块504，用于选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理。
任务状态修改模块506，用于接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。
在一个实施例中，任务分配模块504还用于选取至少一个待分配状态的任务标识；从主数据库中读取每个任务标识对应的任务，当从主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
在一个实施例中，任务处理请求中携带终端的互联网协议地址；任务分配模块504还用于根据互联网协议地址，将选取的任务分配给终端进行处理。
在一个实施例中，任务状态修改模块506还用于接收终端发送的心跳包，以根据心跳包确定终端正在处理选取的任务。
在一个实施例中，任务分配模块504还用于检测是否在预设时长之内接收到任务处理结果，若否，则将选取的任务重新分配给其它发送任务处理请求的终端进行处理。
在一个实施例中，如图6所示，该装置还包括：
排序模块508，用于统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
上述任务分布式处理装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。其中，网络接口可以是以太网卡或无线网卡等。上述各模块可以硬件形式内嵌于或独立于服务器中的处理器中，也可以以软件形式存储于服务器中的存储器中，以便于处理器调用执行以上各个模块对应的操作。该处理器可以为中央处理单元(CPU)、微处理器、单片机等。
在一个实施例中，本申请提供的任务分布式处理装置可以实现为一种计算机程序的形式，该计算机程序可在如图7所示的服务器上运行，该服务器的非易失性存储介质可存储组成该任务分布式处理装置的各个程序模块。比如可包括如图5所示的请求接收模块502、任务分配模块504及任务状态修改模块506。各个程序模块中包括计算机可读指令，该计算机可读指令用于使该服务器执行本说明书中描述的本申请各个实施例的任务分布式处理方法中的步骤，例如，服务器可以通过如图5所示的任务分布式处理装置中的行为请求接收模块502接收终端发送的任务处理请求，所述终端为所述分布式任务调度系统中的执行节点；通过任务分配模块选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理；及通过任务状态修改模块506接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。
在一个实施例中，提供了一个或多个包含计算机可读指令的非易失性计算机可读存储介质，当该计算机可读指令被一个或多个处理器执行时，使得该处理器实现一种任务分布式处理方法，该方法应用于分布式任务调度系统中，该分布式任务调度系统包括服务器和多个终端，该服务器为所述分布式任务调度系统中的主控节点，每个终端为该分布式任务调度系统中的执行节点，该处理器执行所述指令时执行上述各个实施例所提供的任务分布式处理方法的步骤。
在一个实施例中，当该计算机可读指令被一个或多个处理器执行时，使得该处理器执行以下步骤：接收终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理；接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。
在一个实施例中，当该计算机可读指令被一个或多个处理器执行时，使得该处理器执行的选取至少一个待分配状态的任务，包括：选取至少一个待分配状态的任务标识；从主数据库中读取每个任务标识对应的任务，当从主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
在一个实施例中，任务处理请求中携带终端的互联网协议地址；当该计算机可读指令被一个或多个处理器执行时，使得该处理器执行的将选取的任务分配给终端进行处理，包括：根据互联网协议地址，将选取的任务分配给终端进行处理。
在一个实施例中，当该计算机可读指令被一个或多个处理器执行时，使得该处理器在执行将选取的任务分配给终端进行处理之后，还包括执行以下步骤：接收终端发送的心跳包，以根据心跳包确定终端正在处理选取的任务。
在一个实施例中，当该计算机可读指令被一个或多个处理器执行时，使得该处理器在执行将选取的任务分配给终端进行处理之后，还包括执行以下步骤：检测是否在预设时长之内接收到任务处理结果，若否，则将选取的任务重新分配给其它发送任务处理请求的终端进行处理。
在一个实施例中，当该计算机可读指令被一个或多个处理器执行时，使得该处理器还执行以下步骤：统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
在一个实施例中，提供了一种应用于分布式任务调度系统中的服务器，包括存储器及处理器，存储器中储存有计算机可读指令，该分布式任务调度系统包括所述服务器和多个终端，该服务器为所述分布式任务调度系统中的主控节点，每个终端为所述分布式任务调度系统中的执行节点，该指令被处 理器执行时，使得处理器执行上述各个实施例所提供的任务分布式处理方法的步骤。
在一个实施例中，该指令被处理器执行时，使得处理器执行以下步骤：接收终端发送的任务处理请求；选取至少一个待分配状态的任务，将选取的任务分配给终端进行处理；接收终端反馈的任务处理结果；根据任务处理结果修改对应任务的任务状态。
在一个实施例中，如图7所示，为一个实施例中服务器的内部结构示意图。该服务器包括通过系统总线连接的包括通过系统总线连接的处理器、存储器和网络接口。其中，该服务器的处理器用于提供计算和控制能力，支撑整个终端的运行。存储器用于存储数据、指令代码等，网络接口用于与终端进行网络通信。比如，可向终端发送所选取的任务等。存储器上存储至少一个计算机可读指令，该计算机可读指令可被处理器执行，以实现本申请实施例中提供的适用于服务器的任务分布式处理方法。存储器可包括磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)等非易失性存储介质等。例如，在一个实施例中，存储器包括非易失性存储介质及内存储器。服务器的非易失性存储介质存储有操作系统和计算机可读指令。该计算机可读指令可被处理器所执行，以用于实现以上各个实施例所提供的任务分布式处理方法。终端中的内存储器为非易失性存储介质中的操作系统、数据库和计算机可读指令提供高速缓存的运行环境。网络接口可以是以太网卡或无线网卡等，用于与外部的终端或服务器进行通信。服务器可以用独立的服务器或者是多个服务器组成的服务器集群来实现。本领域技术人员可以理解，图7中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的服务器的限定，具体的服务器可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，该指令被处理器执行时，使得处理器执行的选取至少一个待分配状态的任务，包括：选取至少一个待分配状态的任务标识；从主数据库中读取每个任务标识对应的任务，当从主数据库读取失败时，从备用 数据库中读取每个任务标识对应的任务。
在一个实施例中，任务处理请求中携带终端的互联网协议地址；该指令被处理器执行时，使得处理器执行的将选取的任务分配给终端进行处理，包括：根据互联网协议地址，将选取的任务分配给终端进行处理。
在一个实施例中，该指令被处理器执行时，使得处理器在执行将选取的任务分配给终端进行处理之后，还包括执行以下步骤：接收终端发送的心跳包，以根据心跳包确定终端正在处理选取的任务。
在一个实施例中，该指令被处理器执行时，使得处理器在执行将选取的任务分配给终端进行处理之后，还包括执行以下步骤：检测是否在预设时长之内接收到任务处理结果，若否，则将选取的任务重新分配给其它发送任务处理请求的终端进行处理。
在一个实施例中，该指令被处理器执行时，使得处理器还执行以下步骤：统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机可读指令来指令相关的硬件来完成，所述的计算机可读指令可存储于一非易失性计算机可读取存储介质中，该计算机可读指令在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)等。
以上所述实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (20)

  1. 一种任务分布式处理方法，应用于分布式任务调度系统中，其特征在于，所述分布式任务调度系统包括服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点；所述方法包括：
    所述服务器接收所述终端发送的任务处理请求；
    选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；
    接收所述终端反馈的任务处理结果；及
    根据所述任务处理结果修改对应任务的任务状态。
  2. 根据权利要求1所述的方法，其特征在于，所述选取至少一个待分配状态的任务，包括：
    选取至少一个待分配状态的任务标识；及
    从主数据库中读取每个任务标识对应的任务，当从所述主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
  3. 根据权利要求1所述的方法，其特征在于，所述任务处理请求中携带所述终端的互联网协议地址；
    所述将选取的任务分配给所述终端进行处理，包括：
    根据所述互联网协议地址，将选取的任务分配给所述终端进行处理。
  4. 根据权利要求1所述的方法，其特征在于，在所述将选取的任务分配给所述终端进行处理之后，包括：
    接收所述终端发送的心跳包，以根据所述心跳包确定所述终端正在处理所述选取的任务。
  5. 根据权利要求1所述的方法，其特征在于，在所述将选取的任务分配给所述终端进行处理之后，包括：
    检测是否在预设时长之内接收到所述任务处理结果，若否，则将所述选取的任务重新分配给其它发送任务处理请求的终端进行处理。
  6. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
  7. 一种应用于分布式任务调度系统中的任务分布式处理装置，包括：
    请求接收模块，用于接收终端发送的任务处理请求，所述终端为所述分布式任务调度系统中的执行节点；
    任务分配模块，用于选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；及
    任务状态修改模块，用于接收所述终端反馈的任务处理结果；根据所述任务处理结果修改对应任务的任务状态。
  8. 根据权利要求7所述的装置，其特征在于，所述任务分配模块还用于选取至少一个待分配状态的任务标识；从主数据库中读取每个任务标识对应的任务，当从所述主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
  9. 一个或多个存储有计算机可读指令的计算机可读非易失性存储介质，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行一种任务分布式处理方法，所述方法应用于分布式任务调度系统中，所述分布式任务调度系统包括服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点；所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：
    接收所述终端发送的任务处理请求；
    选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处理；
    接收所述终端反馈的任务处理结果；及
    根据所述任务处理结果修改对应任务的任务状态。
  10. 根据权利要求9所述的存储介质，其特征在于，所述计算机可读指令被一个或多个处理器执行时，还使得所述一个或多个处理器执行以下步骤：
    选取至少一个待分配状态的任务标识；
    从主数据库中读取每个任务标识对应的任务，当从所述主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
  11. 根据权利要求9所述的存储介质，其特征在于，所述任务处理请求中携带所述终端的互联网协议地址；
    所述计算机可读指令被一个或多个处理器执行时，还使得所述一个或多个处理器执行以下步骤：根据所述互联网协议地址，将选取的任务分配给所述终端进行处理。
  12. 根据权利要求9所述的存储介质，其特征在于，所述计算机可读指令被一个或多个处理器执行时，还使得所述一个或多个处理器执行以下步骤：
    接收所述终端发送的心跳包，以根据所述心跳包确定所述终端正在处理所述选取的任务。
  13. 根据权利要求9所述的存储介质，其特征在于，所述计算机可读指令被一个或多个处理器执行时，还使得所述一个或多个处理器执行以下步骤：
    检测是否在预设时长之内接收到所述任务处理结果，若否，则将所述选取的任务重新分配给其它发送任务处理请求的终端进行处理。
  14. 根据权利要求9所述的存储介质，其特征在于，所述计算机可读指令被一个或多个处理器执行时，还使得所述一个或多个处理器执行以下步骤：
    统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
  15. 一种应用于分布式任务调度系统中的服务器，包括存储器及处理器，所述存储器中储存有计算机可读指令，所述分布式任务调度系统包括所述服务器和多个终端，所述服务器为所述分布式任务调度系统中的主控节点，每个所述终端为所述分布式任务调度系统中的执行节点；所述指令被所述处理器执行时，使得所述处理器执行以下步骤：
    接收所述终端发送的任务处理请求；
    选取至少一个待分配状态的任务，将选取的任务分配给所述终端进行处 理；
    接收所述终端反馈的任务处理结果；及
    根据所述任务处理结果修改对应任务的任务状态。
  16. 根据权利要求15所述的处理器，其特征在于，所述指令被所述处理器执行时，还使得所述处理器执行以下步骤：
    选取至少一个待分配状态的任务标识；及
    从主数据库中读取每个任务标识对应的任务，当从所述主数据库读取失败时，从备用数据库中读取每个任务标识对应的任务。
  17. 根据权利要求15所述的处理器，其特征在于，所述任务处理请求中携带所述终端的互联网协议地址；
    所述指令被所述处理器执行时，还使得所述处理器执行以下步骤：
    根据所述互联网协议地址，将选取的任务分配给所述终端进行处理。
  18. 根据权利要求15所述的处理器，其特征在于，所述指令被所述处理器执行时，还使得所述处理器执行以下步骤：
    接收所述终端发送的心跳包，以根据所述心跳包确定所述终端正在处理所述选取的任务。
  19. 根据权利要求15所述的处理器，其特征在于，所述指令被所述处理器执行时，还使得所述处理器执行以下步骤：
    检测是否在预设时长之内接收到所述任务处理结果，若否，则将所述选取的任务重新分配给其它发送任务处理请求的终端进行处理。
  20. 根据权利要求15所述的处理器，其特征在于，所述指令被所述处理器执行时，还使得所述处理器执行以下步骤：
    统计每个终端完成的任务数量，根据完成的任务数量对终端进行排序。
PCT/CN2017/104970 2017-07-24 2017-09-30 任务分布式处理方法、装置、存储介质和服务器 Ceased WO2019019400A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710608537.XA CN107688496B (zh) 2017-07-24 2017-07-24 任务分布式处理方法、装置、存储介质和服务器
CN201710608537.X 2017-07-24

Publications (1)

Publication Number Publication Date
WO2019019400A1 true WO2019019400A1 (zh) 2019-01-31

Family

ID=61152957

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/104970 Ceased WO2019019400A1 (zh) 2017-07-24 2017-09-30 任务分布式处理方法、装置、存储介质和服务器

Country Status (2)

Country Link
CN (1) CN107688496B (zh)
WO (1) WO2019019400A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110661844A (zh) * 2019-08-16 2020-01-07 北京旷视科技有限公司 自动发布调度系统、方法和存储介质
CN115629854A (zh) * 2022-10-12 2023-01-20 神策网络科技（北京）有限公司 分布式任务调度方法、系统、电子设备和存储介质

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110309391A (zh) * 2018-03-21 2019-10-08 腾讯科技（深圳）有限公司 内容抓取任务的调度、处理方法、装置、存储介质和设备
CN109120679B (zh) * 2018-07-27 2021-09-28 平安科技（深圳）有限公司 任务分配方法及装置
CN110825532B (zh) * 2018-08-08 2022-08-05 深圳市优必选科技有限公司 分布式任务处理方法、装置、系统、机器人和存储介质
CN111126733A (zh) * 2018-10-31 2020-05-08 北京国双科技有限公司 爬取任务的分配方法及装置
CN109669773B (zh) * 2018-11-12 2024-03-08 平安科技（深圳）有限公司 金融数据处理方法、装置、设备和存储介质
CN109582451A (zh) * 2018-11-21 2019-04-05 金色熊猫有限公司 任务调度方法、系统、设备以及可读介质
CN109634730B (zh) * 2018-11-26 2024-05-31 平安科技（深圳）有限公司 任务调度方法、装置、计算机设备和存储介质
CN109669775B (zh) * 2018-12-10 2024-06-25 平安科技（深圳）有限公司 分布式任务调度方法、系统及存储介质
CN109684095B (zh) * 2018-12-26 2021-03-23 北京奇艺世纪科技有限公司 一种任务处理方法及任务分配设备
CN110209488B (zh) * 2019-06-10 2021-12-07 北京达佳互联信息技术有限公司 任务执行方法、装置、设备、系统及存储介质
CN110377407A (zh) * 2019-06-19 2019-10-25 北京威努特技术有限公司 一种扫描任务调度方法、装置、存储介质及设备
CN110458407A (zh) * 2019-07-12 2019-11-15 平安普惠企业管理有限公司 一种任务分配方法、装置及计算机可读存储介质
CN110398985B (zh) * 2019-08-14 2022-11-11 北京信成未来科技有限公司 一种分布式自适应无人机测控系统及方法
CN110489222B (zh) * 2019-08-22 2025-01-28 深圳前海微众银行股份有限公司 任务调度方法、系统、集群服务器及可读存储介质
CN110543354B (zh) * 2019-09-05 2023-06-13 腾讯科技（上海）有限公司 任务调度方法、装置、设备及存储介质
CN110691120A (zh) * 2019-09-10 2020-01-14 威富通科技有限公司 一种定时任务状态检测方法、服务器及检测系统
CN110602545A (zh) * 2019-09-26 2019-12-20 杭州米络星科技(集团)有限公司 一种网络直播的分布式录制执行方法
CN110888730A (zh) * 2019-12-06 2020-03-17 深圳前海环融联易信息科技服务有限公司 分布式任务调度方法、装置、计算机设备及存储介质
CN111078379B (zh) * 2019-12-06 2024-01-09 中至数据集团股份有限公司 任务调度方法、系统、移动终端及存储介质
CN111182061B (zh) * 2019-12-30 2023-04-14 苏宁云计算有限公司 任务分发处理方法、系统、计算机设备和存储介质
CN111522641B (zh) * 2020-04-21 2023-11-14 北京嘀嘀无限科技发展有限公司 任务调度方法、装置、计算机设备和存储介质
CN112231108A (zh) * 2020-11-02 2021-01-15 网易（杭州）网络有限公司 任务处理方法、装置、计算机可读存储介质及服务器
CN112561264A (zh) * 2020-12-04 2021-03-26 中广核工程有限公司 企业内容的服务方法、服务系统和存储介质
CN112804334A (zh) * 2021-01-15 2021-05-14 京东方科技集团股份有限公司 分发及获取任务的方法、装置、存储介质、电子设备
CN114245288B (zh) * 2021-10-29 2025-05-30 重庆惠科金渝光电科技有限公司 一种服务设备的任务调度方法、服务设备及存储介质
CN114139999A (zh) * 2021-12-07 2022-03-04 海南电网有限责任公司澄迈供电局 智能派工分值辅助计算系统
CN116737345A (zh) * 2023-08-11 2023-09-12 之江实验室 分布式任务处理系统及方法、装置、存储介质以及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030065709A1 (en) * 2001-10-01 2003-04-03 International Business Machines Corporation Dynamically determining whether to process requests synchronously or asynchronously
CN102279730A (zh) * 2010-06-10 2011-12-14 阿里巴巴集团控股有限公司 一种并行的数据处理方法、装置和并行的数据处理系统
CN102710554A (zh) * 2012-06-25 2012-10-03 深圳中兴网信科技有限公司 分布式消息系统和分布式消息系统的服务状态检测方法
CN103713942A (zh) * 2012-09-28 2014-04-09 腾讯科技（深圳）有限公司 在集群中调度运行分布式计算框架的方法和系统
CN104683171A (zh) * 2013-12-02 2015-06-03 中国移动通信集团广东有限公司 应用软件的实时监控方法及装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20020039921A (ko) * 2000-11-22 2002-05-30 김봉태 인터넷 분산 슈퍼컴퓨팅의 모델 및 구현방법
CN101340331B (zh) * 2007-07-06 2010-12-08 中国电信股份有限公司 P2p网络中利用空闲终端执行系统任务的方法
CN101770402B (zh) * 2008-12-29 2012-12-05 中国移动通信集团公司 MapReduce系统中的Map任务调度方法、设备及系统
CN102096602A (zh) * 2009-12-15 2011-06-15 中国移动通信集团公司 一种任务调度方法及其系统和设备
CN102255926B (zh) * 2010-05-17 2015-11-25 中国移动通信集团公司 MapReduce系统中的任务分配方法、系统及装置
US8732720B2 (en) * 2011-12-22 2014-05-20 Hewlett-Packard Development Company, L.P. Job scheduling based on map stage and reduce stage duration
CN103246550A (zh) * 2012-02-09 2013-08-14 深圳市腾讯计算机系统有限公司 一种基于容量的多任务调度方法及系统
CN103279351B (zh) * 2013-05-31 2016-06-29 北京高森明晨信息科技有限公司 一种任务调度的方法及装置
CN106161643B (zh) * 2016-07-25 2019-09-06 镇江惠通电子有限公司 任务分布式处理方法、装置及云服务器

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030065709A1 (en) * 2001-10-01 2003-04-03 International Business Machines Corporation Dynamically determining whether to process requests synchronously or asynchronously
CN102279730A (zh) * 2010-06-10 2011-12-14 阿里巴巴集团控股有限公司 一种并行的数据处理方法、装置和并行的数据处理系统
CN102710554A (zh) * 2012-06-25 2012-10-03 深圳中兴网信科技有限公司 分布式消息系统和分布式消息系统的服务状态检测方法
CN103713942A (zh) * 2012-09-28 2014-04-09 腾讯科技（深圳）有限公司 在集群中调度运行分布式计算框架的方法和系统
CN104683171A (zh) * 2013-12-02 2015-06-03 中国移动通信集团广东有限公司 应用软件的实时监控方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110661844A (zh) * 2019-08-16 2020-01-07 北京旷视科技有限公司 自动发布调度系统、方法和存储介质
CN115629854A (zh) * 2022-10-12 2023-01-20 神策网络科技（北京）有限公司 分布式任务调度方法、系统、电子设备和存储介质

Also Published As

Publication number Publication date
CN107688496B (zh) 2020-12-04
CN107688496A (zh) 2018-02-13

Similar Documents

Publication Publication Date Title
CN107688496B (zh) 任务分布式处理方法、装置、存储介质和服务器
CN103677752B (zh) 基于分布式数据的并发处理方法和系统
CN106254144B (zh) 故障定位平台、故障定位方法及装置
CN103412786B (zh) 一种高性能服务器架构系统及数据处理方法
CN106850830B (zh) 一种业务请求处理方法、装置、系统和相关服务器
CN106557497A (zh) 一种数据同步方法和装置
CN108268372A (zh) Mock测试处理方法、装置、存储介质和计算机设备
CN105701099B (zh) 用于在分布式环境中执行任务的方法、装置及系统
US20150074264A1 (en) Traffic statistic generation for datacenters
CN106033373A (zh) 一种云计算平台中虚拟机资源调度方法和调度系统
CN111459641B (zh) 一种跨机房的任务调度和任务处理的方法及装置
CN104866378A (zh) 一种协调执行任务的系统及方法
EP3422186A1 (en) Method, device and system for preventing memory data loss
CN105956481A (zh) 一种数据处理方法及其装置
CN103164262B (zh) 一种任务管理方法及装置
CN111913784A (zh) 任务调度方法及装置、网元、存储介质
CN107566214B (zh) 一种性能测试方法和装置
CN110928679A (zh) 一种资源分配方法及装置
US10237148B2 (en) Providing a data set for tracking and diagnosing datacenter issues
WO2014173366A2 (zh) 一种实现电信能力群发的方法、装置及系统
CN108255820B (zh) 分布式系统中数据入库的方法、装置以及电子设备
US20250097162A1 (en) Reducing placement conflicts between concurrent virtual machine allocations
CN103514036B (zh) 一种用于事件触发和批量处理的调度系统和方法
CN113934525A (zh) 一种基于正负反馈负载调度算法的Hadoop集群任务调度方法
CN118689638A (zh) 数据处理方法、装置、设备、存储介质和程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17919020

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17919020

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28.05.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17919020

Country of ref document: EP

Kind code of ref document: A1