WO2017041674A1 - 一种启动备份任务的方法、装置及电子设备 - Google Patents

一种启动备份任务的方法、装置及电子设备 Download PDF

Info

Publication number
WO2017041674A1
WO2017041674A1 PCT/CN2016/098009 CN2016098009W WO2017041674A1 WO 2017041674 A1 WO2017041674 A1 WO 2017041674A1 CN 2016098009 W CN2016098009 W CN 2016098009W WO 2017041674 A1 WO2017041674 A1 WO 2017041674A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
processed
processing
time
processing speed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/098009
Other languages
English (en)
French (fr)
Inventor
才华
王家忙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017041674A1 publication Critical patent/WO2017041674A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/445—Program loading or initiating

Definitions

  • the present application relates to the field of distributed computing technologies, and in particular, to a method for starting a backup task in a distributed data processing system and a device for starting a backup task in a distributed data processing system; device.
  • the processing time of processing a task slice on a node with poor performance is significantly longer than that of a normal machine, which becomes a bottleneck of the entire task processing, which greatly increases the processing time of the entire task, which causes serious long Tail delay effect.
  • the solution to avoid long tail delay is to create a backup task for the long tail task slice and process the same data.
  • the scheme for avoiding long tail delay has the following problems: the user is required to configure parameters, and the user determines the time when the backup task should be started by setting parameters; since the parameters are preset fixed values, Difficult to configure the rationality, even if the user has done a relatively reasonable configuration through a large number of practices, but for each task to be configured, the workload is also very large; and it is difficult to fully consider the real-time status of task processing Because the processing of each task slice is related to the real-time processing of the hardware and software of the entire system.
  • the present application provides a method for starting a backup task in a distributed data processing system and an apparatus for starting a backup task in a distributed data processing system to solve the problem of avoiding long tail delay in the prior art.
  • the application also relates to an electronic device.
  • the application provides a method for starting a backup task in a distributed data processing system, the method comprising:
  • the estimated remaining processing time of the task is obtained according to the processing state of the task and the processing speed of the task;
  • the processing speed of acquiring the task according to the processing state of the task processed on the single node is specifically: acquiring the task according to the processing state of the task processed on the single node according to the preset time interval. Processing speed.
  • the processing status of the task includes: the amount of data that the task has processed and the processed time.
  • the acquiring the processing speed of the task according to the processing state of the task processed on the single node includes:
  • the quotient of the processed data amount of the task and the processed time of the task is taken as the processing speed of the task.
  • the algorithm for obtaining the processing speed of the task according to the processing state of the task processed on the single node is:
  • AvgInstRate represents the processing speed of the task processed on a single node. The larger the value, the faster the processing speed of the task; the dataSize represents the amount of data processed by the task processed on a single node, and the runTime represents the processed task on the single node.
  • the letter N represents the task being processed on a specific single node.
  • the preset processing speed is a task currently processed on all nodes in the processing system. Average processing speed.
  • the preset processing speed is obtained by:
  • the quotient of the amount of data that has been processed by the total task and the time that the total task has been processed is taken as the preset processing speed.
  • the algorithm for the preset processing speed is:
  • AvgRate (dataSize 1 +dataSize 2 +...+dataSize N )/(runTime 1 +runTime 2 +...+runTime N )
  • AvgRate represents the preset processing speed, the larger the value, the faster the processing speed of the task
  • the dataSize represents the amount of data processed by the task processed on the single node
  • the runTime represents the processed time of the task processed on the single node.
  • the numbers and the letters N in the algorithm represent the tasks that are processed on a particular single node.
  • the processing status of the task further includes: an amount of original data when the task is not processed.
  • the obtaining the estimated remaining processing time of the task according to the processing status of the task and the processing speed of the task including:
  • the difference between the original data amount when the task is not processed and the processed data amount of the task is used as the remaining data amount of the task;
  • the quotient of the remaining data amount of the task and the processing speed of the task is taken as the estimated remaining processing time of the task.
  • the algorithm for obtaining the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task is:
  • TimeLast N (AllData N -dataSize N )/AvgInstRate N
  • TimeLast represents the estimated remaining processing time of the task processed on a single node.
  • the estimated processing time obtained according to the processing state of the task and the preset processing speed includes:
  • the startup time includes at least a time for the task application resource and a node that starts processing the task;
  • the algorithm for estimating the processing time obtained according to the processing state of the task and the preset processing speed is:
  • BackupTime represents the estimated processing time of the task processed on a single node. The larger the value, the longer the processing time is when the backup task of the task is processed. AllData represents the original data amount when the task processed on the single node is not processed. AvgRate represents the preset processing speed, the letter N represents the task processed on a specific single node, and the overheadTime represents the time required to start the task processed on the single node, including application resources and node startup.
  • the initiating a backup task for the task includes:
  • the backup task is processed in the node.
  • the present application further provides an apparatus for starting a backup task in a distributed data processing system, the apparatus comprising:
  • a processing speed obtaining unit configured to acquire a processing speed of the task according to a processing state of a task processed on a single node
  • a processing speed determining unit configured to determine whether a processing speed of the task is less than a preset processing speed
  • the estimated remaining processing time obtaining unit is configured to receive the determination result of the processing speed determining unit, and if yes, obtain the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task;
  • a processing time judging unit configured to determine whether the estimated remaining processing time of the task is greater than according to the task The processing status and the estimated processing time obtained by the preset processing speed;
  • a backup task starting unit configured to receive a determination result of the processing time judging unit, and if yes, start a backup task for the task.
  • the processing speed obtaining unit is configured to acquire a processing speed of the task according to a processing state of a task processed on a single node according to a preset time interval.
  • the processing speed obtaining unit is configured to acquire, according to a preset time interval, a processing speed of the task according to the processed data amount of the task processed on the single node and the processed time.
  • the processing speed obtaining unit includes:
  • a data amount time acquisition subunit for acquiring the processed data amount of the task currently processed on the single node and the processed time
  • a processing speed calculation subunit for using a quotient of the processed data amount of the task and the processed time of the task as the processing speed of the task.
  • the processing speed determining unit is specifically configured to determine whether the processing speed of the task is smaller than an average processing speed of tasks currently processed on all nodes in the processing system.
  • the processing speed determining unit includes:
  • a node data amount calculation subunit configured to calculate a sum of data amounts that have been processed by tasks currently processed on all nodes in the processing system, and to process the tasks currently processed on all nodes in the processing system The sum of the amount of data that has been processed as the total task in the processing system;
  • a node time calculation sub-unit for calculating a sum of times that a task currently processed on all nodes in the processing system has been processed, and a time at which the task currently processed on all nodes in the processing system has been processed And the time that has been processed as the total task in the processing system;
  • a processing speed calculation subunit configured to use, as the preset processing speed, a quotient of the amount of data processed by the total task and the time that the total task has been processed.
  • the processing speed obtaining unit is specifically configured to: according to the preset time interval, according to the processed data amount of the task processed on the single node, the processed time, and the original data when the task is not processed. Quantity, the processing speed of the task is obtained.
  • the estimated remaining processing time acquiring unit includes:
  • Processing a state obtaining subunit configured to acquire an original data amount when the task is not processed, a processed data amount of the task, and a processing speed of the task;
  • a remaining data amount calculation subunit for using the original data amount when the task is not processed and the task being The difference in the amount of data, as the amount of data remaining for the task;
  • the remaining processing time calculation subunit is configured to use a quotient of the remaining data amount of the task and the processing speed of the task as the estimated remaining processing time of the task.
  • the processing time judging unit includes:
  • a raw data quantity obtaining subunit configured to acquire an original data amount when the task is not processed
  • a startup time acquisition subunit configured to acquire a startup time of the task on a node that processes the task; the startup time includes at least a time for the task application resource and a node that starts processing the task;
  • a preliminary estimation subunit configured to use a quotient of the original data amount when the task is not processed and the preset processing speed as a preliminary estimation processing time of the backup task of the task;
  • a processing time calculation subunit configured to use, as a processing state of the task, a sum of a startup time of the task on a node that processes the task and a preliminary estimated processing time of the backup task of the task The estimated processing time obtained by the preset processing speed.
  • the backup task starting unit includes:
  • a backup task creation subunit is configured to create a backup task of the task for the task
  • a node allocation subunit configured to allocate a node for the backup task
  • a backup task processing subunit for processing the backup task in the node.
  • an electronic device including:
  • a memory for storing a backup task startup program, the program, when being read and executed by the processor, performing an operation of: acquiring a processing speed of the task according to a processing state of a task processed on a single node, determining the Whether the processing speed of the task is less than a preset processing speed. If the processing speed of the task is less than, the estimated remaining processing time of the task is obtained according to the processing state of the task and the processing speed of the task, and the estimated remaining processing of the task is determined. Whether the time is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed. If it is greater than, the backup task is started for the task.
  • the present invention provides a method for starting a backup task in a distributed data processing system and a device for starting a backup task in a distributed data processing system and an electronic device by processing according to a task processed on a single node
  • the state acquires the processing speed of the task; determines whether the processing speed of the task is less than a preset processing speed; if less, obtains the estimated remaining of the task according to the processing state of the task and the processing speed of the task
  • the remaining processing time is determined whether the estimated remaining processing time of the task is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed; if greater, the backup task is started for the task.
  • the technical solution solves the problem that the process of avoiding the long tail delay is cumbersome, the operation is inconvenient, and the pre-configuration is unreasonable, especially when the backup task is started, the original task can still be avoided after the backup task is started by the above method. After the backup task is processed, it causes a waste of resources.
  • FIG. 1 shows a flow diagram of a method for initiating a backup task in a distributed data processing system, provided in accordance with an embodiment of the present application
  • FIG. 2 is a flow chart showing the processing speed of acquiring a task according to a processing state of a task processed on a single node according to an embodiment of the present application;
  • FIG. 3 is a flowchart of obtaining a preset processing speed according to an embodiment of the present application.
  • FIG. 4 is a flowchart showing obtaining an estimated remaining processing time of the task according to a processing state of the task and a processing speed of the task, according to an embodiment of the present application;
  • FIG. 5 is a flowchart showing an estimated processing time acquired according to a processing state of the task and the preset processing speed, according to an embodiment of the present application;
  • FIG. 6 shows a flow chart of initiating a backup task for the task provided in accordance with an embodiment of the present application
  • FIG. 7 shows a schematic diagram of an apparatus for initiating a backup task in a distributed data processing system, provided in accordance with an embodiment of the present application
  • FIG. 8 shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application.
  • Embodiments of the present application provide a method for starting a backup task in a distributed data processing system and an apparatus for starting a backup task in a distributed data processing system; the present application also relates to an electronic device. Detailed description will be made one by one in the following embodiments.
  • the solution to avoid long tail delay is to create a backup task for the long tail task slice and process the same data.
  • avoiding the long tail delay requires the user to configure the parameters, and the user sets the parameters to determine when the backup task should be started. It can be seen that in the existing scheme for avoiding long tail delay, the process is cumbersome, the user is inconvenient to operate, and the pre-configuration is unreasonable, for example, the long tail problem introduced by the unevenness of the task data slice, the long tail cannot pass the advance
  • the configuration parameters start the backup task to ease. In particular, if the backup task is started without considering the execution speed of the original task, it is easy to start the backup task, but the original task is still processed earlier than the backup task, resulting in waste of resources.
  • the technical solution of the present application determines whether the processing speed of the task is less than a preset processing speed and determines whether the estimated remaining processing time of the task is greater than a processing state according to the task and the preset.
  • the processing speed gets the estimated processing time, thus achieving the function of avoiding long tail delay.
  • tasks are divided into a limited number of subtasks and processed on different nodes.
  • a distributed system execution unit composed of a limited number of subtasks is called a distributed job, but in a large distributed system, The hardware and software configuration of the processing tasks are different, and there are external interference factors. As a result, the processing time of each subtask is different.
  • the processing time of subtasks running on nodes with poor performance is significantly longer than that of other machines.
  • the processing time becomes the bottleneck of the entire task operation, which greatly increases the running time of the entire task, which is the long tail delay phenomenon.
  • the reason for the long tail delay phenomenon is that the system is busy when the number of subtasks running in the machine is too large, so that the subtask cannot be applied to the resource normally and the subtask runs slowly; the subtask is generated by subtasks that depend on other nodes.
  • the data, and these nodes may also affect the slow operation of subtasks due to the busy system; for data transfer intensive subtasks, due to the small network bandwidth, the subtasks transmit network data slowly; the data disks in the machine cannot be normal.
  • Sub-tasks run slowly due to hardware failures such as read and write, slow system read and write, or some failure of the operating system.
  • Embodiments of the present application provide a method for initiating a backup task in a distributed data processing system.
  • a specific task is divided into a limited number of subtasks and processed on different nodes. Therefore, in this embodiment, an implementation manner in which a single subtask starts a backup task on a single node is mainly described, which is convenient for description.
  • a subtask is simply referred to as a task; a task divided into a plurality of subtasks is referred to as a total task.
  • the method implementation for starting a backup task in a distributed data processing system is as follows:
  • FIG. 1 illustrates a flow diagram of a method for launching a backup task in a distributed data processing system, in accordance with an embodiment of the present application.
  • the method for starting a backup task in a distributed data processing system includes:
  • Step S101 Acquire a processing speed of the task according to a processing state of a task processed on a single node.
  • the processing speed of the task is obtained according to the processing state of the task processed on the single node, which may be implemented according to the preset time interval, according to the processing of the task processed on a single node.
  • the state acquires the processing speed of the task.
  • the preset time interval may be any time interval, but the short time interval may waste additional resources in the distributed system, and the excessive time interval may increase the error in subsequent processing, which is ideal in the embodiment.
  • the time interval for example, can be obtained once every 15 seconds, according to the processing state of the task processed on a single node.
  • the processing status of the task processed on a single node includes: the amount of data processed by the task and the processed time.
  • the processing speed of the task processed on a single node is acquired every 15 seconds according to the current amount of data processed and the processed time of the task processed on a single node.
  • step S101 the processing speed of the task is obtained according to the processing state of the task processed on the single node, and specifically includes steps S101-1 to S101-2, which will be further described below with reference to FIG. 2.
  • FIG. 2 illustrates a flow chart of obtaining a processing speed of the task according to a processing state of a task processed on a single node according to an embodiment of the present application.
  • step S101-1 the processed data amount of the task currently processed on the single node and the processed time are acquired.
  • the obtaining the processed data amount of the task currently processed on the single node and the processed time may be implemented by acquiring the processed data amount of the task processed on the single node and processing the processed according to the interval of 15 seconds. time.
  • the amount of processed data for a task currently being processed on a single node is 1 m
  • the processed time is 2 s.
  • step S101-2 the quotient of the processed data amount of the task and the processed time of the task is taken as the processing speed of the task.
  • the amount of processed data of the task currently processed on a single node is 1 m
  • the processed time is 2 s
  • the processed data amount of the task is divided by 1 m by the time that the task has been processed for 2 s.
  • the processing speed of the task is 0.5 m/s.
  • the algorithm for obtaining the processing speed of the task according to the processing state of the task processed on the single node is:
  • AvgInstRate represents the processing speed of the task processed on a single node. The larger the value, the faster the processing speed of the task; the dataSize represents the amount of data processed by the task processed on a single node, and the runTime represents the processed task on the single node.
  • the letter N represents the task being processed on a specific single node.
  • the task is a task processed on the first node, and the processed data amount dataSize1 of the task currently processed on the node is 1 m, and the processed time runTime1 is 2 s, according to the above processing speed algorithm.
  • the calculated processing speed AvgInstRate1 of the task is 0.5 m/s.
  • step S101 the processing speed of the task is obtained, and there are two ways:
  • the amount of data processed by the task processed on the single node and the processed time are obtained according to the interval of 15 seconds, and after the processing speed of the task is calculated according to the algorithm of the processing speed of the task on a single node, Report the processing speed of the task. For example, the processing speed of the task is reported to the master.
  • the master is a parallel computing structure in a distributed system.
  • the master and the slave maintain a heartbeat, and communicate with each other to transmit tasks, and finally the results are collected by the master.
  • the amount of data processed by the task processed on the single node and the processed time are obtained according to the interval of 15 seconds, and the amount of data processed by the task and the processed time are reported by the master.
  • the processing speed of the task is calculated according to an algorithm of processing speed of the task according to the reported amount of data processed by the task and the processed time.
  • step S101 if the processing speed of the task is obtained in the first manner, in the subsequent steps, the processing is performed on a single node and the processed result is reported to the master; For example, if the processing speed of the task is obtained in the second manner, in the subsequent steps, after the processing state of the task is reported, the calculation is processed in the master.
  • the second embodiment is exemplified.
  • Step S103 determining whether the processing speed of the task is less than a preset processing speed.
  • the preset processing speed refers to an average processing speed of tasks currently processed on all nodes in the processing system.
  • step S103 the preset processing speed may be obtained by the following steps, including step S102-1. Up to S102-3, further explained below in conjunction with FIG.
  • FIG. 3 illustrates a flow chart for obtaining a preset processing speed according to an embodiment of the present application.
  • Step S102-1 calculating a sum of data amounts that have been processed by tasks currently processed on all nodes in the processing system, and summing the amount of data processed by the tasks currently processed on all nodes in the processing system The amount of data that has been processed as a total task in the processing system.
  • the amount of data processed is the sum of the amount of data that has been processed by the tasks processed on all current nodes. For example, in a distributed system, a total of three tasks are processed on three different nodes. The amount of data processed on the first node is currently 1m, and the tasks processed on the second node are currently The amount of data processed is 2m, and the amount of data processed by the task processed on the third node is 3m, and the sum of the amount of data processed by the tasks currently processed on all nodes in the processing system is 6m. 6m is the amount of data that has been processed by the total task in the processing system.
  • Step S102-2 calculating a sum of times when the tasks currently processed on all nodes in the processing system have been processed, and summing the time at which the tasks currently processed on all nodes in the processing system have been processed The time at which the total task in the processing system has been processed.
  • the time is calculated as the sum of the times that the tasks processed on all current nodes have been processed. For example, in a distributed system, a total of three tasks are processed on three different nodes. The task processed on the first node is currently processed for 2 s, and the tasks processed on the second node are currently processed.
  • the time is 2s
  • the task processed on the third node is 1s at the current processed time
  • the sum of the times that the tasks currently processed on all nodes in the processing system have been processed is 5s
  • 5s is used as the Handles the time that the total task in the system has been processed.
  • Step S102-3 the quotient of the amount of data that has been processed by the total task and the time that the total task has been processed is taken as the preset processing speed.
  • the total task has processed 6m of data, the total task has been processed for 5s, and the total amount of data processed by the total task is 6m divided by the total task processed time of 5s.
  • the processing speed for obtaining the total task is 1.2 m/s, and 1.2 m/s is taken as the preset processing speed.
  • the preset processing speed algorithm is:
  • AvgRate (dataSize 1 +dataSize 2 +...+dataSize N )/(runTime 1 +runTime 2 +...+runTime N )
  • AvgRate represents the preset processing speed, the larger the value, the faster the processing speed of the task
  • dataSize represents the amount of data that has been processed by a task processed on a single node.
  • RunTime represents the time that the task processed on a single node has been processed.
  • the number and letter N in the algorithm represent the tasks processed on a particular single node.
  • a total of three tasks are processed on three different nodes.
  • the task processed on the first node has a data amount of 1 m in the currently processed data
  • the task processed on the second node is the amount of the amount of data currently processed data dataSize 2 to 2m
  • the current task in the processing system processes all the nodes have been processed in the amount of data currently processed dataSize 1 + dataSize 2 + dataSize 3 and is 6m, the amount of data processing system the general task processed as 6m
  • a second processing task on the first node 2s runTime 1 is currently in the process of time
  • the tasks processed on the nodes are 2s at the current processed time runTime 2
  • the tasks processed on the third node are 1s at the current processed time runTime 3
  • the tasks currently processed on all nodes in the processing system The sum of the processed time runTime 1 + runTime 2 + runTime 3 is 5 s , and the preset processing speed AvgRate
  • the determining whether the processing speed of the task is less than the preset processing speed may be implemented in the following manner: in the receiving step S101, acquiring the task according to the processing state of the task processed on the single node. The processing speed is compared with the preset processing speed to determine whether the processing speed of the task acquired in step S101 is less than the preset processing speed, that is, whether the processing speed of the task acquired in step S101 is determined. Less than the average processing speed of tasks currently being processed on all nodes in the processing system.
  • Step S105 If less than, obtain the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task.
  • step S101 If the processing speed of the task is less than the preset processing speed according to the processing state of the task processed on the single node in step S101, the processing speed of the task is smaller than that currently processed on all nodes in the processing system. The average processing speed of the task, in which case step S105 is performed.
  • the obtaining the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task may be implemented in the following manner: according to the processed data volume of the current task And an original data amount when the task is not processed and a processing speed of the task acquired in step S101, and an estimated remaining processing time of the task is acquired.
  • the processing status of the task further includes: an original data amount when the task is not processed.
  • step S105 the estimated remaining processing time of the task is obtained according to the processing state of the task and the processing speed of the task, which specifically includes steps S105-1 to S105-3, which are further described below with reference to FIG. .
  • FIG. 4 illustrates a processing state according to the task and the method according to an embodiment of the present application.
  • the processing speed of the task obtains a flowchart of the estimated remaining processing time of the task.
  • Step S105-1 acquiring the original data amount when the task is not processed, the processed data amount of the task, and the processing speed of the task.
  • the obtaining the original data amount when the task is not processed, the processed data amount of the task, and the processing speed of the task may be implemented by acquiring the processing on the single node according to the interval of 15 seconds.
  • the amount of raw data when the task is not processed, the amount of processed data of the task, and the processing speed of the task For example, the amount of processed data of a task currently processed on a single node is 1 m, the amount of original data when unprocessed is 9 m, and the processing speed of the task is 0.5 m/s.
  • Step S105-2 the difference between the original data amount when the task is not processed and the processed data amount of the task is used as the remaining data amount of the task.
  • the amount of processed data of a task currently processed on a single node is 1 m
  • the amount of original data when unprocessed is 9 m
  • the amount of original data at the time of unprocessed is 9 m minus the processed value of the task.
  • the amount of data is 1 m
  • the amount of remaining data for obtaining the task is 8 m.
  • step S105-3 the quotient of the remaining data amount of the task and the processing speed of the task is used as the estimated remaining processing time of the task.
  • the remaining data amount of the task currently processed on a single node is 8 m
  • the processing speed of the task acquired in step S101 is 0.5 m/s
  • the remaining data amount of the task is 8 m divided by the The processing speed of the task is 0.5 m/s
  • the estimated remaining processing time of the task is obtained for 16 s.
  • the algorithm for obtaining the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task is:
  • TimeLast N (AllData N -dataSize N )/AvgInstRate N
  • TimeLast represents the estimated remaining processing time of the task processed on a single node.
  • the task is a task processed on the first node, and the processed data amount dataSize 1 of the task currently processed on the node is 1 m, and the original data amount AllData 1 when unprocessed is 9 m.
  • the processing speed of the task acquired in step S101 is 0.5 m/s, and the algorithm is calculated according to the above algorithm for acquiring the estimated remaining processing time of the task according to the processing state of the task and the processing speed of the task.
  • the estimated remaining processing time of the task TimeLast 1 is 16s.
  • Step S107 determining whether the estimated remaining processing time of the task is greater than an estimated processing time acquired according to the processing state of the task and the preset processing speed.
  • the estimated processing time obtained according to the processing state of the task and the preset processing speed refers to: an estimated processing time when the backup task of the task is started to when the backup task is completed. .
  • the step is to estimate the processing time when the backup task is started for the task.
  • the estimated processing time obtained according to the processing state of the task and the preset processing speed includes step S107- 1 to S107-4, which will be further described below in conjunction with FIG.
  • FIG. 5 is a flowchart of an estimated processing time acquired according to a processing state of the task and the preset processing speed, according to an embodiment of the present application.
  • Step S107-1 acquiring the original data amount when the task is not processed.
  • the obtaining the original data amount when the task is not processed may be implemented by acquiring the original data amount when the task processed on a single node is not processed every 15 seconds. For example, the amount of raw data when the task currently processed on a single node is not processed is 9m.
  • Step S107-2 Acquire a startup time of the task on a node that processes the task; the startup time includes at least a time for the task application resource and a node that starts processing the task.
  • the acquiring the start time of the task on the node that processes the task may be implemented by: time for applying the resource when the task is started, and time for starting the node for processing the task. And , as the start time of the task on the node that handles the task. For example, the time for the task to apply for the resource at startup is 2 s, and the time for starting the node for processing the task is 1 s, and the startup time of the task is 3 s.
  • Step S107-3 the quotient of the original data amount when the task is not processed and the preset processing speed is used as a preliminary estimation processing time of the backup task of the task.
  • the original data amount when the task currently processed on a single node is not processed is 9 m
  • the preset processing speed acquired in step S102 is 1.2 m/s
  • the original data amount when the task is not processed is 9m
  • 9m is divided by the preset processing speed of 1.2m/s
  • the preliminary estimated processing time of the obtained backup task of the task is 7.5s.
  • Step S107-4 the startup time of the task on the node that processes the task and the backup task of the task
  • the sum of the preliminary estimated processing times is used as the estimated processing time obtained according to the processing state of the task and the preset processing speed.
  • the startup time of the task currently processed on a single node is 3s
  • the initial estimated processing time of the backup task of the task is 7.5s
  • the startup time of the task is 3s plus the preliminary task of the backup task of the task.
  • the estimated processing time is 7.5 s, as the estimated processing time obtained according to the processing state of the task and the preset processing speed, that is, the startup task of starting the task to the estimation when the backup task is completed.
  • the processing time is 10.5 s.
  • the algorithm for estimating the processing time obtained according to the processing state of the task and the preset processing speed is:
  • BackupTime represents the estimated processing time of the task processed on a single node. The larger the value, the longer the processing time is when the backup task of the task is processed. AllData represents the original data amount when the task processed on the single node is not processed. AvgRate represents the preset processing speed, the letter N represents the task processed on a specific single node, and the overheadTime represents the time required to start the task processed on the single node, including application resources and node startup.
  • the task is a task processed on the first node, and the original data amount AllData 1 when the task currently processed on the node is not processed is 9 m, and the preset processing acquired in step S102 is performed.
  • the speed AvgRate is 1.2 m/s
  • the start time overheadTime of the task on the node that processes the task is obtained in step S107-2, and is obtained according to the processing state of the task and the preset processing speed.
  • the estimated processing time BackupTime 1 obtained according to the processing state of the task and the preset processing speed calculated by the algorithm of the estimated processing time is 10.5 s, that is, the backup task of the task is started to complete the backup.
  • Estimated processing time for tasks is 10.5s
  • the determining whether the estimated remaining processing time of the task is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed may be implemented as follows: S105: Obtain an estimated remaining processing time of the task according to a processing state of the task and a processing speed of the task, and an estimated processing time acquired according to the processing state of the task and the preset processing speed. In contrast, it is determined whether the estimated remaining processing time of the task acquired in step S105 is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed, that is, determining the location acquired in step S105. Whether the estimated remaining processing time of the task is greater than the estimated processing time when the backup task of the task is started to when the backup task is completed.
  • Step S109 if it is greater, initiate a backup task for the task.
  • step S105 the task is obtained according to the processing state of the task and the processing speed of the task.
  • the estimated remaining processing time is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed, indicating whether the estimated remaining processing time of the task is greater than the backup task starting the task to complete the The estimated processing time when the task is backed up.
  • step S109 is performed.
  • step S109 the backup task is started for the task, and specifically includes steps S109-1 to S109-3, which will be further described below in conjunction with FIG. 6.
  • FIG. 6 illustrates a flow chart for starting a backup task for the task according to an embodiment of the present application.
  • Step S109-1 for the task, create a backup task of the task.
  • the backup task refers to a task that has the same processing data as the original task and has the same processing logic.
  • Step S109-2 assigning a node to the backup task.
  • the assigning a node to the backup task may be implemented by: selecting and processing a node different from the original task, and the node is configured to process the backup task.
  • the nodes may be nodes that are not performing processing tasks in the same machine, nodes in other machines under the distributed system.
  • Step S109-3 processing the backup task in the node.
  • the processing of the backup task in the node may be implemented in the following manner: the backup task starts the node, and applies the resources required by the backup task in the node, and runs the backup task.
  • a method for starting a backup task in a distributed data processing system provided by the embodiment of the present application, by determining whether the processing speed of the task is less than a preset processing speed and determining whether the estimated remaining processing time of the task is
  • the estimated processing time obtained according to the processing state of the task and the preset processing speed is simplified, the user pre-configured time is simplified, and the processing speed of the original task is considered, and the backup task for starting the task is avoided. After that, the original task is earlier than the waste of resources caused by the backup task being processed.
  • a method for starting a backup task in a distributed data processing system is provided.
  • the present application further provides a A device for initiating a backup task in a distributed data processing system. Since the embodiment of the device is substantially similar to the embodiment of the method, the description is relatively simple, and the relevant portions can be referred to the description of the method embodiment.
  • the device embodiments described below are merely illustrative.
  • the device implementation for starting a backup task in a distributed data processing system is as follows:
  • FIG. 7 illustrates a schematic diagram of an apparatus for initiating a backup task in a distributed data processing system, in accordance with an embodiment of the present application.
  • the device for starting a backup task in the distributed data processing system includes: a processing speed obtaining unit 701, a processing speed determining unit 703, an estimated remaining processing time obtaining unit 705, a processing time determining list 707, and a backup task starting unit 709. ;
  • the processing speed obtaining unit 701 is configured to acquire a processing speed of the task according to a processing state of a task processed on a single node;
  • the processing speed determining unit 703 is configured to determine whether the processing speed of the task is less than a preset processing speed
  • the estimated remaining processing time obtaining unit 705 is configured to receive the determination result of the processing speed determining unit 205, and if yes, obtain the estimated remaining of the task according to the processing state of the task and the processing speed of the task. Processing time
  • the processing time judging unit 707 is configured to determine whether the estimated remaining processing time of the task is greater than an estimated processing time acquired according to the processing state of the task and the preset processing speed;
  • the backup task starting unit 709 is configured to receive the determination result of the processing time determining unit 209, and if so, initiate a backup task for the task.
  • the processing speed obtaining unit 701 is configured to acquire, according to a preset time interval, a processing speed of the task according to a processing state of a task processed on a single node.
  • the processing speed obtaining unit 701 is configured to acquire, according to a preset time interval, a processing speed of the task according to the processed data amount of the task processed on the single node and the processed time.
  • the processing speed obtaining unit 701 includes: a data amount time acquiring subunit and a processing speed calculating subunit;
  • the data amount time acquisition subunit is configured to acquire the processed data amount of the task currently processed on the single node and the processed time;
  • the processing speed calculation subunit is configured to use a quotient of the processed data amount of the task and the processed time of the task as a processing speed of the task.
  • the processing speed determining unit 703 is specifically configured to determine whether the processing speed of the task is smaller than an average processing speed of tasks currently processed on all nodes in the processing system.
  • the processing speed determining unit 703 includes: a node data amount calculating subunit, a node time calculating subunit, and a processing speed calculating subunit;
  • the node data amount calculation subunit is configured to calculate a sum of data amounts that have been processed by tasks currently processed on all nodes in the processing system, and the tasks currently processed on all nodes in the processing system have been Processed The sum of the amount of data that has been processed as the total task in the processing system;
  • the node time calculation subunit is configured to calculate a sum of times that the tasks currently processed on all nodes in the processing system have been processed, and the tasks currently processed on all nodes in the processing system have been processed. The sum of time as the time that the total task in the processing system has been processed;
  • the processing speed calculation subunit is configured to use, as the preset processing speed, a quotient of the amount of data that has been processed by the total task and the time that the total task has been processed.
  • the processing speed obtaining unit 701 is specifically configured to: according to the preset time interval, according to the processed data amount of the task processed on the single node, the processed time, and the original when the task is not processed. The amount of data, the processing speed of the task is obtained.
  • the estimated remaining processing time obtaining unit 705 includes: a processing state acquiring subunit, a remaining data amount calculating subunit, and a remaining processing time calculating subunit;
  • the processing state acquisition subunit is configured to acquire an original data amount when the task is not processed, a processed data amount of the task, and a processing speed of the task;
  • the remaining data amount calculation subunit is configured to use, as the remaining data amount of the task, a difference between the original data amount when the task is not processed and the processed data amount of the task;
  • the remaining processing time calculation subunit is configured to use a quotient of the remaining data amount of the task and the processing speed of the task as the estimated remaining processing time of the task.
  • the processing time judging unit 707 includes: an original data amount acquiring subunit, a startup time acquiring subunit, a preliminary estimating subunit, and a processing time calculating subunit;
  • the original data amount obtaining subunit is configured to acquire an original data amount when the task is not processed
  • the startup time acquisition subunit is configured to acquire a startup time of the task on a node that processes the task; the startup time includes at least a time for the task application resource and a node that starts processing the task;
  • the preliminary estimating subunit is configured to use a quotient of the original data amount when the task is not processed and the preset processing speed as a preliminary estimated processing time of the backup task of the task;
  • the processing time calculation subunit is configured to use, as the processing status of the task, a sum of a startup time of the task on a node that processes the task and a preliminary estimation processing time of the backup task of the task. And an estimated processing time obtained by the preset processing speed.
  • the backup task starting unit 709 includes:
  • the backup task creation subunit is configured to create a backup task of the task for the task
  • the node allocation subunit is configured to allocate a node for the backup task
  • the backup task processing subunit is configured to process the backup task in the node.
  • a method for starting a backup task in a distributed data processing system and a device for starting a backup task in a distributed data processing system are provided, and further, the present application provides a Electronic device; the electronic device implementation is as follows:
  • FIG. 8 shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application.
  • the electronic device includes: a display 801, a processor 803, and a memory 805;
  • the memory 805 is configured to store a backup task startup program, where the program, when being read and executed by the processor, performs an operation of: acquiring a processing speed of the task according to a processing state of a task processed on a single node, Determining whether the processing speed of the task is less than a preset processing speed. If the processing speed is less than, the estimated remaining processing time of the task is obtained according to the processing state of the task and the processing speed of the task, and the pre-determination of the task is determined. The estimated remaining processing time is greater than the estimated processing time obtained according to the processing state of the task and the preset processing speed. If it is greater than, the backup task is started for the task.
  • a method for starting a backup task in a distributed data processing system a device for starting a backup task in a distributed data processing system, and an electronic device are provided.
  • the present application A method for data processing of a distributed processing system is also provided; the method for data processing of the distributed processing system includes:
  • At least one task running on the node in the system that employs a method for initiating a backup task in a distributed data processing system as described above.
  • the present application provides a method for data processing of a distributed processing system, in which a distributed system includes a plurality of tasks processed on each node, and each task processed on each node
  • the backup task of the corresponding task can be started by referring to the description in the method for starting the backup task in the distributed data processing system in the foregoing embodiment, and details are not described herein again.
  • a total of two tasks are respectively processed on two different nodes, and reference may be made to the description in the method for starting a backup task in the distributed data processing system in the above embodiment, according to each node.
  • the processing status of the processed task acquires the processing speed of each corresponding task.
  • a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media including both permanent and non-persistent, removable and non-removable media may be implemented by any method or technology.
  • the information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • embodiments of the present application can be provided as a method, system, or computer program product.
  • the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment in combination of software and hardware.
  • the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Multi Processors (AREA)

Abstract

用于分布式数据处理系统中启动备份任务的方法、装置及电子设备。其中，用于分布式数据处理系统中启动备份任务的方法包括：根据在单节点上处理的任务的处理状态获取所述任务的处理速度（S101），判断所述任务的处理速度是否小于预设的处理速度（S103），若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间（S105），判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间（S107），若大于，为所述任务启动备份任务（S109）。所述技术方案解决了避免长尾延迟的过程繁琐，预先配置不合理的问题，特别是在启动备份任务时，也能避免备份任务启动后，原任务早于备份任务处理完毕，资源浪费的问题。

Description

一种启动备份任务的方法、装置及电子设备
本申请要求2015年09月10日递交的申请号为201510573758.9、发明名称为“一种启动备份任务的方法、装置及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及分布式计算技术领域，具体涉及一种用于分布式数据处理系统中启动备份任务的方法以及一种用于分布式数据处理系统中启动备份任务的装置；本申请同时涉及一种电子设备。
背景技术
目前互联网应用已经成为人们生活的一部分，这些应用往往要为上亿用户服务，而支撑拥有海量用户互联网应用的各种分布式计算环境也成为不可或缺的基础。在分布式环境下，任务会被划分多个任务切片由许多节点进行分步处理，并且会被部署到多个服务器上。尽管在分布式环境下能够显著提高资源利用率，但同时也会带来任务之间相互干扰的问题。例如在分布式环境下，长尾延迟效应会被进一步放大。
在分布式环境下，处理在性能较差的节点上的任务切片的处理时间会明显长于正常机器，变成整个任务处理的瓶颈，大大增加了整个任务的处理时间，这就会出现严重的长尾延迟效应。
目前，避免长尾延迟的方案为：给长尾的任务切片建立一个备份任务，处理同样的数据。但是在现有技术下，避免长尾延迟的方案存在如下问题：需要用户来配置参数，用户通过设置参数来决定在何时应该启动备份任务；由于参数是预先设定好的固定的数值，很难做到配置的合理性，即使用户通过大量的实践做了相对合理的配置，但是对于每个任务都要做配置，工作量也是非常巨大的；并且很难全面地考虑到任务处理的实时状态，因为每个任务切片的处理情况跟整个系统的软硬件的实时处理情况具有关联。
由此可见，在现有避免长尾延迟的方案下，过程比较繁琐，用户操作起来不方便，预先配置不合理，例如任务数据切片不均匀引入的长尾问题，这种长尾是无法通过预先配置参数启动备份任务来缓解的。特别是如果启动备份任务的时候没有考虑原任务的执行速度，很容易出现备份任务虽然启动了，但是原任务仍然早于备份任务处理完毕，导 致了资源的白白浪费。
发明内容
本申请提供一种用于分布式数据处理系统中启动备份任务的方法以及一种用于分布式数据处理系统中启动备份任务的装置,以解决现有技术中避免长尾延迟的问题。本申请同时涉及一种电子设备。
本申请提供了一种用于分布式数据处理系统中启动备份任务的方法，所述方法包括：
根据在单节点上处理的任务的处理状态获取所述任务的处理速度；
判断所述任务的处理速度是否小于预设的处理速度；
若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间；
判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间；
若大于，为所述任务启动备份任务。
可选的，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度具体是，按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
可选的，所述任务的处理状态，包括：所述任务已处理的数据量以及已处理的时间。
可选的，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度，包括：
获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间；
将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
可选的，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度的算法为：
AvgInstRateN＝dataSizeN/runTimeN
其中，AvgInstRate代表单节点上处理的任务的处理速度，其数值越大说明任务的处理速度越快；dataSize代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，字母N代表某一具体单节点上处理的任务。
可选的，所述预设的处理速度是，当前在所述处理系统中所有节点上处理的任务的 平均处理速度。
可选的，所述预设的处理速度，通过如下方式获取：
计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的数据量的和作为所述处理系统中总任务已处理的数据量；
计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间；
将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
可选的，所述预设的处理速度的算法为：
AvgRate＝(dataSize1+dataSize2+…+dataSizeN)/(runTime1+runTime2+…+runTimeN)
其中，AvgRate代表预设的处理速度，其数值越大说明任务的处理速度越快；dataSize代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，该算法中的数字和字母N代表某一具体单节点上处理的任务。
可选的，所述任务的处理状态，还包括：所述任务未处理时的原始数据量。
可选的，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，包括：
获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度；
将所述任务未处理时的原始数据量和所述任务的已处理的数据量的差，作为所述任务的剩余数据量；
将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
可选的，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间的算法为：
TimeLastN＝(AllDataN-dataSizeN)/AvgInstRateN
其中，TimeLast代表单节点上处理的任务的预估剩余处理时间，其数值越大说明处理完任务还需要的时间就越多；dataSize代表单节点上处理的任务已处理的数据量，AllData代表单节点上处理的任务未处理时的原始数据量，AvgInstRate代表单节点上处 理的任务的处理速度，字母N代表某一具体单节点上处理的任务。
可选的，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，包括：
获取所述任务未处理时的原始数据量；
获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间；
将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间；
将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
可选的，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的算法为：
BackupTimeN＝AllDataN/AvgRate+overheadTime
其中，BackupTime代表单节点上处理的任务的预估处理时间，其数值越大说明处理该任务的备份任务时处理时间就越长，AllData代表单节点上处理的任务未处理时的原始数据量，AvgRate代表预设的处理速度，字母N代表某一具体单节点上处理的任务，overheadTime代表启动单节点上处理的任务所需要的时间，包括申请资源和节点启动等。
可选的，所述为所述任务启动备份任务，包括：
为所述任务，创建该任务的备份任务；
为所述备份任务分配节点；
在所述节点中处理该备份任务。
相应的，本申请还提供了一种用于分布式数据处理系统中启动备份任务的装置，所述装置包括：
处理速度获取单元，用于根据在单节点上处理的任务的处理状态获取所述任务的处理速度；
处理速度判断单元，用于判断所述任务的处理速度是否小于预设的处理速度；
预估剩余处理时间获取单元，用于接收所述处理速度判断单元的判断结果，若是，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间；
处理时间判断单元，用于判断所述任务的预估剩余处理时间是否大于按照所述任务 的处理状态和所述预设的处理速度获取的预估处理时间；
备份任务启动单元，用于接收所述处理时间判断单元的判断结果，若是，为所述任务启动备份任务。
可选的，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
可选的，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量以及已处理的时间，获取所述任务的处理速度。
可选的，所述处理速度获取单元，包括：
数据量时间获取子单元，用于获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间；
处理速度计算子单元，用于将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
可选的，所述处理速度判断单元，具体用于判断所述任务的处理速度是否小于当前在所述处理系统中所有节点上处理的任务的平均处理速度。
可选的，所述处理速度判断单元，包括：
节点数据量计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的数据量的和作为所述处理系统中总任务已处理的数据量；
节点时间计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间；
处理速度计算子单元，用于将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
可选的，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量、已处理的时间以及所述任务未处理时的原始数据量，获取所述任务的处理速度。
可选的，所述预估剩余处理时间获取单元，包括：
处理状态获取子单元，用于获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度；
剩余数据量计算子单元，用于将所述任务未处理时的原始数据量和所述任务的已处 理的数据量的差，作为所述任务的剩余数据量；
剩余处理时间计算子单元，用于将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
可选的，所述处理时间判断单元，包括：
原始数据量获取子单元，用于获取所述任务未处理时的原始数据量；
启动时间获取子单元，用于获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间；
初步预估子单元，用于将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间；
处理时间计算子单元，用于将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
可选的，所述备份任务启动单元，包括：
备份任务创建子单元，用于为所述任务，创建该任务的备份任务；
节点分配子单元，用于为所述备份任务分配节点；
备份任务处理子单元，用于在所述节点中处理该备份任务。
此外，本申请还提供了一种电子设备，包括：
显示器；
处理器；
存储器，用于存储备份任务启动程序,所述程序在被所述处理器读取执行时,执行如下操作:根据在单节点上处理的任务的处理状态获取所述任务的处理速度，判断所述任务的处理速度是否小于预设的处理速度，若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，若大于，为所述任务启动备份任务。
与现有技术相比，本申请具有以下优点：
本申请提供的一种用于分布式数据处理系统中启动备份任务的方法和一种用于分布式数据处理系统中启动备份任务的装置以及电子设备，通过根据在单节点上处理的任务的处理状态获取所述任务的处理速度；判断所述任务的处理速度是否小于预设的处理速度；若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩 余处理时间；判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间；若大于，为所述任务启动备份任务。所述技术方案解决了避免长尾延迟的过程比较繁琐，操作起来不方便，预先配置不合理的问题，特别是在启动备份任务时，也能通过上述方法避免备份任务启动后，原任务仍然早于备份任务处理完毕，导致了资源的白白浪费的问题。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请中记载的一些实施例，对于本领域普通技术人员来讲，还可以根据这些附图获得其他的附图。
图1示出了根据本申请的实施例提供的用于分布式数据处理系统中启动备份任务的方法的流程图；
图2示出了根据本申请的实施例提供的根据在单节点上处理的任务的处理状态获取所述任务的处理速度的流程图；
图3示出了根据本申请的实施例提供的获取预设的处理速度的流程图；
图4示出了根据本申请的实施例提供的根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间的流程图；
图5示出了根据本申请的实施例提供的按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的流程图；
图6示出了根据本申请的实施例提供的为所述任务启动备份任务的流程图；
图7示出了根据本申请的实施例提供的用于分布式数据处理系统中启动备份任务的装置的示意图；
图8示出了根据本申请的实施例提供的电子设备的示意图。
具体实施方式
为了能够更清楚地理解本申请的上述目的、特征和优点，下面结合附图和具体实施方式对本申请进行进一步的详细描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本申请。但是，本申请能够以 很多不同于在此描述的其它方式来实施，本领域技术人员可以在不违背本申请内涵的情况下做类似推广，因此，本申请不受下面公开的具体实施的限制。
本申请的实施例提供了一种用于分布式数据处理系统中启动备份任务的方法以及一种用于分布式数据处理系统中启动备份任务的装置；本申请同时涉及一种电子设备。在下面的实施例中逐一进行详细说明。
目前，避免长尾延迟的方案为：给长尾的任务切片建立一个备份任务，处理同样的数据。但是，避免长尾延迟时需要用户来配置参数，用户通过设置参数来决定在何时应该启动备份任务。由此可见，在现有避免长尾延迟的方案下，过程比较繁琐，用户操作起来不方便，预先配置不合理，例如任务数据切片不均匀引入的长尾问题，这种长尾是无法通过预先配置参数启动备份任务来缓解的。特别是如果启动备份任务的时候没有考虑原任务的执行速度，很容易出现备份任务虽然启动了，但是原任务仍然早于备份任务处理完毕，导致了资源的白白浪费。针对这一问题，本申请的技术方案通过判断所述任务的处理速度是否小于预设的处理速度和判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，从而实现了避免长尾延迟的功能。
在详细描述本实施例的具体步骤之前，先对本技术方案涉及的长尾延迟作简要说明。
在分布式系统中，将任务分为有限数量的子任务，并在不同节点上处理，由有限数量的子任务组成的分布式系统执行单元称为分布式作业，但是在大型分布式系统中，处理任务的机器软硬件配置各不相同，还具有外界干扰因素，因此导致每个子任务的处理时间不同，运行在性能较差的节点上的子任务的处理时间会明显长于其他机器处理子任务的处理时间，变成整个任务运行的瓶颈，大大增加了整个任务的运行时间，这就是长尾延迟现象。产生长尾延迟现象的原因有：在机器中运行的子任务较多时导致的系统繁忙，使该子任务无法正常申请到资源而导致子任务运行缓慢；子任务由于依赖其他节点的子任务所产生的数据，而这些节点也可能由于系统繁忙导致影响子任务的运行缓慢；对于数据传输密集型的子任务，由于网络带宽较小，导致子任务传输网络数据的速度缓慢；机器中数据磁盘无法正常读写、系统盘读写缓慢等硬件故障，或者操作系统的某些故障，导致子任务运行缓慢。
本申请的实施例提供了一种用于分布式数据处理系统中启动备份任务的方法。由于分布式系统中，将具体的任务分为有限数量的子任务，并在不同节点上处理，因此在本实施例中重点描述单一子任务在单一节点上启动备份任务的实施方式，为了描述方便， 在下文实施例的描述中对子任务简称为任务；对划分为多个子任务的任务称为总任务。所述用于分布式数据处理系统中启动备份任务的方法实施例如下：
请参考图1，其示出了根据本申请的实施例提供的用于分布式数据处理系统中启动备份任务的方法的流程图。
所述用于分布式数据处理系统中启动备份任务的方法包括：
步骤S101，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
在本实施例中，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度，可以采用如下方式实现：按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。所述预设的时间间隔可以是任意时间间隔，但是时间间隔过短会使分布式系统浪费额外的资源，时间间隔过长会增加后续处理中的误差，在本实施例中提供了较理想的时间间隔，例如：可以每间隔15秒，根据在单节点上处理的任务的处理状态，获取一次所述任务的处理速度。
需要说明的是，所述在单节点上处理的任务的处理状态，包括：所述任务已处理的数据量以及已处理的时间。
可以理解的，每间隔15秒根据在单节点上处理的任务的当前已处理的数据量以及已处理的时间，获取所述在单节点上处理的任务的处理速度。
在步骤S101中，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度，具体包括步骤S101-1至S101-2，下面结合图2作进一步说明。
请参考图2，其示出了根据本申请的实施例提供的根据在单节点上处理的任务的处理状态获取所述任务的处理速度的流程图。
步骤S101-1，获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间。
所述获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间，可以采用如下方式实现：根据每间隔15秒获取在单节点上处理的任务已处理的数据量以及已处理的时间。例如：当前在单节点上处理的任务的已处理的数据量为1m，已处理的时间为2s。
步骤S101-2，将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
例如：当前在单节点上处理的任务的已处理的数据量为1m，已处理的时间为2s，将所述任务的已处理的数据量1m除以所述任务已处理的时间2s，获取的所述任务的处理速度为0.5m/s。
在具体实施时，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度的算法为：
AvgInstRateN＝dataSizeN/runTimeN
其中，AvgInstRate代表单节点上处理的任务的处理速度，其数值越大说明任务的处理速度越快；dataSize代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，字母N代表某一具体单节点上处理的任务。
例如：所述任务是在第一个节点上处理的任务，则当前在节点上处理的所述任务的已处理的数据量dataSize1为1m，已处理的时间runTime1为2s，根据上述处理速度的算法计算出的所述任务的处理速度AvgInstRate1为0.5m/s。
在步骤S101中，获取所述任务的处理速度，可以有两种方式：
一是，根据每间隔15秒获取在单节点上处理的任务已处理的数据量以及已处理的时间，在单节点上根据所述任务的处理速度的算法计算出所述任务的处理速度后，上报所述任务的处理速度。例如：将所述任务的处理速度上报给master。
需要说明的是，master是一种在分布式系统中并行计算结构，master与slave之间保持心跳，并且互相通信传送任务，最后由master汇集结果。
二是，在单节点上，根据每间隔15秒获取在单节点上处理的任务已处理的数据量以及已处理的时间，上报所述任务已处理的数据量以及已处理的时间后，由master根据上报的所述任务已处理的数据量以及已处理的时间，根据所述任务的处理速度的算法计算出所述任务的处理速度。
需要说明的是，在本步骤S101中，若采用第一种方式获取所述任务的处理速度，则在后续的步骤中，就都在单节点上处理并将处理后的结果上报master；同样的道理，若采用第二种方式获取所述任务的处理速度，则在后续的步骤中，就都上报所述任务的处理状态之后，在master中处理计算。在本实施例中，以第二种方式进行举例说明。
步骤S103，判断所述任务的处理速度是否小于预设的处理速度。
在本实施例中，所述预设的处理速度是指：当前在所述处理系统中所有节点上处理的任务的平均处理速度。
可以理解的，由于分布式系统中，将任务分为有限数量的子任务，并在不同节点上处理，因此在每一节点上处理的任务的数据量和处理时间都不相同，则每一节点上处理的任务的处理速度都不相同。
在步骤S103中，所述预设的处理速度可以通过如下步骤获取，具体包括步骤S102-1 至S102-3，下面结合图3作进一步说明。
请参考图3，其示出了根据本申请的实施例提供的获取预设的处理速度的流程图。
步骤S102-1，计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的数据量的和作为所述处理系统中总任务已处理的数据量。
根据上报给master的每一节点上处理的任务已处理的数据量，获取在分布式系统中每一节点上处理的任务已处理的数据量，并根据已获取的每一节点上处理的任务已处理的数据量计算在当前所有节点上处理的任务已处理的数据量的和。例如：在分布式系统中，共有3个任务分别在3个不同节点上处理，第一个节点上处理的任务在当前已处理的数据量为1m，第二个节点上处理的任务在当前已处理的数据量为2m，第三个节点上处理的任务在当前已处理的数据量为3m，则当前在所述处理系统中所有节点上处理的任务已处理的数据量的和为6m，将6m作为所述处理系统中总任务已处理的数据量。
步骤S102-2，计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间。
根据上报给master的每一节点上处理的任务已处理的时间，获取在分布式系统中每一节点上处理的任务已处理的时间，并根据已获取的每一节点上处理的任务已处理的时间计算在当前所有节点上处理的任务已处理的时间的和。例如：在分布式系统中，共有3个任务分别在3个不同节点上处理，第一个节点上处理的任务在当前已处理的时间为2s，第二个节点上处理的任务在当前已处理的时间为2s，第三个节点上处理的任务在当前已处理的时间为1s，则当前在所述处理系统中所有节点上处理的任务已处理的时间的和为5s，将5s作为所述处理系统中总任务已处理的时间。
步骤S102-3，将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
例如：在分布式系统中，总任务已处理的数据量为6m，总任务已处理的时间为5s，将所述总任务已处理的数据量6m除以所述总任务已处理的时间5s，获取所述总任务的处理速度为1.2m/s，将1.2m/s作为所述预设的处理速度。
在具体实施时，所述预设的处理速度的算法为：
AvgRate＝(dataSize1+dataSize2+…+dataSizeN)/(runTime1+runTime2+…+runTimeN)
其中，AvgRate代表预设的处理速度，其数值越大说明任务的处理速度越快；dataSize 代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，该算法中的数字和字母N代表某一具体单节点上处理的任务。
例如：在分布式系统中，共有3个任务分别在3个不同节点上处理，第一个节点上处理的任务在当前已处理的数据量dataSize1为1m，第二个节点上处理的任务在当前已处理的数据量dataSize2为2m，第三个节点上处理的任务在当前已处理的数据量dataSize3为3m，则当前在所述处理系统中所有节点上处理的任务已处理的数据量dataSize1+dataSize2+dataSize3的和为6m，将6m作为所述处理系统中总任务已处理的数据量；第一个节点上处理的任务在当前已处理的时间runTime1为2s，第二个节点上处理的任务在当前已处理的时间runTime2为2s，第三个节点上处理的任务在当前已处理的时间runTime3为1s，则当前在所述处理系统中所有节点上处理的任务已处理的时间runTime1+runTime2+runTime3的和为5s，根据上述预设的处理速度的算法计算出的预设的处理速度AvgRate为1.2m/s。
在本实施例中，所述判断所述任务的处理速度是否小于预设的处理速度，可以采用如下方式实现：接收步骤S101中，根据在单节点上处理的任务的处理状态获取所述任务的处理速度，与所述预设的处理速度进行对比判断步骤S101中获取的所述任务的处理速度是否小于所述预设的处理速度，即：判断步骤S101中获取的所述任务的处理速度是否小于当前在所述处理系统中所有节点上处理的任务的平均处理速度。
步骤S105，若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间。
如果步骤S101中，根据在单节点上处理的任务的处理状态获取所述任务的处理速度小于预设的处理速度，说明所述任务的处理速度小于当前在所述处理系统中所有节点上处理的任务的平均处理速度，这种情况下执行步骤S105。
在本实施例中，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，可以采用如下方式实现：根据当前所述任务的已处理的数据量、所述任务未处理时的原始数据量以及在步骤S101中获取的所述任务的处理速度，获取所述任务的预估剩余处理时间。在本实施例中，所述任务的处理状态，还包括：所述任务未处理时的原始数据量。
在步骤S105中，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，具体包括步骤S105-1至S105-3，下面结合图4作进一步说明。
请参考图4，其示出了根据本申请的实施例提供的根据所述任务的处理状态和所述 任务的处理速度获取所述任务的预估剩余处理时间的流程图。
步骤S105-1，获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度。
所述获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度，可以采用如下方式实现：根据每间隔15秒获取在单节点上处理的所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度。例如：当前在单节点上处理的任务的已处理的数据量为1m，未处理时的原始数据量为9m，所述任务的处理速度为0.5m/s。
步骤S105-2，将所述任务未处理时的原始数据量和所述任务的已处理的数据量的差，作为所述任务的剩余数据量。
例如：当前在单节点上处理的任务的已处理的数据量为1m，未处理时的原始数据量为9m，将所述未处理时的原始数据量为9m减去所述任务的已处理的数据量为1m，获取所述任务的剩余数据量为8m。
步骤S105-3，将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
例如：当前在单节点上处理的任务的剩余数据量为8m，根据步骤S101中获取到的所述任务的处理速度为0.5m/s，将所述任务的剩余数据量为8m除以所述任务的处理速度0.5m/s，获取所述任务的预估剩余处理时间16s。
在具体实施时，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间的算法为：
TimeLastN＝(AllDataN-dataSizeN)/AvgInstRateN
其中，TimeLast代表单节点上处理的任务的预估剩余处理时间，其数值越大说明处理完任务还需要的时间就越多；dataSize代表单节点上处理的任务已处理的数据量，AllData代表单节点上处理的任务未处理时的原始数据量，AvgInstRate代表单节点上处理的任务的处理速度，字母N代表某一具体单节点上处理的任务。
例如：所述任务是在第一个节点上处理的任务，则当前在节点上处理的所述任务的已处理的数据量dataSize1为1m，未处理时的原始数据量AllData1为9m，在步骤S101中获取到的所述任务的处理速度为0.5m/s，根据上述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间的算法计算出的所述任务的预估剩余处理时间TimeLast1为16s。
步骤S107，判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
在本实施例中，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间是指：启动所述任务的备份任务至完成该备份任务时的预估处理时间。
可以理解的，由于分布式系统中，将任务分为有限数量的子任务，并在不同节点上处理，由于节点会受到机器系统繁忙、网络带宽或数据磁盘系统盘读写缓慢的影响，导致该节点处理任务的处理速度缓慢，造成长尾延迟现象，若将所述任务备份到其他机器中的节点上处理时，就可能避免上述问题的发生。将所述任务备份到其他机器中的节点上处理就是启动备份任务的过程。
本步骤就是对所述任务估算启动备份任务时的处理时间，在步骤S107中，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，具体包括步骤S107-1至S107-4，下面结合图5作进一步说明。
请参考图5，其示出了根据本申请的实施例提供的按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的流程图。
步骤S107-1，获取所述任务未处理时的原始数据量。
所述获取所述任务未处理时的原始数据量，可以采用如下方式实现：每间隔15秒获取在单节点上处理的所述任务未处理时的原始数据量。例如：当前在单节点上处理的任务未处理时的原始数据量为9m。
步骤S107-2，获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间。
在本实施例中，所述获取所述任务在处理该任务的节点上的启动时间，可以采用如下方式实现：将所述任务在启动时申请资源的时间和启动处理所述任务的节点的时间的和，作为所述任务在处理该任务的节点上的启动时间。例如：所述任务在启动时申请资源的时间为2s，启动处理所述任务的节点的时间为1s，则所述任务的启动时间为3s。
步骤S107-3，将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间。
例如：当前在单节点上处理的任务未处理时的原始数据量为9m，在步骤S102中获取的所述预设的处理速度为1.2m/s，将所述任务未处理时的原始数据量9m除以所述预设的处理速度1.2m/s，获取的所述任务的备份任务的初步预估处理时间为7.5s。
步骤S107-4，将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务 的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
例如：当前在单节点上处理的任务的启动时间为3s，所述任务的备份任务的初步预估处理时间为7.5s，将所述任务的启动时间3s加上所述任务的备份任务的初步预估处理时间7.5s，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，即：启动所述任务的备份任务至完成该备份任务时的预估处理时间为10.5s。
在具体实施时，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的算法为：
BackupTimeN＝AllDataN/AvgRate+overheadTime
其中，BackupTime代表单节点上处理的任务的预估处理时间，其数值越大说明处理该任务的备份任务时处理时间就越长，AllData代表单节点上处理的任务未处理时的原始数据量，AvgRate代表预设的处理速度，字母N代表某一具体单节点上处理的任务，overheadTime代表启动单节点上处理的任务所需要的时间，包括申请资源和节点启动等。
例如：所述任务是在第一个节点上处理的任务，则当前在节点上处理的所述任务未处理时的原始数据量AllData1为9m，在步骤S102中获取的所述预设的处理速度AvgRate为1.2m/s，在步骤S107-2中获取所述任务在处理该任务的节点上的启动时间overheadTime为3s，根据上述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的算法计算出的按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间BackupTime1为10.5s，即：启动所述任务的备份任务至完成该备份任务时的预估处理时间为10.5s
在本实施例中，所述判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，可以采用如下方式实现：接收步骤S105中，根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间与按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间进行对比，判断步骤S105中获取的所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，即：判断步骤S105中获取的所述任务的预估剩余处理时间是否大于启动所述任务的备份任务至完成该备份任务时的预估处理时间。
步骤S109，若大于，为所述任务启动备份任务。
如果步骤S105中，根据所述任务的处理状态和所述任务的处理速度获取所述任务的 预估剩余处理时间大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，说明所述任务的预估剩余处理时间是否大于启动所述任务的备份任务至完成该备份任务时的预估处理时间，这种情况下执行步骤S109。
在步骤S109中，所述为所述任务启动备份任务，具体包括步骤S109-1至S109-3，下面结合图6作进一步说明。
请参考图6，其示出了根据本申请的实施例提供的为所述任务启动备份任务的流程图。
步骤S109-1，为所述任务，创建该任务的备份任务。
在本实施例中，所述备份任务是指：与原有的所述任务处理的数据相同，且具有相同处理逻辑的任务。
步骤S109-2，为所述备份任务分配节点。
所述为所述备份任务分配节点，可以采用如下方式实现：选取与处理原有的所述任务不同节点，该节点用于处理所述备份任务。所述节点可以是，在相同机器中未进行处理任务的节点、在分布式系统下的其他机器中的节点。
步骤S109-3，在所述节点中处理该备份任务。
所述在所述节点中处理该备份任务，可以采用如下方式实现：所述备份任务启动所述节点，并在在所述节点中申请该备份任务需要的资源，运行所述备份任务。
通过本申请实施例提供的一种用于分布式数据处理系统中启动备份任务的方法，通过判断所述任务的处理速度是否小于预设的处理速度和判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，简化了用户预先配置的时间，并且考虑了原有任务的处理速度，避免了启动所述任务的备份任务后，原有任务早于备份任务处理完毕从而造成的资源浪费的问题。
在上述的实施例中，提供了一种用于分布式数据处理系统中启动备份任务的方法，与上述用于分布式数据处理系统中启动备份任务的方法相对应的，本申请还提供了一种用于分布式数据处理系统中启动备份任务的装置。由于装置的实施例基本相似于方法的实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。下述描述的装置实施例仅仅是示意性的。所述用于分布式数据处理系统中启动备份任务的装置实施例如下：
请参考图7，其示出了根据本申请的实施例提供的用于分布式数据处理系统中启动备份任务的装置的示意图。
所述用于分布式数据处理系统中启动备份任务的装置，包括：处理速度获取单元701、处理速度判断单元703、预估剩余处理时间获取单元705、处理时间判断单707以及备份任务启动单元709；
所述处理速度获取单元701，用于根据在单节点上处理的任务的处理状态获取所述任务的处理速度；
所述处理速度判断单元703，用于判断所述任务的处理速度是否小于预设的处理速度；
所述预估剩余处理时间获取单元705，用于接收所述处理速度判断单元205的判断结果，若是，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间；
所述处理时间判断单元707，用于判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间；
所述备份任务启动单元709，用于接收所述处理时间判断单元209的判断结果，若是，为所述任务启动备份任务。
可选的，所述处理速度获取单元701，具体用于按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
可选的，所述处理速度获取单元701，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量以及已处理的时间，获取所述任务的处理速度。
可选的，所述处理速度获取单元701，包括：数据量时间获取子单元以及处理速度计算子单元；
所述数据量时间获取子单元，用于获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间；
所述处理速度计算子单元，用于将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
可选的，所述处理速度判断单元703，具体用于判断所述任务的处理速度是否小于当前在所述处理系统中所有节点上处理的任务的平均处理速度。
可选的，所述处理速度判断单元703，包括：节点数据量计算子单元、节点时间计算子单元以及处理速度计算子单元；
所述节点数据量计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的 数据量的和作为所述处理系统中总任务已处理的数据量；
所述节点时间计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间；
所述处理速度计算子单元，用于将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
可选的，所述处理速度获取单元701，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量、已处理的时间以及所述任务未处理时的原始数据量，获取所述任务的处理速度。
可选的，所述预估剩余处理时间获取单元705，包括：处理状态获取子单元、剩余数据量计算子单元以及剩余处理时间计算子单元；
所述处理状态获取子单元，用于获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度；
所述剩余数据量计算子单元，用于将所述任务未处理时的原始数据量和所述任务的已处理的数据量的差，作为所述任务的剩余数据量；
所述剩余处理时间计算子单元，用于将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
可选的，所述处理时间判断单元707，包括：原始数据量获取子单元、启动时间获取子单元、初步预估子单元以及处理时间计算子单元；
所述原始数据量获取子单元，用于获取所述任务未处理时的原始数据量；
所述启动时间获取子单元，用于获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间；
所述初步预估子单元，用于将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间；
所述处理时间计算子单元，用于将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
可选的，所述备份任务启动单元709，包括：
所述备份任务创建子单元，用于为所述任务，创建该任务的备份任务；
所述节点分配子单元，用于为所述备份任务分配节点；
所述备份任务处理子单元，用于在所述节点中处理该备份任务。
在上述的实施例中，提供了一种用于分布式数据处理系统中启动备份任务的方法以及一种用于分布式数据处理系统中启动备份任务的装置，此外，本申请还提供了一种电子设备；所述电子设备实施例如下：
请参考图8，其示出了根据本申请的实施例提供的电子设备的示意图。
所述电子设备，包括：显示器801、处理器803以及存储器805；
所述存储器805，用于存储备份任务启动程序,所述程序在被所述处理器读取执行时,执行如下操作:根据在单节点上处理的任务的处理状态获取所述任务的处理速度，判断所述任务的处理速度是否小于预设的处理速度，若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，若大于，为所述任务启动备份任务。
在上述的实施例中，提供了一种用于分布式数据处理系统中启动备份任务的方法、一种用于分布式数据处理系统中启动备份任务的装置以及一种电子设备，此外，本申请还提供了一种分布式处理系统数据处理的方法；所述分布式处理系统数据处理的方法，包括：
系统中有至少一个在节点上运行的任务，所述任务采用如上述的用于分布式数据处理系统中启动备份任务的方法。
基于上述可以理解的是，本申请提供的一种分布式处理系统数据处理的方法，该方法中在分布式系统中包含多个在各节点上处理的任务，每一在各节点上处理的任务在处理时，都可以参考上述实施例中的用于分布式数据处理系统中启动备份任务的方法中的说明启动对应任务的备份任务，此处不再赘述。
例如：在分布式系统中共有2个任务分别在2个不同节点上处理，可以参考上述实施例中的用于分布式数据处理系统中启动备份任务的方法中的说明，根据在每一节点上处理的任务的处理状态获取对应的每一任务的处理速度。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
1、计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括非暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
2、本领域技术人员应明白，本申请的实施例可提供为方法、系统或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请虽然以较佳实施例公开如上，但其并不是用来限定本申请，任何本领域技术人员在不脱离本申请的精神和范围内，都可以做出可能的变动和修改，因此本申请的保护范围应当以本申请权利要求所界定的范围为准。

Claims (25)

  1. 一种用于分布式数据处理系统中启动备份任务的方法，其特征在于，包括：
    根据在单节点上处理的任务的处理状态获取所述任务的处理速度；
    判断所述任务的处理速度是否小于预设的处理速度；
    若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间；
    判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间；
    若大于，为所述任务启动备份任务。
  2. 根据权利要求1所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度具体是，按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
  3. 根据权利要求1所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述任务的处理状态，包括：所述任务已处理的数据量以及已处理的时间。
  4. 根据权利要求3所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度，包括：
    获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间；
    将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
  5. 根据权利要求4所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述根据在单节点上处理的任务的处理状态获取所述任务的处理速度的算法为：
    AvgInstRateN＝dataSizeN/runTimeN
    其中，AvgInstRate代表单节点上处理的任务的处理速度，其数值越大说明任务的处理速度越快；dataSize代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，字母N代表某一具体单节点上处理的任务。
  6. 根据权利要求3所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述预设的处理速度是，当前在所述处理系统中所有节点上处理的任务的平均 处理速度。
  7. 根据权利要求6所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述预设的处理速度，通过如下方式获取：
    计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的数据量的和作为所述处理系统中总任务已处理的数据量；
    计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间；
    将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
  8. 根据权利要求7所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述预设的处理速度的算法为：
    AvgRate＝(dataSize1+dataSize2+…+dataSizeN)/(runTime1+runTime2+…+runTimeN)
    其中，AvgRate代表预设的处理速度，其数值越大说明任务的处理速度越快；dataSize代表单节点上处理的任务已处理的数据量，runTime代表单节点上处理的任务已处理的时间，该算法中的数字和字母N代表某一具体单节点上处理的任务。
  9. 根据权利要求3所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述任务的处理状态，还包括：所述任务未处理时的原始数据量。
  10. 根据权利要求9所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，包括：
    获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度；
    将所述任务未处理时的原始数据量和所述任务的已处理的数据量的差，作为所述任务的剩余数据量；
    将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
  11. 根据权利要求10所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩 余处理时间的算法为：
    TimeLastN＝(AllDataN-dataSizeN)/AvgInstRateN
    其中，TimeLast代表单节点上处理的任务的预估剩余处理时间，其数值越大说明处理完任务还需要的时间就越多；dataSize代表单节点上处理的任务已处理的数据量，AllData代表单节点上处理的任务未处理时的原始数据量，AvgInstRate代表单节点上处理的任务的处理速度，字母N代表某一具体单节点上处理的任务。
  12. 根据权利要求9所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，包括：
    获取所述任务未处理时的原始数据量；
    获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间；
    将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间；
    将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
  13. 根据权利要求12所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间的算法为：
    BackupTimeN＝AllDataN/AvgRate+overheadTime
    其中，BackupTime代表单节点上处理的任务的预估处理时间，其数值越大说明处理该任务的备份任务时处理时间就越长，AllData代表单节点上处理的任务未处理时的原始数据量，AvgRate代表预设的处理速度，字母N代表某一具体单节点上处理的任务，overheadTime代表启动单节点上处理的任务所需要的时间，包括申请资源和节点启动等。
  14. 根据权利要求1所述的用于分布式数据处理系统中启动备份任务的方法，其特征在于，所述为所述任务启动备份任务，包括：
    为所述任务，创建该任务的备份任务；
    为所述备份任务分配节点；
    在所述节点中处理该备份任务。
  15. 一种用于分布式数据处理系统中启动备份任务的装置，其特征在于，包括：
    处理速度获取单元，用于根据在单节点上处理的任务的处理状态获取所述任务的处理速度；
    处理速度判断单元，用于判断所述任务的处理速度是否小于预设的处理速度；
    预估剩余处理时间获取单元，用于接收所述处理速度判断单元的判断结果，若是，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间；
    处理时间判断单元，用于判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间；
    备份任务启动单元，用于接收所述处理时间判断单元的判断结果，若是，为所述任务启动备份任务。
  16. 根据权利要求15所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的处理状态获取所述任务的处理速度。
  17. 根据权利要求15所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量以及已处理的时间，获取所述任务的处理速度。
  18. 根据权利要求17所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度获取单元，包括：
    数据量时间获取子单元，用于获取当前在单节点上处理的任务的已处理的数据量以及已处理的时间；
    处理速度计算子单元，用于将所述任务的已处理的数据量和所述任务的已处理的时间的商，作为该任务的处理速度。
  19. 根据权利要求17所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度判断单元，具体用于判断所述任务的处理速度是否小于当前在所述处理系统中所有节点上处理的任务的平均处理速度。
  20. 根据权利要求19所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度判断单元，包括：
    节点数据量计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的数据量的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的数据 量的和作为所述处理系统中总任务已处理的数据量；
    节点时间计算子单元，用于计算当前在所述处理系统中所有节点上处理的任务已处理的时间的和，将所述当前在所述处理系统中所有节点上处理的任务已处理的时间的和作为所述处理系统中总任务已处理的时间；
    处理速度计算子单元，用于将所述总任务已处理的数据量和所述总任务已处理的时间的商，作为所述预设的处理速度。
  21. 根据权利要求17所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理速度获取单元，具体用于按照预设的时间间隔，根据在单节点上处理的任务的已处理的数据量、已处理的时间以及所述任务未处理时的原始数据量，获取所述任务的处理速度。
  22. 根据权利要求21所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述预估剩余处理时间获取单元，包括：
    处理状态获取子单元，用于获取所述任务未处理时的原始数据量、所述任务的已处理的数据量以及所述任务的处理速度；
    剩余数据量计算子单元，用于将所述任务未处理时的原始数据量和所述任务的已处理的数据量的差，作为所述任务的剩余数据量；
    剩余处理时间计算子单元，用于将所述任务的剩余数据量和所述任务的处理速度的商，作为所述任务的预估剩余处理时间。
  23. 根据权利要求21所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述处理时间判断单元，包括：
    原始数据量获取子单元，用于获取所述任务未处理时的原始数据量；
    启动时间获取子单元，用于获取所述任务在处理该任务的节点上的启动时间；所述启动时间至少包括所述任务申请资源和启动处理该任务的节点的时间；
    初步预估子单元，用于将所述任务未处理时的原始数据量和所述预设的处理速度的商，作为所述任务的备份任务的初步预估处理时间；
    处理时间计算子单元，用于将所述任务在处理该任务的节点上的启动时间和所述任务的备份任务的初步预估处理时间的和，作为所述按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间。
  24. 根据权利要求15所述的用于分布式数据处理系统中启动备份任务的装置，其特征在于，所述备份任务启动单元，包括：
    备份任务创建子单元，用于为所述任务，创建该任务的备份任务；
    节点分配子单元，用于为所述备份任务分配节点；
    备份任务处理子单元，用于在所述节点中处理该备份任务。
  25. 一种电子设备，其特征在于，所述电子设备包括：
    显示器；
    处理器；
    存储器，用于存储备份任务启动程序,所述程序在被所述处理器读取执行时,执行如下操作:根据在单节点上处理的任务的处理状态获取所述任务的处理速度，判断所述任务的处理速度是否小于预设的处理速度，若小于，则根据所述任务的处理状态和所述任务的处理速度获取所述任务的预估剩余处理时间，判断所述任务的预估剩余处理时间是否大于按照所述任务的处理状态和所述预设的处理速度获取的预估处理时间，若大于，为所述任务启动备份任务。
PCT/CN2016/098009 2015-09-10 2016-09-05 一种启动备份任务的方法、装置及电子设备 Ceased WO2017041674A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510573758.9A CN106528189B (zh) 2015-09-10 2015-09-10 一种启动备份任务的方法、装置及电子设备
CN201510573758.9 2015-09-10

Publications (1)

Publication Number Publication Date
WO2017041674A1 true WO2017041674A1 (zh) 2017-03-16

Family

ID=58239110

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/098009 Ceased WO2017041674A1 (zh) 2015-09-10 2016-09-05 一种启动备份任务的方法、装置及电子设备

Country Status (2)

Country Link
CN (1) CN106528189B (zh)
WO (1) WO2017041674A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113037528A (zh) * 2019-12-25 2021-06-25 中国移动通信集团山东有限公司 一种告警服务节点的弹性扩缩容方法、装置

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI624757B (zh) * 2017-05-24 2018-05-21 財團法人工業技術研究院 資料處理方法、資料處理系統與電腦程式產品
CN107656805A (zh) * 2017-10-20 2018-02-02 广东电网有限责任公司电力调度控制中心 一种基于Hadoop平台的电力数据作业调度方法
CN108199894B (zh) * 2018-01-15 2020-02-14 华中科技大学 一种数据中心功率管理及服务器部署方法
CN110084473A (zh) * 2019-03-19 2019-08-02 平安普惠企业管理有限公司 项目监管方法、装置、计算机设备及存储介质
CN111258724B (zh) * 2020-01-14 2024-02-06 平安银行股份有限公司 一种分布式系统的数据处理方法、装置、设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609303A (zh) * 2012-01-18 2012-07-25 华为技术有限公司 MapReduce系统的慢任务调度方法和装置
CN103440167A (zh) * 2013-09-04 2013-12-11 福州大学 Hadoop多作业环境下自学习反馈的任务调度方法
CN103473345A (zh) * 2013-09-23 2013-12-25 深圳市融创天下科技股份有限公司 一种预估时长的方法
CN104199739A (zh) * 2014-08-26 2014-12-10 浪潮(北京)电子信息产业有限公司 一种基于负载均衡的推测式Hadoop调度方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7092976B2 (en) * 2003-06-24 2006-08-15 International Business Machines Corporation Parallel high speed backup for a storage area network (SAN) file system
CN101741885A (zh) * 2008-11-19 2010-06-16 珠海市西山居软件有限公司 分布式系统及分布式系统处理任务流的方法
CN102073546B (zh) * 2010-12-13 2013-07-10 北京航空航天大学 一种云计算环境中分布式计算模式下的任务动态调度方法
US9247033B2 (en) * 2012-12-26 2016-01-26 Google Inc. Accessing payload portions of client requests from client memory storage hardware using remote direct memory access
CN104486129B (zh) * 2014-12-24 2017-11-03 中国科学院计算技术研究所 分布式环境下保障应用服务质量的方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609303A (zh) * 2012-01-18 2012-07-25 华为技术有限公司 MapReduce系统的慢任务调度方法和装置
CN103440167A (zh) * 2013-09-04 2013-12-11 福州大学 Hadoop多作业环境下自学习反馈的任务调度方法
CN103473345A (zh) * 2013-09-23 2013-12-25 深圳市融创天下科技股份有限公司 一种预估时长的方法
CN104199739A (zh) * 2014-08-26 2014-12-10 浪潮(北京)电子信息产业有限公司 一种基于负载均衡的推测式Hadoop调度方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113037528A (zh) * 2019-12-25 2021-06-25 中国移动通信集团山东有限公司 一种告警服务节点的弹性扩缩容方法、装置

Also Published As

Publication number Publication date
CN106528189B (zh) 2019-05-28
CN106528189A (zh) 2017-03-22

Similar Documents

Publication Publication Date Title
TWI752994B (zh) 應用的擴容方法、裝置和系統
WO2017041674A1 (zh) 一种启动备份任务的方法、装置及电子设备
CN104583960B (zh) 在线服务的事务级健康监控
CN106776998B (zh) 一种数据库服务提供方法及服务器
JP2017511940A5 (zh)
US8271644B2 (en) System and method to manage information handling system resources in virtual machine infrastructure
Calatrava et al. Self-managed cost-efficient virtual elastic clusters on hybrid Cloud infrastructures
CN106371889B (zh) 一种调度镜像的高性能集群系统实现方法及装置
WO2018082451A1 (zh) 一种应用链路的扩容方法、装置和系统
CN107968810A (zh) 一种服务器集群的资源调度方法、装置和系统
CN107251007B (zh) 集群计算服务确保装置和方法
WO2018086467A1 (zh) 一种云环境下应用集群资源分配的方法、装置和系统
WO2016041446A1 (zh) 一种资源分配方法、装置及设备
JP2021504780A (ja) 分散コンピューティング環境における自動対角スケーリングためのアプリケーションの優先順位付け
US9619288B2 (en) Deploying software in a multi-instance node
US9727394B2 (en) Establishing causality order of computer trace records
JP2021504777A (ja) 分散コンピューティング環境における作業負荷の自動対角スケーリング
JP5521644B2 (ja) ハイブリッド並列処理システム、方法、及びプログラム
CN109829678B (zh) 一种回滚处理方法、装置以及电子设备
CN107241370A (zh) IaaS云环境下资源操作的回滚方法及系统
CN117009063A (zh) 一种任务执行方法和装置
CN111435320B (zh) 一种数据处理方法及其装置
US10725815B2 (en) Representative-based approach to store historical resource usage data
CN110928738A (zh) 性能分析方法、装置和设备
US12395633B1 (en) Systems and methods for video decoder management

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16843612

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16843612

Country of ref document: EP

Kind code of ref document: A1