CN111290841B - Task scheduling method, device, computing equipment and storage medium - Google Patents
Task scheduling method, device, computing equipment and storage medium Download PDFInfo
- Publication number
- CN111290841B CN111290841B CN201811501565.2A CN201811501565A CN111290841B CN 111290841 B CN111290841 B CN 111290841B CN 201811501565 A CN201811501565 A CN 201811501565A CN 111290841 B CN111290841 B CN 111290841B
- Authority
- CN
- China
- Prior art keywords
- data
- task
- processing
- group
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 224
- 239000012634 fragment Substances 0.000 claims abstract description 143
- 230000005540 biological transmission Effects 0.000 claims description 19
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000011218 segmentation Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Abstract
The application discloses a task scheduling method, a task scheduling device, computing equipment and a storage medium. The task scheduling method comprises the following steps: executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of a processing task of a plurality of data fragments of the data to be processed, and the plurality of data fragments are stored in a cluster; and selecting one task scheduling plan with the shortest execution duration from the task scheduling plans as a task execution mode for the data to be processed. The task scheduling scheme can discover various task scheduling plans in a heuristic manner, and enable the locally executed task amounts in different task scheduling plans to be different, so that the task scheduling plan with the highest execution efficiency is selected as a task execution mode, and further load balance of data to be processed in cluster processing is improved, and task execution efficiency is improved.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a task scheduling method, a task scheduling device, a computing device, and a storage medium.
Background
With the development of data processing technology, clusters can be used to store data and to process data in a distributed manner. For example, some applications may utilize clusters to store and transcode video. To improve data processing efficiency, clusters may apply task scheduling policies to allocate data processing tasks. Clusters can be divided into homogeneous and heterogeneous modes. A cluster in isomorphic mode refers to a cluster in which the computing power of the computing nodes is substantially uniform. A cluster of heterogeneous modes may include compute nodes of different computing capabilities. The data processing scheme may be applied in clusters in heterogeneous or homogeneous modes. However, the load balancing of existing task scheduling strategies is to be improved.
Disclosure of Invention
According to one aspect of the present application, there is provided a task scheduling method, the method including: executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of a processing task of a plurality of data fragments of the data to be processed, and the plurality of data fragments are stored in a cluster; and selecting one task scheduling plan with the shortest execution duration from the plurality of task scheduling plans as a task execution mode for the data to be processed. Wherein each time the generating operation is executed, the generating operation comprises: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different secondary generation operations is different; the processing task of any one data slice in the first data slice group is distributed to one computing node storing the data slice in the cluster, and the task distribution mode of the data slice in the first data slice group is used as a first task distribution plan; distributing the processing task of any one data slice in the second data slice group to one computing node which does not store the data slice in the cluster, and taking the task distribution mode of the data slice in the second data slice group as a second task distribution plan; and taking an allocation mode comprising the first task allocation plan and the second task allocation plan as a task scheduling plan generated by the current generating operation.
In some embodiments, the data to be processed is video to be transcoded, the plurality of data slices is a plurality of video slices, and the processing task amount of each of the plurality of data slices is the transcoding task amount of each video slice.
In some embodiments, the method further comprises: before generating the task scheduling plans, acquiring the processing task quantity of each data slice in the data slices of the data to be processed; and acquiring the computing capacity of each computing node in the cluster and the data transmission bandwidth among the computing nodes, wherein the computing capacity of each computing node is used for describing the task amount processed by each computing node in unit time.
In some embodiments, dividing the plurality of data slices into a first data slice group and a second data slice group comprises: in a first generation operation of the multiple generation operations, taking the total task amount of the plurality of data fragments as the total task amount of the first data fragment group, dividing the plurality of data fragments into a first data fragment group and a second data fragment group according to the total task amount of the first data fragment group, wherein the first data fragment group comprises the plurality of data fragments, and the second data fragment group is empty; in any generating operation except the first generating operation of the multiple generating operations, setting the total task amount of the first data slicing group to be reduced by an amount threshold value from the total task amount of the first data slicing group in the last generating operation, and dividing the data slicing into the first data slicing group and the second data slicing group according to the set total task amount of the first data slicing group.
In some embodiments, the dividing the plurality of data slices into the first data slice group and the second data slice group according to the set total task amount of the first data slice group includes: selecting a set of data slices from the plurality of data slices as the first data slice group based on the set total task amount of the first data slice group; and adding the data fragments which do not belong to the first data fragment group in the plurality of data fragments to the second data fragment group.
In some embodiments, the allocating the processing task of any one of the first data shards to one computing node in the cluster storing the data shard includes: and distributing the processing task of any one data slice in the first data slice group to one computing node capable of finishing the processing task of the data slice earliest in a plurality of computing nodes storing the data slice according to the processing task quantity of the data slice in the first data slice group and the computing capacity of the computing nodes in the cluster.
In some embodiments, the allocating the processing task of any one of the first data shards to one of the plurality of computing nodes storing the data shard that completes the processing task of the data shard earliest according to the processing task amount of each data shard in the first data shard group and the computing power of each computing node in the cluster includes: and for the data fragments in the first data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of each data fragment according to the descending order of the processing task quantity. For any one of the first data slices, determining a computing node corresponding to a processing task of the data slice includes: determining a plurality of computing nodes in the cluster that store copies of the data shards; determining an expected completion time point of each computing node of the plurality of computing nodes storing the copy of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragment to the computing capacity of the computing node and the sum of the starting time of the computing node to the processing task of the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragments as the computing node corresponding to the processing task of the data fragments.
In some embodiments, the allocating the processing task of any one of the second data shards to one computing node in the cluster that does not store the data shard includes: and distributing the processing task of any one data slice in the second data slice group to one computing node which can finish the processing task of the data slice earliest in a plurality of computing nodes which do not store the data slice according to the processing task quantity of the data slice in the second data slice group, the computing capacity of the computing nodes in the cluster and the data transmission bandwidth among the computing nodes.
In some embodiments, the allocating the processing task of any one of the second data slices to one of the plurality of computing nodes that does not store the data slice that completes the processing task of the data slice earliest according to the amount of processing tasks of the data slice in the second data slice group, the computing power of the computing nodes in the cluster, and the transmission bandwidth between the computing nodes includes: and for the data fragments in the second data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of each data fragment according to the descending order of the processing task quantity. For any one of the second data slices, determining a computing node corresponding to a processing task of the data slice includes: determining a plurality of computing nodes in the cluster that do not store the data fragment; determining an expected completion time point of each computing node of the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragments to the computing capacity of the computing node, the transmission time of the data fragments to the computing node and the starting time of the computing node to the processing task of the data fragments; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
In some embodiments, the selecting, from the plurality of task scheduling plans, a task scheduling plan with a shortest execution duration as a task execution manner for the data to be processed includes: for a task scheduling plan generated by any generating operation, determining the completion time of each computing node in the cluster for executing the task according to the task scheduling plan; selecting the maximum completion time length from the completion time lengths corresponding to the computing nodes as the execution time length corresponding to the task scheduling plan; and selecting a task scheduling plan with the shortest execution duration as a task execution mode for the data to be processed.
According to an aspect of the present application, there is provided a task scheduling device, including:
a plan generating unit configured to perform a plurality of generating operations, and generate a plurality of task scheduling plans for data to be processed, where each task scheduling plan is configured to describe an execution plan for a processing task for a plurality of data slices of the data to be processed, the plurality of data slices being stored in a cluster; and
and the plan selecting unit is used for selecting one task scheduling plan with the shortest execution duration from the plurality of task scheduling plans as a task execution mode for the data to be processed.
Wherein the generating operation executed by the plan generating unit each time includes: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different secondary generation operations is different; the processing task of any one data slice in the first data slice group is distributed to one computing node storing the data slice in the cluster, and the task distribution mode of the data slice in the first data slice group is used as a first task distribution plan; distributing the processing task of any one data slice in the second data slice group to one computing node which does not store the data slice in the cluster, and taking the task distribution mode of the data slice in the second data slice group as a second task distribution plan; and taking an allocation mode comprising the first task allocation plan and the second task allocation plan as a task scheduling plan generated by the current generating operation.
According to an aspect of the present application, there is provided a computing device comprising: a processor; a memory; and one or more programs stored in the memory and configured to be executed by the processor, the one or more programs including instructions for performing a task scheduling method according to the present application.
According to an aspect of the present application, there is provided a storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a task scheduling method according to the present application.
In summary, according to the task scheduling scheme of the application, various task scheduling plans can be discovered in a heuristic manner, and the locally executed task amounts in different task scheduling plans are different, so that the task scheduling plan with the highest execution efficiency is selected as a task execution mode, and further, the load balance of data to be processed in cluster processing is improved, and the task execution efficiency is improved. In particular, the task scheduling scheme fully considers the transmission time of data among different computing nodes, so that the modeling accuracy of the task scheduling scheme is improved, and the execution efficiency of the data to be processed is further improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1A illustrates a schematic diagram of an application scenario 100 according to some embodiments of the present application;
FIG. 1B illustrates a schematic diagram of a hierarchy of a video according to some embodiments of the present application;
FIG. 2 illustrates a flow chart of a task scheduling method 200 according to some embodiments of the present application;
FIG. 3 illustrates a flowchart of a method 300 of performing a generating operation according to some embodiments of the present application;
FIG. 4 illustrates a flow chart of a method 400 of partitioning a group of data slices, according to some embodiments of the present application;
FIG. 5 illustrates a flow chart of a method 500 of determining computing nodes corresponding to processing tasks of a data shard according to some embodiments of the present application;
FIG. 6 illustrates a flow chart of a method 600 of determining computing nodes corresponding to processing tasks of a data shard according to some embodiments of the present application;
FIG. 7 illustrates a flow chart of a method 700 of selecting a task scheduling plan according to some embodiments of the present application;
FIG. 8 illustrates a flow chart of a task scheduling method 800 according to some embodiments of the present application;
FIG. 9 illustrates a schematic diagram of a task scheduler 900 according to some embodiments of the present application; and
FIG. 10 illustrates a component block diagram of a computing device according to some embodiments of the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In some application scenarios, a cluster may store various data to be processed, and may perform processing tasks for the data to be processed. The task scheduling scheme can be utilized by the cluster to balance the task load of each computing node, so that the data processing efficiency of the cluster is improved. Here, the data to be processed is, for example, video to be transcoded, but is not limited thereto. In some embodiments, the data to be processed may include a plurality of data slices. The data slices may be, for example, video slices partitioned from the video content. The data fragments may be stored in clusters in a distributed manner. Clusters may allocate processing tasks for data slices based on a task scheduling scheme in order to balance the load. An application scenario of the task scheduling scheme according to the present application will be exemplarily described below taking a video processing scenario as an example. Fig. 1A illustrates a schematic diagram of an application scenario 100 according to some embodiments of the present application. As shown in fig. 1A, the application scenario includes a video storage device 110, a segmentation device 120, a prediction device 130, a cluster 140, and a merging device 150.
The video storage device 110 may be, for example, a user terminal device or a server capable of providing video to be transcoded. Here, the user terminal device may include, for example, but is not limited to, a video camera, a palm top computer, a wearable computing device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a desktop computer, a mobile phone, a smart phone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, a game console, a television, or a combination of any two or more of these data processing devices or other data processing devices. The server may be, for example, a hardware independent server node or a virtual server, etc. The video storage device 110 may provide various video content such as movies, television shows, sports games, and user live broadcasts.
The splitting device 120 may split the video content from the video storage device 110 to obtain a plurality of video slices. For an item of video content, the segmentation device 120 may employ various suitable segmentation strategies to control the number and size of video segments. Here, the splitting apparatus 120 may be implemented as a server or a user terminal apparatus. FIG. 1B illustrates a schematic diagram of a hierarchy of a video in accordance with some embodiments of the present application. The segmentation device 120 may divide the video into a sequence of video slices. The video slice sequence may include video slices 1, 2, and 3, for example. Video slice 2 may include header information of the video slice and a plurality of groups of pictures (Group of pictures, abbreviated GOP), including groups of pictures 1, 2, and 3, for example. The image group 1 may include, for example, header information of the image group and a plurality of image frames including, for example, image frames 1, 2, and 3. Image frame 1 may comprise a plurality of groups of macroblocks (abbreviated GOBs), for example comprising groups of macroblocks 1, 2, 3. The Macroblock group 3 may include, for example, a plurality of macroblocks (abbreviated MB), such as macroblocks 1, 2, 3, 4, and 5. Each macroblock may include a luminance pixel block and a chrominance pixel block.
The prediction device 130 may predict the amount of transcoding tasks for each video slice from the segmentation device 120. The amount of transcoding tasks may also be referred to herein as transcoding complexity. The transcoding complexity of a video slice may be expressed as the product of the computing power of a computing device processing the video slice and the required transcoding time. The computing capabilities of a computing device are used to describe the computing speed of the computing device in processing computing tasks. The prediction period 130 may be implemented as a server or a user terminal device.
Cluster 140 may include a plurality of server nodes. The cluster 140 may be a variety of server cluster architectures, which is not limited in this application. From a software perspective, the cluster 140 may be, for example, a Ha Dupu (Hadoop) based server cluster. In addition, from a hardware perspective, the cluster 140 may be a cluster of a homogeneous mode (i.e., the computing power of different computing nodes is consistent) or a cluster of a heterogeneous mode (i.e., the computing power of different computing nodes may be different). The cluster 140 may store video slices of the video to be transcoded in a distributed manner, and may also manage description information of the video to be transcoded. Here, the description information may include the amount of transcoding tasks for each video slice of the video to be transcoded. The cluster 140 may utilize server nodes for transcoding operations. In some embodiments, server nodes in cluster 140 may include a management node 141 and a plurality of computing nodes (e.g., 142, 143, and 144). The compute node may be used to store and transcode video slices. In this way, cluster 140 may generate video slices in various code rate formats. From a definition perspective, the code rate format may be, for example, low definition, standard definition, 720P, 1080P, and so on. The management node 141 may generate a transcoding scheduling policy according to the amount of transcoding tasks of each video slice, and control the computing node to perform transcoding operation.
The merge device 150 may splice the video slices transcoded by the cluster 140 into a complete video. The merge device 150 may also transmit the complete video to the video storage device 110. The combining device 150 may be implemented as a server or a user terminal device.
Devices in application scenario 100 may communicate over one or more networks (not shown). Examples of one or more networks include a Local Area Network (LAN), a Wide Area Network (WAN), and the like. Embodiments of the present application may implement one or more networks using any network protocol, including various wired or wireless protocols, such as Ethernet, universal Serial Bus (USB), FIREWIRE, global System for Mobile communications (GSM), enhanced Data GSM Environment (EDGE), code Division Multiple Access (CDMA), time Division Multiple Access (TDMA), bluetooth, wiFi, voice Over IP (VOIP), wi-MAX, or any other suitable communication protocol.
FIG. 2 illustrates a flow chart of a task scheduling method 200 according to some embodiments of the present application. The method 200 may be performed by, but is not limited to, various clusters that perform data processing. Taking the application scenario of fig. 1 as an example, the method 200 may be performed by the management node 141 in the cluster 140.
In step S201, a plurality of generation operations are performed, and a plurality of task scheduling plans for data to be processed are generated. Each generation operation may generate a task scheduling plan. Wherein each task scheduling plan is used to describe an execution plan of a processing task for a plurality of data slices of data to be processed. Multiple data slices may be stored in cluster 140. In some embodiments, the data to be processed is, for example, video to be transcoded. The plurality of data slices is a plurality of video slices. The processing task amount of each data slice is the transcoding task amount of each video slice. To improve storage security, the cluster 140 may store multiple copies of each video slice.
In addition, step S201 may sequentially perform each generation operation, or may perform multiple generation operations in parallel, which is not limited in this application. In some embodiments, the manner in which step S201 performs the generating operation at any one time may be implemented as the method 300.
As shown in fig. 3, in step S301, a plurality of data slices of data to be processed are divided into a first data slice group and a second data slice group. Wherein, the total task amount of the first data fragment group corresponding to different secondary generation operations is different. The amount of processing tasks for one data slice may also be referred to as computational complexity. A data-slice computational complexity may be expressed as the product of the computational power of a computing device that handles the data slice and the required computation time. In some embodiments, cluster 140 may obtain descriptive information from prediction device 130. The descriptive information may include the amount of transcoding tasks for the video slices. In some embodiments, cluster 140 may analyze the transcoding task amounts for video slices to obtain the transcoding task amounts for each video slice in the video to be transcoded. Additionally, embodiments of the present application may employ various partitioning strategies to partition the total amount of tasks of the first data slice group in different secondary generation operations. For example, embodiments of the present application may enable: the total task amount of the plurality of first data slice groups corresponding to the multiple generation operations satisfies the arithmetic progression when ordered. The difference value of the arithmetic progression is, for example, the smallest processing task amount among the processing task amounts of the plurality of data slices.
In step S302, a processing task of any one of the data slices in the first data slice group is allocated to one of the computing nodes storing the data slice in the cluster 140, and a task allocation manner of the data slice in the first data slice group is used as a first task allocation plan. Step S302 may employ various load balancing policies to distribute processing tasks of the data shards in the first data shard group to computing nodes in cluster 140.
In step S303, the processing task of any one of the data slices in the second data slice group is allocated to one of the computing nodes in the cluster 140 that does not store the data slice, and the task allocation manner of the data slice in the second data slice group is used as the second task allocation plan. Step S303 may employ various load balancing policies to distribute processing tasks of the data slices in the first data slice group to computing nodes in the cluster 140.
In step S304, the allocation scheme including the first task allocation plan and the second task allocation plan is used as one task scheduling plan generated in the current generation operation.
In summary, through the combination of steps S302 and S303, the method 300 may perform local execution of the processing task corresponding to the first data slice group, and perform remote execution of the processing task corresponding to the second data slice group. On this basis, through the combination of steps S301 to S303, step S201 may acquire a plurality of task scheduling plans by using a plurality of generating operations, and make the amounts of tasks locally executed in different task scheduling plans different (i.e., the total amounts of tasks of the first data slice group different).
In step S202, one task scheduling plan having the shortest execution time length is selected from the plurality of task scheduling plans as a task execution mode for the data to be processed.
In summary, through the combination of steps S201 and S202, the task scheduling method 200 may heuristically find various task scheduling plans, and make the locally executed task amounts in different task scheduling plans different, so as to select the task scheduling plan with the highest execution efficiency as a task execution mode, thereby improving load balance of the cluster 140 for processing the data to be processed and improving task execution efficiency.
In some embodiments, the dividing operation manner of the first data slice group and the second data slice group in the step S301 may include steps S3011 and S3012. In the first generating operation of the above-described multiple generating operation, step S301 may execute step S3011 to divide the plurality of data slices into the first data slice group and the second data slice group according to the total task amount of the first data slice group, taking the total task amount of the plurality of data slices as the total task amount of the first data slice group. Wherein the first data slice group includes the plurality of data slices and the second data slice group is empty. In other words, the data slices to be processed are all divided into the first data slice group.
In any one of the generating operations other than the first generating operation described above, step S301 may execute step S3012 to set the total task amount of the first data slice group to be reduced by an amount threshold from the total task amount of the first data slice group in the previous generating operation. The volume threshold may be used as a step value to adjust the total task volume of the first data slice group when the generating operation is iteratively performed a plurality of times. For example, step S201 may iteratively perform 4 times of generation operations. The total task amount of the first data slice group set in each generation operation is C 1 、C 2 、C 3 、C 4 . The quantity threshold may be expressed as C step For example, the smallest processing task amount among the processing task amounts of the plurality of data slices to be processed. C (C) 1 And C 2 Is C step . Similarly C 2 And C 3 Is also C step . On the basis of this, step S3012 divides the plurality of data slices into the first data slice group and the second data slice group according to the set total task amount of the first data slice group. In some embodiments, the total task size of the first data burst may be set to zero (i.e., the data bursts are all allocated into the second data burst).
In some embodiments, step S3012 is implemented as method 400. In step S401, a set of data slices is selected from the plurality of data slices as the first data slice group based on the set total task amount of the first data slice group. In other words, the set total task amount of the first data slice group is the reference value, and S401 may make the actual total task amount of the selected data slice close to the set total task amount. For example, step S401 may cause the task volume of the set of selected data slices to be as close as possible to and not exceed the set total task volume.
In step S402, data fragments that do not belong to the first data fragment group among all data fragments corresponding to the data to be processed are added to the second data fragment group. In order to more visually explain the operation of step S3012, an explanation follows. For example, the data slices to be processed include a total of N data slices. Set of data slices v= (V) 1 ，v 2 ，v 3 ，…，v N ) The set of data sizes for a data slice may be represented as s= (S) 1 ，s 2 ，s 3 ，…，s N ) The set of processing task amounts for each data slice may be represented as C (C 1 ，c 2 ，c 3 ，…，c N ). Wherein the data slicing v i The data size of (2) is s i The processing task amount is c i . i is an integer, and the value range is [1, N]. The first data slice group may be denoted as L and the first data slice group may be denoted as R. The total task size of the first data slice group L may be denoted as C L The total task size of the second data slice group R may be denoted as C R 。
The total task amount of all data fragments corresponding to the data to be processed can be expressed as C sum ：
In some embodiments, a more specific implementation of step S302 is: and distributing the processing task of any one data slice in the first data slice group to one computing node capable of finishing the processing task of the data slice earliest in a plurality of computing nodes for storing the data slice according to the processing task quantity of the data slice in the first data slice group and the computing power of the computing nodes in the cluster 140. Here, step S302 may enable the processing tasks of the data slices in the first data slice group to be performed locally and load-balance the computing nodes in the cluster 140 that include the data slices in the first data slice group by distributing the processing tasks of the data slices to the computing nodes that complete the data earliest.
In some embodiments, for the data slices in the first data slice group, step S302 may sequentially determine the computing nodes corresponding to the processing tasks of each data slice in descending order of the amount of processing tasks. Here, by descending the processing task amount, step S302 may avoid the processing idle state after the partial computing nodes in the cluster 140 complete the task processing, thereby improving the load balancing of the cluster 140. For any one of the first data slices, step S302 may determine a computing node corresponding to a processing task of the data slice by executing method 500.
As shown in fig. 5, in step S501, a plurality of computing nodes in the cluster 140 that store copies of the data shards are determined. For example, data fragment A includes 3 copies. The 3 copies are stored at compute nodes b1, b2, and b3, respectively, of cluster 140. Step S501 may determine, for data fragment a, computing nodes b1, b2, and b3 as candidate nodes for processing tasks that process data fragment a.
In step S502, an expected completion time point of a processing task of the data shard by each of a plurality of computing nodes storing copies of the data shard is determined. For example, if the computing nodes b1, b2 and b3 perform processing tasks of the data slice a, respectively. Step S502 may determine the expected completion times of each of the computing nodes b1, b2, and b3. The expected completion time point of each computing node to the processing task of the data fragment is as follows: the computing node performs a time value obtained by adding a task processing duration of the processing task of the data slice to a starting time point of the processing task of the data slice. The task processing duration of each computing node is as follows: the ratio of the processing task amount of the data slicing to the computing power of the computing node, and the data slicing by the computing node The sum of the start-up durations of the processing tasks of (a) is determined. Taking the data slice a as an example, the starting time point t of the computing node b1 to the data slice a may be according to the total task amount C to be executed by the computing node b1 before executing the processing task of the data slice a t Is determined by the total execution duration T of (a). Start time point t=t 0 +T。t 0 The start time of the first processing of the data slice by the computing node b1 is indicated.
Calculating the pixel processing time length T of the node b1 on the data fragment A A ＝c A /p b1 +O b1 Wherein c A Indicating the amount of processing tasks for data fragment a. P is p b1 Representing the computing power of the computing node b 1. O (O) b1 Representing the start-up time of a processing task for a compute node to process a data slice.
In step S503, one of the computing nodes storing the copy of the data slice, which is the earliest expected completion time point, is selected as the computing node corresponding to the processing task of the data slice. Upon determining that the computing node b2 can complete the processing task of the data fragment a earliest, step S503 may allocate the processing task of the data fragment a to the computing node b2, i.e., select the computing node b2 as the computing node corresponding to the processing task of the data fragment a.
In summary, the method 500 may allocate processing tasks of the data slices in the first data slice group to the local computing node for execution, and may determine a task allocation manner of each data slice according to a manner of minimum completion time (i.e., a manner of selecting a computing node that completes the data slice earliest in step S503), so as to maintain load balancing of the cluster 140.
In some embodiments, step S303 may allocate the processing task of any one data slice in the second data slice group to one computing node, among the plurality of computing nodes that do not store the data slice, capable of earliest completing the processing task of the data slice, according to the amount of processing tasks of the data slice in the second data slice group, the computing power of the computing nodes in the cluster 140, and the data transmission bandwidth between the computing nodes.
In some embodiments, for the data slices in the second data slice group, step S303 may sequentially determine the computing nodes corresponding to the processing tasks of each data slice in descending order of the transcoding task amount. By determining the computing nodes corresponding to the data slices in the second data slice group according to the descending order of the transcoding task amount, in step S303, the idle state of processing after the task processing of part of the computing nodes in the cluster 140 is avoided, thereby improving the load balancing of the cluster 140. For any one of the second data slices, step S303 may determine a computing node corresponding to a processing task of the data slice by executing method 600.
As shown in fig. 6, in step S601, a plurality of computing nodes in the cluster 140 that do not store the data shards are determined. For example, the compute node set j= (J1, J2, …, jm) of the cluster 140, m is a positive integer. For example, the compute nodes in J that store data shard B are J2, J3, and J4. Step S601 may take the remaining compute nodes in set J excluding compute nodes J2, J3, and J4 as alternative compute nodes for processing tasks of the carded data patch B.
In step S602, an expected completion time point for each of a plurality of computing nodes that did not store the data shard for a processing task of the data shard is determined. The expected completion time point of each computing node to the processing task of the data fragment is as follows: the computing node performs a time value obtained by adding a task processing duration of the processing task of the data slice to a starting time point of the processing task of the data slice. The task processing duration of each computing node is as follows: the ratio of the processing task amount of the data fragments to the computing capacity of the computing node, the transmission time of the data fragments to the computing node and the starting time of the computing node to the transcoding task of the data fragments.
To calculate node j 1 Processing task for processing data slice B is taken as an example, and node j is calculated 1 Task processing duration T of (2) B ＝c B /p j1 +O j1 +d B,j1 Wherein c B Representing the amount of processing tasks, p, of data fragment B j1 Representing the computing power, O, of computing node j1 j1 Representing the processing of data partition B by compute node j1The duration of the start of the transaction. Here, the start-up duration of any one computing node for different data slices may be a constant. d, d B,j1 Representing the transmission duration of the data fragment B to the computing node j 1.
S B The data size of the data slice B is represented, and W represents the transmission bandwidth between one computing node jx and the computing node j1 storing the data slice B. d is a reference coefficient. When jx and compute node j1 are co-located, d takes a value of, for example, 1. When jx is different from the enclosure of compute node j1, d takes a value of, for example, 2.
In step S603, one of the computing nodes that has not stored the data slice, which is the earliest expected completion time point, is selected as the computing node corresponding to the processing task of the data slice.
In summary, the method 600 may consider the data transmission time of the data slices when allocating the processing tasks of the data slices in the second data slice group according to the minimum completion time manner, so as to improve the accuracy of predicting the completion time of the task scheduling plan (i.e., improve the modeling accuracy of the task scheduling plan), and further improve the load balancing of the cluster and the processing efficiency of the data to be processed.
In some embodiments, step S202 may be implemented as method 700.
As shown in fig. 7, in step S701, for a task scheduling plan generated by any one generation operation, a completion time period for each computing node in the cluster 140 to execute a task according to the task scheduling plan is determined. For example, the compute node set j= (J1, J2, …, jm) of the cluster 140, the total task amount allocated on the compute node ji is C ji . i is a positive integer, and the value range is [1, m]. Compute node ji execution C ji The completion time of (2) is T ji 。
In step S702, the maximum completion time length is selected from the completion time lengths corresponding to the respective computing nodes as the execution time length corresponding to the task scheduling plan. Here, the execution duration refers to the cluster 140, the processing time length of the task scheduling plan is completed. For example, for the task scheduling plan P, the computing node with the shortest completion time in the set J is J5. Completion time length of calculation node j5 is T j5 . Step S702 may compare T j5 As the execution duration of the task scheduling plan P.
In step S703, a task scheduling plan having the shortest execution time length is selected as the task execution mode of the data to be processed. In summary, the method 700 may compare execution durations of the plurality of task scheduling plans, thereby selecting a task scheduling plan with the shortest execution duration as a task execution mode of the data to be processed, and improving processing efficiency of the data to be processed.
Fig. 8 illustrates a flow chart of a task scheduling method 800 according to some embodiments of the present application. The task scheduling method 800 may be performed by the cluster 140, for example.
As shown in fig. 8, in step S801, the processing task amount of each of a plurality of data slices of the data to be processed is acquired.
In step S802, the computing power of each computing node in the cluster 140 and the data transmission bandwidth between each computing node are obtained, where the computing power of each computing node is used to describe the task amount processed by each computing node in a unit time.
In step S803, a plurality of generation operations are performed, and a plurality of task scheduling plans for the data to be processed are generated. Each generation operation may generate a task scheduling plan. A more specific embodiment of step S803 corresponds to step S201, and will not be described here.
In step S804, one task scheduling plan having the shortest execution time length is selected from the plurality of task scheduling plans as a task execution mode for the data to be processed. A more specific embodiment of step S804 corresponds to step S202, and will not be described here.
Fig. 9 illustrates a schematic diagram of a task scheduler 900 according to some embodiments of the present application. The management node 141 of the cluster 140 may comprise, for example, a task scheduler 900. The apparatus 900 may comprise a plan generation unit 901 and a plan selection unit 902.
The plan generating unit 901 may perform a plurality of generating operations to generate a plurality of task scheduling plans for data to be processed. Wherein each task scheduling plan is used to describe an execution plan for processing tasks for a plurality of data slices of the data to be processed. The plurality of data slices are stored in a cluster.
In some embodiments, the generating operation performed by the plan generating unit 901 each time includes: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different secondary generation operations is different; the processing task of any one data slice in the first data slice group is distributed to one computing node storing the data slice in the cluster, and the task distribution mode of the data slice in the first data slice group is used as a first task distribution plan; distributing the processing task of any one data slice in the second data slice group to one computing node which does not store the data slice in the cluster, and taking the task distribution mode of the data slice in the second data slice group as a second task distribution plan; and taking an allocation mode comprising the first task allocation plan and the second task allocation plan as a task scheduling plan generated by the current generating operation.
In some embodiments, the data to be processed is video to be transcoded, the plurality of data slices is a plurality of video slices, and the processing task amount of each of the plurality of data slices is the transcoding task amount of each video slice.
In some embodiments, the apparatus 900 may further include a first acquisition unit 903 and a second acquisition unit 904. The first obtaining unit 903 may obtain the processing task amount of each of the plurality of data slices of the data to be processed before generating the plurality of task scheduling plans. The second obtaining unit 904 may obtain the computing power of each computing node in the cluster 140 and the data transmission bandwidth between each computing node. The computing power of each computing node is used for describing the task amount processed by each computing node in unit time.
In some embodiments, the plan generating unit 901 may divide the plurality of data slices into the first data slice group and the second data slice group according to the total task amount of the first data slice group, taking the total task amount of the plurality of data slices as the total task amount of the first data slice group in the first generating operation of the plurality of generating operations. Wherein the first data slice group comprises the plurality of data slices, and the second data slice group is empty. In any one generation operation other than the first generation operation of the plurality of generation operations, the plan generation unit 901 may set the total task amount of the first data slicing group to be reduced by an amount threshold from the total task amount of the first data slicing group in the previous generation operation, and divide the plurality of data slices into the first data slicing group and the second data slicing group according to the set total task amount of the first data slicing group.
In some embodiments, the plan generating unit 901 selects a set of data slices from the plurality of data slices as the first data slice group based on the set total task amount of the first data slice group. The plan generating unit 901 may further add a data slice which does not belong to the first data slice group among the plurality of data slices to the second data slice group.
In some embodiments, the plan generating unit 901 allocates the processing task of any one of the data slices in the first data slice group to one of the plurality of computing nodes storing the data slice capable of earliest completing the processing task of the data slice according to the amount of the processing tasks of the data slice in the first data slice group and the computing power of the computing nodes in the cluster.
In some embodiments, for the data slices in the first data slice group, the plan generating unit 901 sequentially determines the computing nodes corresponding to the processing tasks of each data slice in descending order of the processing task amount. For any one of the first data slices, the plan generating unit 901 may determine a computing node corresponding to a processing task of the data slice according to the following manner: determining a plurality of computing nodes in the cluster that store copies of the data shards; determining an expected completion time point of each computing node of the plurality of computing nodes storing the copy of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragment to the computing capacity of the computing node and the sum of the starting time of the computing node to the processing task of the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragments as the computing node corresponding to the processing task of the data fragments.
In some embodiments, the plan generating unit 901 may allocate the processing task of any one of the data slices in the second data slice group to one of the computing nodes that can complete the processing task of the data slice earliest among the computing nodes that do not store the data slice according to the amount of processing tasks of the data slice in the second data slice group, the computing power of the computing nodes in the cluster, and the data transmission bandwidth between the computing nodes.
In some embodiments, for the data slices in the second data slice group, the plan generating unit 901 may sequentially determine the computing nodes corresponding to the processing tasks of each data slice in descending order of the amount of processing tasks. For any one of the second data slices, the plan generating unit 901 may determine a computing node corresponding to a processing task of the data slice according to the following manner: determining a plurality of computing nodes in the cluster that do not store the data fragment; determining an expected completion time point of each computing node of the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragments to the computing capacity of the computing node, the transmission time of the data fragments to the computing node and the starting time of the computing node to the processing task of the data fragments; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
The schedule selecting unit 902 may select, from the plurality of task scheduling schedules, a task scheduling schedule with a shortest execution duration as a task execution manner for the data to be processed.
In some embodiments, for a task scheduling plan generated by any one generation operation, the plan selection unit 902 determines a completion time for each computing node in the cluster to execute a task according to the task scheduling plan. The plan selection unit 902 may also select the maximum completion time length from the completion time lengths corresponding to the respective computing nodes as the execution time length corresponding to the task scheduling plan. The schedule selecting unit 902 may also select a task scheduling schedule with the shortest execution duration as a task execution mode for the data to be processed.
In summary, the task scheduling device 900 according to the present application may heuristically find various task scheduling plans, and make the locally executed task amounts in different task scheduling plans different, so as to select the task scheduling plan with the highest execution efficiency as a task execution mode, thereby improving load balance of the data to be processed in cluster processing and improving task execution efficiency. In particular, the task scheduling device 900 of the present application fully considers the transmission time of data between different computing nodes, so as to improve the modeling accuracy of the task scheduling scheme, and further improve the execution efficiency of the data to be processed.
FIG. 10 illustrates a block diagram of the components of a computing device. Here, the computing device may be implemented, for example, as a management node 141 of the cluster 140. As shown in fig. 10, the computing device includes one or more processors (CPUs) 1002, a communication module 1004, a memory 1006, a user interface 1010, and a communication bus 1008 for interconnecting these components.
The processor 1002 may receive and transmit data via the communication module 1004 to enable network communication and/or local communication.
The user interface 1010 includes one or more output devices 1012, including one or more speakers and/or one or more visual displays. The user interface 1410 also includes one or more input devices 1014. The user interface 1010 may receive an instruction of a remote controller, for example, but is not limited thereto.
Memory 1006 may be a high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; or non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory 1006 stores a set of instructions executable by the processor 1002, including: an operating system 1016 and applications 1018.
The operating system 1016 includes programs for handling various basic system services and for performing hardware related tasks. In some embodiments, application 1018 may include task scheduler 900 shown in FIG. 9.
In addition, each of the embodiments of the present application may be implemented by a data processing program executed by a data processing apparatus such as a computer. Obviously, the data processing procedure constitutes the present application.
In addition, a data processing program typically stored in one storage medium is executed by directly reading the program out of the storage medium or by installing or copying the program into a storage device (such as a hard disk and/or a memory) of the data processing apparatus. Therefore, such a storage medium also constitutes the present application. The storage medium may use any type of recording means, such as paper storage medium (e.g., paper tape, etc.), magnetic storage medium (e.g., floppy disk, hard disk, flash memory, etc.), optical storage medium (e.g., CD-ROM, etc.), magneto-optical storage medium (e.g., MO, etc.), etc.
The present application thus also discloses a non-volatile storage medium in which a data processing program is stored for performing any one of the embodiments of the task scheduling method described above in the present application.
In addition, the method steps described herein may be implemented by hardware, such as logic gates, switches, application Specific Integrated Circuits (ASIC), programmable logic controllers, embedded microcontrollers, etc., in addition to data processing programs. Such hardware that can implement the methods described herein may also constitute the present application.
The foregoing description of the exemplary embodiments of the present application is not intended to limit the present application, but is intended to cover any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the present application.
Claims (13)
1. A method of task scheduling, the method comprising:
executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of a processing task of a plurality of data fragments of the data to be processed, and the plurality of data fragments are stored in a cluster; and
selecting a task scheduling plan with the shortest execution time from the task scheduling plans as a task execution mode of the data to be processed;
wherein each time the generating operation is executed, the generating operation comprises:
Dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different secondary generation operations is different;
the processing task of any one data slice in the first data slice group is distributed to one computing node storing the data slice in the cluster, and the task distribution mode of the data slice in the first data slice group is used as a first task distribution plan;
distributing the processing task of any one data slice in the second data slice group to one computing node which does not store the data slice in the cluster, and taking the task distribution mode of the data slice in the second data slice group as a second task distribution plan; and
and taking an allocation mode comprising the first task allocation plan and the second task allocation plan as a task scheduling plan generated by the current generation operation.
2. The method of claim 1, wherein the data to be processed is video to be transcoded, the plurality of data slices is a plurality of video slices, and the amount of processing tasks for each of the plurality of data slices is the amount of transcoding tasks for each video slice.
3. The method of claim 1, wherein the method further comprises:
Before generating the task scheduling plans, acquiring the processing task quantity of each data slice in the data slices of the data to be processed;
and acquiring the computing capacity of each computing node in the cluster and the data transmission bandwidth among the computing nodes, wherein the computing capacity of each computing node is used for describing the task amount processed by each computing node in unit time.
4. The method of claim 1, wherein dividing the plurality of data slices into a first data slice group and a second data slice group comprises:
in a first generation operation of the multiple generation operations, taking the total task amount of the plurality of data fragments as the total task amount of the first data fragment group, dividing the plurality of data fragments into a first data fragment group and a second data fragment group according to the total task amount of the first data fragment group, wherein the first data fragment group comprises the plurality of data fragments, and the second data fragment group is empty;
in any generating operation except the first generating operation of the multiple generating operations, setting the total task amount of the first data slicing group to be reduced by an amount threshold value from the total task amount of the first data slicing group in the last generating operation, and dividing the data slicing into the first data slicing group and the second data slicing group according to the set total task amount of the first data slicing group.
5. The method of claim 4, wherein the dividing the plurality of data slices into the first data slice group and the second data slice group according to the set total task volume of the first data slice group comprises:
selecting a set of data slices from the plurality of data slices as the first data slice group based on the set total task amount of the first data slice group;
and adding the data fragments which do not belong to the first data fragment group in the plurality of data fragments to the second data fragment group.
6. The method of any of claims 1-5, wherein the assigning processing tasks for any one of the first set of data slices to one of the computing nodes in the cluster that stores the data slice comprises:
and distributing the processing task of any one data slice in the first data slice group to one computing node capable of finishing the processing task of the data slice earliest in a plurality of computing nodes storing the data slice according to the processing task quantity of the data slice in the first data slice group and the computing capacity of the computing nodes in the cluster.
7. The method of claim 6, wherein the assigning the processing task of any one of the first data shards to one of the plurality of computing nodes storing the data shard that completed the processing task of the data shard earliest according to the amount of processing tasks of each of the data shards in the first data shard group and the computing power of each of the computing nodes in the cluster comprises:
For the data fragments in the first data fragment group, sequentially determining computing nodes corresponding to the processing tasks of each data fragment according to the descending order of the processing task quantity;
for any one of the first data slices, determining a computing node corresponding to a processing task of the data slice includes:
determining a plurality of computing nodes in the cluster that store copies of the data shards;
determining an expected completion time point of each computing node of the plurality of computing nodes storing the copy of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragment to the computing capacity of the computing node and the sum of the starting time of the computing node to the processing task of the data fragment;
and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragments as the computing node corresponding to the processing task of the data fragments.
8. The method of any of claims 1-5, wherein the assigning processing tasks of any one of the second set of data slices to one of the computing nodes in the cluster that does not store the data slice comprises:
and distributing the processing task of any one data slice in the second data slice group to one computing node which can finish the processing task of the data slice earliest in a plurality of computing nodes which do not store the data slice according to the processing task quantity of the data slice in the second data slice group, the computing capacity of the computing nodes in the cluster and the data transmission bandwidth among the computing nodes.
9. The method of claim 8, wherein the allocating the processing task of any one of the second data shards to one of the plurality of computing nodes that did not store the data shard that completed the processing task of the data shard earliest according to the amount of processing tasks of the data shards in the second data shard group, the computing power of the computing nodes in the cluster, and the transmission bandwidth between the computing nodes, comprises:
for the data fragments in the second data fragment group, sequentially determining computing nodes corresponding to the processing tasks of each data fragment according to the descending order of the processing task quantity;
For any one of the second data slices, determining a computing node corresponding to a processing task of the data slice includes:
determining a plurality of computing nodes in the cluster that do not store the data fragment;
determining an expected completion time point of each computing node of the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is: the time value obtained by adding the task processing duration of the processing task of the data fragment to the starting time point of the processing task of the data fragment by the computing node, wherein the task processing duration of each computing node is as follows: the ratio of the processing task quantity of the data fragments to the computing capacity of the computing node, the transmission time of the data fragments to the computing node and the starting time of the computing node to the processing task of the data fragments;
and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
10. The method of claim 1, wherein selecting, from the plurality of task scheduling plans, a task scheduling plan having a shortest execution duration as a task execution manner for the data to be processed, comprises:
For a task scheduling plan generated by any generating operation, determining the completion time of each computing node in the cluster for executing the task according to the task scheduling plan;
selecting the maximum completion time length from the completion time lengths corresponding to the computing nodes as the execution time length corresponding to the task scheduling plan;
and selecting a task scheduling plan with the shortest execution duration as a task execution mode for the data to be processed.
11. A task scheduling device, the device comprising:
a plan generating unit configured to perform a plurality of generating operations, and generate a plurality of task scheduling plans for data to be processed, where each task scheduling plan is configured to describe an execution plan for a processing task for a plurality of data slices of the data to be processed, the plurality of data slices being stored in a cluster; and
a plan selecting unit, configured to select, from the plurality of task scheduling plans, a task scheduling plan with a shortest execution duration as a task execution manner for the data to be processed;
wherein the generating operation executed by the plan generating unit each time includes:
dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different secondary generation operations is different;
The processing task of any one data slice in the first data slice group is distributed to one computing node storing the data slice in the cluster, and the task distribution mode of the data slice in the first data slice group is used as a first task distribution plan;
distributing the processing task of any one data slice in the second data slice group to one computing node which does not store the data slice in the cluster, and taking the task distribution mode of the data slice in the second data slice group as a second task distribution plan; and
and taking an allocation mode comprising the first task allocation plan and the second task allocation plan as a task scheduling plan generated by the current generation operation.
12. A computing device, comprising:
a processor;
a memory; and
one or more programs stored in the memory and configured to be executed by the processor, the one or more programs comprising instructions for performing the method of any of claims 1-10.
13. A storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811501565.2A CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811501565.2A CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111290841A CN111290841A (en) | 2020-06-16 |
| CN111290841B true CN111290841B (en) | 2024-04-05 |
Family
ID=71020760
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811501565.2A Active CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111290841B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813524B (en) * | 2020-07-09 | 2023-09-08 | 北京奇艺世纪科技有限公司 | Task execution method and device, electronic equipment and storage medium |
| CN112346845B (en) * | 2021-01-08 | 2021-04-16 | 腾讯科技（深圳）有限公司 | Method, device and equipment for scheduling coding tasks and storage medium |
| CN114510540A (en) * | 2022-04-19 | 2022-05-17 | 北京微芯感知科技有限公司 | Data processing method, calculation storage separation system and block chain network architecture |
| CN114866334B (en) * | 2022-06-09 | 2023-11-24 | 中国工商银行股份有限公司 | Data fusion processing method and device |
| CN114862606B (en) * | 2022-06-13 | 2023-05-09 | 新疆益盛鑫创展科技有限公司 | Insurance information processing method and device based on cloud service |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7693813B1 (en) * | 2007-03-30 | 2010-04-06 | Google Inc. | Index server architecture using tiered and sharded phrase posting lists |
| CN103226467A (en) * | 2013-05-23 | 2013-07-31 | 中国人民解放军国防科学技术大学 | Data parallel processing method and system as well as load balancing scheduler |
| CN104182279A (en) * | 2014-02-26 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Task scheduling method, device and system |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
| WO2017008477A1 (en) * | 2015-07-14 | 2017-01-19 | 杭州海康威视数字技术股份有限公司 | Cluster video analysis method and system |
| CN108391142A (en) * | 2018-03-30 | 2018-08-10 | 腾讯科技（深圳）有限公司 | A kind of method and relevant device of video source modeling |
-
2018
- 2018-12-10 CN CN201811501565.2A patent/CN111290841B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7693813B1 (en) * | 2007-03-30 | 2010-04-06 | Google Inc. | Index server architecture using tiered and sharded phrase posting lists |
| CN103226467A (en) * | 2013-05-23 | 2013-07-31 | 中国人民解放军国防科学技术大学 | Data parallel processing method and system as well as load balancing scheduler |
| CN104182279A (en) * | 2014-02-26 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Task scheduling method, device and system |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| WO2017008477A1 (en) * | 2015-07-14 | 2017-01-19 | 杭州海康威视数字技术股份有限公司 | Cluster video analysis method and system |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
| CN108391142A (en) * | 2018-03-30 | 2018-08-10 | 腾讯科技（深圳）有限公司 | A kind of method and relevant device of video source modeling |
Non-Patent Citations (1)
| Title |
|---|
| 基于节点集计算能力差异的Hadoop自适应任务调度算法;朱洁;李雯睿;王江平;赵红;;计算机应用;20160410(04);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111290841A (en) | 2020-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111290841B (en) | Task scheduling method, device, computing equipment and storage medium | |
| US10298969B2 (en) | Architecture and method for high performance on demand video transcoding | |
| Li et al. | Cost-efficient and robust on-demand video transcoding using heterogeneous cloud services | |
| CN107045456B (en) | Resource allocation method and resource manager | |
| US9407944B1 (en) | Resource allocation optimization for cloud-based video processing | |
| KR102499076B1 (en) | Graph data-based task scheduling method, device, storage medium and apparatus | |
| CN107885595B (en) | Resource allocation method, related equipment and system | |
| CN109189563B (en) | Resource scheduling method and device, computing equipment and storage medium | |
| US9489231B2 (en) | Selecting provisioning targets for new virtual machine instances | |
| JP2017050001A (en) | System and method for use in efficient neural network deployment | |
| CN112148492B (en) | Service deployment and resource allocation method considering multi-user mobility | |
| US11775350B2 (en) | Compute resource estimation for function implementation on computing platform | |
| US20230024699A1 (en) | System for high performance on-demand video transcoding | |
| Meskar et al. | Fair multi-resource allocation with external resource for mobile edge computing | |
| JP7006607B2 (en) | Distributed processing system, distributed processing method, and recording medium | |
| WO2015042904A1 (en) | Method, apparatus and system for scheduling resource pool in multi-core system | |
| Cao et al. | A deep reinforcement learning approach to multi-component job scheduling in edge computing | |
| US20230379268A1 (en) | Resource scheduling method and system, electronic device, computer readable storage medium | |
| JP6721800B2 (en) | Cooperative distributed system, cooperative distributed management device, cooperative distributed method, and program | |
| CN110399210B (en) | Task scheduling method and device based on edge cloud | |
| Mavromoustakis et al. | Dynamic cloud resource migration for efficient 3D video processing in mobile computing environments | |
| CN115840634B (en) | Service execution method, device, equipment and storage medium | |
| Lee et al. | Quality-aware transcoding task allocation under limited power in live-streaming systems | |
| CN115361284B (en) | Deployment adjustment method of virtual network function based on SDN | |
| CN116112976B (en) | Equipment calculation migration method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |