CN105718244A - Streamline data shuffle Spark task scheduling and executing method - Google Patents
Streamline data shuffle Spark task scheduling and executing method Download PDFInfo
- Publication number
- CN105718244A CN105718244A CN201610029211.7A CN201610029211A CN105718244A CN 105718244 A CN105718244 A CN 105718244A CN 201610029211 A CN201610029211 A CN 201610029211A CN 105718244 A CN105718244 A CN 105718244A
- Authority
- CN
- China
- Prior art keywords
- task
- shuffling
- transmission
- node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 230000005540 biological transmission Effects 0.000 claims description 67
- 238000013507 mapping Methods 0.000 claims description 44
- 230000006870 function Effects 0.000 claims description 15
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 14
- 238000013480 data collection Methods 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 claims description 3
- 230000001960 triggered effect Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001737 promoting effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
- G06F9/3867—Concurrent instruction execution, e.g. pipeline or look ahead using instruction pipelines
- G06F9/3869—Implementation aspects, e.g. pipeline latches; pipeline synchronisation and clocking
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Multi Processors (AREA)
Abstract
The invention discloses a streamline data shuffle Spark task scheduling and executing method. According to the method, stages and tasks in the stages are submitted and executed from back side to front side; meanwhile, a predecessor task is used for sending an execution result to a memory of a successor task; and under the conditions that a user interface is not changed, and the completeness and the fault tolerance of the stages are also not damaged, the problem of disk accessing expenditure of the shuffle of the original Spark in different stages is solved, so that the operation time of a distributed calculation program on the Spark is reduced.
Description
Technical field
The present invention relates to computer distribution type Computational frame field.Specifically, on the basis of distributed computing framework Spark, his Task Scheduling Mechanism is mainly changed, thus promoting the performance of this Computational frame.
Background technology
Spark, as current most widely used distributed computing framework, has been deployed in countless data centers.The distributed rebound data collection (ResilientDistributedDataset, RDD) that it proposes makes carrying out in internal memory of the calculating process maximum possible of big data.Performing in logic, Spark generates RDD from front to back according to the logic of user program, and each RDD can have the dependence of oneself.When the program of user needs final output result, Spark will find forward from last RDD recurrence, and divides the stage (Stage) according to transmission dependence (ShuffleDependency) of shuffling wherein existed.After having divided the stage, Spark will presentation stage from front to back, first submit the stage not lacking dependence to, successively backward.This scheduling logic makes the calculative position of data automatic stream, and allows and calculate being saved in internal memory of intermediate object program maximum possible.
However, to ensure that segmentation between the stage (Stage) and the fault-tolerance of framework own, when the transmission of shuffling dividing each stage relies on (ShuffleDependency), the intermediate object program that prodromic phase produces can be stored in disk by Spark, then start to distribute the task in next stage, gone the data in long-range reading disk by the task in stage more afterwards, be then calculated.
Being much more slowly than in the reality of internal memory at current disk speed, the reading and writing data of this part becomes the maximum bottleneck of restriction Spark performance boost.But for fear of the integrity of division in stage and fault-tolerance, remain without the patch for optimizing this part bottleneck at present or solution occurs.
Summary of the invention
The present invention is directed to the bottleneck of reading writing harddisk when Spark data shuffle transmission, propose a kind of streamlined data to shuffle the Spark method for scheduling task of transmission (Shuffle), by changing the submission order of Spark task, a task is made to start, prior to his predecessor task, the distribution that is scheduled, adopt predecessor task to send simultaneously and perform result to the method in the internal memory of subsequent tasks, do not changing user interface, not while the integrity of failure stage and fault-tolerance, solve originally Spark to shuffle in different phase (Stage) the disk read-write expense of data transmission (Shuffle).Making predecessor task send the data to subsequent tasks by network while the generation intermediate object program performed, thus avoiding the read-write of disk I/O, promoting the performance of Spark distributed computing framework.
The technical solution of the present invention is as follows:
A kind of streamlined data are shuffled Spark task scheduling and the execution method of transmission, comprise the steps:
Step 1: when Spark submit to a task and this task be divided into multiple stage submit to time, first find user to perform task and generate the last stage of result；
Step 2: from the last stage, it is judged that whether this stage comprises the prodromic phase being not fully complete:
If the prodromic phase in this stage has all performed, then this stage is submitted to perform；
If there being prodromic phase not to be performed, then by this phased markers for waiting, submit to this stage to perform simultaneously, and recurrence submits the prodromic phase in this stage to；
Step 3: after submitting to a stage to perform, whether this stage resolutions is become multiple task by scheduler, and judges this stage: be loitering phase:
If this stage is marked as wait, then scheduler performs node to the free time that explorer request is identical with task number, after scheduler obtains the corresponding execution node performing task, the dependence recurrence of the distributed rebound data collection comprised according to this stage is found forward transmission of shuffling and is relied on, scheduler often find one shuffle transmission rely on will to mapping output trace table register this time shuffle transmission flowing water information, after registration is complete, scheduler is also notified that each execution node that namely will run this task gets out corresponding internal memory and carrys out the intermediate object program that their predecessor task of buffer memory sends；After each execution node receives the log-on message of scheduler, can in local cache newly-built one with shuffle transmission rely on ID for index, value is the key-value pair of the always several buffer memory array of conventions data block, also can rely on ID for index at local newly-built one with transmission of shuffling simultaneously, value is the key-value pair of the semaphore data structure of conventions data block sum, wherein each semaphore comprises the duty mapping sum of shuffling relied on of shuffling specifically；
Otherwise, it is directly entered next step；
Step 4: when the set of tasks of scheduler encapsulated phase, it is judged that whether this stage is a mapping phase of shuffling:
If this stage is a mapping phase of shuffling, then the transmission of shuffling that each task in this stage arranges correspondence relies on ID；
Otherwise, it is directly entered next step；
Step 5: packaged task is distributed to each and performs node by scheduler；
Step 6: when performing on the executor that task is assigned to each execution node, executor can judge that this task is whether for mapping tasks of shuffling:
If, the transmission of shuffling then comprised according to this task relies on ID, to mapping the aggregate information that output trace table asks the execution node of stipulations task corresponding for this ID, then, the corresponding stipulations information of mapping tasks of shuffling is set, conventions data block number in the combining information received and remote address are packaged into a Hash table and pass to this mapping tasks of shuffling, and enter step 7；
If executor judges that this task is stipulations task, the function calling this stipulations task corresponding is calculated, and enters step 11；
Step 7: when a mapping tasks of shuffling starts to perform, can check the need for the output of streamlined data；
The need to, intermediate object program key-value pair is calculated the conventions data block number of his correspondence by the grader of the grader first specified according to user or Spark acquiescence according to key, according to the data block number arranged and remote address Hash table, the execution node that the data result obtained is sent to the responsible follow-up stipulations task of correspondence will be calculated, the information sent includes: transmission of shuffling relies on ID, conventions data block number, the key-value pair of data result；While sending data, executor is to disk write data, and enters step 8；Meanwhile, after the execution node of responsible stipulations task receives pipelined data, can, using the data dependence ID that shuffles as index, preserve in the buffer memory of conventions data block number of buffer memory array corresponding for this ID, enter step 8；
If it is not required, then result write disk directly will be performed, enter step 8；
Step 8: executor completes mapping tasks of shuffling；
Step 9: when one shuffle mapping tasks end of run time, flowing water ending message will be sent to the execution node of all of responsible stipulations task, this information includes: transmission of shuffling relies on ID, the mapping data block number that this task is responsible for, corresponding with this information performs the conventions data block number that node is responsible for；
Step 10: after the execution node of responsible stipulations task receives the information that flowing water terminates, can rely on ID as index according to transmission of shuffling, find semaphore array corresponding for this ID, subtract one by wherein conventions data block number CountDownLatch.If this semaphore is reduced to 0, then it represents that the data that this conventions data block relies on map whole ends of transmission；
Step 11: when executor performs the specified function of stipulations task, can call corresponding stipulations function, and this function, when the task of execution reads data, can ask an iterator reading in data to performing node；
Step 12: to performing node inquiry shuffles to transmit whether have local cache specifically, namely whether can be transmitted by streamlined data when generating iterator:
If it is, call the acquisition caching method performing node, rely on conventions data block number responsible with this task for ID to performing node request buffer memory according to the transmission of shuffling of stipulations task, and enter step 13；
Otherwise, read teledata, enter step 15；
Step 13: perform after node receives and obtain the calling of buffer memory, to rely on ID for index with transmission of shuffling, finds buffer memory array corresponding in buffer memory, and returns conventions data block number the asynchronous of buffer memory and quote；
Step 14: iterator receive buffer memory asynchronous quote time, start waiting for, until the CountDownLatch semaphore in the conventions data block number of the transmission dependence of shuffling of this required by task becomes 0, represent that the mapping data block that this conventions data block relies on is fully completed, enter step 15；
Step 15: executor performs to specify stipulations function.
Compared with prior art, the invention has the beneficial effects as follows: can significantly shorten data in Spark tasks carrying process and shuffle the time transmitted, shorten the execution time of distributed computing task.
Accompanying drawing explanation
Fig. 1. configuration diagram
Fig. 2. scheduler schedules method flow diagram
Fig. 3. perform node flow chart
Fig. 4. flowing water log-on message
Fig. 5. flowing water announcement information
Fig. 6. flowing water transmission information
Fig. 7. flowing water ending message
Fig. 8. perform nodal cache framework
Fig. 9. perform node signal and control framework
Specific implementation method
Below with reference to accompanying drawing, embodiments of the invention are elaborated.The present embodiment is carried out under the premise of technical solution of the present invention and algorithm, and provides detailed embodiment and specific operation process, but is suitable for platform and is not limited to following embodiment.As long as the Spark of cluster compatibility master just can run the present invention.The small-sized cluster that the concrete operations platform of this example is made up of two common servers, equipped with UbuntuServer14.04.1LTS64bit on each server, and is equipped with 8GB internal memory.The concrete exploitation of the present invention is based on the source code version of ApacheSpark1.5.
As shown in Figure 1, the present invention includes scheduler (DAGScheduler) based on the original framework of Spark, explorer (BlockMangerMaster), map output trace table (MapOutputTracker) and perform node (BlockManger) and executor (Executor), by changing task scheduling algorithm and performing flow process and realize the streamlined that the data of shuffling in not the completing property of failure stage and the basis of fault-tolerance are transmitted.Scheduler and execution node are respectively according to accompanying drawing 2 and the process flow operation in accompanying drawing 3, thus realizing the lifting of Distributed Calculation performance.
All deploying the Spark comprising the present invention on every station server, wherein a station server is as the Master of Spark cluster, and an other machine is as the Slave of cluster.It should be noted that the performance in order to ensure the present invention, the cluster response of deployment configures the internal memory bigger than original Spark cluster, and concrete memory size is depending on the data volume size of operation.
Namely Distributed Calculation application can be run according to the method for operation of Spark after having disposed.Change the user for using Spark Computational frame fully transparent.
A kind of streamlined data are shuffled Spark task scheduling and the execution method of transmission, comprise the steps:
Step 1: when Spark submit to a task and this task be divided into multiple stage submit to time, first find user to perform task and generate the last stage of result；
Step 2: from the last stage, it is judged that whether this stage comprises the prodromic phase being not fully complete:
If the prodromic phase in this stage has all performed, then this stage is submitted to perform；
If there being prodromic phase not to be performed, then by this phased markers for waiting, submit to this stage to perform simultaneously, and recurrence submits the prodromic phase in this stage to；
Step 3: after submitting to a stage to perform, whether this stage resolutions is become multiple task by scheduler, and judges this stage: be loitering phase:
If this stage is marked as wait, then scheduler performs node to the free time that explorer request is identical with task number, after scheduler obtains the corresponding execution node performing task, the dependence recurrence of the distributed rebound data collection comprised according to this stage is found forward transmission of shuffling and is relied on, scheduler often find one shuffle transmission rely on will to mapping output trace table register this time shuffle transmission flowing water information, after registration is complete, scheduler is also notified that each execution node that namely will run this task gets out corresponding internal memory and carrys out the intermediate object program that their predecessor task of buffer memory sends；After each execution node receives the log-on message of scheduler, can in local cache newly-built one with shuffle transmission rely on ID for index, value is the key-value pair of the always several buffer memory array of conventions data block, also can rely on ID for index at local newly-built one with transmission of shuffling simultaneously, value is the key-value pair of the semaphore data structure of conventions data block sum, wherein each semaphore comprises the duty mapping sum of shuffling relied on of shuffling specifically；
Otherwise, it is directly entered next step；
Step 4: when the set of tasks of scheduler encapsulated phase, it is judged that whether this stage is a mapping phase of shuffling:
If this stage is a mapping phase of shuffling, then the transmission of shuffling that each task in this stage arranges correspondence relies on ID；
Otherwise, it is directly entered next step；
Step 5: packaged task is distributed to each and performs node by scheduler；
Step 6: when performing on the executor that task is assigned to each execution node, executor can judge that this task is whether for mapping tasks of shuffling:
If, the transmission of shuffling then comprised according to this task relies on ID, to mapping the aggregate information that output trace table asks the execution node of stipulations task corresponding for this ID, then, the corresponding stipulations information of mapping tasks of shuffling is set, conventions data block number in the combining information received and remote address are packaged into a Hash table and pass to this mapping tasks of shuffling, and enter step 7；
If executor judges that this task is stipulations task, the function calling this stipulations task corresponding is calculated, and enters step 11；
Step 7: when a mapping tasks of shuffling starts to perform, can check the need for the output of streamlined data；
The need to, intermediate object program key-value pair is calculated the conventions data block number of his correspondence by the grader of the grader first specified according to user or Spark acquiescence according to key, according to the data block number arranged and remote address Hash table, the execution node that the data result obtained is sent to the responsible follow-up stipulations task of correspondence will be calculated, the information sent includes: transmission of shuffling relies on ID, conventions data block number, the key-value pair of data result；While sending data, executor is to disk write data, and enters step 8；Meanwhile, after the execution node of responsible stipulations task receives pipelined data, can, using the data dependence ID that shuffles as index, preserve in the buffer memory of conventions data block number of buffer memory array corresponding for this ID, enter step 8；
If it is not required, then result write disk directly will be performed, enter step 8；
Step 8: executor completes mapping tasks of shuffling；
Step 9: when one shuffle mapping tasks end of run time, flowing water ending message will be sent to the execution node of all of responsible stipulations task, this information includes: transmission of shuffling relies on ID, the mapping data block number that this task is responsible for, corresponding with this information performs the conventions data block number that node is responsible for；
Step 10: after the execution node of responsible stipulations task receives the information that flowing water terminates, can rely on ID as index according to transmission of shuffling, find semaphore array corresponding for this ID, subtract one by wherein conventions data block number CountDownLatch.If this semaphore is reduced to 0, then it represents that the data that this conventions data block relies on map whole ends of transmission；
Step 11: when executor performs the specified function of stipulations task, can call corresponding stipulations function, and this function, when the task of execution reads data, can ask an iterator reading in data to performing node；
Step 12: to performing node inquiry shuffles to transmit whether have local cache specifically, namely whether can be transmitted by streamlined data when generating iterator:
If it is, call the acquisition caching method performing node, rely on conventions data block number responsible with this task for ID to performing node request buffer memory according to the transmission of shuffling of stipulations task, and enter step 13；
Otherwise, read teledata, enter step 15；
Step 13: perform after node receives and obtain the calling of buffer memory, to rely on ID for index with transmission of shuffling, finds buffer memory array corresponding in buffer memory, and returns conventions data block number the asynchronous of buffer memory and quote；
Step 14: iterator receive buffer memory asynchronous quote time, start waiting for, until the CountDownLatch semaphore in the conventions data block number of the transmission dependence of shuffling of this required by task becomes 0, represent that the mapping data block that this conventions data block relies on is fully completed, enter step 15；
Step 15: executor performs to specify stipulations function.
When performing above step, one step of any of which is made mistakes and all can be triggered fault-tolerance mechanism: if mistake betides the predecessor task of the transmission of shuffling of a streamlined, namely shuffle the step of mapping tasks, and the either step before this step, so his follow-up all can be marked as failure and resubmit, and continues executing with the transmission of shuffling of streamlined；If mistake betides the subsequent tasks of the transmission of shuffling of a streamlined, the i.e. execution step of stipulations task, so predecessor task can't be affected, and the subsequent tasks of failure can be resubmited, the subsequent tasks now resubmited will go to read required data from the disk of predecessor task.
Due to the node of the pending stipulations tasks such as the present embodiment transmits data to while mapping tasks is shuffled in execution, thus the waiting time of transmission of being shuffled by script Spark stashes.When mapping tasks of shuffling terminates, stipulations task just can start within the more shorter time, thus accelerating the speed of whole Distributed Calculation.
By the benchmark program of the relevant Spark such as WordCount on the basis of this embodiment, demonstrating the correctness of the present invention, the present invention has lifting in various degree compared to the Spark of master in performance in different benchmark programs simultaneously.
The preferred embodiment of the present invention described in detail above.Should be appreciated that the ordinary skill of this area just can make many modifications and variations according to the design of the present invention without creative work.Therefore, all technical staff in the art, all should in the protection domain being defined in the patent claims under this invention's idea on the basis of existing technology by the available technical scheme of logical analysis, reasoning, or a limited experiment.
Claims (6)
1. streamlined data are shuffled the Spark task scheduling of transmission and execution method, it is characterised in that comprise the steps:
Step 1: when Spark submit to a task and this task be divided into multiple stage submit to time, first find user to perform task and generate the last stage of result；
Step 2: from the last stage, it is judged that whether this stage comprises the prodromic phase being not fully complete:
If the prodromic phase in this stage has all performed, then this stage is submitted to perform；
If there being prodromic phase not to be performed, then by this phased markers for waiting, submit to this stage to perform simultaneously, and recurrence submits the prodromic phase in this stage to；
Step 3: after submitting to a stage to perform, whether this stage resolutions is become multiple task by scheduler, and judges this stage: be loitering phase:
If this stage is marked as wait, then scheduler performs node to the free time that explorer request is identical with task number, after scheduler obtains the corresponding execution node performing task, the dependence recurrence of the distributed rebound data collection comprised according to this stage is found forward transmission of shuffling and is relied on, scheduler often find one shuffle transmission rely on will to mapping output trace table register this time shuffle transmission flowing water information, after registration is complete, scheduler is also notified that each execution node that namely will run this task gets out corresponding internal memory and carrys out the intermediate object program that their predecessor task of buffer memory sends；After each execution node receives the log-on message of scheduler, can in local cache newly-built one with shuffle transmission rely on ID for index, value is the key-value pair of the always several buffer memory array of conventions data block, also can rely on ID for index at local newly-built one with transmission of shuffling simultaneously, value is the key-value pair of the semaphore data structure of conventions data block sum, wherein each semaphore comprises the duty mapping sum of shuffling relied on of shuffling specifically；
Otherwise, it is directly entered next step；
Step 4: when the set of tasks of scheduler encapsulated phase, it is judged that whether this stage is a mapping phase of shuffling:
If this stage is a mapping phase of shuffling, then the transmission of shuffling that each task in this stage arranges correspondence relies on ID；
Otherwise, it is directly entered next step；
Step 5: packaged task is distributed to each and performs node by scheduler；
Step 6: when performing on the executor that task is assigned to each execution node, executor can judge that this task is whether for mapping tasks of shuffling:
If, the transmission of shuffling then comprised according to this task relies on ID, to mapping the aggregate information that output trace table asks the execution node of stipulations task corresponding for this ID, then, the corresponding stipulations information of mapping tasks of shuffling is set, conventions data block number in the combining information received and remote address are packaged into a Hash table and pass to this mapping tasks of shuffling, and enter step 7；
If executor judges that this task is stipulations task, the function calling this stipulations task corresponding is calculated, and enters step 11；
Step 7: when a mapping tasks of shuffling starts to perform, can check the need for the output of streamlined data；
The need to, intermediate object program key-value pair is calculated the conventions data block number of his correspondence by the grader of the grader first specified according to user or Spark acquiescence according to key, according to the data block number arranged and remote address Hash table, the execution node that the data result obtained is sent to the responsible follow-up stipulations task of correspondence will be calculated, the information sent includes: transmission of shuffling relies on ID, conventions data block number, the key-value pair of data result；While sending data, executor is to disk write data, and enters step 8；Meanwhile, after the execution node of responsible stipulations task receives pipelined data, can, using the data dependence ID that shuffles as index, preserve in the buffer memory of conventions data block number of buffer memory array corresponding for this ID, enter step 8；
If it is not required, then result write disk directly will be performed, enter step 8；
Step 8: executor completes mapping tasks of shuffling；
Step 9: when one shuffle mapping tasks end of run time, flowing water ending message will be sent to the execution node of all of responsible stipulations task, this information includes: transmission of shuffling relies on ID, the mapping data block number that this task is responsible for, corresponding with this information performs the conventions data block number that node is responsible for；
Step 10: after the execution node of responsible stipulations task receives the information that flowing water terminates, can rely on ID as index according to transmission of shuffling, find semaphore array corresponding for this ID, subtract one by wherein conventions data block number CountDownLatch.If this semaphore is reduced to 0, then it represents that the data that this conventions data block relies on map whole ends of transmission；
Step 11: when executor performs the specified function of stipulations task, can call corresponding stipulations function, and this function, when the task of execution reads data, can ask an iterator reading in data to performing node；
Step 12: to performing node inquiry shuffles to transmit whether have local cache specifically, namely whether can be transmitted by streamlined data when generating iterator:
If it is, call the acquisition caching method performing node, rely on conventions data block number responsible with this task for ID to performing node request buffer memory according to the transmission of shuffling of stipulations task, and enter step 13；
Otherwise, read teledata, enter step 15；
Step 13: perform after node receives and obtain the calling of buffer memory, to rely on ID for index with transmission of shuffling, finds buffer memory array corresponding in buffer memory, and returns conventions data block number the asynchronous of buffer memory and quote；
Step 14: iterator receive buffer memory asynchronous quote time, start waiting for, until the CountDownLatch semaphore in the conventions data block number of the transmission dependence of shuffling of this required by task becomes 0, represent that the mapping data block that this conventions data block relies on is fully completed, enter step 15；
Step 15: executor performs to specify stipulations function.
2. streamlined data according to claim 1 are shuffled Spark task scheduling and the execution method of transmission, it is characterized in that, in described step 1 to 15, an arbitrary step is made mistakes and all can be triggered fault-tolerance mechanism: if mistake betides the predecessor task of the transmission of shuffling of a streamlined, namely shuffle the step of mapping tasks, and the either step before this step, so his follow-up all can be marked as failure and resubmit, and continues executing with the transmission of shuffling of streamlined；If mistake betides the subsequent tasks of the transmission of shuffling of a streamlined, the i.e. execution step of stipulations task, so predecessor task can't be affected, and the subsequent tasks of failure can be resubmited, the subsequent tasks now resubmited will go to read required data from the disk of predecessor task.
3. streamlined data according to claim 1 are shuffled Spark task scheduling and the execution method of transmission, it is characterized in that, in described step 3, scheduler performs node to the free time that explorer request is identical with task number, takes the random strategy performing node obtaining the free time.
4. streamlined data according to claim 1 are shuffled Spark task scheduling and the execution method of transmission, it is characterised in that transmission flowing water information of shuffling in described step 3, rely on the information of the execution node set of ID and correspondence including transmission of shuffling.
5. streamlined data according to claim 4 are shuffled Spark task scheduling and the execution method of transmission, it is characterized in that, the information of described execution node set comprises: this performs conventions data block number and the metamessage that node is responsible for, and this metamessage includes nodename and address.
6. streamlined data according to claim 1 are shuffled Spark task scheduling and the execution method of transmission, it is characterized in that, in described step 3 after registration is complete, scheduler is also notified that each node that performs that namely will run this task gets out corresponding internal memory and carrys out the intermediate object program that their predecessor task of buffer memory sends, and this information comprises: the ID of transmission that shuffles specifically, forerunner shuffle the conventions data block number and conventions data block sum that the piecemeal sum of mapping tasks, this this subtask of execution node be responsible for.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610029211.7A CN105718244B (en) | 2016-01-18 | 2016-01-18 | A kind of streamlined data are shuffled Spark task schedulings and the execution method of transmission |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610029211.7A CN105718244B (en) | 2016-01-18 | 2016-01-18 | A kind of streamlined data are shuffled Spark task schedulings and the execution method of transmission |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105718244A true CN105718244A (en) | 2016-06-29 |
| CN105718244B CN105718244B (en) | 2018-01-12 |
Family
ID=56147869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610029211.7A Active CN105718244B (en) | 2016-01-18 | 2016-01-18 | A kind of streamlined data are shuffled Spark task schedulings and the execution method of transmission |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105718244B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106168963A (en) * | 2016-06-30 | 2016-11-30 | 北京金山安全软件有限公司 | Real-time streaming data processing method and device and server |
| CN106371919A (en) * | 2016-08-24 | 2017-02-01 | 上海交通大学 | Shuffle data caching method based on mapping-reduction calculation model |
| CN107612886A (en) * | 2017-08-15 | 2018-01-19 | 中国科学院大学 | A kind of Spark platforms Shuffle process compresses algorithm decision-making techniques |
| CN107885587A (en) * | 2017-11-17 | 2018-04-06 | 清华大学 | A kind of executive plan generation method of big data analysis process |
| CN109951556A (en) * | 2019-03-27 | 2019-06-28 | 联想(北京)有限公司 | A kind of Spark task processing method and system |
| CN110083441A (en) * | 2018-01-26 | 2019-08-02 | 中兴飞流信息科技有限公司 | A kind of distributed computing system and distributed computing method |
| CN110109747A (en) * | 2019-05-21 | 2019-08-09 | 北京百度网讯科技有限公司 | Method for interchanging data and system, server based on Apache Spark |
| CN110134714A (en) * | 2019-05-22 | 2019-08-16 | 东北大学 | A kind of distributed computing framework caching index suitable for big data iterative calculation |
| CN110750341A (en) * | 2018-07-24 | 2020-02-04 | 深圳市优必选科技有限公司 | Task scheduling method, device, system, terminal equipment and storage medium |
| CN111061565A (en) * | 2019-12-12 | 2020-04-24 | 湖南大学 | Two-stage pipeline task scheduling method and system in Spark environment |
| CN111258785A (en) * | 2020-01-20 | 2020-06-09 | 北京百度网讯科技有限公司 | Data shuffling method and device |
| CN111782367A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Distributed storage method and device, electronic equipment and computer readable medium |
| CN112269648A (en) * | 2020-11-13 | 2021-01-26 | 北京轩宇信息技术有限公司 | Parallel task allocation method and device for multi-stage program analysis |
| CN113364603A (en) * | 2020-03-06 | 2021-09-07 | 华为技术有限公司 | Fault recovery method of ring network and physical node |
| CN113495679A (en) * | 2020-04-01 | 2021-10-12 | 孟彤 | Optimization method for large data storage access and processing based on nonvolatile storage medium |
| CN114448660A (en) * | 2021-12-16 | 2022-05-06 | 国网江苏省电力有限公司电力科学研究院 | Internet of things data access method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130346988A1 (en) * | 2012-06-22 | 2013-12-26 | Microsoft Corporation | Parallel data computing optimization |
| CN103605576A (en) * | 2013-11-25 | 2014-02-26 | 华中科技大学 | Multithreading-based MapReduce execution system |
| US20140358977A1 (en) * | 2013-06-03 | 2014-12-04 | Zettaset, Inc. | Management of Intermediate Data Spills during the Shuffle Phase of a Map-Reduce Job |
| CN104750482A (en) * | 2015-03-13 | 2015-07-01 | 合一信息技术（北京）有限公司 | Method for constructing dynamic script execution engine based on MapReduce |
-
2016
- 2016-01-18 CN CN201610029211.7A patent/CN105718244B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130346988A1 (en) * | 2012-06-22 | 2013-12-26 | Microsoft Corporation | Parallel data computing optimization |
| US20140358977A1 (en) * | 2013-06-03 | 2014-12-04 | Zettaset, Inc. | Management of Intermediate Data Spills during the Shuffle Phase of a Map-Reduce Job |
| CN103605576A (en) * | 2013-11-25 | 2014-02-26 | 华中科技大学 | Multithreading-based MapReduce execution system |
| CN104750482A (en) * | 2015-03-13 | 2015-07-01 | 合一信息技术（北京）有限公司 | Method for constructing dynamic script execution engine based on MapReduce |
Non-Patent Citations (2)
| Title |
|---|
| WEIKUAN YU等: "Virtual shuffling for efficient data movement in mapduce", 《IEEE TRANSACTIONS ON COMPUTERS》 * |
| XUANHUA SHI等: "Mammoth：Gearing Hadoop towards memory-intensive mapreduce applications", 《IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS》 * |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106168963B (en) * | 2016-06-30 | 2019-06-11 | 北京金山安全软件有限公司 | Real-time streaming data processing method and device and server |
| CN106168963A (en) * | 2016-06-30 | 2016-11-30 | 北京金山安全软件有限公司 | Real-time streaming data processing method and device and server |
| CN106371919A (en) * | 2016-08-24 | 2017-02-01 | 上海交通大学 | Shuffle data caching method based on mapping-reduction calculation model |
| CN106371919B (en) * | 2016-08-24 | 2019-07-16 | 上海交通大学 | It is a kind of based on mapping-reduction computation model data cache method of shuffling |
| CN107612886B (en) * | 2017-08-15 | 2020-06-30 | 中国科学院大学 | Spark platform Shuffle process compression algorithm decision method |
| CN107612886A (en) * | 2017-08-15 | 2018-01-19 | 中国科学院大学 | A kind of Spark platforms Shuffle process compresses algorithm decision-making techniques |
| CN107885587A (en) * | 2017-11-17 | 2018-04-06 | 清华大学 | A kind of executive plan generation method of big data analysis process |
| CN107885587B (en) * | 2017-11-17 | 2018-12-07 | 清华大学 | A kind of executive plan generation method of big data analysis process |
| CN110083441A (en) * | 2018-01-26 | 2019-08-02 | 中兴飞流信息科技有限公司 | A kind of distributed computing system and distributed computing method |
| CN110083441B (en) * | 2018-01-26 | 2021-06-04 | 中兴飞流信息科技有限公司 | Distributed computing system and distributed computing method |
| CN110750341A (en) * | 2018-07-24 | 2020-02-04 | 深圳市优必选科技有限公司 | Task scheduling method, device, system, terminal equipment and storage medium |
| CN109951556A (en) * | 2019-03-27 | 2019-06-28 | 联想(北京)有限公司 | A kind of Spark task processing method and system |
| CN110109747A (en) * | 2019-05-21 | 2019-08-09 | 北京百度网讯科技有限公司 | Method for interchanging data and system, server based on Apache Spark |
| CN110109747B (en) * | 2019-05-21 | 2021-05-14 | 北京百度网讯科技有限公司 | Apache Spark-based data exchange method, system and server |
| CN110134714B (en) * | 2019-05-22 | 2021-04-20 | 东北大学 | Distributed computing framework cache index method suitable for big data iterative computation |
| CN110134714A (en) * | 2019-05-22 | 2019-08-16 | 东北大学 | A kind of distributed computing framework caching index suitable for big data iterative calculation |
| CN111061565B (en) * | 2019-12-12 | 2023-08-25 | 湖南大学 | Two-section pipeline task scheduling method and system in Spark environment |
| CN111061565A (en) * | 2019-12-12 | 2020-04-24 | 湖南大学 | Two-stage pipeline task scheduling method and system in Spark environment |
| CN111258785A (en) * | 2020-01-20 | 2020-06-09 | 北京百度网讯科技有限公司 | Data shuffling method and device |
| CN113364603B (en) * | 2020-03-06 | 2023-05-02 | 华为技术有限公司 | Fault recovery method of ring network and physical node |
| CN113364603A (en) * | 2020-03-06 | 2021-09-07 | 华为技术有限公司 | Fault recovery method of ring network and physical node |
| CN113495679A (en) * | 2020-04-01 | 2021-10-12 | 孟彤 | Optimization method for large data storage access and processing based on nonvolatile storage medium |
| CN113495679B (en) * | 2020-04-01 | 2022-10-21 | 北京大学 | Optimization method for large data storage access and processing based on nonvolatile storage medium |
| CN111782367A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Distributed storage method and device, electronic equipment and computer readable medium |
| CN111782367B (en) * | 2020-06-30 | 2023-08-08 | 北京百度网讯科技有限公司 | Distributed storage method and device, electronic equipment and computer readable medium |
| CN112269648A (en) * | 2020-11-13 | 2021-01-26 | 北京轩宇信息技术有限公司 | Parallel task allocation method and device for multi-stage program analysis |
| CN112269648B (en) * | 2020-11-13 | 2024-05-31 | 北京轩宇信息技术有限公司 | Parallel task allocation method and device for multi-stage program analysis |
| CN114448660A (en) * | 2021-12-16 | 2022-05-06 | 国网江苏省电力有限公司电力科学研究院 | Internet of things data access method |
| CN114448660B (en) * | 2021-12-16 | 2024-06-04 | 国网江苏省电力有限公司电力科学研究院 | Internet of things data access method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105718244B (en) | 2018-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105718244A (en) | Streamline data shuffle Spark task scheduling and executing method | |
| US11694066B2 (en) | Machine learning runtime library for neural network acceleration | |
| Almeida et al. | ChainReaction: a causal+ consistent datastore based on chain replication | |
| CN111309649B (en) | Data transmission and task processing method, device and equipment | |
| CN109690510A (en) | Multicast device and method for multiple receivers by data distribution into high-performance calculation network and network based on cloud | |
| US20230153158A1 (en) | Method, apparatus, system, and storage medium for performing eda task | |
| US9418181B2 (en) | Simulated input/output devices | |
| CN111104224B (en) | FPGA-based dynamic graph processing method | |
| US20170085653A1 (en) | Method, device and system for message distribution | |
| CN110457033A (en) | Device and method for generating dynamic trace data on GPU | |
| CN105022288A (en) | Simulation system of industrial electronic embedded system | |
| US11275661B1 (en) | Test generation of a distributed system | |
| CN117312215B (en) | Server system, job execution method, device, equipment and medium | |
| EP2672388B1 (en) | Multi-processor parallel simulation method, system and scheduler | |
| CN111897826A (en) | Parameter information updating method and device, electronic equipment and readable storage medium | |
| US8688428B2 (en) | Performance evaluation device, performance evaluation method and simulation program | |
| CN116611375A (en) | Software and hardware collaborative simulation platform and software and hardware testing method | |
| US10162913B2 (en) | Simulation device and simulation method therefor | |
| US9892010B2 (en) | Persistent command parameter table for pre-silicon device testing | |
| CN115391181A (en) | Verification method of SOC chip | |
| CN111950219B (en) | Method, apparatus, device and medium for realizing simulator | |
| US11526490B1 (en) | Database log performance | |
| CN111258748B (en) | Distributed file system and control method | |
| CN114444423B (en) | Data processing method and system based on verification platform and electronic equipment | |
| CN116151308A (en) | Deep neural network checkpoint optimization system and method based on nonvolatile memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |