US8898422B2 - Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration - Google Patents

Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration Download PDF

Info

Publication number
US8898422B2
US8898422B2 US13/595,088 US201213595088A US8898422B2 US 8898422 B2 US8898422 B2 US 8898422B2 US 201213595088 A US201213595088 A US 201213595088A US 8898422 B2 US8898422 B2 US 8898422B2
Authority
US
United States
Prior art keywords
data processing
partition
partitions
mapping result
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US13/595,088
Other languages
English (en)
Other versions
US20130227244A1 (en
Inventor
Myung-june Jung
Ju-Pyung Lee
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Assigned to SAMSUNG ELECTRONICS CO., LTD. reassignment SAMSUNG ELECTRONICS CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: JUNG, MYUNG-JUNE, LEE, JU-PYUNG
Publication of US20130227244A1 publication Critical patent/US20130227244A1/en
Application granted granted Critical
Publication of US8898422B2 publication Critical patent/US8898422B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/22—Microcontrol or microprogram arrangements
    • G06F9/28—Enhancement of operational speed, e.g. by using several microcontrol devices operating in parallel
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/501—Performance criteria

Definitions

  • the following description relates to a workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration.
  • a distributed parallel processing programming model supports distributed parallel computation of a large amount of data stored in a cluster that is formed of a large number of nodes at a low cost.
  • the distributed parallel processing programming model includes two steps: a “Map step” based on a map function made by a user and a “Reduce step” based on a reduce function. These two steps are performed in turn.
  • Map step based on a map function made by a user
  • Reduce step based on a reduce function
  • a data processing apparatus including a memory buffer including partitions.
  • the data processing apparatus further includes a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme.
  • the data processing apparatus further includes a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result.
  • the data processing apparatus may further include a job controller configured to measure a processing performance of the reduce node, and establish the partition proportion scheme based on the processing performance.
  • the mapping result may include a key.
  • the partition unit may include a main processor configured to generate an intermediate code with a fixed size based on the key.
  • the partition unit may further include a proportion controller configured to determine the partitions based on the intermediate code and the partition proportion scheme, generate partition codes corresponding to the respective partitions.
  • the data processing apparatus may further include a data transfer controller configured to write the mapping result in the partitions based on the partition codes.
  • the data processing apparatus may further include a sorter configured to sort the content, and output the sorted content to the reduce node.
  • the data processing apparatus may further include a memory buffer management table configured to store memory buffer management information.
  • the memory buffer management information may include a start address of the memory buffer, or a number of partitions of the memory buffer, or a number of records remaining to be processed, or a memory section size, or information of a memory section header, or a partition lookup table, or any combination thereof.
  • the data processing apparatus may further include a mapping node configured to process input data in parallel to generate the mapping result.
  • the mapping result may include a key.
  • the mapping node may be further configured to discard a bit other than a least significant bit of the key to update the key if a length of the key is greater than the least significant bit, and add a padding bit to the least significant bit to update the key if the length of the key is less than the least significant bit.
  • the data processing apparatus may be implemented through hardware acceleration on a field programmable gate array (FPGA).
  • FPGA field programmable gate array
  • a data processing method including distributing a mapping result to partitions of a memory buffer based on a partition proportion scheme.
  • the data processing method further includes determining content of a corresponding one of the partitions.
  • the data processing method further includes performing a reduction operation on the content to generate a reduce result.
  • the data processing method may further include measuring a processing performance of a reduce node.
  • the data processing method may further include establishing the partition proportion scheme based on the processing performance.
  • the data processing method may further include extracting a job environment.
  • the data processing method may further include acquiring a list of reduce nodes from the job environment.
  • the data processing method may further include determining the reduce node to be measured for the processing performance based on the list of the reduce nodes.
  • the measuring of the processing performance may include extracting first information about a number of records being processed by the reduce node.
  • the measuring of the processing performance may further include extracting second information about the number of records being processed by the reduce node after a predetermined period of time.
  • the measuring of the processing performance may further include determining the processing performance based on the first information, the second information, and the predetermined period of time.
  • the establishing of the partition proportion scheme may include determining a proportion for each of the partitions based on the processing performance.
  • the mapping result may include a key.
  • the distributing of the mapping result may include generating an intermediate code with a fixed size based on the key.
  • the distributing of the mapping result may further include determining the partitions based on the intermediate code and the partition proportion scheme.
  • the distributing of the mapping result may further include generating partition codes corresponding to the respective partitions.
  • the distributing of the mapping result may further include writing the mapping result in the partitions based on the partition codes.
  • the determining of the content may include sorting the content.
  • the data processing method may further include processing input data in parallel to generate the mapping result.
  • the data processing method may be implemented through hardware acceleration on a field programmable gate array (FPGA).
  • FPGA field programmable gate array
  • FIG. 1 is a diagram illustrating an example of a distributed data processing apparatus.
  • FIG. 2 is a diagram illustrating an example of a partition unit of a distributed data processing apparatus.
  • FIG. 3 is a diagram illustrating an example of a memory buffer management table of a distributed data processing apparatus.
  • FIG. 4 is a flowchart illustrating an example of a distributed data processing method.
  • FIG. 5 is a flowchart illustrating an example of establishing a partition proportion plan.
  • FIG. 6 is a flowchart illustrating an example of distributing at least one mapping result to partitions.
  • FIG. 7 is a flowchart illustrating an example of sorting at least one mapping result.
  • FIG. 1 illustrates an example of a distributed data processing apparatus 1 .
  • the distributed data processing apparatus 1 may be implemented through, for example, hardware acceleration on a field programmable gate array (FPGA).
  • the distributed data processing apparatus 1 may be configured to include all worker nodes disposed within a FPGA chip, and may be configured to include some worker nodes disposed on a separate FPGA chip.
  • the distributed data processing apparatus 1 includes at least one mapping node 100 , at least one shuffle node 200 , and at least one reduce node 300 .
  • the distributed data processing apparatus 1 performs inter-node parallelization using these nodes, thereby allowing an effective distributed processing of large data.
  • the mapping node 100 receives input data, and processes the input data in parallel to generate and output at least one mapping result in the form of a key-value pair, to the shuffle node 200 .
  • the shuffle node 200 receives the mapping result output from the mapping node 100 , and distributes the mapping result to the reduce node 300 .
  • the reduce node 300 receives the mapping result distributed from the shuffle node 200 , and performs a reduction operation on the mapping result to generate and output at least one final result (e.g., reduce result). For example, in the reduction operation, the reduce node 300 may remove duplicate data from the mapping result.
  • the distributed data processing apparatus 1 further includes a job controller node 400 , e.g., a job controller.
  • the job controller node 400 includes a job buffer configured to manage working environment information of each of the nodes 100 , 200 , and 300 .
  • the job controller node 400 monitors all jobs (e.g., the mapping results) to be processed, and measures and manages a status of resources (e.g., workloads, computing capabilities, and/or other resources known to one of ordinary skill in the art) of the reduce node 300 .
  • the job controller node 400 establishes a partition proportion plan 410 (e.g., a partition proportion scheme) based on the shuffle node 200 distributing jobs to the reduce node 300 . Operations of the measurement of the performance of the reduce node 300 and the establishment of the partition proportion plan 410 will be described later in detail with reference to FIG. 5 .
  • the shuffle node 200 includes a partition unit 210 , a memory buffer management unit 220 , a memory buffer 230 , and a sort unit 240 , e.g., a sorter.
  • the partition unit 210 receives the mapping result output from the mapping node 100 , for example, at least one respective key of the mapping result structured in a key-value pair.
  • the partition unit 210 allocates at least one corresponding partition (e.g., section) of the memory buffer 230 based on the key.
  • FIG. 2 illustrates an example of the partition unit 210 of the distributed data processing apparatus 1 of FIG. 1 .
  • the key of the mapping result is variable, and thus, for effective hardware-based processing, a least significant bit M in -bit of the key is input into the partition unit 210 .
  • the partition unit 210 includes a main processing unit 211 (e.g., a main processor) and a proportion control unit 212 (e.g., a proportion controller).
  • the main processing unit 211 receives the least significant bit M in -bit output from the mapping node 100 , and performs an operation on the least significant bit M in -bit to generate and output an intermediate code with a fixed size.
  • the main processing unit 211 may perform a cryptographic hash function on an input of an arbitrary length, or perform a hash function on an input of a fixed length, to generate and output a hash code of a fixed size.
  • the cryptographic hash function and the hash function are only examples, and other functions of generating an output of a fixed length based on an input of any length may be used, as known to one of ordinary skill in the art.
  • the proportion control unit 212 receives the intermediate code output from the main processing unit 211 and the partition proportion plan 410 established by the job controller node 400 .
  • the proportion control unit 212 generates and outputs at least one partition code M out -bit corresponding to the partition(s) to which the mapping result is distributed or allocated based on the intermediate code output and the partition proportion plan 410 .
  • the job controller node 400 of FIG. 1 may measure a resource status of each of three reduce nodes (e.g., the reduce node 300 ), and may establish the partition proportion plan 410 based on measurement results.
  • reduce node 3 (corresponding to partition 3 ) including the most resources is allocated 50% of a job
  • the partitions to which the mapping result is distributed may be recognized from partition codes output from the proportion control unit 212 , and accordingly, 30% of the mapping result is distributed to partition 1 , 20% is distributed to partition 2 , and 50% of the mapping result is distributed to partition 3 .
  • the memory buffer management unit 220 includes a memory buffer management table 221 and a data transfer control unit 222 , e.g., a data transfer controller.
  • the memory buffer management table 221 stores information required by the partition unit 210 and the sort unit 240 to use the memory buffer 230 .
  • the data transfer control unit 222 receives the mapping result output from the mapping node 100 and the partition code output from the partition unit 210 , and outputs (e.g., writes) the mapping result to the memory buffer 230 based on the partition code.
  • the data transfer control unit 222 further outputs data (e.g., the mapping result) of the memory buffer 230 to the sort unit 240 .
  • the shuffle node 200 performs data transmission through the data transfer control unit 222 without a central processing unit's (CPU's) directive and iterative load/store-based I/O intensive operation.
  • FIG. 3 illustrates an example of the memory buffer management table 221 of the distributed data processing apparatus 1 of FIG. 1 .
  • the memory buffer management table 221 stores a variety of memory buffer management information required for use of the memory buffer 230 .
  • the memory buffer management information includes a start address of the memory buffer 230 , a total number of partitions (i.e., sections) of the memory buffer 230 , a number of records remaining to be processed, a mapping result flag, a maximum number of records, a section size, a partition lookup table, and section header information.
  • this information are only examples, and other information known to one of ordinary skill in the art may be used.
  • the number of remaining records to be processed is information about a number of records waiting to be processed by the shuffle node 200 among all records of the mapping result.
  • the mapping result flag is information indicating that there is no remaining mapping result to be processed by the shuffle node 200 .
  • the maximum number of records is a maximum number of records that can be written in each section of the memory buffer 230 .
  • the section size is information of a size of each section of the memory buffer 230 in bytes.
  • the partition lookup table includes at least one partition code and at least one respective section header link, i.e., a list of pairs of a partition code and a section header link. Based on the partition lookup table, a header of a corresponding section of the memory buffer 230 can be directly-accessed via the section header link.
  • the section header manages information of the memory buffer section.
  • the information of the memory buffer section includes a start address of the section that is used as a base address of the section, a sort flag indicating whether the section is sorted, a next write address offset by predetermined bytes from the start address, and information about a number of records written in the section so far.
  • the memory buffer 230 includes one or more partitions (e.g., sections), each corresponding to the respective reduce node 300 .
  • Each partition may include a different distribution proportion in proportion to a processing performance of the corresponding reduce node 300 .
  • the reduce node 300 reads content (e.g., the mapping result) of a corresponding partition, removes a duplicate key, and generates and outputs a final result based on the content.
  • content e.g., the mapping result
  • reduce nodes process jobs at different distribution proportions, a processing delay due to the occurrence of a bottleneck event in the reduce nodes can be prevented.
  • the sort unit 240 sorts unsorted content (e.g., record) of each partition of the memory buffer 230 , and outputs the sorted content to the reduce node 300 .
  • the sort unit 240 may check the sort flag of the memory buffer management table 221 , sort unsorted content of a corresponding partition if the sort flag is set, and output the sorted content to the reduce node 300 .
  • FIG. 4 illustrates an example of a distributed data processing method.
  • the distributed data processing method will be described with reference to FIG. 4 in conjunction with FIG. 1 .
  • the distributed data processing method may be implemented through hardware acceleration, and the hardware acceleration may be implemented by, for example, implementing all functions of each node on a field programmable gate array (FPGA) chip.
  • FPGA field programmable gate array
  • the job controller node 400 measures a resource status (e.g., a processing performance) of each reduce node 300 , and establishes a partition proportion plan based on measurement results.
  • the mapping node 100 processes input data in parallel to generate at least one mapping result, and outputs the mapping result.
  • the mapping result is in the form of a key-value pair, and as shown in FIG. 2 , a least significant bit (M in -bit) of a key of the mapping result is input into the partition unit 210 of the shuffle node 200 .
  • the partition unit 210 distributes the mapping result output from the mapping node 100 to the partitions of the memory buffer 230 based on the partition proportion plan 410 , and the sort unit 240 sorts content (e.g., the mapping result) of the partitions written in the memory buffer 230 , and outputs the sorted content.
  • the reduce node 300 receives the content output from the sort unit 240 , and generates and outputs at least one final result by performing a reduction operation on the received content, such as removal of a duplicate key from the received content.
  • FIG. 5 illustrates an example of establishing a partition proportion plan. Operation 510 of FIG. 4 in which the job controller node 400 establishes the partition proportion plan will be described in detail with reference to FIG. 5 .
  • the job controller node 400 extracts a job environment from a job buffer, and acquires a list of reduce nodes (e.g., the reduce node 300 ) based on the job environment.
  • the job controller node 400 determines which reduce node is to be measured for its current performance.
  • the job controller node 400 extracts information ‘rec_rd_ 1 ’ about the number of records being processed by the determined reduce node, from, e.g., the determined reduce node.
  • the job controller node 400 extracts information ‘rec_rd_ 2 ’ about the number of records being processed by the determined reduce node, from, e.g., the determined reduce node.
  • FIG. 6 illustrates an example of distributing at least one mapping result to partitions.
  • operation 530 shown in FIG. 5 in which the shuffle node 200 distributes the mapping result to the partitions will be described in detail.
  • the partition unit 210 of the shuffle node 200 checks whether there is a mapping result record to be processed, i.e., present. For example, the partition unit 210 may check the memory buffer management table 221 to identify a number of records remaining to be processed, and if the number is 0, the partition unit 210 may determine that there is no record to be processed. If there is no record to be processed, in operation 539 , the partition unit 210 and/or the memory buffer management unit 220 set a mapping result flag in the memory buffer management table 221 and a sort flag in every section header of the memory buffer management table 221 .
  • the sort unit 240 checks the sort flag of a section being processed, and if the sort flag is set, the sort unit 240 sorts content (e.g., the mapping result) of the section in the memory buffer 230 , as described later in FIG. 7 .
  • content e.g., the mapping result
  • mapping node 100 and/or the partition unit 210 optimizes a key of the remaining record to a length of a least significant bit (M in -bit).
  • main processing unit 211 of the partition unit 210 generates an intermediate code with a fixed size using the the least significant bit M in -bit.
  • the proportion control unit 212 of the partition unit 210 determines partitions of the memory buffer 230 to which the mapping result is distributed using the intermediate code and the partition proportion plan 410 , and generates at least one partition code M out -bit corresponding to each respective partition or section.
  • the data transfer control unit 222 of the memory buffer management unit 220 writes the mapping result (structured in a key-value pair) in a corresponding section of the partition code.
  • the memory buffer management unit 220 updates a number of records written in a section header of the current section in the memory buffer management table 221 , and checks whether the number of records written reaches the maximum number of records in the memory buffer management table 221 . If the number of records written reaches the maximum number of records, in operation 537 , the memory buffer management unit 220 sets a sort flag in the section header of the current section. Otherwise, in operation 538 , the memory buffer management unit 220 updates a next write address field in the section header.
  • the shuffle node 200 iteratively performs the above operations (after operation 531 ) until there is no remaining record to be processed, and thereafter, the sort unit 240 proceeds with the sorting operation.
  • FIG. 7 illustrates an example of sorting at least one mapping result.
  • operation 530 shown in FIG. 5 in which the shuffle node 200 sorts the mapping result will be described in detail.
  • the sort unit 240 checks whether a sort flag in a corresponding section header of the memory buffer management table 221 is set in 631 . If the sort flag is not set, in operation 637 , the sort unit 240 extracts information of a next section header from the the memory buffer management table 221 .
  • the sort unit 240 determines whether a number of the current records is less than a maximum number of records, that is, whether there is an empty slot in the sort unit 240 . If there is an empty slot, in operation 634 , the sort unit 240 inputs an arbitrary number, for example, the largest number, to the empty slot. This is to increase a hardware processing efficiency of the sort unit 240 and to locate the largest number at the very last position of the sorting result, thereby allowing the number to be easily deleted.
  • the sort unit 240 is operated to sort the current records.
  • the sort unit 240 determines whether all records in the current section are completely-sorted. If the sorting is not complete, operations after 632 are iteratively-performed to sort the remaining records. If the sorting of the records is complete, in operation 637 , the sort unit 240 extracts information of the next section header from the memory buffer management table 221 .
  • the sort unit 240 determines whether the current section is the last section. If the current section is not the last section, operations after 631 are iteratively-performed. Otherwise, the sort unit 240 ends the sorting operation.
  • the units described herein may be implemented using hardware components and software components.
  • the hardware components may include microphones, amplifiers, band-pass filters, audio to digital converters, and processing devices.
  • a processing device may be implemented using one or more general-purpose or special purpose computers, such as, for example, a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner.
  • the processing device may run an operating system (OS) and one or more software applications that run on the OS.
  • the processing device also may access, store, manipulate, process, and create data in response to execution of the software.
  • OS operating system
  • a processing device may include multiple processing elements and multiple types of processing elements.
  • a processing device may include multiple processors or a processor and a controller.
  • different processing configurations are possible, such a parallel processors.
  • the software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct or configure the processing device to operate as desired.
  • Software and data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, computer storage medium or device, or in a propagated signal wave capable of providing instructions or data to or being interpreted by the processing device.
  • the software also may be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion.
  • the software and data may be stored by one or more computer readable recording mediums.
  • the computer readable recording medium may include any data storage device that can store data which can be thereafter read by a computer system or processing device.
  • non-transitory computer readable recording medium examples include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices.
  • ROM read-only memory
  • RAM random-access memory
  • CD-ROMs compact disc-read only memory
  • magnetic tapes magnetic tapes
  • floppy disks optical data storage devices.
  • functional programs, codes, and code segments accomplishing the examples disclosed herein can be easily construed by programmers skilled in the art to which the examples pertain based on and using the flow diagrams and block diagrams of the figures and their corresponding descriptions as provided herein.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
US13/595,088 2012-02-27 2012-08-27 Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration Active 2032-12-29 US8898422B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020120019709A KR101867286B1 (ko) 2012-02-27 2012-02-27 작업 부하를 고려한 하드웨어 가속화 기반의 대규모 데이터의 분산 처리 장치 및 방법
KR10-2012-0019709 2012-02-27

Publications (2)

Publication Number Publication Date
US20130227244A1 US20130227244A1 (en) 2013-08-29
US8898422B2 true US8898422B2 (en) 2014-11-25

Family

ID=49004580

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/595,088 Active 2032-12-29 US8898422B2 (en) 2012-02-27 2012-08-27 Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration

Country Status (2)

Country Link
US (1) US8898422B2 (ko)
KR (1) KR101867286B1 (ko)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9900378B2 (en) * 2016-02-01 2018-02-20 Sas Institute Inc. Node device function and cache aware task assignment
US10678690B2 (en) 2017-08-29 2020-06-09 Qualcomm Incorporated Providing fine-grained quality of service (QoS) control using interpolation for partitioned resources in processor-based systems

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9116860B2 (en) 2012-12-14 2015-08-25 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Cascading failover of blade servers in a data center
US9122652B2 (en) * 2012-12-17 2015-09-01 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Cascading failover of blade servers in a data center
CN103929608B (zh) * 2014-04-16 2017-06-16 浙江宇视科技有限公司 一种动态分配缓存容量的方法以及装置
CN104021161B (zh) 2014-05-27 2018-06-15 华为技术有限公司 一种聚簇存储方法及装置
US20170017394A1 (en) * 2015-07-15 2017-01-19 Futurewei Technologies, Inc. SYSTEM AND METHOD FOR DATA WAREHOUSE AND FINE GRANULARITY SCHEDULING FOR SYSTEM ON CHIP (SoC)
KR101641179B1 (ko) 2015-12-31 2016-07-20 아이씨티웨이주식회사 대용량 공간데이터 분산 처리 방법 및 이를 위한 분산 처리 서버
KR102424962B1 (ko) * 2017-11-15 2022-07-25 삼성전자주식회사 병렬 연산 처리를 수행하는 메모리 장치 및 이를 포함하는 메모리 모듈
CN108763299A (zh) * 2018-04-19 2018-11-06 贵州师范大学 一种大规模数据处理计算加速系统
US11061609B2 (en) * 2018-08-02 2021-07-13 MemVerge, Inc Distributed memory object method and system enabling memory-speed data access in a distributed environment
KR102251863B1 (ko) * 2019-11-01 2021-05-14 경희대학교 산학협력단 차량 클라우드 컴퓨팅을 위한 작업 할당 방법 및 이를 수행하는 서버 및 차량 데이터 처리 시스템

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100907533B1 (ko) 2007-12-17 2009-07-14 한국전자통신연구원 작업 분산 병렬 처리 시스템 및 방법
JP2010097489A (ja) 2008-10-17 2010-04-30 Nec Corp 分散データ処理システム、分散データ処理方法および分散データ処理用プログラム
US7853639B2 (en) * 2006-09-12 2010-12-14 International Business Machines Corporation Performing process migration with allreduce operations
KR101013073B1 (ko) 2007-12-17 2011-02-14 한국전자통신연구원 태스크 분배 및 병렬 처리 시스템과 그 방법
US20110154350A1 (en) 2009-12-18 2011-06-23 International Business Machines Corporation Automated cloud workload management in a map-reduce environment
US8225259B1 (en) * 2004-09-15 2012-07-17 Altera Corporation Apparatus and methods for time-multiplex field-programmable gate arrays with multiple clocks
US8321873B2 (en) * 2009-06-09 2012-11-27 Yahoo! Inc. System and method for offline data generation for online system analysis
US8392482B1 (en) * 2008-03-31 2013-03-05 Amazon Technologies, Inc. Versioning of database partition maps

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3559021B2 (ja) * 2001-10-31 2004-08-25 株式会社リコー データ変換装置，画像処理装置およびデータ変換方法
KR100717216B1 (ko) * 2005-09-05 2007-05-11 주식회사 태울엔터테인먼트 클러스터 시스템을 제어하는 방법

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8225259B1 (en) * 2004-09-15 2012-07-17 Altera Corporation Apparatus and methods for time-multiplex field-programmable gate arrays with multiple clocks
US7853639B2 (en) * 2006-09-12 2010-12-14 International Business Machines Corporation Performing process migration with allreduce operations
KR100907533B1 (ko) 2007-12-17 2009-07-14 한국전자통신연구원 작업 분산 병렬 처리 시스템 및 방법
KR101013073B1 (ko) 2007-12-17 2011-02-14 한국전자통신연구원 태스크 분배 및 병렬 처리 시스템과 그 방법
US8392482B1 (en) * 2008-03-31 2013-03-05 Amazon Technologies, Inc. Versioning of database partition maps
JP2010097489A (ja) 2008-10-17 2010-04-30 Nec Corp 分散データ処理システム、分散データ処理方法および分散データ処理用プログラム
US8321873B2 (en) * 2009-06-09 2012-11-27 Yahoo! Inc. System and method for offline data generation for online system analysis
US20110154350A1 (en) 2009-12-18 2011-06-23 International Business Machines Corporation Automated cloud workload management in a map-reduce environment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9900378B2 (en) * 2016-02-01 2018-02-20 Sas Institute Inc. Node device function and cache aware task assignment
US10678690B2 (en) 2017-08-29 2020-06-09 Qualcomm Incorporated Providing fine-grained quality of service (QoS) control using interpolation for partitioned resources in processor-based systems

Also Published As

Publication number Publication date
US20130227244A1 (en) 2013-08-29
KR20130097973A (ko) 2013-09-04
KR101867286B1 (ko) 2018-06-15

Similar Documents

Publication Publication Date Title
US8898422B2 (en) Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration
CN109218355B (zh) 负载均衡引擎，客户端，分布式计算系统以及负载均衡方法
US9870270B2 (en) Realizing graph processing based on the mapreduce architecture
US9405589B2 (en) System and method of optimization of in-memory data grid placement
KR101827369B1 (ko) 데이터 스트림 분산 병렬 처리 서비스 관리 장치 및 방법
US20150032759A1 (en) System and method for analyzing result of clustering massive data
CN110308984B (zh) 一种用于处理地理分布式数据的跨集群计算系统
Xie et al. Pandas: robust locality-aware scheduling with stochastic delay optimality
US20150067695A1 (en) Information processing system and graph processing method
US9535743B2 (en) Data processing control method, computer-readable recording medium, and data processing control device for performing a Mapreduce process
da Silva et al. Scalability limits of Bag-of-Tasks applications running on hierarchical platforms
US20170048352A1 (en) Computer-readable recording medium, distributed processing method, and distributed processing device
JP2014120097A (ja) 情報処理装置、プログラム、及び、情報処理方法
Cong et al. CPU-FPGA coscheduling for big data applications
US9342564B2 (en) Distributed processing apparatus and method for processing large data through hardware acceleration
Mohamed et al. Accelerating data-intensive genome analysis in the cloud
US10896056B2 (en) Cluster expansion method and apparatus, electronic device and storage medium
US8799619B2 (en) Method and system for providing distributed programming environment using distributed spaces, and computer readable recording medium
CN109947531B (zh) 超融合一体机的存储容量扩展方法、装置及存储介质
Ouyang et al. Mitigate data skew caused stragglers through ImKP partition in MapReduce
Yassir et al. Graph-based model and algorithm for minimising big data movement in a cloud environment
CN110609744A (zh) 处理计算任务的方法、设备和计算机程序产品
CN113934767A (zh) 一种数据处理的方法及装置、计算机设备和存储介质
TW202315360A (zh) 微服務分配方法、電子設備及儲存介質
WO2018222064A1 (en) Systems and methods of hierarchical community detection in graphs

Legal Events

Date Code Title Description
AS Assignment

Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JUNG, MYUNG-JUNE;LEE, JU-PYUNG;REEL/FRAME:028851/0131

Effective date: 20120807

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551)

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8