New! View global litigation for patent families

WO2015096649A1 - Data processing method and related device - Google Patents

Data processing method and related device

Info

Publication number
WO2015096649A1
WO2015096649A1 PCT/CN2014/094071 CN2014094071W WO2015096649A1 WO 2015096649 A1 WO2015096649 A1 WO 2015096649A1 CN 2014094071 W CN2014094071 W CN 2014094071W WO 2015096649 A1 WO2015096649 A1 WO 2015096649A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
data
buffer
block
gpu
set
Prior art date
Application number
PCT/CN2014/094071
Other languages
French (fr)
Chinese (zh)
Inventor
崔慧敏
谢睿
阮功
杨文森
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for programme control, e.g. control unit
    • G06F9/06Arrangements for programme control, e.g. control unit using stored programme, i.e. using internal store of processing equipment to receive and retain programme
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogramme communication ; Intertask communication
    • G06F9/541Interprogramme communication ; Intertask communication via adapters, e.g. between incompatible applications

Abstract

A data processing method and a related device, implementing automatic conversion of data format and automatic splicing of data in a node device by a Hadoop. The method mainly comprises: a data preprocessor reads metadata from a first buffer area of a CPU, reads data of a data collection from the first buffer area on the basis of a memory address indicated by the metadata, converts, on the basis of a preset analytic function, the data of the data collection into a data format indicated by the preset analytic function, and stores data blocks generated with the converted data collection in a second buffer area of the CPU, thus allowing a data splicer to read from the second buffer area the data blocks and to splice same to a GPU.

Description

A data processing method and related apparatus FIELD

The present invention relates to the field of information processing technology, particularly relates to a data processing method and related apparatus.

Background technique

Big data and cloud computing together, for the information technology (IT, Information Technology) has brought a new revolution, cloud computing with powerful computing capabilities of big data, compute very fast, but large data transfer has become one big problem.

MapReduce (temporary there is no uniform Chinese translation of the art) is a well-known cloud computing architecture Google search engine provided by Google for parallel computing on large data sets (greater than 1TB), Hadoop (temporary art is no uniform Chinese translation) is a specific implementation of MapReduce architecture, a master node and slave node in a Hadoop cluster carve equipment. Wherein MapReduce using Map function provided in the master node device according to the data set size is divided into M pieces of data pieces, the data piece allocated to a plurality of parallel processing done from the node device. Specifically, each node apparatus from acquiring data pieces from the key-value pairs, the processor node from the device (Central Processing Unit, abbreviated CPU) allocated buffer, after splicing value stored from the buffer reading the key-value pair parsed, for example, convert the data format of the key-value pairs, then the value obtained by analyzing an application programming interface (API, application programming Interface) to a stitching device from the graphics processor node ( the GPU, buffer Graphics processing Unit) allocated for storing data, the calculation processing performed by the GPU.

Skill in implementing the present invention found that the above-described embodiment, since the MapReduce framework does not provide analytic function, when the key-value pair parsed, programmers need to rely on the respective programs written; the same time, since the CPU stores key distribution buffer value of the buffer size allocated to the GPU store data may be inconsistent, and the MapReduce framework does not provide a method for determining, if the function corresponding to rely on the same programmers write buffer of the CPU and GPU It is the same judgment, reducing the efficiency of the device from the node.

SUMMARY

For the above disadvantages, the present invention provides a data processing method and related apparatus, applied in MapReduce Hadoop cluster architecture can improve the working efficiency Hadoop cluster node apparatus, simplify the programming of the programmer, in favor subsequent optimization MapReduce architecture.

A first aspect, the present invention provides a data processing method, applied in MapReduce framework Hadoop clusters, said cluster comprising Hadoop master node device and a slave device, the processor CPU and a graphics processor GPU from the node device comprising, from the node device acquires the data piece from the master node device, the CPU is provided with a data processor and a data pre-splicing, said method comprising:

The data preprocessor reads metadata from a first buffer of said CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the first buffer header adding metadata to said set of data, the metadata comprising data of the data set in said first buffer memory address;

The preprocessor reads the data stored in the address indicated by the metadata from the data buffer of the first set;

The data processor according to a preset pre-analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and the transformed data set to generate the data block is stored in the CPU's the second buffer so that the data registration reads the data block from said second buffer to said splicing GPU.

With the first aspect, in a first possible implementation manner, the metadata index comprises an array of addresses, said address data includes an index array with the set of data one to one data element, the data element for instructing said data set in said first memory address buffer, and then the processor reads data from the first pre-buffer according to the memory address indicated by metadata in the data set data comprising: the data read from the data pre-processor element array index indicating the address of the first buffer memory address until the address of the next instruction or a data element of the first tip end of the read buffer take data.

With the first aspect, in a second possible implementation, the converted data in the data set into a predetermined data format indicated by analytic functions comprising: a preprocessor according to the preset data parsing function data of the data set into the analysis function satisfies logical operation specified data format.

Binding a second possible implementation of the first aspect of the embodiment, in a third possible implementation manner, if the storage format of the first buffer of the GPU and the data set stored in the data format of the data after inconsistent, the converted data is set to generate data blocks comprising: a data pre-processor converts the data in the data blocks to store the format of the GPU.

With the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, or a third possible implementation of the first aspect, in a fourth possible implementations, the data values ​​of specific sub-set of splicing a plurality of key-value pairs of sheets composed of the data.

With the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, or a third possible implementation of the first aspect, in a fifth possible implementations, the first buffer and said second buffer is automatically assigned by the CPU and recovery, the first buffer is a life cycle processing time slice data, said second buffer the life cycle is the processing time of one data set.

A second aspect of the present invention provides a data processing method, applied in MapReduce framework Hadoop clusters, said cluster comprising Hadoop master node device and a slave device, the processor CPU and a graphics processor GPU from the node device comprising the said apparatus acquired from the node from the master node device data slice, the CPU is provided with a pre-processor data and data splice, said method comprising:

The data read from the second buffer splice the CPU of the block data generated by the preprocessor;

The data splicer to splice the data block is allocated to the GPU work memory block buffers.

Binding a second aspect, in a first possible implementation manner, when the data splicer to splice the data block when the GPU memory block is allocated buffers fails to work, the data is suspended splicing block, and triggers the GPU processing data blocks stored in the working buffer.

Combination with the second aspect, or the first possible implementation of the second aspect, in a second possible implementation, the data splicer splicing starts from the starting address of the block of data indicated by the cursor parameters, the parameter for indicating the cursor is allocated GPU work buffer stores a data block can be used to store the starting address of the data block.

Binding a second possible implementation of the second aspect, in a third possible implementation, the splicing when the data block is successful, the method further comprising: notifying the splice of the data of the data GPU block size; the data update the cursor splice parameters.

A third aspect of the present invention provides a data pre-processor, comprising:

A first reading unit configured to read the metadata from a first buffer of said CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the first buffer adding header for the data set of metadata, the metadata includes the address of the data set stored in said first buffer;

A second reading unit for reading from the first memory buffer according to the address indicated by the metadata of the data set;

A conversion unit according to a preset analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and the converted data to generate a set of data blocks;

A storage unit for the data block is stored in the second buffer of the CPU, so that the data registration reads the data block from said second buffer to said splicing GPU.

Combination with the third aspect, in a first possible implementation, the metadata index comprises an array of addresses, said address data includes an index array with the set of data one to one data element, the data element data for indicating a set of data stored in the address of the first buffer, the second reading unit further comprising: a data reading unit, for said address from the data element of index indication array in a first starts reading the buffer memory address, the address is stored until the next data element indicates the end of the buffer or the first end of the read data.

Binding a third aspect, or the first possible implementation of the third aspect, in a second possible implementation, the parsing unit comprises: a data format conversion unit for parsing function preset by the data set into the analysis function satisfies logical operation specified data format generating means for converting data to generate a set of data blocks.

Combination with the third aspect, in a third possible implementation, the analysis unit further comprises: a format conversion unit for storing said first format when the data buffer is set in the data with the GPU of inconsistent data storage format, converting the data into the data block is stored in the format of the GPU.

A fourth aspect of the present invention provides a data splicer, comprising:

Third read means for reading from the second buffer of the CPU data pre-processor generates the data block;

Splicing means for splicing the data block is allocated to the GPU work memory block buffers.

Binding a fourth aspect, in a first possible implementation, the data splicer further comprising: a trigger processing unit, when the data for the data block splicer splice is allocated to the data stored in the GPU work buffers block fails, the data block is suspended splicing, and triggers the GPU processing data blocks stored in the working buffer.

Binding fourth aspect, or the first possible implementation of the fourth aspect, in a second possible implementation manner, the splicing unit is configured from the start address indicated by the cursor splicing said parameter data block , the cursor for indicating the parameter is allocated GPU work memory block can be used to buffer the start address of the memory block.

Binding a second possible implementation of the fourth aspect, in a third possible implementation, the data splicer further comprising: a notification unit for notifying the GPU size of the data block; update unit, for updating the cursor parameters.

A fifth aspect of the present invention provides a processor may comprise a data pre-processor according to the third aspect and the fourth data splicer aspect.

Binding fifth aspect, in a first possible implementation, the automatic dispensing and recovering said first buffer and said second buffer, said first buffer life cycle is the processing time of a data slice , the processing time of the second buffer for the lifetime of a data set.

A sixth aspect of the present invention provides a method according to the fifth aspect processor CPU, a graphics processor GPU and the slave device, may comprise; wherein said CPU the data pre-processor for data pieces acquired from data set converts the data format, the data format and converts data set to generate a data block, said CPU through the data splicer to splice the data block in the work buffer of the GPU's allocated memory block; the GPU said data blocks for the processing result obtained by processing, then returns the processing result to the CPU.

As can be seen from the above technical solutions, the embodiments of the present invention has the following advantages:

In one aspect, the embodiments of the preprocessor and the setting data from data splicer node device, from the first buffer to read the metadata from the data pre-processor of the CPU, since the metadata in the data set are stored into the present invention when a buffer for generating a set of data, data representing the data set in the memory address of the first buffer, then the data pre-processor can read the data from the first set of metadata buffer, then in accordance with analytic function preset data format conversion is set, then the converted data to generate a set of data blocks, the data blocks stored in the second buffer of the CPU to complete the data block by the data splicing and splice GPU . Compared with the prior art, by storing the data set into a first buffer, the data added to the data set in the embodiment includes a metadata storage address, data can be automatically read from the pre-processor embodiment of the present invention, the first buffer data fetch data set does not depend on the respective programmer write. Further, the data processor may be according to a preset pre parsing function parses the data collection, processing efficiency of CPU, but also facilitates the subsequent optimization MapReduce framework;

On the other hand, the data read out data blocks from the second buffer splice splicing is allocated to the GPU work memory block buffer, the stitching fails, the GPU is allocated the remaining data blocks stored in the working buffer not enough memory to complete the splicing block, the data block of the splicing is temporarily stopped, to turn the trigger data block is GPU operation. And the data block will be temporarily stored in the second buffer, then the next splice. Compared with the prior art, need not rely upon a program written by the programmer, the data can be done by splicing automatically splicing block, prevent loss of the data block, the data block to improve the efficiency of splicing.

BRIEF DESCRIPTION

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following embodiments of the invention will be required to be used in the drawings are briefly introduced hereinafter, described in the following drawings are only some embodiments of the invention, for ordinary skill in the art, without creative efforts, can obtain other drawings based on these drawings.

1 data processing method provided in the embodiment of the present invention, a schematic diagram;

Figure 2 provides a data processing method of the embodiment of flow diagram of another embodiment of the present invention;

3 embodiment provides a data processing method flow diagram of an embodiment of the present invention;

The data processing method provided in the present invention. FIG. 4 a schematic view of another embodiment;

FIG 5-a schematic structural diagram of a data pre-processor according to an embodiment of the present invention;

Another embodiment of a data structure provided by the pre-processor of FIG. 5-b a schematic view of the present invention;

FIG. 5-c a schematic structural diagram of a data pre-processor according to another embodiment of the present invention;

A schematic structural diagram of a data pre-processor provided in FIG. 5-d a further embodiment of the present invention;

FIGS. 6-a schematic structural diagram of data of a splicer embodiment of the present invention;

FIG. 6-b diagram of another embodiment of a data structure of the splicer of the present invention is provided;

FIGS. 6-c a schematic view of another embodiment of a data structure of the splicer of the present invention is provided;

FIG 7 is a schematic structural diagram of an embodiment of the present invention, the processor;

FIG. 8-a schematic configuration from the node device according to an embodiment of the present invention;

FIG. 8-b schematic interaction between the CPU and GPU provided from a node device of the present embodiment of the invention;

FIG 9 is a schematic structure of a data processing apparatus according to an embodiment of the present invention.

detailed description

Below in conjunction with the accompanying drawings embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described, obviously, the described embodiments are merely part of embodiments of the present invention rather than all embodiments. Based on the embodiments of the present invention, those of ordinary skill in the art to make all other embodiments without creative work obtained by, it falls within the scope of the present invention.

Embodiment provides a data processing method and related apparatus, applied in MapReduce Hadoop cluster architecture to achieve Hadoop automatically switch from the node data and the device data format automatic splicing, simplify the programming of the programmer, it facilitates the subsequent optimization of the present invention MapReduce architecture.

As shown, an aspect of the present invention to provide a data processing method, comprising:

S110, data preprocessor reads metadata from a first buffer of the CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the head portion of the first buffer adding said data set metadata, the metadata comprising data of the data set in said first buffer memory address;

Example Hadoop cluster is applied in the embodiment of the present invention MapReduce framework, the Hadoop cluster includes a master node device and a slave node device comprises a processor CPU and the GPU graphics processor, the sub-slave device from the master node node device acquires data from device sheet, provided with a pre-processor and the data in the CPU data splicer.

Allocating a first buffer in the CPU, for storing a set of data acquired from the data slice, and when the data set is stored into the first buffer, then the set of metadata is added in a first data buffer header, meta the main data comprising address data stored in the first buffer of the data set.

S120, the processor reads data pre-stored according to the address indicated by the metadata from the data buffer of the first set;

Since the meta data comprises a first set of data in the buffer memory address, the data pre-processor can be directly read from the metadata according to an instruction buffer in a first set of data, without having to rely on additional programmers write a program to read the data.

S130, the data processor according to a preset pre-analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and generating a set of converted data stored in said data block the second buffer of the CPU, so that the data registration reads the data block from said second buffer to splice GPU.

Further, in MapReduce framework preset in analytic function, data pre-processor may parse the data set according to a preset first buffer analytic function, converted into the corresponding analytic function of the preset data format, and then converts after the data set to generate a data block. Meanwhile, the CPU is also assigned a second buffer for storing data blocks. Stitcher data blocks may be spliced ​​into the GPU reads data from the second buffer.

Embodiments of the present invention, due to the CPU 110 in the data set stored in a first buffer, is added to the data set metadata, the metadata includes a set of data in a first memory address buffer, and therefore, data preprocessing after the start of the first read buffer metadata, the metadata according to the data storage address indicated by a first set of data read from the buffer, and then use analytic function converts data preset data format, are converted to the format after the data set to generate a data block stored in the second CPU in the buffer, so as to realize a first data buffer read automatically by the data pre-processor, and data parsing operation, program no longer dependent on additional the programmers, for programmers to provide more comprehensive MapReduce architecture, but also conducive to follow-up on the MapReduce architecture optimization.

It will be appreciated that, in MapReduce framework, specify a mapping Map function for mapping the input key into a new key-value pair; re-specified concurrent simplification Reduce function, used to ensure that all mappings of key each key share the same group. In the Map function key mapped to an input of the new key pair, the Hadoop cluster master node device, according to the data size of all the new key-value pairs into different data fragments, according to the data arrangement fragmentation corresponding to each node from the arithmetic processing device.

From the CPU to the node device, based RecordReader call data pieces acquired key value, and the value of the key is extracted together into data sets. CPU data set in its memory allocation DirectBuffer, collection of data required to be stored into the DirectBuffer DirectBuffer format, wherein, when the data set is stored into the DirectBuffer, adding metadata to be set in the data DirectBuffer head. Meanwhile, in MapReduce architecture, is preset on the preset data set of analytic function of parsing, a preset specific analytic functions to convert the data into a data format satisfying the specified logical operations. Provided in the CPU data pre-processor, the data processor to perform the pre-read data from the meta data DirectBuffer, and automatically through a preset data format conversion analytic function. Specifically, description will be provided in FIG. 1 for the embodiment described in detail, referring to FIG 2, a data processing method, comprising:

S210, data is read from the pre-processor DirectBuffer metadata, wherein the metadata index comprises an array address, said address data includes an index array with the set of data one to one data element, the data element data for indicating the storage address in said DirectBuffer data set;

Specifically, when set into DirectBuffer, adding metadata to the head DirectBuffer storing data, data for indicating the data set in the memory address of DirectBuffer. It will be appreciated that the metadata may include an address index array, when data is stored into DirectBuffer set, based on the position data set in the DirectBuffer will add the address stored in the array index address data. The index array with the address data and the data elements of the one-set data, data indicating a set of data elements stored in the address DirectBuffer. Generally, the data are stored DirectBuffer data set is the same data format, text format or binary format logic operation is not performed.

S220, the data pre-processor, read data from the data set according to the data elements DirectBuffer the metadata address index array;

In particular, the data preprocessor in accordance with the address stored in the address data in the array element indexed indicated in DirectBuffer start reading data from the corresponding memory address, the address is stored until the next data element is indicated or read DirectBuffer terminal end, the read data into a data set, and then continue to read the next data, the data in the data set until DirectBuffer read last.

S230, the data processor according to a preset pre-analytic function, data converted into the data set of analytic functions of the preset logical operation to meet specified data format;

Data stored into the data set in the DirectBuffer logical operation is typically not a data format, and performs a logic operation before transmission to the GPU, the format need to be converted into a logical operation can be performed. Thus, in the default schema MapReduce analytic function, data preprocessor automatically according to a preset data format conversion analytic function, converted into the analytic function satisfies the logical operation specified data format.

Alternatively, the preset analytic functions specified data format, logical operations may be required when the GPU data format. Specifically, the predetermined data format parsing may be a function of specified logical operations, the data may be plastic, floating-point data, string data and the like.

S240, the pre-processor converts the data format of the data set to generate a data block;

After the data processor according to a preset pre-analytic function, data is automatically converted into each preset data format for a logical operation specified analytic functions, in order to facilitate the subsequent splicing data between the CPU and the GPU, the data format conversion after the data set to generate a data block.

S250, the pre-processor to store the data blocks into LaunchingBuffer, the splice so that the data read from the data block in the LaunchingBuffer spliced ​​to GPU.

Specifically, the CPU is still in memory allocation data block LaunchingBuffer to temporarily store data format conversion, wherein the pre-processor data stored in the data blocks LaunchingBuffer, followed by the splice completion of reading the data from the data block to the splicing LaunchingBuffer GPU.

It will be appreciated that the data stored in the DirectBuffer CPU data and GPU to be treated may differ in the storage format, i.e., inconsistency in processing the size of the mantissa problem, wherein little-endian storage format refers to the high upper storing data address, lower bank address data in the low; big-endian format storage means for storing data in a high-low-address, the status data stored in the high-address. Thus, the data pre-processor also need to address the problem of data block size of the mantissa.

In DirectBuffer CPU assigned comes with a member variable, the variable indicating the data member is big-endian or little endian format stored in the format of the DirectBuffer, indicating whether there are also stored into LaunchingBuffer upon storage format needs to be converted, given the need to convert National Cheng Kung University tail end of a small format or suggested format. For example, the data in the data set stored big endian format DirectBuffer, whereas GPU for storing data, but are stored in little endian format, then the data block is stored when the LaunchingBuffer, the data block is in little-endian storage format stored in LaunchingBuffer in. Thereafter, the data can be read directly stitcher from the data block LaunchingBuffer spliced ​​to the GPU, the CPU LaunchingBuffer ensure consistent and GPU data storage format, can be read to ensure that the correct data block GPU performs arithmetic processing, to avoid high data read into low, or the status of the data read operation to cause high error.

In an embodiment of the present invention, the data read by the preprocessor DirectBuffer start address index into the array, the data set corresponding to read the data from the DirectBuffer indexed array element address, and thereafter, according to the preset analytic functions for the data in the data set data format conversion, data that can meet the data format conversion logic operations. Generating a set of data stored in the LaunchingBuffer data block, the read block of data transferred to the data by the GPU from the LaunchingBuffer splicer. Embodiment of the present invention is completed by the CPU in the data preprocessor alone, automatically parse the data preset by analytic functions, it provides convenient operation of the GPU data block, using a pre-processor to simplify programming data from the node device with help to optimize future.

Automatic CPU distribution and recovery WorkingBuffer and LaunchingBuffer, wherein a lifetime WorkingBuffer processing time for a data slice, a LaunchingBuffer life cycle is the processing time of a data set. In addition, the assigned CPU ResultBuffer, the operation returns to the calculation result stored in the GPU, after the calculation result as input in MapReduce Reduce tasks.

As shown in FIG. 3, an embodiment of another aspect of the present invention to provide a data processing method, comprising:

S310, reads data stitching block data from the pre-processor generates a second buffer of the CPU;

Example Hadoop cluster is applied in the embodiment of the present invention MapReduce framework, the Hadoop cluster includes a master node device and a slave node device comprises a processor CPU and the GPU graphics processor, the sub-slave device from the master node node device acquires data from device sheet, provided with a pre-processor and the data in the CPU data splicer.

Wherein the data processor for performing pre-set data from the CPU to read a first data buffer, the data converts the data format, the data set to generate a data block stored in the second buffer. The main data registration is completed and the data block is spliced ​​from the CPU to GPU.

S320, the data block of the data splicer to splice the GPU memory block is allocated buffers work.

Embodiments of the present invention reads data stitching a second data block buffer from the CPU, the block of data from the second buffer splicing work buffer of the CPU to the GPU. By splicing data registration example of the embodiment of the present invention, the data, the programmer is no longer dependent on the programming, thereby simplifying the programming of the programmer, but also benefit the entire subsequent MapReduce architecture optimization.

Example 3 The following embodiments will be provided in detail in FIG introduction, a data processing method shown in Figure 4, may include:

S410, the data reading data blocks from the splice LaunchingBuffer;

CPU in memory have been assigned LaunchingBuffer, mainly used to store the data block needs to splicing the GPU.

S420, the data splicer splicing starts from the starting address of the block of data indicated by the parameter cursor, said cursor parameter indicating the start address of the GPU is allocated to store data blocks that can be used to store data WorkingBuffer block ;

GPU memory assigned WorkingBuffer, mainly used to store the LaunchingBuffer splicing from the CPU over the data; WorkingBuffer the GPU memory size is determined by itself, and in the CPU DirectBuffer memory size is determined by the operating environment of java. Generally speaking, WorkingBuffer memory size on the GPU is much larger than the CPU memory DirectBuffer supported by the java, and therefore, WorkingBuffer may be stored come from at least one DirectBuffer data blocks stored in a certain block of data, WorkingBuffer remaining memory may no longer continue to store blocks of data, by the data processing for the splicer to make the right data block.

Specifically, the data management with a splicer cursor parameter indicating parameter WorkingBuffer cursor start address of store data, after each data block to WorkingBuffer splicing, the cursor accordingly updated parameters to the next can be stored accurately know WorkingBuffer start address of data. When the need for data block transfer to WorkingBuffer, the cursor from the start address indicated by the parameter, to the splicing block WorkingBuffer.

S430, when the data splicer to splice the data block to the WorkingBuffer fails, the data block is suspended splicing, and triggers the GPU processing block WorkingBuffer stored.

Wherein the data read from the splicer LaunchingBuffer data block is a logical operation can be performed directly, and satisfies the requirements of the GPU data storage format. The calling application program interface (API, Application Programming Interface) splicing data in the data block to WorkingBuffer. If the remaining memory WorkingBuffer finished splice can be read from the data blocks of the CPU LaunchingBuffer, then the entire data block is spliced ​​to the WorkingBuffer; WorkingBuffer if the remaining memory not completely spliced ​​data block read from the CPU LaunchingBuffer, it will be suspended splicing the data block, the data block remains stored in LaunchingBuffer, an additional trigger for all the data blocks start GPU WorkingBuffer the arithmetic processing.

Embodiment, the CPU data splicer embodiment of the present invention, for solving the problem when splicing block remaining memory size inconsistency WorkingBuffer DirectBuffer the CPU and the GPU. Splicing data blocks by the data from the splicing directly into WorkingBuffer LaunchingBuffer, if the remaining memory WorkingBuffer memory block size can not meet when the time splicing operation is temporarily stopped, the next WorkingBuffer if more memory can be joined to the remaining finish, again LaunchingBuffer spliced ​​into the read data block in WorkingBuffer. Since the data block has LaunchingBuffer meet the needs of data processing when a GPU, GPU after receiving the data block can be directly arithmetic processing, improve the efficiency of the GPU.

It will be appreciated that the data after the splicer transport blocks successfully, perform the following further steps:

B1, the data registration unit notifies the data block size of the GPU;

B2, the data registration update the cursor parameters.

Wherein the data splicer after each data block successfully joined to the GPU, the data block size are notified to GPU, GPU can be used directly without re-calculation of data block size, it is possible to reduce the workload of the GPU.

Further, in the above-described DirectBuffer the CPU to indicate the data indexed by an address in the memory array address DirectBuffer same, the GPU may be added to a look-up index in the array is a data block header WorkingBuffer, lookup index array with the data block data correspondence data elements, data elements used to indicate the memory address in WorkingBuffer data. After the splice stitcher data over a data block, i.e., data elements to be added for each data block index in the lookup array, for subsequent data from the GPU WorkingBuffer quickly find and read data operation.

The above-described steps B1 and B2 in no particular order, this is not limited.

Since the CPU, each of the received last data piece may generate a plurality of data blocks, the GPU WorkingBuffer allocated in units of blocks to be stored, the survival time period is finished processing a data slice. After the data splicer entire data fragments are transmitted, then the data splicer the flag value is transferred successful return, in order to inform the main distribution node device one data slice; after the data registration and transmission data slice fails, the return transmission failed flag value, in order to inform the master device to suspend a data distribution node slice.

Similarly partitioned ResultBuffer GPU memory, the results are stored for ResultBuffer operation, after calling the API interface, and the calculation result is stored in the CPU returns ResultBuffer CPU allocation among tasks in MapReduce Reduce as input.

The CPU for storing data set DirectBuffer, LaunchingBuffer data blocks stored in the data format conversion and ResultBuffer for storing operation results are returned by the GPU automatic CPU distribution and recovery, wherein, LaunchingBuffer life cycle of a data block is processing time; ResultBuffer the life cycle; the GPU for storing the received data block and WorkingBuffer ResultBuffer stored operation results are automatically assigned by the GPU and recovery, wherein WorkingBuffer life cycle is the processing time of a data slice Like WorkingBuffer life cycle. CPU and GPU automatically synchronize the buffer, for example, the CPU distribution and recovery ResultBuffer achieve synchronization with the GPU WorkingBuffer, ResultBuffer.

Shown in FIG. 5-a, embodiments of the present invention further provides a data pre-processor 500, may include:

A first reading unit 510 for reading metadata from a first buffer of said CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the first buffer header portion for the data set of add metadata, the metadata includes the address of the data set stored in said first buffer;

The second reading unit 520, according to a preset analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and the converted data to generate a set of data blocks;

Conversion unit 530 for parsing the data, and generates a set of parsed data blocks;

A storage unit 540 for the data block is stored in the second buffer of the CPU, so that the data registration reads the data block from said second buffer to said splicing GPU.

Example Hadoop cluster is applied in the embodiment of the present invention MapReduce framework, the data preprocessor 500 is provided at Hadoop cluster node apparatus from the CPU, wherein the CPU is further provided with a splicer data, and each node device also from the GPU, acquired from the node device from the master node device Hadoop cluster data piece, and the first divided data into the data buffer value spliced ​​sheet set in the key-value pairs stored in the memory allocated to the CPU 110, since the first buffer memory area division data may not be all keys of slice values ​​stored once finished, therefore, the key data pieces of multiple values ​​may be spliced ​​into the sub-data sets.

When the data set is stored into the first buffer, the data set metadata is added to the head of the first buffer, the metadata includes a set of data stored in the first address buffer. Thereafter, a first read by the reading unit 510 from the first metadata buffer, and then the second reading unit 520 reads data from the first data set according to the buffer storage address indicated by the metadata, and then by the conversion unit 530 converts the data format of data, the entire data set and generate a data format conversion block, the block storage unit 540 stores the second buffer to the CPU, the second buffer allocated to the CPU in the main memory storing data blocks in order to splice the data block transfer can read data to the GPU work buffer from the second buffer. Embodiments of the present invention, the preprocessor automatically performed by the data conversion of the data read and data format, then the programmer need not write the corresponding reduced programming programmer, more conducive to subsequent optimization M, apReduce architecture, improve the efficiency of the CPU.

Further, the metadata index comprises an array of addresses, said address data includes an index array with the set of data elements one to one data, the data elements of said data indicative of said first set of data in a buffer memory address, and further shown in FIG. 5-b, the second reading unit 520 may include:

Data reading unit 5210, a data element read from the array index indicating said address in the memory address of the first buffer, or until the end of the first buffer memory address of the next data element indicating the end of the reading data.

Specifically, data reading unit 5210 according to the index address stored in the address data in the array elements indicated in the first buffer to start reading data from the corresponding memory address, the address is stored until the next data element is indicated, or reading a first end to the end of the buffer, to read a set of data, and then continue to read the next data until the completion of the data set read data in the first buffer.

As shown in FIG. 5-c, the analyzing unit 530 includes:

Data format converting unit 5310, for parsing through a preset function to convert the data into a data set specified by the analytic function to satisfy the logical operation of the data format;

Generating unit 5320, the converted data is used to generate a set of data blocks.

In MapReduce architecture, analytic functions specified preset data format, logical operations may be required when the GPU data format. Specifically, the predetermined data format parsing may be a function of specified logical operations, the data may be plastic, floating-point data, string data and the like.

Shown in FIG. 5-d, the analyzing unit 530 may further comprises:

Format conversion unit 5330, when the storage format for the first GPU to the data buffer the data set storage format inconsistent data, converts the data into the data block in the GPU the storage format.

A first buffer of the CPU and the GPU may require the storage format inconsistent data, i.e., agree on the size of the mantissa processing problems, which means little-endian format stored in high address, high-order data of the lower memory for storing data in the lowest address; big-endian format storage means for storing data in a high-low-address, the status data stored in the high-address.

In the first buffer allocated CPU comes with a member variable, the variable indicating the data member is big-endian or little endian format stored in the format of the first buffer, indicating whether there are also stored into the second buffer while storage format conversion, and gives the required format or converted to upper end of the prompt little endian format. For example, the set of data stored in a first buffer big endian format, while the GPU for storing data, but are stored in little endian format, then the format conversion unit 5330 converts the data blocks into a small tail format, the second buffer memory Area. Thereafter, the data can be read directly stitcher from the second buffer to the splicing block GPU, to ensure consistency of the second buffer of the CPU and GPU data storage format, to ensure that the correct data can be read GPU performs arithmetic processing block avoid high to low read data, or the data is read into the high position result in an operation error.

As illustrated, Example 6-a of the present invention further provides a data splicer 600, may include:

The third reading unit 610 for reading the block data from the pre-processor generates a second CPU in the buffer;

Stitching processing unit 620, the data block for splicing is allocated to the GPU work memory block buffers.

Embodiment is applied to a data pre-processor in the cluster the MapReduce framework Hadoop embodiment of the present invention, the data splicer 600 is provided from the CPU in the node device, wherein the CPU is further provided at Hadoop clusters are shown in FIG. 5-a 500 , and each slave device further comprises the GPU, to obtain data slice from a master node device from Hadoop cluster node device, and the key values ​​of the data points in the piece of storage into data sets together into CPU memory allocated a first buffer, the first buffer memory may not be all the data pieces stored in the primary key values ​​of the finished, therefore, the key data pieces of multiple values ​​may be spliced ​​into the sub-data sets.

Data pre-processor 500 in accordance with data read from the metadata of the first buffer, and then converts the data format, and then converts the data format of the entire data set to generate a data block, the CPU stores the second buffer, then , by the data registration to a third reading unit 610 reads data blocks from the second buffer of the CPU, the read processing unit 620 by the splicing data blocks are allocated to the splicing block of data stored in the GPU work buffers in.

Wherein, in the node apparatus from the CPU, the data pre-processor 500 for data format conversion, and the splice is completed by the data block splicing, no longer dependent on the respective programmer write, programmers can simplify programming, and by automatic operation of the data pre-processor 500 and the splicer, the working efficiency of the CPU can be improved, but also facilitates the subsequent optimization of MapReduce pair.

Data splicer 600 manages a parameter cursor, the cursor parameter indicates the GPU work buffer start address of store data, after each data block is spliced ​​to the work buffer of the GPU, the cursor accordingly updated parameters, so that the next can accurately know the starting address of the GPU work buffer can store data. When the need for data block transfer to the GPU work buffers, stitching processing unit 620 according to the start address indicated by the cursor parameters, the data block is spliced ​​to the work buffer of the GPU.

Accordingly, the above-described stitching processing unit 620 is configured to start stitching the data blocks from the starting address parameter indicative of the cursor, the cursor for indicating the parameter GPU memory block is allocated in the working buffer available for storing data the starting address of the block.

As shown in FIG. 6-b, splice said data further comprises:

Trigger processing unit 630, when the data for the data block splicer splice is allocated to the GPU work memory block buffer fails, the data block is suspended splicing, and triggers the processing of the GPU the working data block stored in the buffer.

The data block read data of the third splice unit 610 reads from the second buffer 600 may be made directly to the logic operation, and to meet the requirements of the GPU data storage format. API calls splicing data in the data block to the GPU work buffer. If the GPU work memory buffers remaining block of data can be completed splice read from the second buffer of the CPU, the entire data block is spliced ​​to the GPU work buffer; if the GPU work memory not remaining buffer After splicing the data block read from the second buffer of the CPU, i.e., splicing block fails, then the data block will be suspended spliced, is still stored in the second data block buffer, the processing unit 630 further triggers the trigger starts GPU work buffer all data blocks performs arithmetic processing.

Further, as shown in FIG. 6-c, the above-described data registration unit 600 may further comprises:

A notification unit 640 for notifying the data block size of the GPU;

Updating unit 650 for updating the cursor parameters.

After each data block successfully joined to the GPU, the notification unit 640 notifies all the data block size to the GPU, GPU can be used directly without re-calculation of data block size, it is possible to reduce the workload of the GPU. Further, the parameter update unit 650 more cursors.

7, embodiments of the present invention provides a processor 700, including a data pre-processor shown in FIG. 5-a and 500 shown in FIG. 6-a data splicer 600, reference may be made to the specific data pre-processor 500 and the presentation data splicer 600, will not be repeated here.

Wherein automatically allocating and deallocating the first buffer and the second buffer in the CPU, the first buffer life cycle is the processing time of a data slice, a second buffer for the processing time of the life cycle of a data block; Similarly, the work buffer allocated automatically in the GPU, the survival time of the work buffer to a data processing time slice.

As shown in FIG. 8-a, embodiments of the present invention further provides a slave device, may include:

The processor CPU-700 shown above in FIG. 7, and a graphics processor GPU-800;

Wherein, CPU-700 as described above, are not repeated here.

Data data set data format conversion Specifically, the CPU-700 pre-processor for the data from the data acquisition slices, and converts the data format to generate a set of data blocks, the data in the CPU-700 by the splicer to splice the data block of the GPU-800 work buffers allocated to store data block;

The GPU-800 data blocks for the processing result obtained by processing, then returns the processing result to the CPU-700.

In practice, the CPU-700 and also automatically assigns a recovery ResultBuffer, similarly, automatically assigned and recovered in a ResultBuffer GPU-800, the same as the life cycle of the CPU-700 ResultBuffer and the GPU-800 ResultBuffer, have storing a calculation result is obtained. If the actual application, the first CPU-700 buffer assigned to DirectBuffer, the second buffer is LaunchingBuffer, work buffers assigned to GPU-800 WorkingBuffer, then as shown, in FIG. 8-b 8-b as a schematic view of the interaction between a CPU-700 with GPU-800 provided from the node device embodiment of the present invention. As shown in FIG. 8-b, setting data pre-processor 500 and the data registration unit 600 CPU-700. Further, the CPU-700 are assigned DirectBuffer, LaunchingBuffer and ResultBuffer, DirectBuffer set of data stored in the data format to be converted, the data set includes data from the key-value pair splicing thereof, and adding metadata to DirectBuffer, Yuan the main data includes data collection, the pre-processor 500 can read the stored metadata in accordance with the address DirectBuffer DirectBuffer data from the data set, and then automatically converted to a data format specified by the preset data analytic function, conversion after the data set to generate a data block, the last data pre-processor 500 stores the data blocks into LaunchingBuffer. If when storing into LaunchingBuffer, storage formats need to convert data in the data blocks, the conversion of the storage format, to ensure that the data storage format in the LaunchingBuffer WorkingBuffer GPU-800 in the same. Data splicer 600 reads data blocks from LaunchingBuffer spliced ​​into the GPU-800 WorkingBuffer, when the stitching fails, the memory block can not WorkingBuffer, the first GPU trigger data blocks WorkingBuffer stored in the arithmetic processing, the arithmetic GPU the result is stored in the ResultBuffer to it, call the API interface operation will result to the CPU ResultBuffer.

Please refer to FIG. 9, embodiments of the present invention further provides a data processing apparatus, may include: at least one memory 910 and processor 920 (in FIG. 9 to a processor, for example). EXAMPLES Some embodiments of the present invention, the memory 910 and processor 920 may be connected by a bus or other means, which are connected by a bus to FIG. 9 for example.

Wherein, the processor 920 may perform the following steps: said data preprocessor reads metadata from a first buffer of said CPU; wherein, when the data acquired from the slice data set into said first buffer storage time , is added in the first set for the data buffer header metadata, the metadata comprising data storage address of said data set in said first buffer; the data preprocessor in accordance with the said metadata storage address indicated by the read data from the buffer of the first set of data; said data processor according to a preset pre-analytic function, the data set of data into the preset parse after the function indicated by the data format, and the converted data to generate a set of data blocks stored in the second buffer of the CPU, so that the data read splice splice the data block from said second buffer to the GPU.

or

The data read from the second buffer splice the CPU data pre-processor generates the data block; the data splicer to splice the data block is allocated to the GPU work memory block buffer.

In some embodiments of the present invention, processor 920 may also perform the following steps: data pre-processor of the data elements from said address array index indicating the start of a first read address in the buffer memory until the next data element or memory address indicated by the end of the first end of the read data buffer.

In some embodiments of the present invention, processor 920 may also perform the following steps: the data preprocessor in accordance with the preset data parsing function satisfies the set of data is converted into the specified logical operation function of the parsed Data Format.

In some embodiments of the present invention, processor 920 may also perform the following steps: data pre-processor to said data block into a storage format of the GPU.

In some embodiments of the present invention, processor 920 may also perform the step of: when the data splicer to splice the data block when the GPU memory block is allocated buffers fails to work, the stitching is suspended said data blocks and said trigger GPU processing data blocks stored in the working buffer.

In some embodiments of the present invention, processor 920 may also perform the following steps: the data splicer splicing starts from the starting address of the block of data indicated by the parameter cursor, the cursor indicating the parameter is assigned the GPU working buffer stores a data block can be used to store the starting address of the data block.

In some embodiments of the present invention, processor 920 may also perform the following steps: the data notifies the GPU splicing the size of the data block; data splicer said parameter updating the cursor.

In some embodiments of the present invention, the memory 910 may be used to store a set of data, metadata, and data blocks;

In some embodiments of the present invention, the memory 910 may also be used to store the address of the indexed array.

In some embodiments of the present invention, the memory 910 may also be used to store cursor parameters.

In some embodiments of the present invention, the memory 910 may also be used to store operation results.

Those of ordinary skill in the art may understand that the above embodiments of the method steps may be all or part by a program instructing relevant hardware, the program may be stored in a computer-readable storage medium storing the above-mentioned medium may be a read-only memory, magnetic or optical disk.

A data processing method of the foregoing and related apparatus according to the present invention provides a detailed description, those of ordinary skill in the art, the idea of ​​the embodiment according to the present invention, there are changes in the embodiments and the scope of application, Therefore, the specification shall not be construed as limiting the present invention.

Claims (22)

  1. A data processing method, wherein, in MapReduce applied Hadoop cluster architecture, said master node device comprising a Hadoop cluster and the slave device, said slave node device comprising a processor CPU and the GPU graphics processor, the slave the node device acquires device data from the master node slice, the CPU is provided with a data processor and a data pre-splicing, said method comprising:
    The data preprocessor reads metadata from a first buffer of said CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the first buffer header adding metadata to said set of data, the metadata comprising data of the data set in said first buffer memory address;
    The preprocessor reads the data stored in the address indicated by the metadata from the data buffer of the first set;
    The data processor according to a preset pre-analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and the transformed data set to generate the data block is stored in the CPU's the second buffer so that the data registration reads the data block from said second buffer to said splicing GPU.
  2. The method according to claim 1, wherein the metadata index comprises an array address, said address data includes an index array with the set of data one to one data element, the data element for data indicating that the data set in the memory address of the first buffer, and then the data pre-processor reads data from the data set comprises the first buffer memory according to the address indicated by the metadata :
    The data pre-processor data elements from said address array index indicating the start of a first read address in the memory buffer, memory address until the next data element indicates the end of the buffer or the first end of the read data.
  3. The method according to claim 1, wherein said data of said data set into said predetermined data format indicated by the parse function comprising:
    The data preprocessor into the parsing function to satisfy the logical operation specified data format according to the preset data of analytic functions of the data set.
  4. The method according to claim 3, characterized in that the method further comprises:
    When the storage format inconsistent with the first data buffer of said data set stored in the format of the data to the GPU, the data converted to generate a set of data blocks comprising:
    The data pre-processor converts the data into data blocks stored in the format of the GPU.
  5. The method according to any one of claims 1 to 4, characterized in that,
    The specific sub-set of data values ​​to splice a plurality of key-sheet consists of the data.
  6. The method according to any one of claims 1 to 4, characterized in that,
    Said first buffer and said second buffer and recovered from the automatically assigned the CPU, the first buffer life cycle is the processing time of a data slice, the existence period of the second buffer is a processing time of data collection.
  7. A data processing method, wherein, in MapReduce applied Hadoop cluster architecture, said master node device comprising a Hadoop cluster and the slave device, said slave node device comprising a processor CPU and the GPU graphics processor, the slave the node device acquires device data from the master node slice, the CPU is provided with a data processor and a data pre-splicing, said method comprising:
    The data read from the second buffer splice the CPU of the block data generated by the preprocessor;
    The data splicer to splice the data block is allocated to the GPU work memory block buffers.
  8. The method according to claim 7, wherein said method further comprises:
    When the data splicer to splice the data block is allocated to the GPU work memory block buffer fails, the data block is suspended splicing, and triggers the processing of the GPU work stored in the buffer data block.
  9. The method according to claim 78, wherein said data splicer to splice the data block is allocated to the GPU work memory block buffer comprising:
    The data splicer splicing starts from the starting address of the block of data indicated by the parameter cursor, said cursor parameter indicating the start address of the GPU memory block is allocated in the working buffer for storing data blocks may be .
  10. The method according to claim 9, wherein, when the data block successfully splicing, said method further comprising:
    The splice notifies the data size of the data block of the GPU;
    The data registration update the cursor parameters.
  11. A data pre-processor, characterized by comprising:
    A first reading unit configured to read the metadata from a first buffer of said CPU; wherein, when the first data set is stored into the buffer from the data acquired slice in the first buffer adding header for the data set of metadata, the metadata includes the address of the data set stored in said first buffer;
    A second reading unit for reading from the first memory buffer according to the address indicated by the metadata of the data set;
    A conversion unit according to a preset analytic function to convert data in the data set into the predetermined data format indicated by the parse function, and the converted data to generate a set of data blocks;
    A storage unit for the data block is stored in the second buffer of the CPU, so that the data registration reads the data block from said second buffer to said splicing GPU.
  12. The data pre-processor of claim 11, wherein the metadata index comprises an array address, said address data includes an index array with the set of data elements one to one data, the data data elements for indicating the set of data in a memory address of the first buffer, the second reading unit further comprises:
    Data reading means for reading the start address of the data elements from the array index indicating the address of the first buffer memory, or until the end of the first buffer memory address of the next data element indicates the end of the read data .
  13. The data pre-processor 11 or claim 12, wherein the parsing means comprises:
    Data format conversion means for parsing through a preset function to convert the data into a data set specified by the analytic function to satisfy the logical operation of the data format
    Generating means for converting the data set to generate a data block.
  14. The data pre-processor of claim 11, wherein the analysis unit further comprises:
    Format conversion means for, when said first buffer storage format inconsistent with the data set of data stored in the GPU, the data format, converts the data into the data block in the GPU storage format.
  15. A data splicer, wherein, comprising:
    Third read means for reading from the second buffer of the CPU data pre-processor generates the data block;
    Splicing means for splicing the data block is allocated to the GPU work memory block buffers.
  16. Data splicer according to claim 15, wherein said data splicer further comprises:
    Trigger processing unit, when the data for the data block splicer to splice failure is when the GPU work memory buffers allocated data block, the data block is suspended splicing, and triggering processing of the GPU said working data blocks stored in the buffer.
  17. According to the splicer 15 or claim 16, wherein the processing unit is specifically for splicing the splicing block starts from the start address parameter indicative of the cursor, the cursor for indicating the parameter GPU memory block is allocated buffers may be used to work the start address of the memory block.
  18. Data splicer according to claim 17, wherein said data splicer further comprises:
    A notification unit for notifying the size of the data block of the GPU;
    Updating means for updating the cursor parameters.
  19. A processor, comprising: a data pre-processor of claim 11 and as claimed in claim in claim 15, said data splicer.
  20. The processor according to claim 19, wherein said processor further comprises:
    Automatic distribution and recovering said first buffer and said second buffer, said first buffer life cycle of a data processing time slice, the second buffer is a data set for the lifetime of processing time.
  21. From one of the node device, wherein the device is a slave node from the node device Hadoop cluster, further comprising the Hadoop cluster master node device, the node device from receiving data from the Hadoop cluster fragments, the said slave device comprises: a processor of the CPU 19 and the GPU graphics processor as claimed in claim;
    Wherein the data set data format conversion, the data pre-processor of the CPU for data pieces acquired from the data format and converts the data to generate a set of data blocks, the CPU of the data by the splicer splicing data blocks into the working buffer the data stored in the GPU allocation block;
    The GPU for processing the data blocks to obtain a processing result, after the processing result is returned to the CPU.
  22. From the node device according to claim, 21 wherein said, the GPU further comprises:
    Automatic allocation and deallocation of the working buffer, the lifetime of the working buffer to process a time-sliced ​​data.
PCT/CN2014/094071 2013-12-23 2014-12-17 Data processing method and related device WO2015096649A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN 201310719857 CN104731569A (en) 2013-12-23 2013-12-23 Data processing method and relevant equipment
CN201310719857.4 2013-12-23

Publications (1)

Publication Number Publication Date
WO2015096649A1 true true WO2015096649A1 (en) 2015-07-02

Family

ID=53455495

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/094071 WO2015096649A1 (en) 2013-12-23 2014-12-17 Data processing method and related device

Country Status (2)

Country Link
CN (1) CN104731569A (en)
WO (1) WO2015096649A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105159610B (en) * 2015-09-01 2018-03-09 浪潮(北京)电子信息产业有限公司 Large-scale data processing system and method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050140682A1 (en) * 2003-12-05 2005-06-30 Siemens Medical Solutions Usa, Inc. Graphics processing unit for simulation or medical diagnostic imaging
CN102662639A (en) * 2012-04-10 2012-09-12 南京航空航天大学 Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method
CN102708088A (en) * 2012-05-08 2012-10-03 北京理工大学 CPU/GPU (Central Processing Unit/ Graphic Processing Unit) cooperative processing method oriented to mass data high-performance computation

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050140682A1 (en) * 2003-12-05 2005-06-30 Siemens Medical Solutions Usa, Inc. Graphics processing unit for simulation or medical diagnostic imaging
CN102662639A (en) * 2012-04-10 2012-09-12 南京航空航天大学 Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method
CN102708088A (en) * 2012-05-08 2012-10-03 北京理工大学 CPU/GPU (Central Processing Unit/ Graphic Processing Unit) cooperative processing method oriented to mass data high-performance computation

Also Published As

Publication number Publication date Type
CN104731569A (en) 2015-06-24 application

Similar Documents

Publication Publication Date Title
US20120137098A1 (en) Virtual storage migration method, virtual storage migration system and virtual machine monitor
US20140157251A1 (en) Software version management
US20140317387A1 (en) Method for performing dual dispatch of blocks and half blocks
US20140282601A1 (en) Method for dependency broadcasting through a block organized source view data structure
US20140281411A1 (en) Method for dependency broadcasting through a source organized source view data structure
US20140281426A1 (en) Method for populating a source view data structure by using register template snapshots
US20140281436A1 (en) Method for emulating a guest centralized flag architecture by using a native distributed flag architecture
US20150046683A1 (en) Method for using register templates to track interdependencies among blocks of instructions
US20140281428A1 (en) Method for populating register view data structure by using register template snapshots
US20110153957A1 (en) Sharing virtual memory-based multi-version data between the heterogenous processors of a computer platform
CN102541605A (en) Method and device for picture loading based on Android system
CN101901207A (en) Operating system of heterogeneous shared storage multiprocessor system and working method thereof
CN102609252A (en) Intelligent configuration method for logic menu resources on basis of measurement instruments
US20090248620A1 (en) Interacting methods of data extraction
US7958083B2 (en) Interacting methods of data summarization
CN101221504A (en) Test script processing method and its scheduling and executing system
US20120311589A1 (en) Systems and methods for processing hierarchical data in a map-reduce framework
CN103177027A (en) Method and system for obtaining dynamic feed index
US20060036616A1 (en) Suspending a result set and continuing from a suspended result set for scrollable cursors
US20150067005A1 (en) Path resolver for client access to distributed file systems
US20130054546A1 (en) Hardware-based array compression
US20120303640A1 (en) Self-Parsing XML Documents to Improve XML Processing
CN101694617A (en) Multi-language support implementing method based on resource identifier
CN102226894A (en) Mobile device memory management method and device
US20140157231A1 (en) Modifying a Middleware

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14873198

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14873198

Country of ref document: EP

Kind code of ref document: A1