CN105630690B - The method and device that big data is handled in isomerization hardware - Google Patents
The method and device that big data is handled in isomerization hardware Download PDFInfo
- Publication number
- CN105630690B CN105630690B CN201410608826.6A CN201410608826A CN105630690B CN 105630690 B CN105630690 B CN 105630690B CN 201410608826 A CN201410608826 A CN 201410608826A CN 105630690 B CN105630690 B CN 105630690B
- Authority
- CN
- China
- Prior art keywords
- host
- data
- target data
- address
- address information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The embodiment of the invention discloses the method and devices that big data in a kind of isomerization hardware is handled, it is related to field of communication technology, it can solve isomerization hardware processing big data application, the data of loading and the data needed for program operation are inconsistent, the problem of occupying the memory of isomerization hardware, and influencing the normal operation of program.The method of the present invention includes：Receive program and address information that the Host is sent；Establish the mapping relations of virtual address and described address information that the source data is locally stored；During the isomerization hardware runs described program when invocation target data, target data request is sent to the Host；After receiving the target data that the Host is replied, described program is run using the target data.The present invention is suitable for isomerization hardware.
Description
Technical field
The present invention relates to the method and devices that big data in field of communication technology more particularly to a kind of isomerization hardware is handled.
Background technology
Isomerization hardware be interconnected by PCI-E (pci Express, universal serial bus) as expansion card and host Host it is hard
Part can provide the abilities such as more powerful vector processing, floating-point operation or multi-threaded parallel for Host.Memory on isomerization hardware
Smaller and be difficult extension, when handling data, data can not disposably be loaded into different isomerization hardware by the applications of some big datas
Structure hardware executes, the case where being much smaller than data processing amount for the memory on this isomerization hardware, and the method for generally use is, will
The memory of isomerization hardware is loaded into after big data section in batches, i.e., after last consignment of data are completed in isomerization hardware processing, then by second
Batch data is loaded on isomerization hardware, for example, the data of 16G to be divided into the data block of 4 4G, is loaded into isomerization hardware in batches
Memory on handled.
At least there are the following problems in the prior art：In the data handling procedure of some big datas application, program logarithm
According to demand be it is dynamic, i.e., program operation before not can determine which partial data it needs, so isomerization hardware handles this
Big data is planted in application, data to be loaded into isomerization hardware memory in batches, can be made needed for data and the present procedure operation of loading
Data it is inconsistent, present procedure, which runs unwanted data and is stored in isomerization hardware, occupies a large amount of memory, not only wastes
The memory headroom of isomerization hardware also affects the normal operation of program.
Invention content
The embodiment of the present invention provides the method and device that big data is handled in a kind of isomerization hardware, and it is hard can to solve isomery
Part handle this big data in application, data and the present procedure operation for being loaded into isomerization hardware memory it is required data it is inconsistent,
Not only waste the memory headroom of isomerization hardware, the problem of also affecting the normal operation of program.
In order to achieve the above objectives, the embodiment of the present invention adopts the following technical scheme that：
In a first aspect, the embodiment of the present invention provides a kind of method that big data is handled in isomerization hardware, isomerization hardware with
Host Host connections, including：
Program and address information that the Host is sent are received, described program is the program run on isomerization hardware, institute
It is the address information that is stored in the Host of source data to state address information, and the source data is runs institute on the isomerization hardware
State the data needed for program；
Establish the mapping relations of virtual address and described address information that the source data is locally stored；
During the isomerization hardware runs described program when invocation target data, target data is sent to the Host
Request, the target data request carries the address information of the target data, in order to which the Host is according to the number of targets
According to address information search the target data, the address information of the target data is the isomerization hardware according to the mapping
The virtual address of relationship and the target data determines that the target data belongs to the source data；
After receiving the target data that the Host is replied, described program is run using the target data.
With reference to first aspect, in the first possible realization method of first aspect, when the source data is stored in institute
When stating the memory of Host, described address information is the physical address that the Host stores the source data；
When the source data is stored in the disk of the Host, described address information is that the Host stores the source number
According to virtual address.
With reference to first aspect, in second of possible realization method of first aspect, the number of targets is used described
Before operation described program, the method further includes：
The target data is stored in the memory of the isomerization hardware；
The mapping being locally stored between the virtual address of the target data and the address information of the target data is closed
System is revised as that the virtual address of the target data and target data physics in the isomerization hardware memory is locally stored
Mapping relations between address.
Second of possible realization method with reference to first aspect, in the third possible realization method of first aspect
In, when the source data is stored in the disk of the Host, the target data is stored in the isomerization hardware described
Memory in before, the method further includes：
Receive the physical address that the target data that the Host is replied stores in the memory of the Host, the object
Reason address is that the Host is stored in the physical address in the memory of the Host after transferring the target data in disk；
The mapping being locally stored between the virtual address of the target data and the address information of the target data is closed
System is revised as being locally stored between the virtual address of the target data and the physical address stored in the memory of the Host
Mapping relations；
It is asked to the Host transmitting portions target data, the partial target request of data carries partial target data and exists
The physical address stored in the memory of the Host, in order to the Host according to the partial target data the Host's
The physical address stored in memory searches the partial target data；
It is then described after receiving the target data that the Host is replied, run described program packet using the target data
It includes：
After receiving the partial target data that the Host is replied, the partial target data run described program is used.
With reference to first aspect, in the 4th kind of possible realization method of first aspect, work as the isomerization hardware described
During operation described program when invocation target data, before being asked to the Host transmission datas, the method further includes：
It adds and identifies for described address information, the mark expression described address information is the data in the Host
The address information of storage.
Second aspect, the embodiment of the present invention provide the device that big data is handled in a kind of isomerization hardware, including：
Receiving unit, the program and address information sent for receiving the Host, described program are on isomerization hardware
The program of operation, described address information are the address information that source data stores in the Host, and the source data is described different
The data needed for described program are run on structure hardware；
Unit is established, the mapping for establishing virtual address and described address information that the source data is locally stored is closed
System；
Request unit, for during running described program when the isomerization hardware when invocation target data, to described
Host sends target data request, and the target data request carries the address information of the target data, in order to described
Host searches the target data according to the address information of the target data, and the address information of the target data is described different
Structure hardware determines that the target data belongs to the source number according to the virtual address of the mapping relations and the target data
According to；
It using unit, is used for after receiving the target data that the Host is replied, described in target data operation
Program.
In conjunction with second aspect, in the first possible realization method of second aspect, when the source data is stored in institute
When stating the memory of Host, described address information is the physical address that the Host stores the source data；
When the source data is stored in the disk of the Host, described address information is that the Host stores the source number
According to virtual address.
In conjunction with second aspect, in second of possible realization method of second aspect, described device further includes：
Storage unit, for the target data to be stored in the memory of the isomerization hardware；
Change unit, the address information of virtual address and the target data for the target data will to be locally stored
Between mapping relations be revised as that the virtual address of the target data is locally stored and the target data is hard in the isomery
Mapping relations in part memory between physical address.
In conjunction with second of possible realization method of second aspect, in the third possible realization method of second aspect
In, the receiving unit is additionally operable to receive the object that the target data that the Host is replied stores in the memory of the Host
Address is managed, the physical address is that the Host is stored in after transferring the target data in disk in the memory of the Host
Physical address；
The modification unit is additionally operable to that the ground of the virtual address and the target data of the target data will be locally stored
Mapping relations between the information of location are revised as the virtual address of the target data being locally stored and in the memory of the Host
Mapping relations between the physical address of storage；
The request unit is additionally operable to ask to the Host transmitting portions target data, the partial target request of data
The physical address that partial target data store in the memory of the Host is carried, in order to which the Host is according to the part mesh
The physical address that mark data store in the memory of the Host searches the partial target data；
It is described to be additionally operable to after receiving the partial target data that the Host is replied using unit, use the partial target
Data run described program.
In conjunction with second aspect, in the 4th kind of possible realization method of second aspect, described device further includes：
Adding device, for being identified for the addition of described address information, the mark indicates that described address information is the number
According to the address information stored in the Host.
In a kind of isomerization hardware provided in an embodiment of the present invention big data handle method and device, the present invention in isomery it is hard
The address information that source data stores in Host needed for the program to be run and the program operation that part receiving host Host is sent, ground
Location information is the address information of source data storage；The mapping for establishing virtual address and address information that source data is locally stored is closed
System；When isomerization hardware runs the target data called in program process in source data, isomerization hardware is according to mapping relations and mesh
The virtual address of mark data determines the address information of target data, and the data for carrying target data address information are sent to Host
Request, in order to which Host searches target data according to the address information of target data；After receiving the target data that Host is replied,
Program is run using target data.In this way, when isomerization hardware handles big data in application, isomerization hardware need not be loaded data into
Memory, it is only necessary to be loaded into the address information that data store in Host, while establishing data in LVA local virtual address and address
The mapping of information, when isomerization hardware needs to call data during running program, can by LVA local virtual address and
Mapping determines the address information that data store in Host, and the request for carrying address information is then sent to Host, is returned in Host
After returning data, program is continued to run with using data, both isomerization hardware can be loaded into avoid redundant data, and waste the interior of isomerization hardware
It deposits, and does not influence the normal operation of program.
Description of the drawings
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to needed in the embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for ability
For the those of ordinary skill of domain, without creative efforts, it can also be obtained according to these attached drawings other attached
Figure.
Fig. 1 is the method flow diagram that one embodiment of the invention provides；
Fig. 2 is the method flow diagram that further embodiment of this invention provides；
Fig. 3 is the method flow diagram that further embodiment of this invention provides；
Fig. 4, Fig. 5 are the apparatus structure schematic diagram that further embodiment of this invention provides.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts all other
Embodiment shall fall within the protection scope of the present invention.
One embodiment of the invention provides a kind of method that big data is handled in isomerization hardware, is used for isomerization hardware, and isomery is hard
Part is linked with Host, as shown in Figure 1, method includes：
101, the program and address information that isomerization hardware receiving host Host is sent.
Wherein, program is the program run on isomerization hardware, and described address information is that source data is deposited in the Host
The address information of storage, the source data are the data needed for operation described program on the isomerization hardware.
Optionally, when source data is stored in the memory of Host, address information is the physical address that Host stores source data；
When source data is stored in the disk of Host, address information is the virtual address that Host stores source data, and void is stored in Host
The correspondence for the physical address that quasi- address is stored with source data in disk, Host can be looked by virtual address and correspondence
Find the source data in disk.
102, the virtual address of source data and the mapping relations of address information is locally stored in isomerization hardware foundation.
103, when isomerization hardware runs invocation target data in program process, isomerization hardware is asked to Host transmission datas
It asks.
Wherein, request of data carries the address information of target data, in order to which Host is according to the address information of target data
Search target data.The address information of target data is that isomerization hardware is true according to the virtual address of mapping relations and target data
Fixed, target data is a part for source data.
104, after receiving the target data that Host is replied, isomerization hardware runs program using target data.
Optionally, before isomerization hardware is using target data, target data can also be stored in isomery by isomerization hardware
In the memory of hardware, then it will be locally stored between the virtual address of the target data and the address information of the target data
Mapping relations be revised as being locally stored the virtual address of the target data with the target data in the isomerization hardware
Deposit the mapping relations between physical address.In this way, isomerization hardware is in running program process, it would be desirable to data be loaded into itself
Memory, avoid extra data, save memory headroom.
Further, when source data is stored in the disk of Host, in the memory that target data is stored in isomerization hardware
In before, if the data that isomerization hardware is transferred are larger, target data after disk is transferred, can be first stored in Host by Host
Memory in, the physical address of storage is then sent to isomerization hardware, isomerization hardware receives the target data that Host is replied and exists
The physical address stored in the memory of Host, and the virtual address of the target data and the target data will be locally stored
Mapping relations between address information are revised as that the virtual address of the target data and the memory in the Host is locally stored
Mapping relations between the physical address of middle storage, then isomerization hardware from the memory of Host transfer target data by several times, i.e., to
Host, which is sent, carries the partial target request of data that partial target data store physical address in the memory of Host, in order to
Host searches the partial target data according to the physical address that partial target data store in the memory of Host.
It should be noted that when isomerization hardware transfers the data in disk file, it will usually once to by relevant data
It all transfers, but these data are not moved while being used, it is possible to the data disposably transferred first are stored in Host's
In memory, called data is being implemented according to the operating condition of program, need not mass data disposably called in into isomerization hardware, kept away
Exempt from memory waste.
Optionally, when isomerization hardware invocation target data, before being asked to Host transmission datas, isomerization hardware also needs
To be address information addition mark, mark indicates that address information is the address information that data store in Host, isomerization hardware
It can determine whether data are stored in the memory of isomerization hardware according to mark.
In a kind of isomerization hardware provided in an embodiment of the present invention big data handle method and device, the present invention in isomery it is hard
The address information that source data stores in Host needed for the program to be run and the program operation that part receiving host Host is sent, ground
Location information is the address information of source data storage；The mapping for establishing virtual address and address information that source data is locally stored is closed
System；When calling the target data in source data during isomerization hardware operation described program, isomerization hardware is according to mapping relations
The address information of target data is determined with the virtual address of target data, and is sent to Host and carried target data address information
Request of data, in order to which Host searches target data according to the address information of target data；In the number of targets for receiving Host replies
According to rear, target data is used to run program.In this way, when isomerization hardware handles big data in application, need not load data into different
The memory of structure hardware, it is only necessary to be loaded into the address information that data store in Host, while establish data in LVA local virtual address
It can pass through local virtual when isomerization hardware needs to call data during running program with the mapping of address information
Address and mapping determine the address information that data store in Host, and the request for carrying address information is then sent to Host,
After Host returned datas, program is continued to run with using data, isomerization hardware can be both loaded into avoid redundant data, waste isomery is hard
The memory of part, and the normal operation of program is not influenced.
Further embodiment of this invention provides a kind of method that big data is handled in isomerization hardware, big for isomerization hardware processing
The scene of data, isomerization hardware are interconnected with Host by PCI-E, and big data is stored in the memory of Host, as shown in Fig. 2, institute
The method of stating includes：
201, Host extracts the pointer for the big data for needing to transmit from big data program segment, and is translated into Host
On physical address.
Wherein, big data is source data, and big data program segment is to need the program run on isomerization hardware in Host,
Big data is data needed for program operation, and Host, which can be extracted by corresponding program from big data program segment, to be needed to pass
The pointer for the big data passed, i.e. virtual address, then convert virtual address to physical address, and physical address is in Host memories
The address of big data storage.
202, Host creates the state table for safeguarding big data lock status.
Wherein, the state of data lock status illustrates the use state of the Physical Page corresponding to current data, for example, with
" 0 " indicates to be used, and " 1 " indicates not use.In follow-up data processing procedure, variation that state table is stored according to data
It is updated.
203, the physical address of program and Host storage data is sent to isomerization hardware by Host.
Wherein, Host will need the pointer of data needed for the program run on isomerization hardware and program operation to send
To isomerization hardware, isomerization hardware receives its being executed for task that determines after program, after reception pointer, can store number by Host
According to physical address obtain data.
204, isomerization hardware creates task after receiving program.
205, isomerization hardware stores the local page table of physical address establishment of data according to Host, establishes virtual in local page table
The mapping relations of address and the physical address of data storage on Host.
Wherein, isomerization hardware stores the local page table of physical address establishment of data according to Host, is established in local page table
The mapping relations of the virtual address and the physical address of data storage on Host of data are locally stored.
206, isomerization hardware is the physical address addition mark that Host stores data in local page table.
207, when isomerization hardware, which is run, to be needed to call data in program process, it is hard to judge whether the data are stored in isomery
The memory of part obtains data if the data are stored in the memory of isomerization hardware from memory, executes step 214；When the data
It is not stored in the memory of isomerization hardware, thens follow the steps 208.
Wherein, isomerization hardware can learn that the data whether there is in different by judging that the address in local page table identifies
In the physical memory of structure hardware.
208, isomerization hardware generates interruption, is asked to Host transmission datas.
Wherein, isomerization hardware can obtain the data depositing in Host according to the mapping relations recorded from local page table
Storage address carries the physical address that the data of acquisition store in Host when being asked to Host transmission datas.
209, after Host receives request of data, judge whether the address of the data is lock-out state by lock status table, if
The address of the data is not lock-out state, thens follow the steps 210；If the address of the data is lock-out state, step is continued to execute
Rapid 209.
Wherein, after Host receives request of data, judge whether the corresponding Physical Page of the data is locking by lock status table
State, if the corresponding Physical Page of the data is lock-out state, the address of the data is lock-out state, needs to wait for its unlock,
Continue to execute step 209；If the corresponding Physical Page of the data is not lock-out state, the address of the data is not lock-out state,
Then follow the steps 210.
210, Host judges whether isomerization hardware is copy to the request method of data, if request of the isomerization hardware to data
Mode is copy, thens follow the steps 211；If isomerization hardware is not copy to the request method of data, 212 are thened follow the steps.
Wherein, isomerization hardware can be copy to the request method of data, can also be to call in.
211, Host returns to its data asked to isomerization hardware.
It should be noted that after having executed this step, step 214 is executed.
212, Host returns to its data asked to isomerization hardware, and discharges the memory of data occupancy.
213, isomerization hardware storage receives the data that Host is returned, and changes local page table.
Wherein, it after isomerization hardware receives the data that Host is returned, stores data in local memory, and by native page
The storage address of data is revised as the address in local memory in table, cancels mark.
214, isomerization hardware continues to execute program using the data of calling.
In the prior art, isomerization hardware handles this big data in application, data to be loaded into isomerization hardware memory in batches,
Data and the data needed for program operation of loading can be made inconsistent, in this way, not only taking up the memory of isomerization hardware, can also be influenced
To the normal operation of program.Compared with prior art, when isomerization hardware handles big data in application, need not be by number in the present invention
According to the memory for being loaded into isomerization hardware, it is only necessary to be loaded into the address information that data store in Host, while establish data in local
The mapping of virtual address and address information can pass through when isomerization hardware needs to call data during running program
LVA local virtual address and mapping determine the address information that data store in Host, are then sent to Host and carry address information
Request, after Host returned datas, program is continued to run with using data, and isomerization hardware, wave can be both loaded into avoid redundant data
Take the memory of isomerization hardware, and does not influence the normal operation of program.
Further embodiment of this invention provides a kind of method that big data is handled in isomerization hardware, big for isomerization hardware processing
The scene of data, isomerization hardware are interconnected with Host by PCI-E, the disk text that big data is mapped by maps virtual page mmap
Part, big data are source data, i.e., the data only have virtual address in Host, are not stored in Host memories, but are stored
In disk file, as shown in figure 3, the method includes：
301, Host calls mmap mapped files, and it is literary to disk to establish in Host the virtual address for storing data by page table
The mapping relations of the physical address of data are stored in part.
Wherein, the physical address for storing data in Host in the virtual address to disk file of storage data is recorded in page table
Mapping relations.
302, program and page table are sent to isomerization hardware by Host.
Wherein, Host will need the program run on isomerization hardware and page table to be sent to isomerization hardware, and isomerization hardware receives
Its being executed for task is determined after program, after receiving page table, can obtain data by page table.
303, isomerization hardware creates task after receiving program.
304, isomerization hardware receives the local page table that the virtual address in page table is added to after page table isomerization hardware.
Wherein, after isomerization hardware receives the page table of Host, local page table is changed according to the page table of Host, in local page table
The middle mapping relations for establishing the virtual address of storage data in the virtual address that data are locally stored and Host.
305, isomerization hardware is the virtual address addition mark of storage data in the Host in page table.
Wherein, mark indicates that this virtual address is the virtual address of certain process on Host, object corresponding with virtual address
It manages on the disk that address is Host.
306, when isomerization hardware, which is run, to be needed to call data in program process, it is hard to judge whether the data are stored in isomery
The memory of part obtains data if the data are stored in the memory of isomerization hardware from memory, executes step 313；When the data
It is not stored in the memory of isomerization hardware, thens follow the steps 307.
Wherein, isomerization hardware can learn that the data whether there is in different by judging that the address in local page table identifies
In the physical memory of structure hardware.
307, isomerization hardware generates interruption, is asked to Host transmission datas.
Wherein, isomerization hardware can obtain the data and stored in Host according to the mapping relations recorded from local page table
The virtual address of data, when being asked to Host transmission datas, the data for carrying acquisition store the virtual of data in Host
Address.
308, Host is according to the virtual address and mmap mapped files of storage data, by the data of isomerization hardware request from magnetic
Disk is called in and is stored in memory.
309, Host adds the state of the data in the state table for safeguarding lock status.
Wherein, after data storage to memory, by the addition of its lock status in the state table for safeguarding lock status.If there is no dimensions
The state table for protecting lock status then creates state table to safeguard the lock status of the data.
310, the physical address that Host is stored to isomerization hardware returned data in Host memories.
311, the local page table of physical address modification that isomerization hardware is stored according to data in Host memories.
Wherein, after isomerization hardware receives the physical address that data store in Host memories, by data in local page table
The corresponding address of virtual address be revised as the physical address stored in Host memories.
312, the physical address called data that isomerization hardware is stored according to data in Host memories.
Wherein, the process for the physical address called data that isomerization hardware is stored according to data in Host memories and Fig. 2 institutes
Show that the step 207 of embodiment is identical to step 212 process, details are not described herein.
It should be noted that the data that isomerization hardware is transferred in this step can be isomerization hardware number of request in step 307
According to whole, can also be a part for isomerization hardware request data in step 307.
313, isomerization hardware continues to execute program using the data of calling.
It should be noted that the scene of isomerization hardware processing big data, big data can also partly be stored in the interior of Host
In depositing, be partly stored in the disk of Host, in such cases, isomerization hardware by flow chart of data processing shown in Fig. 2 as shown in figure 3
Flow chart of data processing combine.The data in disk can also be called in memory before Host sends program to isomerization hardware
Storage, then directly executes flow chart of data processing shown in Fig. 2, and data are called from the disk of Host to save isomerization hardware
Time.
In the prior art, isomerization hardware handles this big data in application, data to be loaded into isomerization hardware memory in batches,
Data and the data needed for program operation of loading can be made inconsistent, in this way, not only taking up the memory of isomerization hardware, can also be influenced
To the normal operation of program.Compared with prior art, when isomerization hardware handles big data in application, need not be by number in the present invention
According to the memory for being loaded into isomerization hardware, it is only necessary to be loaded into the address information that data store in Host, while establish data in local
The mapping of virtual address and address information can pass through when isomerization hardware needs to call data during running program
LVA local virtual address and mapping determine the address information that data store in Host, are then sent to Host and carry address information
Request, after Host returned datas, program is continued to run with using data, and isomerization hardware, wave can be both loaded into avoid redundant data
Take the memory of isomerization hardware, and does not influence the normal operation of program.
Further embodiment of this invention provides the device 40 that big data is handled in a kind of isomerization hardware, as shown in figure 4, the dress
Setting 40 includes：
Receiving unit 41, the program and address information sent for receiving the Host, described program are in isomerization hardware
The program of upper operation, described address information are the address information that source data stores in the Host, and the source data is described
The data needed for described program are run on isomerization hardware；
Unit 42 is established, the mapping for establishing virtual address and described address information that the source data is locally stored is closed
System；
Request unit 43, for during running described program when the isomerization hardware when invocation target data, to described
Host sends target data request, and the target data request carries the address information of the target data, in order to described
Host searches the target data according to the address information of the target data, and the address information of the target data is described different
Structure hardware determines that the target data belongs to the source number according to the virtual address of the mapping relations and the target data
According to；
Using unit 44, for after receiving the target data that the Host is replied, institute to be run using the target data
State program.
Wherein, when the source data is stored in the memory of the Host, described address information stores institute for the Host
State the physical address of source data；When the source data is stored in the disk of the Host, described address information is the Host
Store the virtual address of the source data.
Further, as shown in figure 5, described device 40 can also include：
Storage unit 45, for the target data to be stored in the memory of the isomerization hardware；
Unit 46 is changed, the address for virtual address and the target data that the target data will be locally stored is believed
Mapping relations between breath are revised as that the virtual address of the target data is locally stored with the target data in the isomery
Mapping relations in hardware memory between physical address.
Further, as shown in figure 5, the receiving unit 41 is additionally operable to receive the target data that the Host is replied
The physical address stored in the memory of the Host, the physical address are that the Host transfers the number of targets from disk
The physical address being stored in after in the memory of the Host；
The modification unit 46 is additionally operable to that the virtual address of the target data and the target data will be locally stored
Mapping relations between address information are revised as that the virtual address of the target data and the memory in the Host is locally stored
Mapping relations between the physical address of middle storage；
The request unit 43 is additionally operable to ask to the Host transmitting portions target data, and the partial target data are asked
It asks and carries the physical address that partial target data store in the memory of the Host, in order to which the Host is according to the part
The physical address that target data stores in the memory of the Host searches the partial target data；
It is described to be additionally operable to after receiving the partial target data that the Host is replied using unit 44, use the part mesh
Mark data run described program.
Further, as shown in figure 5, described device 40 can also include：
Adding device 47, for being identified for the addition of described address information, the mark indicates that described address information is described
The address information that data store in the Host.
In the prior art, isomerization hardware handles this big data in application, data to be loaded into isomerization hardware memory in batches,
Data and the data needed for program operation of loading can be made inconsistent, in this way, not only taking up the memory of isomerization hardware, can also be influenced
To the normal operation of program.Compared with prior art, the program to be run that 40 receiving host Host of device is sent in the present invention
The address information that required source data stores in Host is run with program, address information is the address information of data source storage；It builds
Deposit the mapping relations of the virtual address and address information of source storage data in vertical local；Source number is called in program process when device 40 is run
When target data in, device 40 determines that the address of target data is believed according to the virtual address of mapping relations and target data
Breath, and the request of data for carrying target data address information is sent to Host, in order to which Host believes according to the address of target data
Breath searches target data；After receiving the target data that Host is replied, program is run using target data.In this way, working as device 40
Big data is handled in application, the memory of device 40 need not be loaded data into, it is only necessary to be loaded into the ground that data store in Host
Location information, while mapping of the data in LVA local virtual address and address information is established, when device 40 is during running program
When needing to call data, the address information that data store in Host can be determined by LVA local virtual address and mapping, then
It is sent to Host and carries the request of address information and continue to run with program using data after Host returned datas, it both can be to avoid
Redundant data device for loading 40, wastes the memory of device 40, and does not influence the normal operation of program.
The method that above-mentioned offer may be implemented in the device that big data is handled in isomerization hardware provided in an embodiment of the present invention is real
Example is applied, concrete function realizes the explanation referred in embodiment of the method, and details are not described herein.Isomery provided in an embodiment of the present invention
The method and device that big data is handled in hardware can be adapted for isomerization hardware, but be not limited only to this.
Each embodiment in this specification is described in a progressive manner, identical similar portion between each embodiment
Point just to refer each other, and each embodiment focuses on the differences from other embodiments.Especially for equipment reality
For applying example, since it is substantially similar to the method embodiment, so describing fairly simple, related place is referring to embodiment of the method
Part explanation.
One of ordinary skill in the art will appreciate that realizing all or part of flow in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in a computer read/write memory medium
In, the program is when being executed, it may include such as the flow of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, all answer by the change or replacement that can be readily occurred in
It is included within the scope of the present invention.Therefore, protection scope of the present invention should be subject to the protection scope in claims.
Claims (10)
1. a kind of method that big data is handled in isomerization hardware, isomerization hardware are connect with host Host, which is characterized in that including：
Program and address information that the Host is sent are received, described program is the program run on isomerization hardware, describedly
Location information is the address information that source data stores in the Host, and the source data is to run the journey on the isomerization hardware
Data needed for sequence；
Establish the mapping relations of virtual address and described address information that the source data is locally stored；
During the isomerization hardware runs described program when invocation target data, sends target data to the Host and ask
It asks, the target data request carries the address information of the target data, in order to which the Host is according to the target data
Address information search the target data, the address information of the target data is that the isomerization hardware is closed according to the mapping
The virtual address of system and the target data determines that the target data belongs to the source data；
After receiving the target data that the Host is replied, described program is run using the target data.
2. according to the method described in claim 1, it is characterized in that, when the source data is stored in the memory of the Host,
The address information of the source data is the physical address that the Host stores the source data；
When the source data is stored in the disk of the Host, the address information of the source data is described in Host storages
The virtual address of source data.
3. according to the method described in claim 1, it is characterized in that, it is described using the target data operation described program it
Before, the method further includes：
The target data is stored in the memory of the isomerization hardware；
Mapping relations between the virtual address that the target data is locally stored and the address information of the target data are repaiied
It is changed to that the virtual address of the target data and target data physical address in the isomerization hardware memory is locally stored
Between mapping relations.
4. according to the method described in claim 3, it is characterized in that, when the source data is stored in the disk of the Host,
It is described the target data is stored in the memory of the isomerization hardware before, the method further includes：
Receive the physical address that the target data that the Host is replied stores in the memory of the Host, it is described physically
Location is that the Host is stored in the physical address in the memory of the Host after transferring the target data in disk；
Mapping relations between the virtual address that the target data is locally stored and the address information of the target data are repaiied
It is changed to that reflecting between the virtual address of the target data and the physical address stored in the memory of the Host is locally stored
Penetrate relationship；
It is asked to the Host transmitting portions target data, the partial target request of data carries partial target data described
The physical address stored in the memory of Host, in order to the Host according to the partial target data the Host memory
The physical address of middle storage searches the partial target data；
It is then described after receiving the target data that the Host is replied, include using target data operation described program：
After receiving the partial target data that the Host is replied, the partial target data run described program is used.
5. according to the method described in claim 1, it is characterized in that, described when the isomerization hardware runs described program process
When middle invocation target data, before being asked to the Host transmission datas, the method further includes：
It adds and identifies for described address information, the mark indicates that described address information is that the data store in the Host
Address information.
6. the device that big data is handled in a kind of isomerization hardware, which is characterized in that including：
Receiving unit, program and address information for receiving Host transmissions, described program is the journey run on isomerization hardware
Sequence, described address information are the address information that source data stores in the Host, and the source data is on the isomerization hardware
Run the data needed for described program；
Unit is established, the mapping relations for establishing virtual address and described address information that the source data is locally stored；
Request unit, for when invocation target data, being sent out to the Host during running described program when the isomerization hardware
Send target data to ask, target data request carries the address information of the target data, in order to the Host according to
The address information of the target data searches the target data, and the address information of the target data is the isomerization hardware root
It is determined according to the virtual address of the mapping relations and the target data, the target data belongs to the source data；
Using unit, for after receiving the target data that the Host is replied, described program to be run using the target data.
7. device according to claim 6, which is characterized in that when the source data is stored in the memory of the Host,
The address information of the source data is the physical address that the Host stores the source data；
When the source data is stored in the disk of the Host, the address information of the source data is described in Host storages
The virtual address of source data.
8. device according to claim 6, which is characterized in that described device further includes：
Storage unit, for the target data to be stored in the memory of the isomerization hardware；
Unit is changed, between the virtual address and the address information of the target data for the target data will to be locally stored
Mapping relations be revised as being locally stored the virtual address of the target data with the target data in the isomerization hardware
Deposit the mapping relations between physical address.
9. device according to claim 8, which is characterized in that the receiving unit is additionally operable to receive what the Host was replied
The physical address that the target data stores in the memory of the Host, the physical address are that the Host is adjusted from disk
Take the physical address being stored in after the target data in the memory of the Host；
Believe the address that the modification unit is additionally operable to the virtual address and the target data that the target data will be locally stored
The virtual address that mapping relations between breath are revised as being locally stored the target data is stored in the memory of the Host
Physical address between mapping relations；
The request unit is additionally operable to ask to the Host transmitting portions target data, and the partial target request of data carries
The physical address that partial target data store in the memory of the Host, in order to which the Host is according to the partial target number
The partial target data are searched according to the physical address stored in the memory of the Host；
It is described to be additionally operable to after receiving the partial target data that the Host is replied using unit, use the partial target data
Run described program.
10. device according to claim 6, which is characterized in that described device further includes：
Adding device, for being identified for the addition of described address information, the mark indicates that described address information is that the data exist
The address information stored in the Host.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410608826.6A CN105630690B (en) | 2014-10-31 | 2014-10-31 | The method and device that big data is handled in isomerization hardware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410608826.6A CN105630690B (en) | 2014-10-31 | 2014-10-31 | The method and device that big data is handled in isomerization hardware |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105630690A CN105630690A (en) | 2016-06-01 |
| CN105630690B true CN105630690B (en) | 2018-10-12 |
Family
ID=56045659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410608826.6A Active CN105630690B (en) | 2014-10-31 | 2014-10-31 | The method and device that big data is handled in isomerization hardware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105630690B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110737608B (en) * | 2018-07-03 | 2024-03-15 | 阿里巴巴集团控股有限公司 | Data operation method, device and system |
| CN112306718B (en) * | 2020-11-18 | 2023-02-28 | 山东云海国创云计算装备产业创新中心有限公司 | Communication method, system and related device between local equipment and heterogeneous equipment |
| CN115454681B (en) * | 2022-11-10 | 2023-01-20 | 维塔科技(北京)有限公司 | Batch processing program execution method, device and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5907864A (en) * | 1995-06-07 | 1999-05-25 | Texas Instruments Incorporated | Data processing device with time-multiplexed memory bus |
| CN101013404A (en) * | 2007-01-26 | 2007-08-08 | 浙江大学 | Heterogeneous multi-core system-oriented management method of paging memory space |
| CN101430651A (en) * | 2007-11-05 | 2009-05-13 | 中兴通讯股份有限公司 | Access method for peripheral devices in heterogeneous multi-core system |
-
2014
- 2014-10-31 CN CN201410608826.6A patent/CN105630690B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5907864A (en) * | 1995-06-07 | 1999-05-25 | Texas Instruments Incorporated | Data processing device with time-multiplexed memory bus |
| CN101013404A (en) * | 2007-01-26 | 2007-08-08 | 浙江大学 | Heterogeneous multi-core system-oriented management method of paging memory space |
| CN101430651A (en) * | 2007-11-05 | 2009-05-13 | 中兴通讯股份有限公司 | Access method for peripheral devices in heterogeneous multi-core system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105630690A (en) | 2016-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104679669B (en) | The method of cache cache accumulator systems and access cache row cache line | |
| CN103544269B (en) | Methods and node controllers for storing and enquiring directories | |
| US7870536B2 (en) | Computer implemented method and system for sharing resources among hierarchical containers of resources | |
| US20200364106A1 (en) | Storage system with coordinated recovery across multiple input-output journals of different types | |
| CN103559319B (en) | The cache synchronization method and equipment of distributed cluster file system | |
| CN105630690B (en) | The method and device that big data is handled in isomerization hardware | |
| CN109491928A (en) | Buffer control method, device, terminal and storage medium | |
| CN105408895A (en) | Latch-free, log-structured storage for multiple access methods | |
| JP2001526815A (en) | File system primitives that provide native file system support for remote storage | |
| CN102693164B (en) | Equipment and method for preventing buffer overflow | |
| CN103885894A (en) | Information Coherency Maintenance Systems And Methods | |
| CN102141925A (en) | Applicability detection using third party target state | |
| CN110019112A (en) | Data transactions method, apparatus and electronic equipment | |
| CN109656886B (en) | Key value pair-based file system implementation method, device, equipment and storage medium | |
| CN108595503A (en) | Document handling method and server | |
| CN103677878A (en) | Method and device for patching | |
| CN105049530A (en) | Adaption device and method for plurality of distributed cache systems | |
| CN106250061A (en) | File download processing method and processing device | |
| CN105446889B (en) | A kind of EMS memory management process, device and Memory Controller Hub | |
| CN109656889A (en) | File system implementation method, device, equipment and storage medium based on block chain | |
| CN105701020B (en) | A kind of method of internal storage access, relevant apparatus and system | |
| CN106649146A (en) | Memory release method and apparatus | |
| WO2015130282A1 (en) | Communication between integrated graphics processing units | |
| JP2004362570A (en) | Shadow paging | |
| US7653797B1 (en) | Optimizing a marking phase in mark-sweep garbage collectors by reducing paging activity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |