CN115421804B - Data management method, system and device based on KPU unified interface - Google Patents
Data management method, system and device based on KPU unified interface Download PDFInfo
- Publication number
- CN115421804B CN115421804B CN202210908007.8A CN202210908007A CN115421804B CN 115421804 B CN115421804 B CN 115421804B CN 202210908007 A CN202210908007 A CN 202210908007A CN 115421804 B CN115421804 B CN 115421804B
- Authority
- CN
- China
- Prior art keywords
- data
- kpu
- management module
- virtual address
- hardware
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44594—Unloading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/10—Program control for peripheral devices
- G06F13/102—Program control for peripheral devices where the programme performs an interfacing function, e.g. device driver
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/40—Bus structure
- G06F13/4063—Device-to-bus coupling
- G06F13/4068—Electrical coupling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F2213/0026—PCI express
Abstract
The invention provides a KPU unified interface-based data management method, a KPU unified interface-based data management system and a KPU unified interface-based data management device.A plurality of drivers are loaded through a data synchronization module to connect and call KPU hardware devices of different types or functions, a virtual address management module is used for storing a Hash table to inquire data attribute pointers corresponding to different KPU hardware devices, data storage standard information in a data attribute management module is called according to the data attribute pointers, and data required by corresponding KPU hardware devices are inquired and stored according to the data storage standard information. And the data in the memory of the host computer is downloaded to the corresponding KPU hardware equipment for operation through a uniform interface of the data synchronization module, so that the system performance is improved.
Description
Technical Field
The invention relates to the technical field of data management, in particular to a data management method, a system and a device based on a KPU unified interface.
Background
The KPU (Neural Network Processor) board, namely a general Neural Network Processor and a KPU-Conflux database acceleration product, unloads the operation intensive in calculation in the relational database to an acceleration board card for execution, thereby solving the operation pressure and bottleneck of a CPU (Processor) under the low-delay and high-throughput scene, and greatly improving the overall performance and throughput of the system. KPU hardware equipment can adopt FPGA (Field Programmable Gate Array), and data query software calls a data pointer to access and control a data storage unit (namely a board memory) of hardware through PCIe (peripheral component interconnect Express) driving, so as to realize the management work of data on a hardware board, and PCIe (PCI Express) is a higher-speed serial communication system standard constructed by using the existing PCI programming concept and signal standard.
Firstly, each data query software needs to develop a hardware memory management device adaptive to the KPU board card, and the device is complex in design and implementation, high in technical barrier and unfriendly to application layer developers; secondly, the memory resource of the KPU board card is more limited relative to the memory resource of the host, namely more precious; the problem is how to more efficiently store the data of the current query calculation by using the limited memory resources of the board card; thirdly, in a scenario where multiple users use the KPU boards to perform data query at the same time (i.e., a multi-user usage scenario), it is likely that data is written in at the same location at the same time, and the data is read at the same time, which cannot ensure the accuracy, security, and uniqueness of the data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data management method, system and apparatus based on a KPU unified interface, so as to eliminate or improve one or more defects in the prior art, and solve the problem of unreasonable application of the memory resource of the KPU board card.
The technical scheme of the invention is as follows:
in one aspect, the present invention provides a data management method based on KPU unified interface, where the method is used to run on a unified data management module, where the unified data management module includes a data virtual address management module, a data attribute management module, and a data synchronization module, where the data virtual address management module is connected to the data attribute management module and the data synchronization module, and the data synchronization module loads multiple drivers and is connected to multiple KPU hardware devices through a hardware management module, and the method includes:
receiving a call demand of a preset KPU hardware device, requesting to create a virtual address identifier as a main key from the virtual address management module according to the call demand, creating a data attribute pointer of the preset KPU hardware device as a value, storing the main key and the value into a hash table, and receiving the virtual address identifier returned by the virtual address management module; the data attribute pointer points to data storage standard information recorded in the data attribute management module, and the data storage standard information at least comprises a host memory storage address and a data offset;
receiving a data writing request aiming at the preset KPU hardware equipment, controlling the virtual address management module to inquire the hash table according to the virtual address identifier, and acquiring the data attribute pointer corresponding to the preset KPU hardware equipment; inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and writing a target data value into a host memory according to the data storage standard information;
receiving a calculation call request aiming at the preset KPU hardware equipment, and controlling the virtual address management module to inquire the hash table according to the virtual address identifier to acquire the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and extracting the target data value from the host memory according to the data storage standard information;
and sending the target data value to the preset KPU hardware equipment for operation through the data synchronization module and the hardware management module.
In some embodiments, after the sending the target data value to the preset KPU hardware device for operation by the data synchronization module and the hardware management module, the method further includes:
controlling the virtual address management module to acquire the virtual address identifier corresponding to the preset KPU hardware equipment;
inquiring the hash table according to the virtual address identifier to obtain the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
and synchronously writing the operation result data into the main machine memory according to the data storage standard information through the data synchronization module and the hardware management module.
In some embodiments, the method further comprises:
when the target data value in the host memory reaches a set condition, controlling the virtual address management module to query the hash table according to the virtual address identifier to acquire the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
releasing and clearing the virtual address identifier, the data attribute pointer, the data storage standard information and the target data value;
wherein the setting condition includes: the target data value is not called when the set duration is reached; or, a clear instruction for the target data value is received.
In some embodiments, the method further comprises:
when the memory resource of the preset KPU hardware equipment is exhausted, controlling the virtual address management module to query the hash table according to the virtual address identifier to acquire the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
and the data synchronization module and the hardware management module transfer all data in the KPU hardware equipment to the host memory according to the data storage standard information.
In some embodiments, the method further comprises:
when the memory of the host computer is exhausted, randomly selecting a virtual address identifier to be optimized in the virtual address management module according to a set proportion, inquiring the hash table according to the virtual address identifier to be optimized, and acquiring a data attribute pointer to be optimized; inquiring corresponding data storage standard information to be optimized in the data attribute management module according to the data attribute pointer to be optimized; the data storage standard information to be optimized at least comprises a host memory storage address and a data offset of the data to be optimized;
and extracting the data to be optimized in the memory of the host according to the data storage standard information to be optimized, transferring the data to be optimized to a hard disk, and releasing and clearing the data storage standard information to be optimized.
In some embodiments, after the data synchronization module and the hardware management module synchronously write the operation result data into the host memory according to the data storage standard information, the method further includes: and acquiring a synchronous writing result of the operation result data, and writing the synchronous writing result into a data synchronous log.
In some embodiments, after the data synchronization module and the hardware management module transfer all data in the KPU hardware device to the host memory according to the data storage standard information, the method further includes:
and acquiring a data unloading result and writing the data unloading result into a data unloading log.
On the other hand, the invention also provides a data management system based on the KPU unified interface, which comprises:
the unified data management module comprises a data virtual address management module, a data attribute management module and a data synchronization module; the data virtual address management module is connected with the data attribute management module and the data synchronization module; the unified data management module is used for executing the data management method based on the KPU unified interface;
the data synchronization module is connected with a plurality of KPU hardware devices through a hardware management module, and the data synchronization module, the hardware management module and each KPU hardware device are connected through a PCIe bus;
and the data synchronization module calls each KPU hardware device based on a universal interface by loading various drivers.
In another aspect, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.
In another aspect, the present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.
The invention has the beneficial effects that:
the invention relates to a data management method, a system and a device based on a KPU unified interface, which loads a plurality of drivers to connect and call KPU hardware devices with different types or functions through a data synchronization module, utilizes a virtual address management module to store a Hash table to inquire data attribute pointers corresponding to different KPU hardware devices, calls data storage standard information in a data attribute management module according to the data attribute pointers, and inquires and stores data required by corresponding KPU hardware devices according to the data storage standard information. And the data in the memory of the host computer is downloaded to the corresponding KPU hardware equipment for operation through a uniform interface of the data synchronization module, so that the system performance is improved.
Furthermore, data in the KPU hardware equipment is synchronously stored to a host memory, and the memory of the KPU hardware equipment is cleared, so that the storage resources of the KPU hardware equipment are efficiently utilized.
Furthermore, the data management method based on the unified data management structure can adapt to the data management scene of heterogeneous computing.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present invention are not limited to the specific details set forth above, and that these and other objects that can be achieved with the present invention will be more clearly understood from the detailed description that follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
fig. 1 is a schematic structural diagram of a data management system based on KPU unified interface according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating a data management method based on KPU unified interface according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and/or processing steps closely related to the scheme according to the present invention are shown in the drawings, and other details not so relevant to the present invention are omitted.
It should be emphasized that the term "comprises/comprising" when used herein
Refers to the presence of features, elements, steps or components but does not preclude the presence or addition of one or more other features, elements, steps or components.
It is also noted herein that the term "coupled," if not specifically stated, may refer herein to not only a direct connection, but also an indirect connection in which an intermediate is present.
In order to adapt to KPU hardware equipment with different types and functions, reasonably utilize the resources of the KPU hardware equipment and optimize the application process of the KPU hardware equipment, the invention provides a data management method, a system and a device based on a KPU unified interface, which improve the usability of a hardware board card for data management and reduce the difficulty of using the hardware board card by data query software, thereby improving the software development efficiency, reducing the development cost and ensuring the uniqueness of data. Meanwhile, limited board card memory resources can be recycled to a certain degree, the utilization rate of the board card memory resources is optimized, and the method is suitable for various big data query and operation scenes.
In one aspect, the present invention provides a data management method based on KPU unified interfaces, as shown in fig. 1, the method is configured to operate on a unified data management module, where the unified data management module includes a data virtual address management module, a data attribute management module, and a data synchronization module, the data virtual address management module is connected to the data attribute management module and the data synchronization module, and the data synchronization module is loaded with multiple drivers and connected to multiple KPU hardware devices through a hardware management module.
It should be noted that the data virtual address management module and the data attribute management module may adopt electronic devices capable of storing and executing computer programs, such as a processor and a single chip, the data synchronization module calls KPU hardware devices of different types and functions by loading multiple PCIe drivers, and the data synchronization module connects multiple KPU hardware devices through the hardware management module by using a general interface. And the hardware management module records and calls physical addresses of different KPU hardware equipment to assist in data transmission.
As shown in fig. 2, the method includes steps S101 to S104:
step S101: receiving a calling demand for a preset KPU hardware device, requesting a virtual address identifier to be created as a main key from a virtual address management module according to the calling demand, creating a data attribute pointer of the preset KPU hardware device as a value, storing the main key and the value into a hash table, and receiving the virtual address identifier returned by the virtual address management module; the data attribute pointer points to data storage standard information recorded in the data attribute management module, and the data storage standard information at least comprises a host memory storage address and a data offset.
Step S102: receiving a data writing request aiming at the preset KPU hardware equipment, and controlling a virtual address management module to inquire a hash table according to a virtual address identifier to obtain a data attribute pointer corresponding to the preset KPU hardware equipment; and inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and writing the target data value into the host memory according to the data storage standard information.
Step S103: receiving a calculation calling request aiming at the preset KPU hardware equipment, and controlling a virtual address management module to inquire a hash table according to a virtual address identifier to obtain a data attribute pointer; and inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and extracting a target data value from the host memory according to the data storage standard information.
Step S104: and sending the target data value to preset KPU hardware equipment for operation through a data synchronization module and a hardware management module.
In step S101, the preset KPU hardware device refers to one of the KPU hardware devices in the system, and in the application process, the call request for the preset KPU hardware device may be obtained by a user directly processing the KPU hardware device in the unified data management module, or may be obtained by receiving an external software and hardware request. Because the functions and hardware settings of different KPU hardware devices are different, the KPU hardware devices operate and execute different tasks based on different drivers, and the adopted data forms are different. In order to realize efficient calling of corresponding KPU hardware devices and writing-in and writing-out of data, data storage standard information of each KPU hardware device needs to be recorded in detail, and the data storage standard information may include information such as addresses, data offsets, data length requirements, format requirements, and encryption forms in a host.
And setting a unique virtual address identifier as a main key and a data attribute pointer as a value for the preset KPU hardware equipment, and storing the main key and the value into a Hash table for memorability query. The data attribute pointer points to the data storage standard information recorded in the data attribute management module and is directly called after being inquired.
In step S102, for the data to be processed, it is first required to write the data into the host memory according to the specified storage format for calling. And inquiring corresponding virtual address identification in a hash table by a virtual address management module according to a data writing request of the preset KPU hardware equipment to obtain a data attribute pointer. And searching data storage standard information through a data attribute pointer to write data.
In step S103 and step S104, according to a method similar to that in step S102, a target data value corresponding to the preset KPU hardware device is called, and after the target data value is synchronized with the preset KPU hardware device by the data synchronization module and the hardware management module and processed, the target data value is fed back for synchronization, so that the calling of the preset KPU hardware device is completed.
In some embodiments, after step S104, that is, after the data synchronization module and the hardware management module send the target data value to the preset KPU hardware device for operation, the method further includes steps S1041 to S1043:
step S1041: and controlling the virtual address management module to acquire a virtual address identifier corresponding to the preset KPU hardware equipment.
Step S1042: inquiring a hash table according to the virtual address identifier to obtain a data attribute pointer; and inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer.
Step S1043: and synchronously writing the operation result data into the main machine memory according to the data storage standard information through the data synchronization module and the hardware management module.
In steps S1041 to S1043, the operation result data obtained after the KPU hardware device is preset to execute the operation task is synchronously written into the host memory for storage. Under the condition that the memory resources of the KPU hardware equipment are enough, the operation result data can be synchronized to the memory of the host computer in a copying mode; under the condition that the memory resources of the preset KPU hardware equipment are insufficient, after the operation result data are synchronized to the main machine memory, the operation result data in the preset KPU hardware equipment can be cleared.
In some embodiments, after the data synchronization module and the hardware management module synchronously write the operation result data into the host memory according to the data storage standard information, the method further includes: and acquiring a synchronous writing result of the operation result data, and writing the synchronous writing result into a data synchronous log.
In some embodiments, the method further comprises steps S105 and S106:
step S105: when a target data value in the memory of the host reaches a set condition, controlling a virtual address management module to query a hash table according to a virtual address identifier to obtain a data attribute pointer; and inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer.
Step S106: and releasing and clearing the virtual address identifier, the data attribute pointer, the data storage standard information and the target data value.
Wherein, the setting conditions include: the target data value is not called when the set duration is reached; alternatively, a clear instruction for the target data value is received.
In steps S105 and S106, the target data values that need to be submitted to the preset KPU hardware devices for processing can be cleared if left unprocessed for a long time, and in the other case, the relevant data can be cleared if the corresponding preset KPU hardware devices have been unloaded.
In some embodiments, the method further comprises step S107 and step S108:
step S107: when the memory resources of the preset KPU hardware equipment are exhausted, controlling a virtual address management module to query a hash table according to the virtual address identifier to obtain a data attribute pointer; and querying corresponding data storage standard information in the data attribute management module according to the data attribute pointer.
Step S108: and the data synchronization module and the hardware management module transfer all data in the KPU hardware equipment to a host memory according to the data storage standard information.
In steps S107 and S108, the memory resources of the preset KPU hardware device are managed, and when the memory resources of the preset KPU hardware device are exhausted, data in the memory of the preset KPU hardware device is synchronized to the host memory in time, so that the working state of the preset KPU hardware device can be ensured.
In some embodiments, after the data synchronization module and the hardware management module transfer all data in the KPU hardware device to the host memory according to the data storage standard information, the method further includes: and acquiring a data unloading result and writing the data unloading result into a data unloading log.
In some embodiments, the method further comprises step S109 and step S110:
step S109: when the memory of the host is exhausted, randomly selecting a virtual address identifier to be optimized in the virtual address management module according to a set proportion, and inquiring a hash table according to the virtual address identifier to be optimized to obtain a data attribute pointer to be optimized; inquiring corresponding data storage standard information to be optimized in the data attribute management module according to the data attribute pointer to be optimized; the data storage standard information to be optimized at least comprises a host memory storage address and a data offset of the data to be optimized.
Step S110: and extracting the data to be optimized in the memory of the host according to the data storage standard information to be optimized, transferring the data to be optimized to the hard disk, and releasing and clearing the data storage standard information to be optimized.
In step S109 and step S110, the memory resource is compared with the hard disk resource, the memory provides an operating space for the system and the software, and the hard disk provides a storage space for the system and the software; the memory space has an upper limit, generally supports 64G, the hard disk space has no upper limit, and the memory space is connected in series to form a group and can exceed 10T; the hard disk is a read-only memory, the data stored after power failure cannot be lost, the memory is a random access memory, and the data can be lost after power failure. Therefore, in this embodiment, the data in the memory of the host is transferred to the hard disk in time, which can ensure data security and optimize resource allocation.
On the other hand, the invention also provides a data management system based on the KPU unified interface, which comprises:
the unified data management module comprises a data virtual address management module, a data attribute management module and a data synchronization module; the data virtual address management module is connected with the data attribute management module and the data synchronization module; the unified data management module is used for executing the data management method based on the KPU unified interface.
The data synchronization module is connected with a plurality of KPU hardware devices through a hardware management module, and the data synchronization module, the hardware management module and each KPU hardware device are connected through a PCIe bus.
The data synchronization module calls each KPU hardware device based on a universal interface by loading various drivers.
The data virtual address management module and the data attribute management module can adopt electronic equipment which can store and execute computer programs, such as a processor, a single chip microcomputer and the like, the data synchronization module calls KPU hardware equipment with different types and functions by loading various PCIe drivers, and the data synchronization module is connected with the KPU hardware equipment by a universal interface through the hardware management module. And the hardware management module records and calls physical addresses of different KPU hardware equipment to assist in data transmission.
In another aspect, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.
In another aspect, the present invention also provides a computer readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the above-mentioned method.
The invention is illustrated below with reference to a specific example:
the data management method based on the KPU unified interface provided in this embodiment is a method for managing data through a virtual address. The system mainly depends on a unified data management module to operate, wherein the unified data management module comprises three modules, a data attribute management module, a data synchronization module and a data virtual address management module, and the data synchronization module is loaded with various drives and is connected with a plurality of KPU hardware devices through a hardware management module.
The data attribute management module is used for recording basic information of the data, and may include a host memory address of the data, a board memory address of the data, a host hardware address of the data, a byte length of the data, a data type, and the like. The module provides simple and easy-to-use function interfaces which can support reading and writing of attribute information of data for the virtual address management module of the data.
The data synchronization module is a pluggable, configurable module. Corresponding data synchronization realizing devices can be provided for KPU hardware equipment of different models and versions, so that data query analysis software can pay more attention to how to use data rather than how to operate the KPU hardware equipment. The module completes the synchronous work of data and a hardware board card through a PCIe drive, and provides a function interface for a data virtual address management module to use.
The virtual data address management module is a core module of the data management method of the whole unified interface, and provides a function interface for data query software to call.
In the data virtual address management module, a virtual address identifier is used as a primary key, and a data attribute pointer is used as a value to form a hash table (the hash table is also called as a hash table, and is a basic computer software data structure).
The data query analysis software runs on the unified data management module, when the data query analysis software needs to call KPU hardware equipment for calculation, the data query management software calls a data virtual address management module to apply for data types supported by certain KPU hardware equipment and line numbers of the type of data, the data virtual address management module can create a corresponding data attribute example according to data attribute information input by the data query software, namely the data storage standard information, and simultaneously create a new data virtual address ID and a data attribute pointer, the newly created data virtual address ID is used as a main key, the data attribute pointer is used as a value to be stored in a hash table, the data attribute example is stored in the data attribute management module, the data attribute pointer points to the data attribute example, and the newly created data virtual address ID is returned to the data query software on the upper layer for future use.
When data query software needs to write data, submitting the data virtual address ID to a data virtual address management module; the data virtual address management module searches a data attribute pointer corresponding to the data virtual address ID in the hash table, calls a data attribute instance pointed by the data attribute pointer, and writes a data value according to information such as the address and the data offset of data in a host memory in the data attribute instance.
When data query software needs data calculation, the data virtual address management module synchronizes data to the KPU hardware board card for calculation through the data synchronization module. When the data calculation is completed, the calculation result is synchronized with the host memory through the data synchronization module.
When the data inquiry software no longer needs the applied data virtual address, the applied data virtual address ID is released by calling the data virtual address management module. When the data virtual address module receives a data virtual address release request, the data attribute pointer is found by searching the hash table, the data attribute instance is inquired according to the data attribute pointer, and the corresponding host machine memory resource, the hard disk resource and/or the KPU hardware equipment memory resource are released.
In some embodiments, when the memory resources of the KPU hardware devices are exhausted, the data resources on all the KPU hardware devices are synchronized to the host, and all the board memory resources are released at one time.
In some embodiments, when the host memory resource is exhausted, 1/4 of the virtual addresses in the data virtual address management module are randomly selected, the data attribute pointer is read through the hash table, the data attribute instance is queried according to the data attribute pointer, data corresponding to the data attribute is written into the hard disk, and the host memory pointer is released.
In summary, the KPU unified interface-based data management method, system and apparatus of the present invention loads various drivers through the data synchronization module to connect and call KPU hardware devices of different types or functions, uses the virtual address management module to store the hash table to query the data attribute pointers corresponding to different KPU hardware devices, retrieves the data storage standard information in the data attribute management module according to the data attribute pointers, and queries and stores the data required by the corresponding KPU hardware devices according to the data storage standard information. And the data in the memory of the host computer is downloaded to the corresponding KPU hardware equipment for operation through a uniform interface of the data synchronization module, so that the system performance is improved.
Furthermore, data in the KPU hardware equipment is synchronously stored to a host memory, and the memory of the KPU hardware equipment is cleared, so that the storage resources of the KPU hardware equipment are efficiently utilized.
Furthermore, the data management method based on the unified data management structure can adapt to the data management scene of heterogeneous computing.
Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein may be implemented as hardware, software, or combinations thereof. Whether this is done in hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
Features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments in the present invention.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes may be made to the embodiment of the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A KPU unified interface based data management method, for running on a unified data management module, said unified data management module comprising a data virtual address management module, a data attribute management module and a data synchronization module, said data virtual address management module connecting said data attribute management module and said data synchronization module, said data synchronization module carrying multiple drivers and connecting multiple KPU hardware devices through a hardware management module, said method comprising:
receiving a call demand of a preset KPU hardware device, requesting to create a virtual address identifier as a main key from the virtual address management module according to the call demand, creating a data attribute pointer of the preset KPU hardware device as a value, storing the main key and the value into a hash table, and receiving the virtual address identifier returned by the virtual address management module; the data attribute pointer points to data storage standard information recorded in the data attribute management module, and the data storage standard information at least comprises a host memory storage address and a data offset;
receiving a data writing request aiming at the preset KPU hardware equipment, controlling the virtual address management module to inquire the hash table according to the virtual address identifier, and acquiring the data attribute pointer corresponding to the preset KPU hardware equipment; inquiring corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and writing a target data value into a host memory according to the data storage standard information;
receiving a calculation call request aiming at the preset KPU hardware equipment, and controlling the virtual address management module to inquire the hash table according to the virtual address identifier to acquire the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer, and extracting the target data value from the host memory according to the data storage standard information;
and sending the target data value to the preset KPU hardware equipment for operation through the data synchronization module and the hardware management module.
2. A KPU unified interface based data management method according to claim 1, wherein after sending the target data value to the preset KPU hardware device for operation by means of a data synchronization module and the hardware management module, further comprising:
controlling the virtual address management module to acquire the virtual address identifier corresponding to the preset KPU hardware equipment;
inquiring the hash table according to the virtual address identifier to obtain the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
and synchronously writing the operation result data into the memory of the host computer according to the data storage standard information through the data synchronization module and the hardware management module.
3. A KPU unified interface based data management method according to claim 1, characterized in that said method further comprises:
when the target data value in the host memory reaches a set condition, controlling the virtual address management module to query the hash table according to the virtual address identifier to acquire the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
releasing and clearing the virtual address identifier, the data attribute pointer, the data storage standard information and the target data value;
wherein the setting condition includes: the target data value is not called when the set duration is reached; or, a clear instruction for the target data value is received.
4. A KPU unified interface based data management method according to claim 1, characterized in that said method further comprises:
when the memory resource of the preset KPU hardware equipment is exhausted, controlling the virtual address management module to inquire the hash table according to the virtual address identifier to obtain the data attribute pointer; inquiring the corresponding data storage standard information in the data attribute management module according to the data attribute pointer;
and the data synchronization module and the hardware management module transfer all data in the KPU hardware equipment to the host memory according to the data storage standard information.
5. A KPU unified interface based data management method according to claim 1, characterized in that said method further comprises:
when the memory of the host computer is exhausted, randomly selecting a virtual address identifier to be optimized in the virtual address management module according to a set proportion, inquiring the hash table according to the virtual address identifier to be optimized, and acquiring a data attribute pointer to be optimized; inquiring corresponding data storage standard information to be optimized in the data attribute management module according to the data attribute pointer to be optimized; the data storage standard information to be optimized at least comprises a host memory storage address and a data offset of the data to be optimized;
and extracting the data to be optimized in the memory of the host according to the data storage standard information to be optimized, transferring the data to be optimized to a hard disk, and releasing and clearing the data storage standard information to be optimized.
6. A KPU unified interface based data management method according to claim 2, wherein after the operation result data is synchronously written into the host memory according to the data storage standard information by the data synchronization module and the hardware management module, further comprising:
and acquiring a synchronous writing result of the operation result data, and writing the synchronous writing result into a data synchronous log.
7. A KPU unified interface based data management method according to claim 4, wherein after said data synchronization module and said hardware management module dump all data in said KPU hardware device into said host memory according to said data storage standard information, further comprising:
and acquiring a data unloading result and writing the data unloading result into a data unloading log.
8. A data management system based on KPU unified interface, characterized in that, includes:
the unified data management module comprises a data virtual address management module, a data attribute management module and a data synchronization module; the data virtual address management module is connected with the data attribute management module and the data synchronization module; the unified data management module is used for executing the KPU unified interface-based data management method according to any one of claims 1 to 7;
the data synchronization module is connected with a plurality of KPU hardware devices through a hardware management module, and the data synchronization module, the hardware management module and each KPU hardware device are connected through a PCIe bus;
and the data synchronization module calls each KPU hardware device based on a universal interface by loading various drivers.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method according to any of claims 1 to 7 are implemented when the processor executes the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210908007.8A CN115421804B (en) | 2022-07-29 | 2022-07-29 | Data management method, system and device based on KPU unified interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210908007.8A CN115421804B (en) | 2022-07-29 | 2022-07-29 | Data management method, system and device based on KPU unified interface |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115421804A CN115421804A (en) | 2022-12-02 |
| CN115421804B true CN115421804B (en) | 2023-02-24 |
Family
ID=84197335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210908007.8A Active CN115421804B (en) | 2022-07-29 | 2022-07-29 | Data management method, system and device based on KPU unified interface |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115421804B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103425700A (en) * | 2012-05-25 | 2013-12-04 | 中广核（北京）仿真技术有限公司 | Data storage, inquiry and updating method for nuclear power plant analog machine |
| CN107770076A (en) * | 2016-08-23 | 2018-03-06 | 中兴通讯股份有限公司 | A kind of processing method of hash-collision, device and switching equipment |
| EP3591563A1 (en) * | 2018-07-04 | 2020-01-08 | Koninklijke Philips N.V. | Verifying the integrity of a plurality of data blocks |
| CN112269784A (en) * | 2020-10-28 | 2021-01-26 | 中科驭数（北京）科技有限公司 | Hash table structure based on hardware realization and inserting, inquiring and deleting method |
| CN112632069A (en) * | 2020-12-22 | 2021-04-09 | 中科驭数（北京）科技有限公司 | Hash table data storage management method, device, medium and electronic equipment |
-
2022
- 2022-07-29 CN CN202210908007.8A patent/CN115421804B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103425700A (en) * | 2012-05-25 | 2013-12-04 | 中广核（北京）仿真技术有限公司 | Data storage, inquiry and updating method for nuclear power plant analog machine |
| CN107770076A (en) * | 2016-08-23 | 2018-03-06 | 中兴通讯股份有限公司 | A kind of processing method of hash-collision, device and switching equipment |
| EP3591563A1 (en) * | 2018-07-04 | 2020-01-08 | Koninklijke Philips N.V. | Verifying the integrity of a plurality of data blocks |
| CN112269784A (en) * | 2020-10-28 | 2021-01-26 | 中科驭数（北京）科技有限公司 | Hash table structure based on hardware realization and inserting, inquiring and deleting method |
| CN112632069A (en) * | 2020-12-22 | 2021-04-09 | 中科驭数（北京）科技有限公司 | Hash table data storage management method, device, medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115421804A (en) | 2022-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110225078B (en) | Application service updating method, system and terminal equipment | |
| KR101856486B1 (en) | Method for bi-direction calling between open system and mainframe system | |
| CN100571160C (en) | Method for upgrading software and device and network element | |
| CN110532106B (en) | Inter-process communication method, device, equipment and storage medium | |
| CN101253500B (en) | Method for managing I/O | |
| KR20210040864A (en) | File directory traversal method, apparatus, device, and medium | |
| CN111352899B (en) | Path aggregation method, access method, communication device, and storage medium | |
| CN115421804B (en) | Data management method, system and device based on KPU unified interface | |
| CN112114983B (en) | Communication method, device and equipment based on shared memory | |
| US8442939B2 (en) | File sharing method, computer system, and job scheduler | |
| CN113177015A (en) | Frame header-based serial port communication method and serial port chip | |
| CN107003880B (en) | Kernel updating method and device and computer equipment | |
| JP2013178685A (en) | Data processing system with asynchronous backup function, front system, backup method and program therefor | |
| CN112230978A (en) | Multi-data-source dynamic switching method, electronic equipment and storage medium | |
| CN101739246A (en) | Smart card and application data processing method | |
| CN107357853B (en) | Method and device for operating REDIS console and computer system | |
| CN113934691B (en) | Method for accessing file, electronic device and readable storage medium | |
| CN113849379B (en) | Method, device, equipment and readable medium for collecting server asset information | |
| US11954069B2 (en) | Find operation for union-mounted systems | |
| CN110704249A (en) | Method, device and system for ensuring application consistency | |
| JPH05250239A (en) | Computer network system | |
| CN114443295A (en) | Heterogeneous cloud resource management scheduling method, device and system | |
| CN112950370A (en) | Service processing method, device, equipment and storage medium | |
| EP3783484A1 (en) | Data processing method and computer device | |
| CN113722147A (en) | Method for keeping service connection and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |