CN116974483A - Computable storage system and storage method based on solid-state disk - Google Patents
Computable storage system and storage method based on solid-state disk Download PDFInfo
- Publication number
- CN116974483A CN116974483A CN202310957383.0A CN202310957383A CN116974483A CN 116974483 A CN116974483 A CN 116974483A CN 202310957383 A CN202310957383 A CN 202310957383A CN 116974483 A CN116974483 A CN 116974483A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- state disk
- solid
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 127
- 238000004891 communication Methods 0.000 claims abstract description 39
- 238000004458 analytical method Methods 0.000 claims description 24
- 230000003993 interaction Effects 0.000 claims description 24
- 230000006399 behavior Effects 0.000 claims description 18
- 239000007787 solid Substances 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 7
- 230000008447 perception Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 abstract description 24
- 238000004364 calculation method Methods 0.000 abstract description 12
- 238000013500 data storage Methods 0.000 abstract description 8
- 230000008569 process Effects 0.000 abstract description 5
- 238000004590 computer program Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 11
- 238000007726 management method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a calculable storage system and a storage method based on a solid-state disk, which are characterized in that a data execution task is generated on a host side, the data execution task is transmitted to a task processing module on the solid-state disk side through a data communication module to be calculated, a data processing strategy is obtained, data corresponding to a data processing request is read and written based on the data processing strategy, and a read-write result is returned to the task generating module, so that the calculation process of data storage of a storage system in large data processing is realized, the data quantity of data transmission is reduced, the data transmission delay is reduced, meanwhile, the calculation load of the host side is reduced, the information sharing between the host side and the solid-state disk side is realized through returning the read-write result to the host side, the performance of flash memory equipment is greatly improved, and the redundancy of the data transmission of the storage system in large data processing is reduced.
Description
Technical Field
The invention relates to the technical field of big data storage, in particular to a calculable storage system and a storage method based on a solid-state disk.
Background
The variety of data application types makes the amount of data explosive. Data-intensive loads have created a need for computer systems that meet the high performance, high reliability, real-time, etc. requirements of large data processing. The large data processing system is utilized to realize the data storage, processing, utilization and distribution of the data intensive application, so that the performance of the data intensive load and the capacity of resource sharing can be effectively improved. International Data Corporation (IDC) predicts that by 2025, the global data volume will reach 175ZB (zebyte) and that nearly 90% of the data is unstructured. Under the big data environment, the research direction of the software and hardware of the prior storage technology mainly comprises the research based on nonvolatile storage media (NAND Flash, PCRAM and the like) as basic media in a storage system.
In addition to the rapid development of the mechanical hard disk technology of magnetic media in recent years, the development speed of a flash memory system in the aspects of performance and cost is higher, and a better storage technology support is provided for the rapid development of big data, which mainly comprises the following steps: solid state disks (solid state drives, SSD) based on Flash memory (NAND Flash) are widely used in storage system design, playing an important role in innovating storage architecture; SSD and flash memory system of PCIe interface, change from Fusion-io ioDriver based on primitive PCI-e agreement into NVMe agreement (NVMe: non-Volatile Memory Express) facing the flash memory device gradually, compare with SATA3 interface based on AHCI agreement traditionally, make the flash memory system have higher throughput and IOPS with their advantages such as the software stack of the light weight, low CPU overhead, direct flash memory access, etc.; the mixed memory of the flash memory and the magnetic disk enables the storage system to improve the performance of the storage system by utilizing the technology of the software and hardware collaborative design.
Although the high performance of the nonvolatile storage medium relieves the problem of data transmission delay of large data processing, the existing software and hardware system architecture based on the novel nonvolatile storage medium still has some defects, wherein the data transmission redundancy is higher, and the performance of the storage system in large data processing is seriously affected, so how to further reduce the redundancy of the data transmission of the storage system in large data processing is a problem to be solved.
Disclosure of Invention
The invention provides a calculable storage system and a storage method based on a solid-state disk, aiming at the technical problems in the prior art, and aims to solve the problem of how to further reduce the redundancy of data transmission of a storage system in large data processing.
In a first aspect of the present invention, there is provided a solid state disk based computable storage system comprising: the system comprises a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module;
the user interaction module is arranged at the host side and is used for analyzing a data processing request sent by a user and sending a data operation instruction corresponding to an analysis result to the task generation module;
the task generating module is arranged on the host side and is used for selecting a task executing end based on the data operation instruction and generating a data executing task based on the task executing end and the data operation instruction;
the task processing module is arranged on the solid-state disk side and is used for acquiring the data execution task through the data communication module when the task execution end is the solid-state disk side, and calculating the data execution task to obtain a data processing strategy;
the flash memory data processing module is arranged at the side of the solid-state disk and is used for reading and writing data corresponding to the data processing request based on the data processing strategy and sending a reading and writing result to the task generating module through the data communication module.
On the basis of the technical scheme, the invention can also make the following improvements.
Preferably, the user interaction module includes: a request analysis unit and an information instruction comparison library;
the request analysis unit is used for analyzing the data processing request sent by the user, acquiring a data operation instruction corresponding to the analysis result based on the information instruction comparison library, and sending the data operation instruction to the task generation module.
Preferably, the field information of the information instruction comparison library comprises a data operation instruction, file granularity, an I/O calling strategy and the number of channels of the flash memory device.
Preferably, the task generating module includes: the device comprises a task generating unit, a device I/O load analysis unit and a task unloading unit;
the task generating unit is used for generating a data execution task based on the data operation instruction;
the device I/O load analysis unit is used for acquiring the I/O load condition of the solid-state disk side based on the data execution task and rearranging the I/O task of the solid-state disk side;
and the task unloading unit is used for sending the data execution task to the solid-state disk side through the data communication module and unloading the data execution task at the host side.
Preferably, the task processing module includes: the system comprises a task acquisition unit, an I/O behavior perception unit and a strategy generation unit;
the task acquisition unit is used for acquiring the data to execute a task through the data communication module;
the I/O behavior perception unit is used for executing tasks based on the data and acquiring I/O behavior characteristics of the data execution tasks;
the policy generation unit is used for generating a data processing policy based on the I/O behavior characteristics.
Preferably, the I/O behavior feature includes an I/O data access feature and an I/O data distribution feature.
Preferably, the system further comprises a flash memory data comparison library;
the flash memory data comparison library is arranged at the side of the solid-state disk and is used for providing key value pair management for data in the solid-state disk.
In a second aspect of the present invention, a solid-state disk-based storage method is provided, and is applied to the solid-state disk-based storage system, where the system includes a user interaction module, a task generation module, a data communication module, a task processing module, and a flash memory data processing module, and the method includes:
the user interaction module at the host side analyzes a data processing request sent by a user and sends a data operation instruction corresponding to an analysis result to the task generation module;
the task generating module selects a task executing end based on the data operation instruction, and generates a data executing task based on the task executing end and the data operation instruction;
when the task processing module at the solid-state disk side is at the solid-state disk side, acquiring the data execution task through the data communication module, and calculating the data execution task to obtain a data processing strategy;
and the flash memory data processing module reads and writes the data corresponding to the data processing request based on the data processing strategy, and sends the read and write result to the task generating module through the data communication module.
In a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor for implementing the steps of the solid-state disk-based computable storage method in the above second aspect when executing a computer management class program stored in the memory.
In a fourth aspect of the present invention, there is provided a computer readable storage medium having stored thereon a computer management class program which when executed by a processor implements the steps of the solid state disk based computable storage method of the above second aspect.
The invention provides a calculable storage system and a storage method based on a solid-state disk, wherein the system comprises: the system comprises a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module; the user interaction module is arranged at the host side and is used for analyzing a data processing request sent by a user and sending a data operation instruction corresponding to an analysis result to the task generation module; the task generating module is arranged on the host side and is used for selecting a task executing end based on the data operation instruction and generating a data executing task based on the task executing end and the data operation instruction; the task processing module is arranged on the solid-state disk side and is used for acquiring the data execution task through the data communication module when the task execution end is the solid-state disk side, and calculating the data execution task to obtain a data processing strategy; the flash memory data processing module is arranged at the side of the solid-state disk and is used for reading and writing data corresponding to the data processing request based on the data processing strategy and sending a reading and writing result to the task generating module through the data communication module. The invention generates the data execution task at the host side, then sends the data execution task to the task processing module at the solid-state disk side through the data communication module for calculation to obtain the data processing strategy, reads and writes the data corresponding to the data processing request based on the data processing strategy, and returns the read and write result to the task generating module, thereby realizing the calculation process of data storage of the storage system in big data processing, migrating to the solid-state disk side, reducing the data quantity of data transmission, reducing the data transmission delay, simultaneously reducing the calculation load at the host side, and realizing the information sharing between the host side and the solid-state disk side by returning the read and write result to the host side, thereby greatly improving the performance of the flash memory device and reducing the redundancy of the data transmission of the storage system in big data processing.
Drawings
FIG. 1 is a schematic diagram of a solid state disk based computable storage system according to the present invention;
FIG. 2 is a schematic diagram of a data storage flow provided by the present invention;
FIG. 3 is a flow chart of a method for computing storage based on a solid state disk according to the present invention;
fig. 4 is a schematic hardware structure of one possible electronic device according to the present invention;
fig. 5 is a schematic hardware structure of a possible computer readable storage medium according to the present invention.
Detailed Description
The following describes in further detail the embodiments of the present invention with reference to the drawings and examples. The following examples are illustrative of the invention and are not intended to limit the scope of the invention.
FIG. 1 is a schematic diagram of a solid-state disk-based computing storage system according to the present invention, where, as shown in FIG. 1, the system includes: the system comprises a user interaction module 100, a task generation module 200, a data communication module 300, a task processing module 400 and a flash memory data processing module 500.
The user interaction module 100 is disposed at the host side, and is configured to parse a data processing request sent by a user, and send a data operation instruction corresponding to a parsing result to the task generation module; the task generating module 200 is configured to be disposed on the host side, and is configured to select a task execution end based on the data operation instruction, and generate a data execution task based on the task execution end and the data operation instruction; the task processing module 400 is configured to, when the task execution end is a solid-state disk side, obtain the data execution task through the data communication module, and calculate the data execution task to obtain a data processing policy; the flash memory data processing module 500 is disposed on the solid-state disk side, and is configured to read and write data corresponding to the data processing request based on the data processing policy, and send a read-write result to the task generating module through the data communication module.
It can be understood that the solid-state disk-based computable storage system in this embodiment is disposed on the host side and the solid-state disk side, respectively, and the host side and the solid-state disk side perform symmetric data interaction through the data communication module.
It should be understood that, because only the task instruction is transmitted between the task generating module and the task executing module, and the task executing module is arranged at the solid-state disk side, the original mode that the host side performs task execution to acquire the data in the solid-state disk first and then write the data back to the solid-state disk according to the calculation result to perform repeated data transmission is changed, and the data transmission delay caused by redundant data transmission is greatly reduced.
Further, the user interaction module includes: a request analysis unit and an information instruction comparison library; the request analysis unit is used for analyzing the data processing request sent by the user, acquiring a data operation instruction corresponding to the analysis result based on the information instruction comparison library, and sending the data operation instruction to the task generation module.
It is understood that the parsing unit is configured to receive a data processing request sent by a user, where the data processing request includes, but is not limited to, acquiring data in the solid-state disk and writing external data to the solid-state disk.
It can be understood that the information instruction comparison library is used for storing data operation instructions corresponding to different data operations, wherein field information of the information instruction comparison library includes the data operation instructions, file granularity, I/O calling strategy and flash memory device end channel number.
Furthermore, the information instruction comparison library is a key-value storage structure, only the key is required to be transmitted in the data transmission process, and the data receiver acquires the corresponding value from the information instruction comparison library after acquiring the key, so that the performance of data transmission, especially data index, is improved.
Further, the task generating module includes: the device comprises a task generating unit, a device I/O load analyzing unit and a task unloading unit.
The task generating unit is used for generating data execution tasks based on the data operation instructions; the device I/O load analysis unit is used for acquiring the I/O load condition of the solid-state disk side based on the data execution task and rearranging the I/O task of the solid-state disk side; and the task unloading unit is used for sending the data execution task to the solid-state disk side through the data communication module and unloading the data execution task at the host side.
It can be understood that the I/O load analysis unit is configured to first determine, according to the I/O load condition of the solid-state disk side, whether the data execution task can be calculated on the solid-state disk side, and when the data execution task is possible, reorder the overall I/O task on the solid-state disk side, and when the data execution task is not possible, directly calculate the data execution task. When the task can be calculated on the solid-state disk side, the data quantity transmitted by the solid-state disk side can be greatly reduced, the storage performance is improved, and meanwhile, the calculation resource on the solid-state disk side is utilized, so that the localization processing capacity of the data is realized to the maximum extent.
Further, the task processing module includes: the system comprises a task acquisition unit, an I/O behavior perception unit and a strategy generation unit.
The task acquisition unit is used for acquiring the data to execute a task through the data communication module; the I/O behavior perception unit is used for executing tasks based on the data and acquiring I/O behavior characteristics of the data execution tasks; the policy generation unit is used for generating a data processing policy based on the I/O behavior characteristics.
It is understood that the above-described I/O behavior features include I/O data access features and I/O data distribution features. The data processing strategies include data distribution and data association strategies.
It should be understood that, based on the above data processing policy, the solid-state disk side may merge the data corresponding to the above data processing request into the same spatial area, so as to change the data distribution, so that the redistribution of the associated data may avoid causing redundant data update in other data areas when the data update in the spatial area is performed, thereby reducing the redundant data update operation and improving the data storage performance.
Further, the system also comprises a flash memory data comparison library; the flash memory data comparison library is arranged at the side of the solid-state disk and is used for providing key value pair management for data in the solid-state disk.
It can be understood that, in order to further improve the effectiveness of data management and reduce the delay of operations such as data query, a key-value system (i.e. the above-mentioned flash memory data comparison library) may be further set on the solid-state disk side, so as to uniformly manage the data in the flash memory space and fully exert the computing capability of the flash memory device.
It will be appreciated that, based on the deficiencies of the prior art, embodiments of the present invention provide a solid state disk based computable storage system. The data executing task is generated on the host side, the data executing task is issued to the task processing module on the solid-state disk side through the data communication module to be calculated, a data processing strategy is obtained, data corresponding to the data processing request is read and written based on the data processing strategy, and a read-write result is returned to the task generating module, so that the calculation process of data storage of the storage system in large data processing is realized, the data is migrated to the solid-state disk side, the data quantity of data transmission is reduced, the data transmission delay is reduced, meanwhile, the calculation load on the host side is reduced, and the information sharing between the host side and the solid-state disk side is realized by returning the read-write result to the host side, thereby greatly improving the performance of the flash memory device and reducing the redundancy of the data transmission of the storage system in large data processing.
In one possible implementation, referring to fig. 2, in fig. 2, the system software and hardware framework provided in this embodiment is divided into a host-side system (host side) and a device-side system (solid-state disk side) in the framework.
The host end simplifies a file system and designs a runtime system based on the existing storage system software stack, analyzes with a user data request and converts the user data request into atomic operations required by a bottom Key-Value management layer, performs data transmission, and improves the data transmission efficiency of the host end; meanwhile, by analyzing the access behavior of the I/O load of the equipment end, the I/O behavior characteristics are obtained, and according to the information of the equipment end and the I/O behavior characteristics, an optimization method of 'I/O request queue reordering' is adopted to reorder the I/O requests to be sent to the equipment end, for example: by acquiring the parallelism information of the equipment end, the host end can reorder the request I/O when sending the request, so that the request I/O is distributed to each parallel channel to the greatest extent, redundant data operation of the equipment end is reduced, and easy operation of a flash memory space is reduced.
Further, referring to fig. 2, when a user starts to generate a data request, the host system performs data analysis, performs data conversion and generates tasks, and meanwhile, the host system performs data interaction with the device side to obtain information of the device side, analyze an IO load, reorder the IO tasks, analyze the correlation of data to be written in the tasks after the tasks reach the device side, divide flash memory data respectively, and finally execute the tasks.
In this embodiment, dynamic state information interaction is performed between the host and the device, so as to adjust a data read-write optimization strategy, thereby reducing transmission of redundant data and improving storage performance of the solid-state disk device.
Referring to fig. 3, fig. 3 is a flowchart of a solid-state disk-based calculable storage method, as shown in fig. 3, applied to the solid-state disk-based calculable storage system, where the system includes a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module, and the method includes:
step S100: the user interaction module at the host side analyzes a data processing request sent by a user and sends a data operation instruction corresponding to an analysis result to the task generation module;
step S200: the task generating module selects a task executing end based on the data operation instruction, and generates a data executing task based on the task executing end and the data operation instruction;
step S300: when the task processing module at the solid-state disk side is at the solid-state disk side, acquiring the data execution task through the data communication module, and calculating the data execution task to obtain a data processing strategy;
step S400: and the flash memory data processing module reads and writes the data corresponding to the data processing request based on the data processing strategy, and sends the read and write result to the task generating module through the data communication module.
It can be understood that the method for storing the solid-state disk-based calculable storage according to the present invention corresponds to the system for storing the solid-state disk-based calculable storage according to the foregoing embodiments, and the relevant technical features of the method for storing the solid-state disk-based calculable storage may refer to the relevant technical features of the system for storing the solid-state disk-based calculable storage, which are not described herein.
Referring to fig. 4, fig. 4 is a schematic diagram of an embodiment of an electronic device according to an embodiment of the invention. As shown in fig. 4, an embodiment of the present invention provides an electronic device including a memory 1310, a processor 1320, and a computer program 1311 stored on the memory 1310 and executable on the processor 1320, the processor 1320 implementing the following steps when executing the computer program 1311:
the user interaction module at the host side analyzes a data processing request sent by a user and sends a data operation instruction corresponding to an analysis result to the task generation module; the task generating module selects a task executing end based on the data operation instruction, and generates a data executing task based on the task executing end and the data operation instruction; when the task processing module at the solid-state disk side is at the solid-state disk side, acquiring the data execution task through the data communication module, and calculating the data execution task to obtain a data processing strategy; and the flash memory data processing module reads and writes the data corresponding to the data processing request based on the data processing strategy, and sends the read and write result to the task generating module through the data communication module.
Referring to fig. 5, fig. 5 is a schematic diagram of an embodiment of a computer readable storage medium according to the present invention. As shown in fig. 5, the present embodiment provides a computer-readable storage medium 1400 having stored thereon a computer program 1411, which computer program 1411, when executed by a processor, performs the steps of:
the user interaction module at the host side analyzes a data processing request sent by a user and sends a data operation instruction corresponding to an analysis result to the task generation module; the task generating module selects a task executing end based on the data operation instruction, and generates a data executing task based on the task executing end and the data operation instruction; when the task processing module at the solid-state disk side is at the solid-state disk side, acquiring the data execution task through the data communication module, and calculating the data execution task to obtain a data processing strategy; and the flash memory data processing module reads and writes the data corresponding to the data processing request based on the data processing strategy, and sends the read and write result to the task generating module through the data communication module.
The invention provides a calculable storage system and a storage method based on a solid-state disk, wherein the system comprises: the system comprises a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module; the user interaction module is arranged at the host side and is used for analyzing a data processing request sent by a user and sending a data operation instruction corresponding to an analysis result to the task generation module; the task generating module is arranged on the host side and is used for selecting a task executing end based on the data operation instruction and generating a data executing task based on the task executing end and the data operation instruction; the task processing module is arranged on the solid-state disk side and is used for acquiring the data execution task through the data communication module when the task execution end is the solid-state disk side, and calculating the data execution task to obtain a data processing strategy; the flash memory data processing module is arranged at the side of the solid-state disk and is used for reading and writing data corresponding to the data processing request based on the data processing strategy and sending a reading and writing result to the task generating module through the data communication module. The invention generates the data execution task at the host side, then sends the data execution task to the task processing module at the solid-state disk side through the data communication module for calculation to obtain the data processing strategy, reads and writes the data corresponding to the data processing request based on the data processing strategy, and returns the read and write result to the task generating module, thereby realizing the calculation process of data storage of the storage system in big data processing, migrating to the solid-state disk side, reducing the data quantity of data transmission, reducing the data transmission delay, simultaneously reducing the calculation load at the host side, and realizing the information sharing between the host side and the solid-state disk side by returning the read and write result to the host side, thereby greatly improving the performance of the flash memory device and reducing the redundancy of the data transmission of the storage system in big data processing.
In the foregoing embodiments, the descriptions of the embodiments are focused on, and for those portions of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (9)
1. A solid state disk-based computable storage system, the system comprising: the system comprises a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module;
the user interaction module is arranged at the host side and is used for analyzing a data processing request sent by a user and sending a data operation instruction corresponding to an analysis result to the task generation module;
the task generating module is arranged on the host side and is used for selecting a task executing end based on the data operation instruction and generating a data executing task based on the task executing end and the data operation instruction;
the task processing module is arranged on the solid-state disk side and is used for acquiring the data execution task through the data communication module when the task execution end is the solid-state disk side, and calculating the data execution task to obtain a data processing strategy;
the flash memory data processing module is arranged at the side of the solid-state disk and is used for reading and writing data corresponding to the data processing request based on the data processing strategy and sending a reading and writing result to the task generating module through the data communication module.
2. The solid state disk based computable storage system of claim 1, wherein the user interaction module comprises: a request analysis unit and an information instruction comparison library;
the request analysis unit is used for analyzing the data processing request sent by the user, acquiring a data operation instruction corresponding to the analysis result based on the information instruction comparison library, and sending the data operation instruction to the task generation module.
3. The solid state disk based computable storage system of claim 2, wherein the information instructions against the library field information includes data manipulation instructions, file granularity, I/O call policy, and flash device side channel number.
4. The solid state disk based computable storage system of claim 1, wherein the task generating module comprises: the device comprises a task generating unit, a device I/O load analysis unit and a task unloading unit;
the task generating unit is used for generating a data execution task based on the data operation instruction;
the device I/O load analysis unit is used for acquiring the I/O load condition of the solid-state disk side based on the data execution task and rearranging the I/O task of the solid-state disk side;
and the task unloading unit is used for sending the data execution task to the solid-state disk side through the data communication module and unloading the data execution task at the host side.
5. The solid state disk based computable storage system of claim 1, wherein the task processing module comprises: the system comprises a task acquisition unit, an I/O behavior perception unit and a strategy generation unit;
the task acquisition unit is used for acquiring the data to execute a task through the data communication module;
the I/O behavior perception unit is used for executing tasks based on the data and acquiring I/O behavior characteristics of the data execution tasks;
the policy generation unit is used for generating a data processing policy based on the I/O behavior characteristics.
6. The solid state disk based computable storage system of claim 5, wherein the I/O behavior features comprise I/O data access features and I/O data distribution features.
7. The solid state disk based computable storage system of claim 5, wherein the data processing policies include data distribution and data association policies.
8. The solid state disk based computable storage system of claim 1, wherein the system further comprises a flash data comparison library;
the flash memory data comparison library is arranged at the side of the solid-state disk and is used for providing key value pair management for data in the solid-state disk.
9. A solid-state disk-based computable storage method, which is applied to the solid-state disk-based computable storage system, the system comprising a user interaction module, a task generation module, a data communication module, a task processing module and a flash memory data processing module, the method comprising:
the user interaction module at the host side analyzes a data processing request sent by a user and sends a data operation instruction corresponding to an analysis result to the task generation module;
the task generating module selects a task executing end based on the data operation instruction, and generates a data executing task based on the task executing end and the data operation instruction;
when the task processing module at the solid-state disk side is at the solid-state disk side, acquiring the data execution task through the data communication module, and calculating the data execution task to obtain a data processing strategy;
and the flash memory data processing module reads and writes the data corresponding to the data processing request based on the data processing strategy, and sends the read and write result to the task generating module through the data communication module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310957383.0A CN116974483A (en) | 2023-07-31 | 2023-07-31 | Computable storage system and storage method based on solid-state disk |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310957383.0A CN116974483A (en) | 2023-07-31 | 2023-07-31 | Computable storage system and storage method based on solid-state disk |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116974483A true CN116974483A (en) | 2023-10-31 |
Family
ID=88479289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310957383.0A Pending CN116974483A (en) | 2023-07-31 | 2023-07-31 | Computable storage system and storage method based on solid-state disk |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116974483A (en) |
-
2023
- 2023-07-31 CN CN202310957383.0A patent/CN116974483A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10191662B2 (en) | Dynamic allocation of segments in a flash storage system | |
| US8521986B2 (en) | Allocating storage memory based on future file size or use estimates | |
| KR20170120489A (en) | Mechanism for ssds to efficiently manage background activity with notify | |
| He et al. | Dash: a recipe for a flash-based data intensive supercomputer | |
| US20130007488A1 (en) | Power management of a storage device including multiple processing cores | |
| JP2017021805A (en) | Interface providing method capable of utilizing data attribute reference data arrangement in nonvolatile memory device and computer device | |
| US20230124520A1 (en) | Task execution method and storage device | |
| CN103999060A (en) | Solid-state storage management | |
| CN112632069B (en) | Hash table data storage management method, device, medium and electronic equipment | |
| US20140115016A1 (en) | Systems and methods for enabling parallel processing of write transactions | |
| US10891150B2 (en) | Storage control method and storage controller for user individual service environment | |
| CN115033188B (en) | Storage hardware acceleration module system based on ZNS solid state disk | |
| US20150033226A1 (en) | Host system and method for managing data consumption rate in a virtual data processing environment | |
| KR20170002866A (en) | Adaptive Cache Management Method according to the Access Chracteristics of the User Application in a Distributed Environment | |
| Shi et al. | A case study of tuning MapReduce for efficient Bioinformatics in the cloud | |
| US20200285510A1 (en) | High precision load distribution among processors | |
| KR20080105390A (en) | Apparatus and method for controlling commands used in flash memory | |
| JPWO2008126202A1 (en) | Storage system load balancing program, storage system load balancing method, and storage management device | |
| US20060242182A1 (en) | System and method for stranded file opens during disk compression utility requests | |
| JP5697195B2 (en) | Management system, program and method for controlling table mirroring based on access prediction | |
| CN109375868B (en) | Data storage method, scheduling device, system, equipment and storage medium | |
| CN107220069B (en) | Shuffle method for nonvolatile memory | |
| US20180196611A1 (en) | Highly scalable computational active ssd storage device | |
| US10936223B2 (en) | Increasing serial read performance | |
| US10521156B2 (en) | Apparatus and method of managing multi solid state disk system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |