CN111061683A - Method and device for storing and operating data stream at high speed - Google Patents
Method and device for storing and operating data stream at high speed Download PDFInfo
- Publication number
- CN111061683A CN111061683A CN201911390260.3A CN201911390260A CN111061683A CN 111061683 A CN111061683 A CN 111061683A CN 201911390260 A CN201911390260 A CN 201911390260A CN 111061683 A CN111061683 A CN 111061683A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- logic unit
- storage
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a method and a device for storing and operating data stream at high speed, wherein the method comprises a data writing step S1: dividing a storage unit into a data storage area and a file index area; a data storage logic unit and a file management logic unit are constructed in the FPGA module, and a file system is used for user access, operation and requirement input; receiving data to be written and partitioning; partitioning the data storage area; writing the block data into a data storage area in sequence block by block; the file management logic unit establishes a data folder and a file name in a file index area; updating the path information of the corresponding data block to a file system; data operation step S2: selecting a target file and generating an operation instruction; transmitting the operation instruction and the data block path information corresponding to the target file to a data storage logic unit; and the data storage logic unit executes data block operation according to the operation instruction. The invention is convenient for data management and can effectively improve the speed of data storage and operation.
Description
Technical Field
The present invention relates to the storage and manipulation of high-speed data streams, and more particularly, to a method and apparatus for storing and manipulating data streams at high speed using a field programmable array (FPGA).
Background
At present, when a data stream is stored and applied at a high speed, a complete data block is generally written into a storage unit by a CPU in sequence to realize storage of the data stream, and when the data block needs to be operated, the CPU directly operates the data block in the storage unit according to an external system instruction (data deletion, data extraction, data combination, data copy, or the like).
In order to efficiently complete the data transfer and control, DMA can be used as a coprocessor to assist a CPU (central processing unit) to carry out mass data transfer, and the CPU, the DMA and a storage unit are mutually connected in a bus form, namely a DMA + CPU architecture; however, the data transfer and processing of the architecture are controlled based on a bus mode, and the data processing is still slow, so that the storage and operation of high-speed data streams cannot be supported.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a method and a device for storing and operating data stream at high speed, wherein a data storage logic unit is constructed to store and read data at high speed based on an FPGA (field programmable gate array), a file management logic unit is constructed to map data into a file for operation, so that the data management is convenient, and the speed of data storage and operation can be effectively improved; because the data storage logic unit and the data management logic unit are both positioned in the FPGA, the connection relationship among the logic units is tight, and the transmission rate and the efficiency are higher.
The purpose of the invention is realized by the following technical scheme: a method of storing and manipulating data streams at high speed includes a data writing step S1 and a data manipulating step S2;
the data writing step S1 includes:
s101, dividing a storage unit into a data storage area and a file index area;
s102, constructing a data storage logic unit and a file management logic unit in the FPGA module, and providing access, operation and requirement input for a user through a file system; the data storage logic unit is used for providing high-speed data storage and reading, and the file management logic unit is used for mapping data into a file for operation;
s103, receiving data to be written by a data storage logic unit, and blocking the data based on user requirements from a file system;
s104, the data storage logic unit divides the data storage area into blocks, and the size of each block of the storage area is equal to that of each block of the data;
s105, the data storage logic unit writes the block data into a data storage area of the storage unit block by block in sequence, and feeds back path information of the data block to the file management logic unit;
s106, after the writing is finished, the file management logic unit establishes a data folder and a file name in the file index area and associates the data folder and the file name with the data block of the data storage area;
s107, after the folders and the file names are established, the file management logic unit updates the path information of the corresponding data blocks to a file system, so that a user can operate the files conveniently;
the data operation step S2 includes:
s201, a user accesses a storage unit through a file system, selects a data folder and a file name as a target file in a file index area, and generates an operation instruction through the file system;
s202, the file system transmits the operation instruction and the data block path information corresponding to the target file to a data storage logic unit;
s203, the data storage logic unit finds the corresponding data block according to the path information and executes data block operation according to the operation instruction of the file system.
The device adopted by the method for storing and operating the data stream at high speed comprises a storage unit and an FPGA module; the FPGA is connected with the storage unit; the storage unit comprises a file index area and a data storage area; the FPGA module comprises a data storage logic unit and a file management logic unit, and is used for user access, operation and requirement input through a file system;
the data storage logic unit is used for receiving data to be written according to user requirements from the file system, performing data blocking and data writing, and operating data blocks in the data storage area according to an operation instruction from the user system;
and the file management logic unit is used for establishing a data folder and a file name in the file index area, associating the data folder and the file name with the data blocks in the data storage area, and updating the path information of the corresponding data blocks to the file system, so that a user can operate the file conveniently.
The invention has the beneficial effects that: based on FPGA, the invention constructs a data storage logic unit to provide high-speed data storage and reading, constructs a file management logic unit to map data into a file for operation, is convenient for data management, and can effectively improve the speed of data storage and operation; because the data storage logic unit and the data management logic unit are both positioned in the FPGA, the connection relationship among the logic units is tight, and the transmission rate and the efficiency are higher; the file management logic unit establishes a data folder and a file name in the file index area, and updates the path information of the corresponding data block to the file system after the file management logic unit is associated with the data block in the data storage area, so that the file system can conveniently and comprehensively manage the data.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 is a schematic block diagram of a corresponding device in which a file system is embedded in an FPGA module;
fig. 3 is a schematic block diagram of a corresponding apparatus for embedding a file system in an external file management device.
Detailed Description
The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings, but the scope of the present invention is not limited to the following.
As shown in fig. 1, a method of storing and manipulating a data stream at high speed includes a data writing step S1 and a data manipulating step S2;
the data writing step S1 includes:
s101, dividing a storage unit into a data storage area and a file index area;
s102, constructing a data storage logic unit and a file management logic unit in the FPGA module, and providing access, operation and requirement input for a user through a file system; the data storage logic unit is used for providing high-speed data storage and reading, and the file management logic unit is used for mapping data into a file for operation;
s103, receiving data to be written by a data storage logic unit, and blocking the data based on user requirements from a file system;
s104, the data storage logic unit divides the data storage area into blocks, and the size of each block of the storage area is equal to that of each block of the data;
s105, the data storage logic unit writes the block data into a data storage area of the storage unit block by block in sequence, and feeds back path information of the data block to the file management logic unit;
s106, after the writing is finished, the file management logic unit establishes a data folder and a file name in the file index area and associates the data folder and the file name with the data block of the data storage area;
s107, after the folders and the file names are established, the file management logic unit updates the path information of the corresponding data blocks to a file system, so that a user can operate the files conveniently;
the data operation step S2 includes:
s201, a user accesses a storage unit through a file system, selects a data folder and a file name as a target file in a file index area, and generates an operation instruction through the file system;
s202, the file system transmits the operation instruction and the data block path information corresponding to the target file to a data storage logic unit;
s203, the data storage logic unit finds the corresponding data block according to the path information and executes data block operation according to the operation instruction of the file system.
In an embodiment of the present application, in step S201, the operation instruction generated by the user through the file system includes a data deletion instruction, a data extraction instruction, a data combination instruction, or a data copying instruction; correspondingly, in step S203, the data block operation performed by the data storage logic unit includes data deletion, data extraction, data combination, or data replication; when data deletion, data combination or data copying is carried out, the data storage logic unit directly operates the corresponding data block in the data storage area according to the operation instruction; when data extraction is carried out, the data storage logic unit reads corresponding data blocks in the data storage area according to the operation instruction and outputs the data blocks to the outside.
The device adopted by the method for storing and operating the data stream at high speed comprises a storage unit and an FPGA module; the FPGA is connected with the storage unit; the storage unit comprises a file index area and a data storage area; the FPGA module comprises a data storage logic unit and a file management logic unit, and is used for user access, operation and requirement input through a file system;
the data storage logic unit is used for receiving data to be written according to user requirements from the file system, performing data blocking and data writing, and operating data blocks in the data storage area according to an operation instruction from the user system;
specifically, in the data storage process, the data storage logic unit receives data to be written, blocks the data based on user requirements from a file system, blocks a data storage area, and sequentially writes the blocks of the data storage area into the data storage area of the storage unit block by block, and feeds back path information of the data blocks to the file management logic unit; in the data operation process, the data storage logic unit finds a corresponding data block according to the path information from the file system and executes data block operation according to an operation instruction of the file system;
and the file management logic unit is used for establishing a data folder and a file name in the file index area, associating the data folder and the file name with the data blocks in the data storage area, and updating the path information of the corresponding data blocks to the file system, so that a user can operate the file conveniently.
As shown in fig. 2, in some embodiments, the file system is embedded in the FPGA module, and a user accesses the file system of the FPGA module through a user interface and performs operation and requirement input; in the embodiment, a user can access the file system in the FPGA module through the external management equipment connected with the user interface to perform operation and demand input;
as shown in fig. 3, in other embodiments, the file system is embedded in an external file management device, a user directly accesses the file system through the external file management device and performs operation and requirement input, and the file system interacts with the FPGA module through a user interface and transmits the operation and the requirement to the FPGA module;
in the above embodiment, the user interface may be a network interface or a hardware interface; the storage logic unit and the file management logic unit are software program units constructed in the FPGA; the file system is a software system installed in the FPGA module or the external management equipment, and is specifically a Windows operating system or a Linux operating system; the storage unit includes but is not limited to a hard disk or an SSD flash memory; the external management device includes, but is not limited to, a computer and a PAD.
Based on FPGA, the invention constructs a data storage logic unit to provide high-speed data storage and reading, constructs a file management logic unit to map data into a file for operation, is convenient for data management, and can effectively improve the speed of data storage and operation; because the data storage logic unit and the data management logic unit are both positioned in the FPGA, the connection relationship among the logic units is tight, and the transmission rate and the efficiency are higher; the file management logic unit establishes a data folder and a file name in the file index area, and updates the path information of the corresponding data block to the file system after being associated with the data block of the data storage area, so that the file system can conveniently and comprehensively manage the data; in the conventional technical scheme, the data stream storage and operation speed is generally in the level of hundreds of megabbps, and in the embodiment of the application, the data stream storage and operation speed can reach about 100 Gbps.
The foregoing is a preferred embodiment of the present invention, it is to be understood that the invention is not limited to the form disclosed herein, but is not to be construed as excluding other embodiments, and is capable of other combinations, modifications, and environments and is capable of changes within the scope of the inventive concept as expressed herein, commensurate with the above teachings, or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (6)
1. A method of high speed storage and manipulation of data streams, characterized by: including a data writing step S1 and a data operating step S2;
the data writing step S1 includes:
s101, dividing a storage unit into a data storage area and a file index area;
s102, constructing a data storage logic unit and a file management logic unit in the FPGA module, and providing access, operation and requirement input for a user through a file system; the data storage logic unit is used for providing high-speed data storage and reading, and the file management logic unit is used for mapping data into a file for operation;
s103, receiving data to be written by a data storage logic unit, and blocking the data based on user requirements from a file system;
s104, the data storage logic unit divides the data storage area into blocks, and the size of each block of the storage area is equal to that of each block of the data;
s105, the data storage logic unit writes the block data into a data storage area of the storage unit block by block in sequence, and feeds back path information of the data block to the file management logic unit;
s106, after the writing is finished, the file management logic unit establishes a data folder and a file name in the file index area and associates the data folder and the file name with the data block of the data storage area;
s107, after the folders and the file names are established, the file management logic unit updates the path information of the corresponding data blocks to a file system, so that a user can operate the files conveniently;
the data operation step S2 includes:
s201, a user accesses a storage unit through a file system, selects a data folder and a file name as a target file in a file index area, and generates an operation instruction through the file system;
s202, the file system transmits the operation instruction and the data block path information corresponding to the target file to a data storage logic unit;
s203, the data storage logic unit finds the corresponding data block according to the path information and executes data block operation according to the operation instruction of the file system.
2. A method of high speed storage and manipulation of data streams according to claim 1, wherein: in step S201, the operation instruction generated by the user through the file system includes a data deletion instruction, a data extraction instruction, a data combination instruction, or a data copy instruction.
3. A method of high speed storage and manipulation of data streams according to claim 1, wherein: in step S203, the data block operation performed by the data storage logic unit includes data deletion, data extraction, data combination, or data replication.
4. A device for high speed storage and manipulation of data streams according to any of claims 1 to 3, wherein: the FPGA module comprises a storage unit and an FPGA module; the FPGA is connected with the storage unit; the storage unit comprises a file index area and a data storage area; the FPGA module comprises a data storage logic unit and a file management logic unit, and is used for user access, operation and requirement input through a file system;
the data storage logic unit is used for receiving data to be written according to user requirements from the file system, performing data blocking and data writing, and operating data blocks in the data storage area according to an operation instruction from the user system;
and the file management logic unit is used for establishing a data folder and a file name in the file index area, associating the data folder and the file name with the data blocks in the data storage area, and updating the path information of the corresponding data blocks to the file system, so that a user can operate the file conveniently.
5. The apparatus of claim 4 for a method of high speed storage and manipulation of data streams, wherein: the file system is a Windows operating system or a Linux operating system.
6. The apparatus of claim 4 for a method of high speed storage and manipulation of data streams, wherein: the storage unit includes, but is not limited to, a hard disk or an SSD flash memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911390260.3A CN111061683B (en) | 2019-12-30 | 2019-12-30 | Method and device for storing and operating data stream at high speed |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911390260.3A CN111061683B (en) | 2019-12-30 | 2019-12-30 | Method and device for storing and operating data stream at high speed |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111061683A true CN111061683A (en) | 2020-04-24 |
| CN111061683B CN111061683B (en) | 2023-03-21 |
Family
ID=70303046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911390260.3A Active CN111061683B (en) | 2019-12-30 | 2019-12-30 | Method and device for storing and operating data stream at high speed |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111061683B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6640233B1 (en) * | 2000-08-18 | 2003-10-28 | Network Appliance, Inc. | Reserving file system blocks |
| CN101122938A (en) * | 2007-09-25 | 2008-02-13 | 北大方正集团有限公司 | Data file safe treatment method and system |
| JP2008262452A (en) * | 2007-04-13 | 2008-10-30 | Matsushita Electric Ind Co Ltd | Cache method of recording device, and recording device |
| CN102156731A (en) * | 2011-04-08 | 2011-08-17 | 传聚互动(北京)科技有限公司 | Data storage method and device for flash memory |
| CN103617250A (en) * | 2013-11-28 | 2014-03-05 | 中经云数据存储科技（北京）有限公司 | File management method based on optical storage array and file format system |
| CN104516929A (en) * | 2013-09-27 | 2015-04-15 | 伊姆西公司 | Method and device for file system |
| US20170168735A1 (en) * | 2015-12-10 | 2017-06-15 | International Business Machines Corporation | Reducing time to read many files from tape |
| CN108647278A (en) * | 2018-05-03 | 2018-10-12 | 中北大学 | A kind of file management method and system |
| CN108664577A (en) * | 2018-05-03 | 2018-10-16 | 中北大学 | A kind of file management method and system based on the free areas FLASH |
| CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
| CN110096456A (en) * | 2019-05-13 | 2019-08-06 | 成都定为电子技术有限公司 | A kind of High rate and large capacity caching method and device |
-
2019
- 2019-12-30 CN CN201911390260.3A patent/CN111061683B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6640233B1 (en) * | 2000-08-18 | 2003-10-28 | Network Appliance, Inc. | Reserving file system blocks |
| JP2008262452A (en) * | 2007-04-13 | 2008-10-30 | Matsushita Electric Ind Co Ltd | Cache method of recording device, and recording device |
| CN101122938A (en) * | 2007-09-25 | 2008-02-13 | 北大方正集团有限公司 | Data file safe treatment method and system |
| CN102156731A (en) * | 2011-04-08 | 2011-08-17 | 传聚互动(北京)科技有限公司 | Data storage method and device for flash memory |
| CN104516929A (en) * | 2013-09-27 | 2015-04-15 | 伊姆西公司 | Method and device for file system |
| CN103617250A (en) * | 2013-11-28 | 2014-03-05 | 中经云数据存储科技（北京）有限公司 | File management method based on optical storage array and file format system |
| US20170168735A1 (en) * | 2015-12-10 | 2017-06-15 | International Business Machines Corporation | Reducing time to read many files from tape |
| CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
| CN108647278A (en) * | 2018-05-03 | 2018-10-12 | 中北大学 | A kind of file management method and system |
| CN108664577A (en) * | 2018-05-03 | 2018-10-16 | 中北大学 | A kind of file management method and system based on the free areas FLASH |
| CN110096456A (en) * | 2019-05-13 | 2019-08-06 | 成都定为电子技术有限公司 | A kind of High rate and large capacity caching method and device |
Non-Patent Citations (6)
| Title |
|---|
| TZER-TA TSENG: "A Flexible and Cost-Effective File-Wise Reliability Scheme for Storage Systems" * |
| 刘剑: "用FPGA实现文件系统及其应用" * |
| 原发杰: "一种新的海量遥感瓦片影像数据存储检索策略" * |
| 吴艳霞: "深度学习FPGA加速器的进展与趋势" * |
| 彭刚;: "一种面向容灾存储系统的重复数据删除方法" * |
| 胡学庆: "集群环境下海量数据存储管理技术的研究" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111061683B (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109085997A (en) | Memory-efficient for nonvolatile memory continues key assignments storage | |
| US10761977B2 (en) | Memory system and non-transitory computer readable recording medium | |
| JP5284905B2 (en) | Data migration method and program | |
| US8639898B2 (en) | Storage apparatus and data copy method | |
| US9606748B2 (en) | Importing pre-existing data of a prior storage solution into a storage pool for use with a new storage solution | |
| US20080195833A1 (en) | Systems, methods and computer program products for operating a data processing system in which a file system's unit of memory allocation is coordinated with a storage system's read/write operation unit | |
| US20120079229A1 (en) | Data storage optimization for a virtual platform | |
| CN106708425A (en) | Distributed multimode storage management | |
| US8924353B1 (en) | Systems and methods for copying database files | |
| JP2016212835A (en) | Memory system | |
| US10296250B2 (en) | Method and apparatus for improving performance of sequential logging in a storage device | |
| JP2011512591A (en) | Apparatus, method, and computer program for providing indirect data addressing in an input / output processing system | |
| US8478933B2 (en) | Systems and methods for performing deduplicated data processing on tape | |
| US10169345B2 (en) | Moving data from linear tape file system storage to cloud storage | |
| US9575669B2 (en) | Programmable solid state drive controller and method for scheduling commands utilizing a data structure | |
| US20130262799A1 (en) | Control method and storage device | |
| US20230281118A1 (en) | Memory system and non-transitory computer readable recording medium | |
| CN107798063B (en) | Snapshot processing method and snapshot processing device | |
| US10503426B2 (en) | Efficient space allocation in gathered-write backend change volumes | |
| EP2669806B1 (en) | Storage system | |
| US9645946B2 (en) | Encryption for solid state drives (SSDs) | |
| US10387267B2 (en) | Fast copy using file system block mappings | |
| CN111061683B (en) | Method and device for storing and operating data stream at high speed | |
| JP6215631B2 (en) | Computer system and data management method thereof | |
| US8892807B2 (en) | Emulating a skip read command |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |