CN115826884A - Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof - Google Patents
Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof Download PDFInfo
- Publication number
- CN115826884A CN115826884A CN202310123448.1A CN202310123448A CN115826884A CN 115826884 A CN115826884 A CN 115826884A CN 202310123448 A CN202310123448 A CN 202310123448A CN 115826884 A CN115826884 A CN 115826884A
- Authority
- CN
- China
- Prior art keywords
- file system
- space
- speed
- storage module
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a lightweight high-speed file system facing an embedded heterogeneous platform and a processing method thereof, wherein the high-speed file system comprises an information table, a record table and an idle table; the information table is used for managing the free table and sorting the continuous space blocks of the residual space of the user data storage module, so that the maximum continuous space block is determined to be used as the space block for storing the record next time; the high-speed file system is used for triggering write operation when the data blocks at the cache end are accumulated to a set value, so that enough continuous linear sector space is allocated in the maximum continuous space block in the idle table according to the size of the data blocks at the cache end for data storage; and the record table is used for recording the start address, the end address and the write data information of the write data stored in the user data storage module each time under the management of the information table. The invention is beneficial to improving the storage efficiency of high-speed data and realizing high-speed storage.
Description
Technical Field
The invention relates to the technical field of embedded heterogeneous platform file systems, in particular to a lightweight high-speed file system facing an embedded heterogeneous platform and a lightweight high-speed file processing method facing the embedded heterogeneous platform.
Background
In a storage System of an FPGA (Field Programmable Gate Array) embedded heterogeneous platform, currently, commonly used File systems FAT (File Allocation Table), extended File Allocation Table (Extended File Allocation Table), and the like all adopt cluster recording management, and when File System management is performed, the management is very complicated by adopting cluster recording, for example, information to be recorded by a File System when data is stored includes: the serial number of each cluster, the serial number of the previous cluster corresponding to each cluster, the serial number of the next cluster corresponding to each cluster, and the file information written in each cluster; moreover, because the number of sectors operated in a single time in the cluster recording management mode is limited, when cluster recording management is adopted, if a file is written, multiple times of partitioned storage are required for data to be written, and sectors which are allowed to operate in a single time are allocated during each time of partitioned storage, so that a complete data to be written can gradually complete the storage process only after multiple times of sector allocation and multiple times of data writing into the sectors, and the requirement of a high-speed data storage scene cannot be met.
Therefore, the file system of the existing embedded heterogeneous platform is influenced by the cluster recording management mode, and the requirement of a high-speed data storage scene cannot be met.
Disclosure of Invention
The invention mainly aims to provide a lightweight high-speed file system facing an embedded heterogeneous platform, and aims to solve the problem that the existing file system of the embedded heterogeneous platform is influenced by a cluster recording management mode and cannot meet a high-speed data storage scene.
In order to achieve the purpose, in the lightweight high-speed file system oriented to the embedded heterogeneous platform, the high-speed file system is in communication connection with the user data storage module and is used for storing the high-speed acquired data to the user data storage module at a high speed; the high-speed file system is arranged at an embedded soft core PS end of the FPGA end;
the high-speed file system comprises an information table, a record table and an idle table;
the information table is used for managing the idle table, so that after the user data storage module finishes storage record or deletion operation each time through the idle table, continuous space blocks of the residual space of the user data storage module are sorted to be sorted according to the size of the storage space of the continuous space blocks, and the largest continuous space block is determined to be used as a space block for storing records next time;
the high-speed file system is used for triggering write operation when the data blocks at the cache end are accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block in the free table according to the size of the data blocks at the cache end for data storage; the system is also used for restoring the sector space corresponding to the deletion operation to the idle table according to a linear relation after the user data storage module executes the deletion operation;
and the record table is used for recording the start address, the end address and the write data information of the write data stored in the user data storage module each time under the management of the information table.
Preferably, the user data storage module includes a plurality of sector spaces, each sector space is divided into a data storage area, a backup area and a reserved area, and each sector space is correspondingly provided with an information table, a record table and a free table;
the data storage area and the backup area are used for synchronously executing write-in operation when the data block of the cache end executes write-in operation to the allocated sector space; and the backup area is also used for deleting the written data blocks after the data blocks at the cache end are written into the allocated sector space.
Preferably, the high-speed file system is also in communication connection with a file system storage module; the file system storage module is used for storing the log update file of the high-speed file system.
Preferably, the FPGA and the CPU communicate with each other via a PCIE-MAP protocol via a PCIE interface;
the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver;
the application layer is used for carrying out FTP or NFS general access by a user through a remote network;
the file system layer is used for calling the general file system conversion drive and the PCIE-MAP drive so as to realize general conversion of file system information recorded by a high-speed file system at the PS end of the embedded soft core, and further realize FTP or NFS general access.
In order to achieve the above object, the present invention further provides a lightweight high-speed file processing method for an embedded heterogeneous platform, which is applied to any one of the lightweight high-speed file systems for an embedded heterogeneous platform; the method comprises the following steps:
after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to arrange the continuous space blocks of the residual space so as to sort according to the size of the storage space of the space blocks, and therefore the largest continuous space block is determined to be used as the space block for storing the record next time;
carrying out high-speed data acquisition through an FPGA end, and detecting the cache data volume of a cache end of the FPGA end;
when the cache data volume reaches a set value, the PS end of the embedded soft core triggers write operation;
when the high-speed file system detects the write operation, allocating enough continuous linear sector space in the maximum continuous space block for data storage in the free table according to the size of the data block of the cache end so as to store the data block of the cache end to the allocated sector space at one time;
after the data block is stored, the information table controls the record table to execute storage records, wherein the storage records comprise the initial address, the end address and the data type of the data block which is recorded and stored in the user data storage module each time;
and after detecting that the user data storage module executes the deletion operation, the high-speed file system restores the sector space corresponding to the deletion operation to the idle table according to a linear relation.
Preferably, the user data storage module includes a plurality of sector spaces, each sector space is divided into a data storage area, a backup area and a reserved area, and each sector space is correspondingly provided with an information table, a record table and a free table; when the high-speed file system detects the write operation, allocating enough continuous linear sector space in the maximum continuous space block for data storage according to the size of the data block at the cache end in the free table, so as to store the cache data block to the allocated sector space at one time, wherein the step comprises the following steps:
when the high-speed file system detects the write operation, the size of the data block of the cache end is obtained, and enough continuous linear sector space is distributed in the maximum continuous space block in the free table according to the size of the data block of the cache end;
simultaneously sending the data blocks of the cache end to the backup area and the data storage area of the allocated sector space;
and deleting the data blocks in the backup area after all the data blocks are written into the backup area and the data storage area.
Preferably, the high-speed file system is also in communication connection with a file system storage module; the method further comprises the following steps:
and sending the log update file generated by the high-speed file system to the file system storage module for storage.
Preferably, the method further comprises:
acquiring a preset log updating mode, wherein the log updating mode comprises updating according to recording time, updating according to writing capacity and establishing updating according to files;
dividing the file system storage module into a plurality of storage areas according to the number of the log updating modes;
the step of sending the log update file generated by the high-speed file system to the file system storage module for storage comprises:
and when the updated log is detected, acquiring a log updating mode corresponding to the updated log, and storing the updated log to a corresponding storage area in the file system storage module according to the corresponding log updating mode.
Preferably, the method further comprises:
dividing each storage area of the file system storage module into a plurality of rollback areas with a set sequence;
the step of sending the log update file generated by the high-speed file system to the file system storage module for storage further comprises:
when a current log to be stored in the storage area is detected, acquiring a last rollback area corresponding to the last log, and storing the current log in a next rollback area corresponding to the last rollback area; and the next rollback area corresponding to the last rollback area is the first rollback area.
Preferably, the FPGA side communicates with the CPU side via a PCIE-MAP protocol through a PCIE interface; the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver; the method further comprises the following steps:
acquiring a general access request initiated to the application layer by a user through a remote network, wherein the general access request is FTP or NFS access;
acquiring file information recorded by a high-speed file system from the embedded soft core PS end of the FPGA end through a PCIE interface, wherein the file information is stored in the user data storage module;
and calling the universal file system conversion driver and the PCIE-MAP driver through the file system layer to carry out universal conversion on the file information, thereby realizing universal access of the file information.
In the technical scheme of the invention, a high-speed file system arranged at an embedded soft core PS end is used for storing high-speed acquired data to the user data storage module at a high speed; the high-speed file system manages a recording list and a free list through an information list; after the user data storage module finishes the storage record or deletion operation each time, the free table sorts the continuous space blocks of the residual space of the user data storage module to be sorted according to the size of the storage space of the continuous space blocks, so that the largest continuous space block is determined to be used as the space block for storing the record next time; therefore, before the time of storing the data blocks of the cache end to the user data storage module, sequencing of all the continuous space blocks is finished, and the largest continuous space block is prepared to the state of the data block to be stored, so that direct storage at the time needing to be stored is facilitated, and storage delay caused by searching for a storage space is avoided. In addition, in a high-speed acquisition scene, the data block cached each time needs enough storage space, the free table arranges the maximum continuous space block as the space block for storing the record next time, so that the sufficient storage space is favorably provided, the condition that the storage space needs to be divided again due to insufficient space is avoided, and the storage speed is also improved.
Further, after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to arrange the continuous space blocks of the residual space of the user data storage module, so as to delete the occupied space from the idle table according to the space in the maximum continuous space block in the previous round, and the sector space corresponding to the deletion operation is restored to the idle table according to the linear relation, thereby reordering each continuous space block and arranging the maximum continuous space block as the next storage space;
different from the cluster recording management mode that the data needs to be stored in a plurality of times in the same cache data block, the high-speed file system is used for triggering write operation when the data block at the cache end is accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block for data storage at one time in the idle table according to the size of the data block at the cache end, and the storage delay caused by the storage in a plurality of times is avoided.
In summary, in the technical solution of the present invention, an information table, a record table, and a free table are used to perform storage management on a user data storage module, and this storage management mode only needs to record a starting storage address and an ending storage address of each data block, and does not need to pay attention to the number of each cluster in the user data storage module, the number of a previous cluster corresponding to each cluster, the number of a next cluster corresponding to each cluster, and file information written in each cluster, so as to achieve lightweight storage of data records.
Drawings
FIG. 1 is a schematic block diagram of an embedded heterogeneous platform according to the present invention;
FIG. 2 is a schematic diagram of a high-speed file system according to the present invention;
FIG. 3 is a schematic diagram of a multi-partition sector spatial distribution of a user data storage module according to the present invention;
fig. 4 is a schematic flowchart of a first embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method according to the present invention.
The objects, features and advantages of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "unit", "means", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "unit", "part" or "unit" may be used mixedly.
Referring to fig. 1 to 2, in order to achieve the above object, the present invention provides an embedded heterogeneous platform-oriented lightweight high-speed file system, wherein the high-speed file system is in communication connection with a user data storage module, so as to perform high-speed storage on high-speed acquired data to the user data storage module; the high-speed file System is arranged at an embedded soft core PS (Processing System) end of the FPGA end.
The high-speed file system comprises an information table, a record table and a free table.
The information table is used for managing the idle table, so that after the user data storage module finishes storage record or deletion operation each time through the idle table, continuous space blocks of the residual space of the user data storage module are sorted to be sorted according to the size of the storage space of the continuous space blocks, and the largest continuous space block is determined to be used as a space block for storing records next time; the maximum continuous space block in the invention is not only a free storage area, but also a continuous and uninterrupted storage area. The largest contiguous block of space belongs to a storage area of the same sector space (rather than a storage area spanning different sector spaces). The continuous space block is used for providing a section of continuous storage area, and the maximum continuous space block is the result obtained by sorting the size of the storage space of each continuous space block in the user data storage module. When the maximum contiguous space block performs a write operation of a data block of the buffer, a part of the maximum contiguous space block is occupied, and therefore, the storage space of the maximum contiguous space block is reduced, and at this time, the remaining space of the maximum contiguous space block may not belong to the maximum contiguous space block any more. Therefore, after the data block with the buffer area is written into the maximum continuous space block every time, the continuous space blocks of the residual space of the user data storage module need to be rearranged, so as to update the sorting result of the size of the storage space corresponding to each continuous space block, so as to obtain the maximum continuous space block corresponding to the latest sorting result, and the maximum continuous space block is used as the space block for storing the record next time.
The high-speed file system is used for triggering write operation when the data blocks at the cache end are accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block in the free table according to the size of the data blocks at the cache end for data storage; and the system is also used for restoring the sector space corresponding to the deletion operation to the idle table according to a linear relation after the user data storage module executes the deletion operation.
And the record table is used for recording the start address, the end address and the write data information of the write data stored in the user data storage module each time under the management of the information table.
In the technical scheme of the invention, a high-speed file system arranged at an embedded soft core PS end is used for storing high-speed acquired data to the user data storage module at a high speed; the high-speed file system manages a recording list and an idle list through an information list; after the user data storage module finishes the storage record or deletion operation each time, the free table sorts the continuous space blocks of the residual space of the user data storage module to be sorted according to the size of the storage space of the continuous space blocks, so that the largest continuous space block is determined to be used as the space block for storing the record next time; therefore, before the time of storing the data blocks of the cache end to the user data storage module, sequencing of all the continuous space blocks is finished, and the largest continuous space block is prepared to the state of the data block to be stored, so that direct storage at the time needing to be stored is facilitated, and storage delay caused by searching for a storage space is avoided. In addition, in a high-speed acquisition scene, the data block cached each time needs enough storage space, the free table arranges the maximum continuous space block as the space block for storing the record next time, so that the sufficient storage space is favorably provided, the condition that the storage space needs to be divided again due to insufficient space is avoided, and the storage speed is also improved.
Further, after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to sort the continuous space blocks of the remaining space of the user data storage module, so as to delete the occupied space from the idle table according to the space in the maximum continuous space block in the previous round, and the sector space corresponding to the deletion operation is restored to the idle table according to the linear relation, thereby reordering each continuous space block and sorting the maximum continuous space block as the next storage space.
Different from the cluster recording management mode that the data needs to be stored in a plurality of times in the same cache data block, the high-speed file system is used for triggering write operation when the data block at the cache end is accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block for data storage at one time in the idle table according to the size of the data block at the cache end, and the storage delay caused by the storage in a plurality of times is avoided.
In summary, in the technical solution of the present invention, an information table, a record table, and a free table are used to perform storage management on a user data storage module, and this storage management mode only needs to record a starting storage address and an ending storage address of each data block, and does not need to pay attention to the number of each cluster in the user data storage module, the number of a previous cluster corresponding to each cluster, the number of a next cluster corresponding to each cluster, and file information written in each cluster, so as to achieve lightweight storage of data records.
The lightweight high-speed file system facing the embedded heterogeneous platform is applied to the FPGA embedded heterogeneous platform, and the heterogeneous platform comprises a Central Processing Unit (CPU) end and an FPGA end which are in communication connection.
Referring to fig. 1, the FPGA side includes a Programmable Logic (PL) side and an embedded soft core PS side, and the embedded soft core PS side is provided with a high-speed file system, where a PCIE-MAP protocol is stored in the high-speed file system, and the FPGA side performs high-speed data acquisition, and accumulates the high-speed acquired data at the cache side to a set data block size, and then performs high-speed storage on the data to the user data storage module. The FPGA logic PL terminal realizes data acquisition, data caching and data storage, and the embedded soft core PS terminal realizes acquisition control, cache space configuration and high-speed file system management.
The invention designs a lightweight high-speed file system under an embedded heterogeneous platform facing to an FPGA and a CPU, which comprises the design of a lightweight high-speed file system architecture, the realization of linear SSD (Solid State Disk) sector management and the realization of a universal file system conversion drive.
As shown in fig. 3, preferably, the user data storage module includes a plurality of sector spaces, each of the sector spaces is divided into a data storage area, a backup area, and a reserved area, and each of the sector spaces is correspondingly provided with an information table, a record table, and a free table;
the data storage area and the backup area are used for synchronously executing write-in operation when the data block of the cache end executes write-in operation to the allocated sector space; and the backup area is also used for deleting the written data blocks after the data blocks at the cache end are written into the allocated sector space.
If abnormal termination (such as power failure) occurs in the process of writing data, a user only needs to check whether the backup area has data or not after the abnormal condition is terminated, and then can determine whether the last writing operation is finished or not. If the backup area has data, the writing operation of the data storage area is not completed because the backup area and the data storage area are synchronously written, and the writing operation is executed again at the moment until the complete storage of the data to be written is terminated. If the backup area has no data, it indicates that the last write operation has been completely finished. This synchronized writing operation is advantageous to ensure that the data storage of the data storage area is complete.
Specifically, the sector space may be one or more than one. Each sector space is correspondingly provided with an information table, a recording table and a free table; each sector space may be further divided into a data storage area, a backup area, and at least one reserved area, and the number of reserved areas may be two.
In the invention, linear SSD sector management is carried out for the user data storage module. The free table is sorted according to the order of the storage spaces of the remaining continuous space blocks (in this embodiment, the continuous space blocks of the data storage area) of each sector space from large to small, the continuous space block with the largest storage space is arranged to the first bit of the free table, and the continuous space block with the second bit of the storage space is arranged to the second bit of the free table until the continuous space block with the smallest storage space is arranged.
When the data block at the cache end is stored in the user data storage module, the continuous linear sector space is divided from the continuous space block with the first bit arranged in the idle table, so that the linear relation that the data block crosses the sector space can be avoided from being stored in different sector spaces, the data block is stored in the same sector space, the rapid storage is facilitated, and the released sector space can be recycled into the same continuous space block of the idle table after the data block is deleted.
Preferably, the high-speed file system is also in communication connection with a file system storage module; the file system storage module is used for storing the log update file of the high-speed file system.
The existing common FPGA embedded heterogeneous platform storage system has the following defects:
firstly, a universal file system FAT, an exFAT and the like are used for cluster recording management, the number of sectors in single operation is limited, large data blocks need to be stored in a multi-time division mode, and the requirement of a high-speed data storage scene cannot be met; the invention solves the technical problem through the information table, the record table and the idle table of the high-speed file system. Specifically, the invention arranges the maximum continuous space block through the free list of the high-speed file system, allocates enough sector space from the maximum continuous space block to the cache data block to be written in each time, does not need to divide the storage for multiple times, can finish the storage in one time, and realizes the high-speed storage.
Secondly, the file system may be damaged and cannot be recovered when the file system is abnormally powered down or the file system is stored incorrectly.
In order to solve the second defect, the high-speed file system of the invention adopts a log type, and designs and realizes a lightweight file system and a log updating mechanism, wherein the high-speed acquired data is stored in a user data storage module in a high-speed storage mode, and the log updated data is separately stored in a file system storage module, so that the log updating does not influence the high-speed storage process, and the file system can be prevented from being damaged and incapable of being recovered (for a specific method, please refer to the following third embodiment to the fifth embodiment).
Third, a common custom file system has no universality, and needs to be matched with upper computer software for data export, so that universal FTP or NFS access cannot be supported.
Preferably, in order to solve the third defect, the invention adopts the following technical scheme:
the FPGA end and the CPU end communicate through a PCIE-MAP protocol through a PCIE interface;
the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver;
the application layer is used for carrying out FTP or NFS general access by a user through a remote network;
the File System layer is used for calling the general File System conversion driver and the PCIE-MAP driver to realize general conversion of File System information recorded by a high-speed File System at the PS end of the embedded soft core, so that FTP (File Transfer Protocol) or NFS (Network File System) general access is realized.
In this embodiment, the FPGA side communicates with the CPU side through the PCIE-MAP custom protocol, the CPU side implements a general file system conversion driver, and the storage system supports general FT or NFS access.
Preferably, the information of the PCIE-MAP protocol includes: port number, information ID, information Type, and buffer.
An embedded soft core PS end of the FPGA end creates and initializes a high-speed file system, and the high-speed file system mainly comprises an information table, a record table and an idle table. The information table management record table and the free table are used for storing the related information of the high-speed file system; the recording list manages files stored in the user data storage module each time, and establishes a recording information linked list and recording linked lists of various data types; after the free list stores records or deletes operation each time, the management of the residual space of the user data storage module is carried out, the largest continuous space block is sorted to prepare for the next operation, and information such as a storage area is provided.
Referring to fig. 4, in order to achieve the above object, a first embodiment of the present invention provides a lightweight high-speed file processing method for an embedded heterogeneous platform, which is applied to any one of the lightweight high-speed file systems for an embedded heterogeneous platform described above; the method comprises the following steps:
step S10, after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to arrange the continuous space blocks of the residual space so as to sort according to the size of the storage space of the space blocks, thereby determining the maximum continuous space block as the space block for the next storage record;
step S20, carrying out high-speed data acquisition through an FPGA end, and detecting the cache data volume of a cache end of the FPGA end;
step S30, when the cache data volume reaches a set value, the PS end of the embedded soft core triggers write operation;
step S40, when the high-speed file system detects the write operation, allocating enough continuous linear sector space in the maximum continuous space block for data storage according to the size of the data block of the cache end in the free table, so as to store the data block of the cache end to the allocated sector space at one time;
step S50, after the data block is stored, the information table controls the record table to execute storage records, and the storage records comprise the initial address, the end address and the data type of the data block which is stored in the user data storage module every time;
step S60, after detecting that the user data storage module executes the deletion operation, the high-speed file system restores the sector space corresponding to the deletion operation to the free list according to the linear relationship.
In the technical scheme of the invention, a high-speed file system arranged at an embedded soft core PS end is used for storing high-speed acquired data to the user data storage module at a high speed; the high-speed file system manages a recording list and an idle list through an information list; after the user data storage module finishes the storage record or deletion operation each time, the free table sorts the continuous space blocks of the residual space of the user data storage module to be sorted according to the size of the storage space of the continuous space blocks, so that the largest continuous space block is determined to be used as the space block for storing the record next time; therefore, before the time of storing the data blocks of the cache end to the user data storage module, sequencing of all the continuous space blocks is finished, and the largest continuous space block is prepared to the state of the data block to be stored, so that direct storage at the time needing to be stored is facilitated, and storage delay caused by searching for a storage space is avoided. In addition, in a high-speed acquisition scene, the data block cached each time needs enough storage space, the free table arranges the maximum continuous space block as the space block for storing the record next time, so that the sufficient storage space is favorably provided, the condition that the storage space needs to be divided again due to insufficient space is avoided, and the storage speed is also improved.
Further, after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to sort the continuous space blocks of the residual space of the user data storage module, so as to delete the occupied space from the idle table according to the maximum continuous space block in the previous round, and restore the sector space corresponding to the deletion operation to the idle table according to the linear relation, thereby reordering each continuous space block and sorting out the maximum continuous space block as the next storage space;
different from the cluster recording management mode that the data needs to be stored in a plurality of times in the same cache data block, the high-speed file system is used for triggering write operation when the data block at the cache end is accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block for data storage at one time in the idle table according to the size of the data block at the cache end, and the storage delay caused by the storage in a plurality of times is avoided.
In summary, in the technical solution of the present invention, an information table, a record table, and a free table are used to perform storage management on a user data storage module, and this storage management mode only needs to record a starting storage address and an ending storage address of each data block, and does not need to pay attention to the number of each cluster in the user data storage module, the number of a previous cluster corresponding to each cluster, the number of a next cluster corresponding to each cluster, and file information written in each cluster, so as to achieve lightweight storage of data records.
In a high-speed data acquisition and storage scene, a data block stored at a cache end usually needs to occupy a larger storage space, and the idle table directly takes the maximum continuous space block in the user data storage module as a space block for storing records, so that the data block can be stored completely in a single storage process without searching for an available storage space for many times.
Based on the first embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method, the second embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method comprises a user data storage module and a user data storage module, wherein the user data storage module comprises a plurality of sector spaces, each sector space is divided into a data storage area, a backup area and a reserved area, and each sector space is correspondingly provided with an information table, a record table and a free table;
the step S40 includes:
step S41, when the high-speed file system detects the write operation, the size of the data block of the cache end is obtained, and enough continuous linear sector space is distributed in the maximum continuous space block in the free table according to the size of the data block of the cache end;
step S42, sending the data block of the cache end to the backup area and the data storage area of the allocated sector space at the same time;
and S43, deleting the data blocks in the backup area after all the data blocks are written into the backup area and the data storage area.
In the cluster recording mode, for a data block at a cache end, a data storage area needs to be divided from a storage space according to a preset single allocation space size, a first part of data blocks are stored, then a second data storage area is divided from the storage space, and a second part of data blocks are stored until the data blocks are completely stored.
According to the method, the maximum continuous space block is distributed with enough continuous linear sector space directly according to the size of the data block, so that the completion of data block storage in a single operation is facilitated.
Based on the first embodiment or the second embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method, in the third embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method, the high-speed file system is further in communication connection with a file system storage module; the method further comprises the following steps:
and step S70, sending the log update file generated by the high-speed file system to the file system storage module for storage.
The existing common FPGA embedded heterogeneous platform storage system has the following defects:
firstly, a universal file system FAT, an exFAT and the like are used for cluster recording management, the number of sectors in single operation is limited, large data blocks need to be stored in a multi-time division mode, and the requirement of a high-speed data storage scene cannot be met; the invention solves the technical problem through the information table, the record table and the idle table of the high-speed file system.
Secondly, the file system may be damaged and cannot be recovered when the file system is abnormally powered down or the file system is stored incorrectly.
In order to solve the second defect, the high-speed file system adopts a log type, designs and realizes a lightweight file system and a log updating mechanism, the high-speed acquired data is stored in a user data storage module in a high-speed storage mode, and the log updated data is separately stored in a file system storage module, so that the log updating does not influence the high-speed storage process, and the file system can be prevented from being damaged and incapable of being recovered.
Based on the third embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, in the fourth embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, the method further includes:
step S80, acquiring a preset log updating mode, wherein the log updating mode comprises updating according to recording time, updating according to writing capacity and establishing updating according to files;
step S90, dividing the file system storage module into a plurality of storage areas according to the number of the log updating modes;
the step S70 includes:
step S71, when the updated log is detected, acquiring a log updating mode corresponding to the updated log, and storing the updated log to a corresponding storage area in the storage module of the file system according to the corresponding log updating mode.
Different log updating modes are set, a user can configure initialization parameters to realize the setting of a log updating mechanism, various modes such as log updating according to recording time, writing capacity and file creation are supported, and the multi-level backup of a file system is realized. The multi-level backup of the log is divided into different storage areas according to different log updating mechanisms, so that different logs can be consulted in different storage areas according to different updating mechanisms.
Based on the fourth embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, in the fifth embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, the method further includes:
step S100, dividing each storage area of the file system storage module into a plurality of rollback areas with a set sequence.
The step S70 further includes:
step S72, when detecting the current log to be stored in the storage area, acquiring a last rollback area corresponding to the previous log, and storing the current log in a next rollback area corresponding to the last rollback area; and the next rollback area corresponding to the last rollback area is the first rollback area.
In the invention, the logs in each storage area are sequentially stored in different rollback areas, for example, when a certain storage area is provided with three rollback areas, the first updated log is stored in the first rollback area, the second updated log is stored in the second rollback area, the third updated log is stored in the third rollback area, and the fourth updated log is stored in the first rollback area.
Therefore, when the log updated for the third time is detected to have abnormality, only the log updated for the second time stored in the second rollback area needs to be traced, and the log updated for the second time does not need to be traced to the earlier updating time. And rollback recovery under abnormal damage conditions is facilitated.
Further, the initialization in the invention supports the allocation of multi-partition and different partition storage capacities, and can specify the number of partitions and the partition storage capacity.
For example, the corresponding storage areas are divided into different storage capacities according to different update modes, or different storage areas are allocated with different partition numbers.
Based on the first to fifth embodiments of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, in the sixth embodiment of the embedded heterogeneous platform-oriented lightweight high-speed file processing method of the present invention, the FPGA side and the CPU side communicate with each other through a PCIE interface using a PCIE-MAP protocol; the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver; the method further comprises the following steps:
step S110, acquiring a general access request initiated to the application layer by a user through a remote network, wherein the general access request is FTP or NFS access;
step S120, acquiring file information recorded by a high-speed file system from the embedded soft core PS end of the FPGA end through a PCIE interface, wherein the file information is stored in the user data storage module;
step S130, the file system layer calls the common file system conversion driver and the PCIE-MAP driver to perform common conversion on the file information, thereby implementing common access to the file information.
Specifically, the FPGA end and the CPU end communicate with each other through a PCIE interface by adopting a self-defined PCIE-MAP protocol, the PCIE-MAP protocol is a driving layer protocol, and protocol information comprises information such as a port number, an information ID, an information Type and a buffer area. The general file system conversion driver at the CPU end realizes PCIE-MAP driver layer protocol analysis, converts file information contained in the file system at the embedded soft core PS end into a general file system, and can be mounted on an FTP or NFS general network node for access.
As shown in fig. 1 to 4, in the embedded heterogeneous platform of FPGA + CPU of the present invention, the FPGA side acquires high-speed data, the embedded soft core PS side runs a high-speed file system to perform high-speed storage management, and the high-speed file system linearly allocates continuous sector spaces of the size of the corresponding cache data blocks in the storage process, so as to satisfy the write operation of the storage maximum performance of the SSD storage array in the user data storage module. And performing log updating of the file system according to a log updating mode initialized by a user, writing the log updating into a file system storage module, wherein the high-speed storage of the user data storage module is not influenced in the file system log updating process. After the acquisition and storage of the high-speed data are finished, a user carries out FTP or NFS and other general accesses through a remote network, a file system layer calls a general file system conversion drive and a PCIE-MAP drive of a CPU end, and the general conversion of file information recorded by a high-speed file system of an embedded soft core PS end is realized through a PCIE-MAP protocol, so that the FTP or NFS general accesses are realized.
The user data storage module and the file system storage module can be solid state disks respectively.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk) as described above, and includes several instructions for enabling a terminal device to enter the method according to the embodiments of the present invention.
In the description herein, references to the description of the term "an embodiment," "another embodiment," "other embodiments," or "first through Xth embodiments," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, method steps, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or system comprising the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A lightweight high-speed file system facing an embedded heterogeneous platform is characterized in that the high-speed file system is in communication connection with a user data storage module and is used for storing high-speed acquired data to the user data storage module at a high speed; the high-speed file system is arranged at an embedded soft core PS end of the FPGA end;
the high-speed file system comprises an information table, a record table and an idle table;
the information table is used for managing the idle table, so that after the user data storage module finishes storage record or deletion operation each time through the idle table, continuous space blocks of the residual space of the user data storage module are sorted to be sorted according to the size of the storage space of the continuous space blocks, and the largest continuous space block is determined to be used as a space block for storing records next time;
the high-speed file system is used for triggering write operation when the data blocks at the cache end are accumulated to a set value, so that enough continuous linear sector space is allocated to the maximum continuous space block in the free table according to the size of the data blocks at the cache end for data storage; the system is also used for restoring the sector space corresponding to the deletion operation to the idle table according to a linear relation after the user data storage module executes the deletion operation;
and the record table is used for recording the start address, the end address and the write data information of the write data stored in the user data storage module each time under the management of the information table.
2. The embedded heterogeneous platform-oriented lightweight high-speed file system according to claim 1, wherein the user data storage module comprises a plurality of sector spaces, each sector space is divided into a data storage area, a backup area and a reserved area, and each sector space is correspondingly provided with an information table, a record table and a free table;
the data storage area and the backup area are used for synchronously executing write-in operation when the data block of the cache end executes write-in operation to the allocated sector space; and the backup area is also used for deleting the written data blocks after the data blocks at the cache end are written into the allocated sector space.
3. The embedded heterogeneous platform-oriented lightweight high-speed file system according to claim 1, wherein the high-speed file system is further in communication connection with a file system storage module; the file system storage module is used for storing the log update file of the high-speed file system.
4. The embedded heterogeneous platform-oriented lightweight high-speed file system according to any one of claims 1 to 3, wherein the FPGA end and the CPU end communicate through a PCIE-MAP protocol via a PCIE interface;
the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver;
the application layer is used for carrying out FTP or NFS general access by a user through a remote network;
the file system layer is used for calling the universal file system conversion driver and the PCIE-MAP driver so as to realize universal conversion of file system information recorded by a high-speed file system at the PS end of the embedded soft core and further realize FTP or NFS universal access.
5. A lightweight high-speed file processing method facing an embedded heterogeneous platform is characterized by being applied to the lightweight high-speed file system facing the embedded heterogeneous platform as claimed in any one of claims 1 to 4; the method comprises the following steps:
after the user data storage module finishes the storage record or deletion operation each time, the information table controls the idle table to arrange the continuous space blocks of the residual space so as to sort according to the size of the storage space of the space blocks, and therefore the largest continuous space block is determined to be used as the space block for storing the record next time;
carrying out high-speed data acquisition through an FPGA end, and detecting the cache data volume of a cache end of the FPGA end;
when the cache data volume reaches a set value, the PS end of the embedded soft core triggers write operation;
when the high-speed file system detects the write operation, allocating enough continuous linear sector space in the maximum continuous space block for data storage in the free table according to the size of the data block of the cache end so as to store the data block of the cache end to the allocated sector space at one time;
after the data block is stored, the information table controls the record table to execute storage records, wherein the storage records comprise the initial address, the end address and the data type of the data block which is recorded and stored in the user data storage module each time;
and after detecting that the user data storage module executes the deletion operation, the high-speed file system restores the sector space corresponding to the deletion operation to the idle table according to a linear relation.
6. The embedded heterogeneous platform-oriented lightweight high-speed file processing method according to claim 5, wherein the user data storage module comprises a plurality of sector spaces, each sector space is divided into a data storage area, a backup area and a reserved area, and each sector space is correspondingly provided with an information table, a record table and a free table; when the high-speed file system detects the write operation, allocating enough continuous linear sector space in the maximum continuous space block for data storage according to the size of the data block at the cache end in the free table, so as to store the cache data block to the allocated sector space at one time, wherein the step comprises the following steps:
when the high-speed file system detects the write operation, the size of the data block of the cache end is obtained, and enough continuous linear sector space is distributed in the maximum continuous space block in the free table according to the size of the data block of the cache end;
simultaneously sending the data block of the cache end to the backup area and the data storage area of the allocated sector space;
and deleting the data blocks in the backup area after all the data blocks are written into the backup area and the data storage area.
7. The embedded heterogeneous platform-oriented lightweight high-speed file processing method according to claim 5, wherein the high-speed file system is further connected with a file system storage module in a communication manner; the method further comprises the following steps:
and sending the log update file generated by the high-speed file system to the file system storage module for storage.
8. The embedded heterogeneous platform-oriented lightweight high-speed file processing method according to claim 7, further comprising:
acquiring a preset log updating mode, wherein the log updating mode comprises updating according to recording time, updating according to writing capacity and establishing updating according to files;
dividing the file system storage module into a plurality of storage areas according to the number of the log updating modes;
the step of sending the log update file generated by the high-speed file system to the file system storage module for storage comprises:
and when the updated log is detected, acquiring a log updating mode corresponding to the updated log, and storing the updated log to a corresponding storage area in the file system storage module according to the corresponding log updating mode.
9. The embedded heterogeneous platform-oriented lightweight high-speed file processing method according to claim 7, further comprising:
dividing each storage area of the file system storage module into a plurality of rollback areas with a set sequence;
the step of sending the log update file generated by the high-speed file system to the file system storage module for storage further comprises:
when a current log to be stored in the storage area is detected, acquiring a last rollback area corresponding to the last log, and storing the current log in a next rollback area corresponding to the last rollback area; and the next rollback area corresponding to the last rollback area is the first rollback area.
10. The embedded heterogeneous platform-oriented lightweight high-speed file processing method according to any one of claims 5 to 9, wherein the FPGA side and the CPU side communicate via a PCIE-MAP protocol via a PCIE interface; the CPU end comprises an application layer, a file system layer, a universal file system conversion driver and a PCIE-MAP driver; the method further comprises the following steps:
acquiring a general access request initiated to the application layer by a user through a remote network, wherein the general access request is FTP or NFS access;
acquiring file information recorded by a high-speed file system from the embedded soft core PS end of the FPGA end through a PCIE interface, wherein the file information is stored in the user data storage module;
and calling the universal file system conversion driver and the PCIE-MAP driver through the file system layer to carry out universal conversion on the file information, thereby realizing universal access of the file information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310123448.1A CN115826884B (en) | 2023-02-16 | 2023-02-16 | Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310123448.1A CN115826884B (en) | 2023-02-16 | 2023-02-16 | Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115826884A true CN115826884A (en) | 2023-03-21 |
| CN115826884B CN115826884B (en) | 2023-04-18 |
Family
ID=85521633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310123448.1A Active CN115826884B (en) | 2023-02-16 | 2023-02-16 | Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115826884B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050187985A1 (en) * | 2001-12-21 | 2005-08-25 | Edwards John K. | File system defragmentation technique via write allocation |
| CN106326384A (en) * | 2016-08-16 | 2017-01-11 | 中国科学院长春光学精密机械与物理研究所 | File storage method suitable for high-speed mass storage based on FPGA (Field Programmable Gate Array) |
| CN106326229A (en) * | 2015-06-17 | 2017-01-11 | 百富计算机技术（深圳）有限公司 | Method and device for file storage of embedded system |
| JP2020135656A (en) * | 2019-02-22 | 2020-08-31 | パナソニックＩｐマネジメント株式会社 | File management system |
| CN114860158A (en) * | 2022-04-11 | 2022-08-05 | 中国电子科技集团公司第五十二研究所 | High-speed data acquisition and recording method |
-
2023
- 2023-02-16 CN CN202310123448.1A patent/CN115826884B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050187985A1 (en) * | 2001-12-21 | 2005-08-25 | Edwards John K. | File system defragmentation technique via write allocation |
| CN106326229A (en) * | 2015-06-17 | 2017-01-11 | 百富计算机技术（深圳）有限公司 | Method and device for file storage of embedded system |
| CN106326384A (en) * | 2016-08-16 | 2017-01-11 | 中国科学院长春光学精密机械与物理研究所 | File storage method suitable for high-speed mass storage based on FPGA (Field Programmable Gate Array) |
| JP2020135656A (en) * | 2019-02-22 | 2020-08-31 | パナソニックＩｐマネジメント株式会社 | File management system |
| CN114860158A (en) * | 2022-04-11 | 2022-08-05 | 中国电子科技集团公司第五十二研究所 | High-speed data acquisition and recording method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115826884B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101055511B (en) | Memory array system and its data operation method | |
| US7694087B2 (en) | Duplicate data storing system, duplicate data storing method, and duplicate data storing program for storage device | |
| US7293133B1 (en) | Performing operations without requiring split mirrors in a multi-class file system | |
| US20050231765A1 (en) | Information recording medium, data processing apparatus and data processing method | |
| US8533420B2 (en) | Thin provisioned space allocation | |
| US5504888A (en) | File updating system employing the temporary connection and disconnection of buffer storage to extended storage | |
| JP2003187203A (en) | File allocation table management by memory controller for memory card | |
| US9043546B2 (en) | Sliding-window multi-class striping | |
| JPH0754478B2 (en) | Data storage space and method for managing data and apparatus for managing data units | |
| JPH09510806A (en) | Data storage management for network interconnect processors | |
| CN107391774A (en) | The rubbish recovering method of JFS based on data de-duplication | |
| CN111797058B (en) | Universal file system and file management method | |
| US6629203B1 (en) | Alternating shadow directories in pairs of storage spaces for data storage | |
| US7200625B2 (en) | System and method to enhance availability of a relational database | |
| EP0694831A2 (en) | Computer system having storage unit provided with data compression function andmethod of management of storage area thereof | |
| US11954346B2 (en) | Using multi-stream storage devices in log structured storage system | |
| CN115826884B (en) | Lightweight high-speed file system facing embedded heterogeneous platform and processing method thereof | |
| CN110750211A (en) | Storage space management method and device | |
| WO2023124423A1 (en) | Storage space allocation method and apparatus, and terminal device and storage medium | |
| US7657719B2 (en) | Controller for a copy operation between a host computer and hard disks, a control method thereof, and a recording medium storing a program for executing the control method | |
| CN103064793B (en) | Simplify configuration capacity recovery method and system | |
| CN114780043A (en) | Data processing method and device based on multilayer cache and electronic equipment | |
| CN114625318A (en) | Data writing method, device and equipment applied to solid state disk | |
| CN111722802B (en) | Storage space allocation method, device and equipment for metadata LSA volume | |
| CN113495807A (en) | Data backup method, data recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |