CN112650448A

CN112650448A - FPGA-based large data volume storage file management method

Info

Publication number: CN112650448A
Application number: CN202011521951.5A
Authority: CN
Inventors: 业瀚青; 唐遒; 李磊; 张健伟; 郭小晨; 田弘博
Original assignee: 8511 Research Institute of CASIC
Current assignee: 8511 Research Institute of CASIC
Priority date: 2020-12-21
Filing date: 2020-12-21
Publication date: 2021-04-13
Anticipated expiration: 2040-12-21
Also published as: CN112650448B

Abstract

The invention discloses a large data volume storage file management method based on FPGA, which adopts FPGA hardware to realize a high-speed solid-state memory interface, fully utilizes the programmable advantage of FPGA, enhances the flexibility portability, and carries a high-speed DDR3 cache chip to solve the requirement of large data burst transmission and balance the fluctuation of the data writing rate of a high-speed solid-state disk memory; the DSP processor realizes the receiving and processing of the file information/solid-state memory data readback; the high-speed eMMC memory chip is added and used for storing file information data, and the purpose of adding the eMMC memory is to enable large data to be uninterruptedly written into the solid-state memory so as to ensure the continuity of data writing.

Description

FPGA-based large data volume storage file management method

Technical Field

The invention belongs to the field of data storage, and particularly relates to a large data volume storage file management method based on an FPGA.

Background

With the increasingly complex and changeable signal environment of modern battlefield, the new system radar, unknown signals and complex waveforms on the battlefield are in endless, and meanwhile, the sampling rate of the front end and the signal resolution of the receiver are improved, so that the electromagnetic data information of the battlefield is increased explosively; sometimes, external electromagnetic environment data cannot be analyzed in real time; the need for data storage and support for post-hoc read-back analysis continues to increase. For this reason, the demand for mass data storage and management is increasing. In the earlier method, a computer mounted disk array is used for data storage, or an FPGA mounted solid state disk is used for data acquisition and direct storage, but many disadvantages are brought, for example, the computer mounted disk array causes very large equipment and cannot meet the requirement of light weight; the FPGA data is directly stored, different data cannot be classified and managed, and data read-back unicity can be caused.

Disclosure of Invention

The invention aims to provide a large-data-volume storage file management method based on an FPGA (field programmable gate array), which meets the requirement of light weight of equipment and the diversity of readback data types. According to the invention, the high bandwidth advantage of FPGA is utilized to complete the collection of big data, special coding characters are typed in through a specific coding module, the characters are used for constructing file information, then flow into a high-speed DDR cache, and are written into a rear-end solid state hard disk when data are read from DDR; the solid-state disk which needs to record the back end simultaneously when writing data writes sector address, write data length, write time, data type and file name information in real time, and the information is recorded in the high-speed eMMC to form a file information entry for storage. The number of the FPGA mounted solid-state disks can be flexibly changed according to the data throughput rate.

The technical solution for realizing the purpose of the invention is as follows: a large data volume storage file management method based on an FPGA (field programmable gate array), wherein the large data volume is larger than 1GB, specifically comprises the following steps:

step 1, receiving original big data by the FPGA, adding special coding characters to a data stream to obtain a coded data stream, and turning to step 2;

step 2, writing the coded data stream into a high-speed DDR3 cache in real time, balancing the fluctuation of the writing rate of the solid-state memory, and turning to step 3;

step 3, reading the data stream coded in the step 2, detecting the coded characters of the data stream, recording the current sector address, updating data length information in the L2/L3 file information after the L2/L3 data is written, generating L1 file information after the storage time is finished, and turning to the step 4;

step 4, writing the data stream containing the special codes in the step 3 into a solid-state memory for storage, and turning to step 5;

step 5, packaging and integrating the file information of the L1, the L2 and the L3 generated in the step 3, sending the information to a DSP processor, and turning to step 6;

step 6, carrying out classification and integration on the file information of L1, L2 and L3 received in the step 5 in the DSP processor, planning a written eMMC storage address, starting eMMC writing operation, and turning to the step 7;

step 7, after receiving the file information synchronization instruction, the DSP processor reports the file information stored in the eMMC to the server, and the step 8 is switched to;

and 8, after receiving the read control instruction, the DSP processor forwards the instruction to the FPGA, the FPGA analyzes the read instruction, extracts a sector initial address and a read total length, reads corresponding data from the solid-state memory and forwards the data to the DSP processor, and the DSP processor forwards and reports the data.

Compared with the prior art, the invention has the remarkable advantages that:

(1) the invention adopts the FPGA to realize the function of file management and realize light weight.

(2) The data read-back is simple and easy to manage, the management of the file information directly corresponds to the management of big data, and the data processing pressure of management software is greatly reduced.

Drawings

Fig. 1 is a data writing flow chart of the large data storage file management method based on the FPGA according to the present invention.

Fig. 2 is a data reading flow chart of the large data storage file management method based on the FPGA according to the present invention.

FIG. 3 is a diagram illustrating a structure of adding special code characters to a data stream according to the present invention.

FIG. 4 is a diagram of a DDR3 read/write state machine control according to the invention.

Detailed Description

The present invention is described in further detail below with reference to the attached drawing figures.

With reference to fig. 1 to 4, a method for managing a large data storage file based on an FPGA includes the following steps:

step 1, receiving original big data by the FPGA, and adding special encoding characters to a data stream by using an encoding and packaging module.

The method for adding special code characters to the original large data stream comprises the following steps:

step 11, dividing the input large data volume into a plurality of segments, as shown in fig. 3, wherein the segments can be divided by a user-defined decision to finally form single file data;

step 12, inputting a data frame header A into each section, wherein the data frame header A contains information such as a creation timestamp, a data type, a creator, a file name and the like; the frame header A comprises easy-to-detect special characters 0xC2C2C2C2, and the length of the data frame header is 1 KB;

step 13, the data frame head A follows the large data flow, namely the data load D, with 1KB as the minimum data granularity and is not sufficient to fill zero; a special character 0xC3C3C3 can be typed into the data load D to form an independent small data segment;

and 14, adding a data frame tail B after the data load D, wherein the data frame tail B comprises the data length of the data load D. The trailer B contains an easily detectable special character 0xD1D1D1D1 (0 xD2D2D2 if the last data), the data trailer length is 1 KB;

after the data stream is encoded, original data is kept, meanwhile, a data file construction basis is achieved for the rear end, 1 file is constructed for every A + D + B, big data are divided into a plurality of files, the files can be continuous or disconnected, and the data stream is flexibly divided.

Step 2, writing the encoded data stream into a high-speed DDR3 cache in real time, balancing the fluctuation of the writing rate of the solid-state memory, and specifically comprising the following steps:

and step 21, buffering the encoded and packaged data in advance through FIFO, and adjusting the time sequence beat to be isolated from the clock.

And 22, the system enters an initialization state after reset, when the pre-cache FIFO data is not 0, detection read-write is carried out, when the pre-cache FIFO data volume in the step 21 is judged to be more than or equal to 1KB, a DDR3 write process is started to be executed, the data in the pre-cache FIFO is transferred to the DDR3, 1KB is written out, and the system returns to the detection read-write state.

Step 23, in the write/read state, when the write idle is used, detecting whether the read DDR3 FIFO can allow transmission, and if so, starting to execute the read process when the data size in the DDR3 is greater than 1 KB. And writing the data read-out in the DDR3 into a read DDR3 FIFO and outputting the data to the outside. When the read operation is finished by 1KB, the read/write state is returned to the detection.

And 24, returning to the initialization state when the data read space is empty when the special characters of 0xD2D2D2 are detected.

Step 3, detecting the read encoded characters in step 2, recording the current sector address, updating the data length information in the L2/L3 file information after the L2/L3 data is written, and finally generating the L1 file information when the storage time length is finished, as shown in fig. 4, specifically comprising the following steps:

step 31, detecting a special character 0xC2C2 in the data frame header a, recording the address of the solid-state memory sector corresponding to the frame header a at this time, extracting information such as a creation time stamp, a data type, a creator and a file name in the frame header a, and caching the information in advance by using a register.

Step 32, receiving a data load D in the data frame, and counting the data amount; if the special character 0xC3C3C3C3 is detected, the sector address at the moment needs to be recorded; while counting begins for each segment 0xC3C 3.

Step 33, detecting the data frame tail B, determining whether the data frame tail B is a special character of 0xD2D2, if not (i.e. 0xD1D 1), returning to step 31; if yes, stopping detection; simultaneously latching the data volume at the moment;

the creation time stamp, data type, creator and filename, and data amount captured in steps 34, 31, and 33 generate file information L2 (corresponding to the detection of 0xC2C 2) or L3 (corresponding to the detection of 0xC3C 3); generating 1 piece of L1 file information including sector initial address, creator, file name and data total amount; meanwhile, the DDR3 cached data is stored in the solid-state memory.

Step 4, writing the big data containing the special codes in the step 3 into a solid-state memory for storage, and concretely comprising the following steps:

step 41, starting write operation on the received write solid-state memory instruction according to the analysis instruction;

step 42, writing the big data into the solid-state memory;

step 43, when detecting 0xD2D2 in the data stream, start the stop write operation.

Step 5, packaging and integrating the L1/L2/L3 file information generated in the step 3 and sending the information to a DSP processor, and specifically comprising the following steps:

step 51, respectively generating corresponding file information messages according to the file information L1/L2/L3 in the step 3, and adding a frame header and a frame tail mark of the message;

step 52, generating L1/L2/L3 level names which are used as level relation definitions to divide the logical folders;

and step 53, sending the message to the DSP through the SRIO interface.

Step 6, the L1/L2/L3 file information received in the step 5 is classified and integrated in the processor, the written eMMC storage address is planned, and the eMMC writing operation is started, and the method specifically comprises the following steps:

step 61, caching the file information of L1/L2/L3 in DSP external DDR3 in advance, and judging the logical hierarchical relationship;

step 62, aiming at the file information of L1/L2/L3, allocating an eMMC memory sector write address;

and step 63, writing the file information preprocessed in the steps 61 and 62 into the eMMC.

7, after receiving the write control instruction, the DSP processor forwards the write control instruction to an FPGA driving module of the solid-state memory; after receiving the file information synchronization instruction, the DSP processor reports the file information stored in the eMMC to the server, and the method specifically comprises the following steps:

step 71, reading out the L1-level file information stored in the eMMC, and simultaneously reading all the L2-level file information corresponding to the current L1 level;

and step 72, summarizing the file information received in the step 71, packaging and sending the file information to a server.

And step 8, data reading process. After receiving the read control instruction, the DSP processor forwards the instruction to the FPGA, the FPGA analyzes the read instruction, extracts information such as a sector initial address and a read total length, reads data corresponding to the solid-state memory from the information and sends the data to the DSP processor, and the DSP processor forwards and reports the data, and the method specifically comprises the following steps:

step 81, the management software reads the specified file, extracts the information in the corresponding file information, including the information of the sector starting address, the total data length and the like, integrates the information into a read instruction and sends the read instruction to the DSP;

step 82, after receiving the read control instruction, the DSP forwards the instruction to the FPGA through the EMIF interface;

step 83, the FPGA decodes the read instruction, extracts the initial address and the length of the sector, reads the data corresponding to the solid-state memory from the initial address and sends the data to the DSP processor;

and step 84, the DSP processor reports the data to the management software after receiving the data.

Claims

1. A large data volume storage file management method based on FPGA is characterized in that the large data volume is larger than 1GB, and specifically comprises the following steps:

2. The method for managing the large data storage file based on the FPGA according to claim 1, wherein in the step 1, a special encoding character is added to the original large data stream, and the steps are as follows:

step 11, dividing the input large data volume into a plurality of sections, and dividing the data of the sections by a user-defined decision to finally form single file data;

step 12, inputting a data frame header A into each section, wherein the data frame header A comprises a creation timestamp, a data type, a creator and a file name; the frame header A comprises easy-to-detect special characters 0xC2C2C2C2, and the length of the data frame header is 1 KB;

step 13, the data frame head A follows the large data flow, namely the data load D, with 1KB as the minimum data granularity and is not sufficient to fill zero; a special character 0xC3C3C3 is typed into the data load D to form an independent small data segment;

step 14, adding a data frame tail B after the data load D, wherein the data frame tail B includes the data length of the data load D, the frame tail B includes a special character 0xD1D1 easy to detect, if the data is the last data, the special character is 0xD2D2, and the data frame tail length is also 1 KB;

after the data stream is encoded, original data is kept, meanwhile, a data file construction basis is achieved for the rear end, 1 file is constructed for every A + D + B, and the big data is divided into a plurality of files.

3. The FPGA-based large data storage file management method as recited in claim 2, wherein the large data is divided into a plurality of files, two adjacent files can be continuous or disconnected, and data streams are flexibly divided.

4. The method for managing the large-data-volume storage file based on the FPGA as recited in claim 2, wherein in the step 2, the encoded data stream is written into a high-speed DDR3 cache in real time, so as to balance fluctuation of a write rate of the solid-state memory, and specifically comprises the following steps:

step 21, pre-buffering the encoded and packaged data stream through FIFO, and adjusting the time sequence beat to be isolated from the clock;

step 22, the system enters an initialization state after being reset, when the pre-cache FIFO data is not 0, detection reading and writing are carried out, when the pre-cache FIFO data volume in the step 21 is judged to be more than or equal to 1KB, a DDR3 writing process is started to be executed, the data in the pre-cache FIFO is transferred to the DDR3, 1KB is written, and the system returns to the detection reading and writing state;

step 23, when executing the write-read state and utilizing the write idle state, detecting whether the read DDR3 FIFO can allow transmission, if yes, and if the data amount in the DDR3 is greater than 1KB, starting to execute the read process; writing the data read-out in the DDR3 into a read DDR3 FIFO and outputting the data to the outside; when the 1KB is read, the state of detecting writing and reading is returned;

5. The method for managing the large-data-volume storage file based on the FPGA according to claim 4, wherein in the step 3, the data stream encoded in the step 2 is read out, and the encoded characters of the data stream are detected, and specifically includes the following steps:

step 31, detecting a special character 0xC2C2C2C2 in the data frame header A, recording a sector address of a solid-state memory corresponding to the frame header A at the moment, extracting a creation time stamp, a data type, a creator and a file name in the frame header A, and caching the information in advance by using a register;

step 32, receiving a data load D in the data frame, and counting the data amount; if the special character 0xC3C3C3C3 is detected, the sector address at the moment needs to be recorded; simultaneously counting each segment of 0xC3C3C3C 3;

step 33, detecting the data frame tail B, judging whether the data frame tail B is a special character of 0xD2D2, if not, namely 0xD1D1, returning to step 31; if so, stopping detection and latching the data volume at the moment;

the creation time stamp, data type, creator and file name and data amount captured in steps 34, 31 and 33 generate file information L2 corresponding to the detection of 0xC2C2, or L3, corresponding to the detection of 0xC3C 3; generating 1 piece of L1 file information including sector initial address, creator, file name and data total amount for each storage task; meanwhile, the DDR3 cached data is stored in the solid-state memory.

6. The method for managing the large-data-volume storage file based on the FPGA according to claim 5, wherein the step 4 includes writing the specially encoded data stream into the solid-state memory for storage, and specifically includes the following steps:

step 41, according to the analyzed instruction, executing write operation on the received instruction for writing the solid-state memory;

step 42, writing the data stream containing the special codes into the solid-state memory;

when detecting 0xD2D2 in the data stream containing the special encoding, step 43, a stop write operation is performed.

7. The FPGA-based large data storage file management method as claimed in claim 6, wherein in step 5, the information of the L1, L2 and L3 files generated in step 3 is packaged, integrated and sent to the DSP processor, and the method specifically comprises the following steps:

step 51, respectively generating corresponding file information messages according to the file information of L1, L2 and L3 in the step 3, and adding a frame header and a frame tail mark of the message;

step 52, generating L1, L2 and L3 hierarchical names which are used as hierarchical relation definitions so as to divide the logical folders;

and step 53, sending the message to the DSP through the SRIO interface.

8. The method as claimed in claim 7, wherein in step 6, the L1, L2 and L3 file information received in step 5 is classified and integrated in the processor, the eMMC storage address for writing is planned, and the eMMC write operation is started, specifically comprising the following steps:

step 61, judging the logical hierarchical relationship aiming at the received file information of L1, L2 and L3, caching in advance and writing into a DSP plug-in DDR3 memory;

step 62, aiming at the file information of L1, L2 and L3, allocating eMMC memory sector write addresses and starting eMMC write operation;

and step 63, writing the file information cached and preprocessed in the steps 61 and 62 into the eMMC.

9. The method for managing the large-data-volume storage file based on the FPGA according to claim 1, wherein the step 7 is a file synchronization operation, and the DSP processor reports the file information stored in the eMMC to the server after receiving a file information synchronization instruction, specifically comprising the steps of:

10. The method for managing the large-data-volume storage file based on the FPGA according to claim 1, wherein step 8 is a data reading flow, the DSP processor receives a read control instruction and then forwards the instruction to the FPGA, the FPGA analyzes the read instruction, extracts information such as a sector start address and a read total length, reads data corresponding to the solid-state memory from the read instruction and sends the data to the DSP processor, and the DSP processor performs forwarding reporting, specifically comprising the following steps:

and step 84, the DSP processor reports the received data.