CN110727402B - High-speed FC data real-time receiving and frame loss-free storage method - Google Patents
High-speed FC data real-time receiving and frame loss-free storage method Download PDFInfo
- Publication number
- CN110727402B CN110727402B CN201910863152.7A CN201910863152A CN110727402B CN 110727402 B CN110727402 B CN 110727402B CN 201910863152 A CN201910863152 A CN 201910863152A CN 110727402 B CN110727402 B CN 110727402B
- Authority
- CN
- China
- Prior art keywords
- data
- frame
- sector
- len
- sectors
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
A high-speed FC data real-time receiving frame-loss-free storage method is characterized in that: correctly analyzing the high-speed FC data frames received in real time, storing the frames into a local file without frame loss, and selecting a logic partition on a hard disk for reading and writing a disk sector; creating a disk file in the selected logical partition, and locking the selected logical partition; moving a pointer to the beginning of a disk sector to be written in a file operation, etc.; the data recording adopts a direct sector read-write technology; the read-write mode abandons the commonly used file format, greatly reduces the addressing time loss caused by repeatedly inquiring the file allocation table and discontinuous sector recording in the file management mode, thereby improving the data recording speed and realizing the requirement of no frame loss of the received data.
Description
Technical Field
The invention relates to a high-speed signal data real-time receiving and frame loss-free storage method, belonging to the technical field of computer data processing.
Background
At present, high-speed signal transmission equipment has large signal transmission data volume and high speed, and when data is transmitted for a long time, application software needs to receive and store the high-speed data in real time in order to check the content of the transmitted high-speed data. The data is generally stored by directly storing the received data into a local file, and the storage process is as shown in fig. 1. When high-speed data is received and stored for a long time, the rate of reading and writing the file by the hard disk is reduced, and the rate of receiving the high-speed FC data far exceeds the rate of storing the data into the file, so that the memory is insufficient, and the real-time frame-loss-free storage of the high-speed data cannot be realized. In summary, this method does not fulfill the practical need of recording large amounts of data continuously in real time at high speed.
Disclosure of Invention
The invention discloses a high-speed FC data real-time receiving and frame-loss-free storage method, which aims to solve the problems that the read-write speed of a hard disk is reduced, the memory is insufficient, and the data real-time receiving and frame-loss-free storage cannot be realized because the received high-speed data is directly written into a file at present.
The technical scheme of the invention is as follows: the method comprises the following steps:
s1: selecting a logic partition on a hard disk for reading and writing a disk sector;
s2: creating a disk file in the selected logical partition, and locking the selected logical partition;
s3: moving a pointer to the beginning of a disk sector to be written in a file operation;
s4: applying for a 1M buffer area in application software;
s5: after receiving data, writing the FC frame into a disk sector after recombining the FC frame;
s6: after stopping receiving the data, if the data still exists in the 1M buffer area, calculating the number of sectors occupied by the data in the 1M buffer area, and writing the content in the 1M buffer area into the disk sectors;
s7: reading and restoring the recombined FC frames from the disk sector in sequence, and analyzing and writing the restored FC frames into a file;
s8: and moving the position of the disk file to the position of the initial written data, starting reading from the FC data of the first frame, and reading one frame of recombined FC data from the disk each time.
Preferably, in S1, to ensure that the high-speed data can be correctly read from and written to the disk sector, no other files can be placed in the selected logical partition on the hard disk.
Preferably, upon receiving the FC frame reassembly, the data occupying one sector is divided into three parts, respectively, an FC frame length, i.e., the 2-byte original FC frame length field added in front of the FC data per frame, an FC frame, i.e., len bytes, and the remaining part.
Preferably, the FC frame reassembly is received, the data occupying multiple sectors is also divided into three parts, which are the FC frame length of the 1 st sector, i.e. the 2-byte original FC frame length field added in front of the FC data of each sector, and the FC frame data between the FC frame length of the 1 st sector and the FC frame data of the N th sector FC frame share len bytes and the remaining space.
Preferably, the method for calculating the number of sectors occupied by each frame of FC data comprises: defining the length of a received FC data frame as FC _ LEN, adding an FC frame length field of two bytes and ensuring that the recombined data occupies an integral number of sectors to be recombined FC _ LEN _ NEW, wherein if FC _ LEN +2 can be divided by 512, FC _ LEN _ NEW is FC _ LEN + 2; otherwise, FC _ LEN _ NEW ═ ((FC _ LEN +2)/512+1) × 512.
Preferably, the number of sectors N occupied by the FC frame data is calculated according to the FC frame length len. The specific calculation method is as follows: if len +2 can be divided by 512, the number of sectors N is (len + 2)/512; otherwise, the sector number N is (len +2)/512+ 1.
Due to the adoption of the technical scheme, the invention has the following beneficial effects:
1. the data recording adopts direct sector read-write technology. The read-write mode abandons the commonly used file format, greatly reduces the addressing time loss caused by repeatedly inquiring the file allocation table and discontinuous sector recording in the file management mode, thereby improving the data recording speed and realizing the requirement of no frame loss of the received data.
2. The data buffer area is specially designed for storing data received in real time, and when the buffer area is full, the content of the buffer area is written into the sector, so that the writing efficiency of the disk sector is improved.
3. Firstly, the received data is written into the sector of the disk in real time, so that the non-frame-loss processing of real-time and high-speed data is realized; after the data is received, the data is read from the sector to be analyzed and stored in a file, so that the correct analysis and storage of the data are realized.
Drawings
FIG. 1 is a flow chart of high-speed FC data real-time reception and no-frame-loss storage;
FIG. 2 shows a storage format of a sector occupied by a reorganized FC frame;
FIG. 3 shows a storage format for occupying multiple sectors after FC frame reassembly;
FIG. 4 is a storage format of reorganized FC frame data in a disk sector.
Detailed Description
The present invention will be explained in detail by the following examples, which are disclosed for the purpose of protecting all technical improvements within the scope of the present invention.
A storage method for receiving high-speed FC data in real time without frame loss comprises the steps that application software receives high-speed FC data from a bottom layer in real time, and the received FC data frames are analyzed and then stored in a local file. The implementation mode of the real-time receiving and frame loss-free storage of the high-speed FC data comprises the following steps:
1) a logical partition or volume is selected on the hard disk, such as selecting the "E" disk, and 400G of space is allocated for the E disk. The E disk cannot store other files;
2) calling a system function to create a disk file on the selected logical partition E disk, and locking the selected logical partition;
3) calling a system function to move a pointer to the beginning of a disk sector to be written in a file operation mode, wherein the pointer is shifted by 4G;
4) applying for a 1M buffer area in application software;
5) calling a bottom layer driving function to start monitoring and start receiving data;
6) FC data is received in the callback function, including FC data per frame and FC data length per frame. Writing the FC frame into a disk sector after being recombined;
7) calling a bottom layer driving function to stop monitoring and stop receiving data;
8) if the 1M buffer zone still contains data, calculating the number of sectors occupied by the data in the 1M buffer zone, writing the content in the 1M buffer zone into the disk sectors, and emptying the 1M buffer zone;
9) reading and restoring the recombined FC frames from the disk sector in sequence, and analyzing and writing the restored FC frames into a file. Firstly, moving a disk file pointer to a position 4G of an E disk where data is written, and starting to read from first frame FC data; FC data is reassembled one frame at a time from disk. Restoring and analyzing each frame of recombined FC data and writing the data into a file;
the specific method for writing the received FC frame into the disk sector in step (6) is as follows: receiving FC frame recombination, adding two bytes of original FC frame length field in front of each frame of FC data, calculating the number of sectors occupied by the recombined FC data, ensuring that each frame of recombined FC data occupies an integral number of sectors, and calculating the number of sectors occupied by each frame of FC data: defining the length of a received FC data frame as FC _ LEN, adding an FC frame length field of two bytes and ensuring that the recombined data occupies an integral number of sectors to be recombined FC _ LEN _ NEW, wherein if FC _ LEN +2 can be divided by 512, FC _ LEN _ NEW is FC _ LEN + 2; otherwise, FC _ LEN _ NEW ═ ((FC _ LEN +2)/512+1) × 512; writing the recombined FC data into a 1M buffer area, and if the 1M buffer area is not full and the remaining available length of the 1M buffer area is longer than the maximum frame length, writing the recombined FC data into the 1M buffer area; the buffer contents are written to disk sectors. If the 1M buffer area is not full and the remaining available length of the 1M buffer area is less than the maximum frame length, calculating the total sector number occupied by the data in the buffer area, writing the content of the buffer area into a disk sector, emptying the 1M buffer area, and restarting to store the recombined FC data.
The specific method for restoring the FC data frame from the disk sector in step (9) is as follows: reading a sector, obtaining the length len of FC frames of the first two bytes in the sector, and calculating the number N of sectors occupied by FC frame data according to the length len of the FC frames, wherein the calculation method comprises the following steps: if len +2 can be divided by 512, the number of sectors N is (len + 2)/512; otherwise, the number of the sectors N is (len +2)/512+1, and the FC frame is restored according to the number of the sectors N occupied by the FC frame. The specific calculation method comprises the following steps: if the number of sectors N is not more than 1 and the storage format of each frame of FC data in the sectors is shown in FIG. 3, directly acquiring data of len bytes from the third byte in the sector 1, namely FC frame data; if the number of sectors N is greater than 1, and the storage format of each frame of FC data in the sectors is shown in fig. 4, the remaining N-1 sectors are continuously read from the disk, starting from the third byte in the sector 1 until the data with the length len in the sector N is the FC frame data.
Through the above steps, the restoration process of the FC data frame is completed.
The present invention is not described in detail in the prior art.
Claims (4)
1. A high-speed FC data real-time receiving frame-loss-free storage method is characterized in that: the method correctly analyzes the high-speed FC data frames received in real time, and stores no lost frames into a local file, and comprises the following steps:
s1: selecting a logic partition on a hard disk for reading and writing a disk sector;
s2: creating a disk file in the selected logical partition, and locking the selected logical partition;
s3: moving a pointer to the beginning of a disk sector to be written in a file operation;
s4: applying for a 1M buffer area in application software;
s5: after receiving data, writing the FC frame into a disk sector after recombining the FC frame;
s6: after stopping receiving the data, if the data still exists in the 1M buffer area, calculating the number of sectors occupied by the data in the 1M buffer area, and writing the content in the 1M buffer area into the disk sectors;
s7: reading and restoring the recombined FC frames from the disk sector in sequence, and analyzing and writing the restored FC frames into a file;
s8: moving the position of a disk file to a position of initial write-in data, starting reading from first frame FC data, and reading a frame of recombined FC data from the disk each time;
writing the recombined FC data into the 1M buffer if the 1M buffer is not full and the remaining available length of the 1M buffer is greater than the maximum frame length; if the 1M buffer area is not full and the remaining available length of the 1M buffer area is less than the maximum frame length, calculating the total sector number occupied by the data in the buffer area, and writing the data content of the buffer area into the disk sector; the 1M buffer is then emptied and storage of the reassembled FC data resumes.
2. The method of claim 1 for storing real-time receiving and no-frame-loss of high-speed FC data, wherein: dividing each frame of received FC data according to sectors, adding a 2-byte original FC frame length field in front of each frame of received FC data, and calculating the number of sectors occupied by the recombined data, wherein the calculation method comprises the following steps: defining the length of a received FC data frame as FC _ LEN, adding an FC frame length field of two bytes and ensuring that the recombined data occupies an integral number of sectors to be recombined FC _ LEN _ NEW, wherein if FC _ LEN +2 can be divided by 512, FC _ LEN _ NEW is FC _ LEN + 2; otherwise, FC _ LEN _ NEW ═ ((FC _ LEN +2)/512+1) × 512.
3. The method of claim 1 for storing real-time receiving and no-frame-loss of high-speed FC data, wherein: in step S7, the restoration of the FC data frame in the disk sector should first read a sector, obtain the FC frame lengths len of the first two bytes in the sector, and calculate the number N of sectors occupied by the FC frame according to the FC frame lengths len, where the calculation method is as follows: if len +2 can be divided by 512, the number of sectors N is (len + 2)/512; otherwise, the sector number N is (len +2)/512+ 1.
4. The method of claim 1 for storing real-time receiving and no-frame-loss of high-speed FC data, wherein: the calculation method for restoring the FC frame according to the number N of sectors occupied by the FC frame comprises the following steps: if the number N of the sectors is not more than 1, directly obtaining data of len bytes from the third byte in the sector 1, namely FC frame data, wherein the storage format of each frame of FC data in the sectors is the storage format occupied by a plurality of sectors after FC frame recombination; if the number of sectors N is greater than 1, the remaining N-1 sectors are continuously read from the third byte in the sector 1 until the data with the length len in the sector N is the FC frame data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910863152.7A CN110727402B (en) | 2019-09-12 | 2019-09-12 | High-speed FC data real-time receiving and frame loss-free storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910863152.7A CN110727402B (en) | 2019-09-12 | 2019-09-12 | High-speed FC data real-time receiving and frame loss-free storage method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110727402A CN110727402A (en) | 2020-01-24 |
| CN110727402B true CN110727402B (en) | 2022-03-15 |
Family
ID=69218978
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910863152.7A Active CN110727402B (en) | 2019-09-12 | 2019-09-12 | High-speed FC data real-time receiving and frame loss-free storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110727402B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6269420B1 (en) * | 1997-12-12 | 2001-07-31 | Olympus Optical Co., Ltd. | Information recording/reproducing apparatus reducing disk access frequency to file management area and sharply accelerating record processing and reproduction processing |
| CN104038327A (en) * | 2014-06-25 | 2014-09-10 | 电子科技大学 | Error retransmission method for FC (fiber channel) network |
| CN104536699A (en) * | 2014-12-11 | 2015-04-22 | 中国科学院声学研究所 | Streamed data write-in method based on embedded file system |
| CN104714918A (en) * | 2013-12-15 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Method for receiving and buffering high-speed FC bus data in host computer environment |
| CN104714832A (en) * | 2013-12-14 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Buffer management method used for airborne data network asynchronous data interaction area |
| CN107276834A (en) * | 2017-08-10 | 2017-10-20 | 电子科技大学 | Optical-fibre channel flow analysis recording method and device under a kind of aviation electronics environment |
-
2019
- 2019-09-12 CN CN201910863152.7A patent/CN110727402B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6269420B1 (en) * | 1997-12-12 | 2001-07-31 | Olympus Optical Co., Ltd. | Information recording/reproducing apparatus reducing disk access frequency to file management area and sharply accelerating record processing and reproduction processing |
| CN104714832A (en) * | 2013-12-14 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Buffer management method used for airborne data network asynchronous data interaction area |
| CN104714918A (en) * | 2013-12-15 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Method for receiving and buffering high-speed FC bus data in host computer environment |
| CN104038327A (en) * | 2014-06-25 | 2014-09-10 | 电子科技大学 | Error retransmission method for FC (fiber channel) network |
| CN104536699A (en) * | 2014-12-11 | 2015-04-22 | 中国科学院声学研究所 | Streamed data write-in method based on embedded file system |
| CN107276834A (en) * | 2017-08-10 | 2017-10-20 | 电子科技大学 | Optical-fibre channel flow analysis recording method and device under a kind of aviation electronics environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110727402A (en) | 2020-01-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6324604B1 (en) | Magnetic disk storage for storing data in disk block size from fixed length of host block in non-integer multiple of the disk block size | |
| US7733892B2 (en) | Buffer management method based on a bitmap table | |
| CN110442560B (en) | Log replay method, device, server and storage medium | |
| JP3763845B2 (en) | Packing variable-length records in fixed blocks | |
| US10120588B2 (en) | Sliding-window multi-class striping | |
| CN109802684B (en) | Method and device for data compression | |
| CN103370691A (en) | Managing buffer overflow conditions | |
| CN115576505B (en) | Data storage method, device and equipment and readable storage medium | |
| CN109739843A (en) | Block chain data read-write method, system, equipment and storage medium | |
| WO2002056169A2 (en) | Method and system of reformatting data blocks for storage as larger size data blocks | |
| CN101174198B (en) | Data storage system and data access method thereof | |
| CN106681659A (en) | Data compression method and device | |
| CN111752987B (en) | Database access method, device, storage medium and computer equipment | |
| CN109375868B (en) | Data storage method, scheduling device, system, equipment and storage medium | |
| US10803018B2 (en) | Compressed data rearrangement to optimize file compression | |
| CN101183298A (en) | SCSI data read-write method, system and device | |
| CN110727402B (en) | High-speed FC data real-time receiving and frame loss-free storage method | |
| CN109213745A (en) | A kind of distributed document storage method, device, processor and storage medium | |
| CN110555890A (en) | Memory management method and system | |
| CN102270179B (en) | For the data Storage and Processing method based on sheet unit and the system thereof of CAD system | |
| JPH0421041A (en) | Dynamic converting system for file format | |
| CN117312201B (en) | Data transmission method and device, accelerator equipment, host and storage medium | |
| CN115794446B (en) | Message processing method and device, electronic equipment and storage medium | |
| CN112783896B (en) | Method for reducing memory usage rate by loading files | |
| US20230004533A1 (en) | Hybrid intermediate stream format |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |