CN110019051A - A kind of high-speed data recording method based on universal document system - Google Patents
A kind of high-speed data recording method based on universal document system Download PDFInfo
- Publication number
- CN110019051A CN110019051A CN201810350049.8A CN201810350049A CN110019051A CN 110019051 A CN110019051 A CN 110019051A CN 201810350049 A CN201810350049 A CN 201810350049A CN 110019051 A CN110019051 A CN 110019051A
- Authority
- CN
- China
- Prior art keywords
- acquisition means
- data
- data acquisition
- processor
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000003139 buffering effect Effects 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 2
- 230000003111 delayed effect Effects 0.000 claims 1
- 238000004080 punching Methods 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 4
- 230000008901 benefit Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of high-speed data recording methods based on universal document system, data acquisition means pass through PCIe bus interconnection together with processor, memory bank, after data are acquired by data acquisition means, being put into data acquisition means, locally plug-in memory is buffered, and notifies that the processor is handled；Then the plug-in memory of data acquisition means is mapped in local spatial, after receiving data acquisition means notice, buffer zone address of the application program usage data buffer address as write-in file；Data, are directly written in memory bank by the DirectIO characteristic for finally utilizing mainstream file system by the plug-in memory of data acquisition means.Whole process of the present invention realizes the zero-copy of data, and higher data recording speed may be implemented, and overall power is reduced, and design based on universal document system, convenient for the use and export of file.
Description
Technical field
The present invention relates to network technique field, in particular to a kind of high-speed data recording side based on universal document system
Method.
Background technique
Current general data recording process mainly includes following steps:
It is cached 1. data are transported in processor memory by data collecting card by internal bus (usually PCIe bus)；
2. processor will be in the caching of data copy to file system；
3. data are buffered by file system and are written in memory bank with document form by processor.
This method major advantage is write data into normative document based on general file system, is existed convenient for data
The use of distinct device and mutually copy；The disadvantage is that being more than given pace in data rate, processor is by data copy to file
When in system cache, a large amount of processor resources can be occupied, when data copy, which is not achieved, in processor processing capacity requires, it may appear that
The bottleneck of data record, while considerably increasing overall power.
In addition the maximum bandwidth between processor and external bus is also the bottleneck for limiting data recording speed.
Currently in order to improving data recording speed mainly with the following method:
In file write-in, universal document system mode is not used, data are directly written to memory bank by processor buffering
In " block ", avoid to occupy a large amount of processor resources progress data copies in this way, but the disadvantage is that do not use general file system
System, data use and export are required to special-purpose software cooperation completion, cause using it is inconvenient with it is not general.This method simultaneously
The limitation of the maximum bandwidth between processor and external bus cannot still be got rid of.
Summary of the invention
The present invention makes to adopt to solve the above problems, provide a kind of high-speed data recording method based on universal document system
Data after collection are directly write in memory bank by original buffer area, without thus getting rid of using processor memory
Processor and external bus maximum bandwidth limit；Whole process realizes the zero-copy of data, and higher data may be implemented
Writing speed, and overall power is reduced, and design based on universal document system, convenient for the use and export of file, finally
The data file of generation uses standard file system format.
In order to achieve the above objectives, the technical solution adopted by the present invention is that: a kind of high-speed data based on universal document system
Recording method, comprising:
S1, data acquisition means pass through PCIe bus interconnection together with processor, memory bank, wherein data acquisition means and storage
Body is as PCIe bus endpoint device；
After S2, data are acquired by data acquisition means, being put into data acquisition means, locally plug-in memory is buffered, and is notified
The processor is handled；
S3, processor are mapped to local spatial after os starting, drive load, by the plug-in memory of data acquisition means
In, and this space is converted into user's space, and as data buffering address；
S4, receive data acquisition means notice after, application program usage data buffer address as be written file buffering
Regional address；
S5, using the DirectIO characteristic of mainstream file system under operating system, it is directly that data are plug-in by data acquisition means
Memory is written in memory bank.
Further, data acquisition means are preferably FPGA device；
Further, operating system is preferably Linux, vxworks operating system；
Further, mainstream file system is to support the file system of DirectIO characteristic, preferably ext4 file system；
Further, data acquisition means notice is realized by processor interrupt response；
Further, memory bank is preferably the storage array of single storage dish or multiple storage dishes composition.
In conclusion the present invention has following advantages:
1, higher data recording speed may be implemented.In universal method, the bottleneck for limiting writing speed is mainly at processor
Manage speed, processor external bus bandwidth.In the present invention, writing speed is no longer limited by above-mentioned bottleneck, as long as selecting more
Memory bank, so that it may obtain bigger data recording speed；
2, recording data files are designed using universal document system, convenient for the use and export of file, while universal document system
It is mature technology, has higher reliability and flexibility than user-defined file system；
3, due to greatly reducing processor load, when so that recording at full speed, whole system power consumption is greatly reduced, reduced power consumption
It is approximately equal to the difference that processor is fully loaded with power consumption and no-load power consumption；
4, since system writing speed is no longer limited by processor ability, so that processor no longer needs to select high-end place when selecting
Device is managed, system cost is reduced.
Detailed description of the invention
Fig. 1 is composition schematic diagram of the present invention.
Specific embodiment
The invention will be further described with reference to the accompanying drawings and embodiments:
Embodiment 1:
A kind of high-speed data recording method based on universal document system, in conjunction with the elaboration of attached drawing 1, data acquisition means and processing
Device, memory bank pass through local bus together, are specifically PCIe bus interconnection, wherein data acquisition means, memory bank conduct
PCIe bus endpoint device, while data acquisition means are preferably FPGA device.
After data collector, i.e. FPGA device acquire data, it is put into its local plug-in memory and is buffered, and at notice
Reason device is handled, and in processor in operating system, preferably Linux or vxworks operating system, starting, drive load
Afterwards, the plug-in memory of data acquisition means is mapped in local spatial, and this space is converted into user's space, and conduct
Data buffering address provision is used with program, and after receiving data acquisition means notice, application program just uses this ground
Buffer zone address of the location as write-in file, it is notable that data acquisition means notice generally passes through implement of interruption function.
When file is written, mainstream file system under operating system preferably supports the file system of DirectIO characteristic, tool
Body can be ext4 file system, and utilize its DirectIO characteristic, directly by data by buffer area, i.e., outside data acquisition means
Memory is hung, is written in memory bank, without passing through file system buffer area.
During this, processor does not need to carry out data handling work, it is only necessary to control storage control in memory bank
Start and stop, to greatly reduce processor load, when so that recording at full speed, whole system power consumption is greatly reduced, and
System writing speed is no longer limited by processor ability.
Certainly, the data format of final entry is standard file system file format, since the data after acquisition are by original
The plug-in memory in buffer area, i.e. data acquisition means, directly writes in memory bank, and whole process realizes the zero-copy of data,
And the data file finally generated uses standard file system format, then writing speed may be up to 6GB/s, and can further expand
Exhibition, that is, select more memory banks, so that it may obtain bigger data recording speed.
The technical concepts and features of above embodiment only to illustrate the invention, its object is to allow be familiar with technique
People cans understand the content of the present invention and implement it accordingly, and it is not intended to limit the scope of the present invention.It is all according to the present invention
The equivalent transformation or modification that Spirit Essence is done, should be covered by the protection scope of the present invention.
Claims (6)
1. a kind of high-speed data recording method based on universal document system characterized by comprising
S1, data acquisition means pass through PCIe bus interconnection together with processor, memory bank, wherein the data acquisition means and
The memory bank is as PCIe bus endpoint device；
After S2, data are acquired by the data acquisition means, being put into the data acquisition means, locally plug-in memory is delayed
Punching, and notify that the processor is handled；
S3, processor are mapped to local after os starting, drive load, by the plug-in memory of the data acquisition means
In space, and this space is converted into user's space, and as data buffering address；
S4, after receiving data acquisition means notice, the application program that runs on processor uses the data buffering address
Buffer zone address as write-in file；
S5, using the DirectIO characteristic of mainstream file system under operating system, it is directly that data are plug-in by data acquisition means
Memory is written in memory bank.
2. a kind of high-speed data recording method based on universal document system according to claim 1, it is characterised in that: described
Data acquisition means are FPGA device.
3. a kind of high-speed data recording method based on universal document system according to claim 2, it is characterised in that: described
Operating system is Linux, vxworks operating system.
4. a kind of high-speed data recording method based on universal document system according to claim 3, it is characterised in that: described
Mainstream file system is to support the file system of DirectIO characteristic.
5. a kind of high-speed data recording method based on universal document system according to claim 4, it is characterised in that: described
Data acquisition means notice is realized by processor interrupt response.
6. a kind of high-speed data recording method based on universal document system according to claim 5, the memory bank is single
The storage array of a storage dish or multiple storage dishes composition.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810350049.8A CN110019051A (en) | 2018-04-18 | 2018-04-18 | A kind of high-speed data recording method based on universal document system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810350049.8A CN110019051A (en) | 2018-04-18 | 2018-04-18 | A kind of high-speed data recording method based on universal document system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019051A true CN110019051A (en) | 2019-07-16 |
Family
ID=67188289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810350049.8A Pending CN110019051A (en) | 2018-04-18 | 2018-04-18 | A kind of high-speed data recording method based on universal document system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019051A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021093249A1 (en) * | 2019-11-13 | 2021-05-20 | 国微集团(深圳)有限公司 | Method for external device to access computer memory |
| CN114780471A (en) * | 2022-04-20 | 2022-07-22 | 湖南艾科诺维科技有限公司 | Data acquisition and storage system and method |
| CN114860158A (en) * | 2022-04-11 | 2022-08-05 | 中国电子科技集团公司第五十二研究所 | High-speed data acquisition and recording method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7330862B1 (en) * | 2003-04-25 | 2008-02-12 | Network Appliance, Inc. | Zero copy write datapath |
| CN101887401A (en) * | 2010-06-24 | 2010-11-17 | 苏州飞鱼星电子技术有限公司 | Apparatus for acquiring and storing high speed data in real time |
| CN102681795A (en) * | 2012-05-02 | 2012-09-19 | 无锡众志和达存储技术有限公司 | Method for data input/output (I/O) read-in in small computer system interface (SCSI) Target mode of Linux system |
| CN103593315A (en) * | 2013-11-20 | 2014-02-19 | 中国船舶重工集团公司第七二四研究所 | Direct multi-hard-disk high-speed parallel reading and writing method based on FPGA |
| CN106528492A (en) * | 2016-10-27 | 2017-03-22 | 济南浪潮高新科技投资发展有限公司 | High-speed large-capacity recording board card realized based on FPGA |
-
2018
- 2018-04-18 CN CN201810350049.8A patent/CN110019051A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7330862B1 (en) * | 2003-04-25 | 2008-02-12 | Network Appliance, Inc. | Zero copy write datapath |
| CN101887401A (en) * | 2010-06-24 | 2010-11-17 | 苏州飞鱼星电子技术有限公司 | Apparatus for acquiring and storing high speed data in real time |
| CN102681795A (en) * | 2012-05-02 | 2012-09-19 | 无锡众志和达存储技术有限公司 | Method for data input/output (I/O) read-in in small computer system interface (SCSI) Target mode of Linux system |
| CN103593315A (en) * | 2013-11-20 | 2014-02-19 | 中国船舶重工集团公司第七二四研究所 | Direct multi-hard-disk high-speed parallel reading and writing method based on FPGA |
| CN106528492A (en) * | 2016-10-27 | 2017-03-22 | 济南浪潮高新科技投资发展有限公司 | High-speed large-capacity recording board card realized based on FPGA |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021093249A1 (en) * | 2019-11-13 | 2021-05-20 | 国微集团(深圳)有限公司 | Method for external device to access computer memory |
| CN114860158A (en) * | 2022-04-11 | 2022-08-05 | 中国电子科技集团公司第五十二研究所 | High-speed data acquisition and recording method |
| CN114780471A (en) * | 2022-04-20 | 2022-07-22 | 湖南艾科诺维科技有限公司 | Data acquisition and storage system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240152274A1 (en) | Apparatus and Method to Provide Cache Move with Non-Volatile Mass Memory System | |
| US9256384B2 (en) | Method and system for reducing write latency in a data storage system by using a command-push model | |
| US10318175B2 (en) | SSD with heterogeneous NVM types | |
| CN102063406B (en) | Network shared Cache for multi-core processor and directory control method thereof | |
| CN109412914A (en) | Flow data and AXI interface communication equipment | |
| CN110019051A (en) | A kind of high-speed data recording method based on universal document system | |
| CN102831088A (en) | Data migration method and device based on mixing memory | |
| US20140006687A1 (en) | Data Cache Apparatus, Data Storage System and Method | |
| CN101794271B (en) | Implementation method and device of consistency of multi-core internal memory | |
| US20130290647A1 (en) | Information-processing device | |
| KR20160104710A (en) | Speculative prefetching of data stored in flash memory | |
| KR20150036281A (en) | Smart memory buffers | |
| EP2565772A1 (en) | Storage array, storage system, and data access method | |
| CN110069443B (en) | UFS storage array system based on FPGA control and data transmission method | |
| CN107430554A (en) | Storage cache performance is improved as the standard that cache is inserted by using the compressibility of data | |
| CN103294836A (en) | PCIE (peripheral component interconnect express) based radar data acquisition displaying and controlling system and method thereof | |
| CN109164976A (en) | Optimize storage device performance using write buffer | |
| US20240361951A1 (en) | Data processing method, storage medium, and processor | |
| WO2015061921A1 (en) | Dynamic caching method and system for data storage system | |
| CN109739696A (en) | Double-control storage array solid state disk cache acceleration method | |
| CN104239231B (en) | A kind of method and device for accelerating L2 cache preheating | |
| CN103488582B (en) | Write the method and device of cache memory | |
| US20180217935A1 (en) | Method and apparatus for reading redundant array of independent disks | |
| CN105915595A (en) | Cluster storage system data accessing method and cluster storage system | |
| CN105138277A (en) | Cache management method for solid-state disc array |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |
|
| RJ01 | Rejection of invention patent application after publication |