CN104407988A - Method and system for improving data storage efficiency - Google Patents
Method and system for improving data storage efficiency Download PDFInfo
- Publication number
- CN104407988A CN104407988A CN201410640923.3A CN201410640923A CN104407988A CN 104407988 A CN104407988 A CN 104407988A CN 201410640923 A CN201410640923 A CN 201410640923A CN 104407988 A CN104407988 A CN 104407988A
- Authority
- CN
- China
- Prior art keywords
- data
- fifo buffer
- write
- high speed
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Storing Facsimile Image Data (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method and a system for improving data storage efficiency. The method comprises the following steps of firstly writing data into a high-speed first in first out buffer, then writing into a temporary first in first out buffer, and buffering the to-be-stored data; when the data is fully written into the first in first out buffer, writing the data in the first in first out buffer into the temporary first in first out buffer, opening a file, and storing the data; after the storage is completed, closing the file. The method has the advantages that the file opening and closing operations of the single data are avoided, and the one-time file opening and closing operations ae centrally performed on one batch of data, so the data storage efficiency is improved; as the data storage efficiency is improved, the limitation on data sampling speed is not decreased, and the speed of collection and storage process is integrally improved.
Description
Technical field
The present invention relates to automation field, particularly relate to a kind of method and system improving data storage efficiency.
Background technology
Automated control technology has been applicable to the every field of social life at present.For Automated condtrol, collection and the storage of data are all necessary, are also very important.Especially to increase work efficiency, to the collection of data and certain requirement must be stored.
At present, in general collection-storing process, all to gather, open file, store the operation of individual data, close file to individual data.In this process, because will open file to each data with the operation of close file, even if thus improve data sampling frequency, time that cycle of whole collection-storages also can consume with close file because of needing constantly to open file in storing process and increasing.
Summary of the invention
The object of this invention is to provide a kind of method and system improving data storage efficiency, to reach the object improving data storage efficiency.
Improve a method for data storage efficiency, comprising:
Image data;
By in described data write high speed fifo buffer;
Judge whether the data in described high speed fifo buffer reach pre-sizing;
If reach pre-sizing, then the data in described high speed fifo buffer are write interim fifo buffer and open file;
Store the data of the interim fifo buffer of said write;
Close described file.
Preferably, also comprise: if the data in described high speed fifo buffer do not reach pre-sizing, then continue write data.
Preferably, described image data comprises: according to predetermined periodicity image data.
Preferably, after described image data, also comprise: digital filtering is carried out to the data gathered.
Improve a system for data storage efficiency, comprising:
Data acquisition module, for image data;
First Data write. module, for writing high speed fifo buffer by described data;
High speed fifo buffer, for data described in buffer memory;
Judging unit, for judging whether described size of data reaches pre-sizing;
Described data if reach described pre-sizing for described data, are then write interim fifo buffer, and open file by the second writing unit;
Interim fifo buffer, for storing the described data of described second writing unit write;
Second judging unit, judges that described second writing unit of described storage writes described data and whether completes, if completed, and close file.
Preferably, also comprise digital filtering unit, for carrying out digital filtering to the data of described collection.
Preferably, the size of described high speed fifo buffer is: A=(B+C) × E × F/D,
Wherein, A is the size of described high speed fifo buffer, and B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and F is the first predetermined coefficient.
Preferably, the size of described interim fifo buffer is: A=(B+C) × E × G/D,
Wherein, A is the size of described interim fifo buffer, and B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and G is the second predetermined coefficient.
Preferably, if the data in described high speed fifo buffer do not reach pre-sizing, then described first Data write. module continues to write the data collected.
Therefore, the present invention has following beneficial effect:
The present invention by data are first write a high speed fifo buffer, then writes the mode of an interim fifo buffer, carries out buffer memory to needing the data stored.After data write full high speed pushup storage, then the data of high speed pushup storage are write interim fifo buffer, open file simultaneously, store data.After storage, close file.Doing so avoids and individual data opened file and the operation of close file, but a collection of data centralization is once opened file and the operation of close file, improve the efficiency storing data.The efficiency storing data improves, and no longer reduces the restriction to data sample rate, also thus entirety improves the speed of collection-storing process.
Accompanying drawing explanation
Fig. 1 is a kind of method flow diagram improving data storage efficiency provided by the invention;
Fig. 2 is the method flow diagram that another kind provided by the invention improves data storage efficiency;
Fig. 3 is a kind of system architecture schematic diagram improving data storage efficiency provided by the invention;
Fig. 4 is the system architecture schematic diagram that another kind provided by the invention improves data storage efficiency.
Embodiment
Core of the present invention is to provide a kind of system and method improving data storage efficiency.
In order to make those skilled in the art person understand the solution of the present invention better, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.
A kind ofly improve in the method for data storage efficiency the invention discloses, see Fig. 1, this embodiment comprises the following steps:
S101, image data.
In process data gathered and record, be first data acquisition.The process gathered depends on the sample frequency of controller.But the time required for whole process gathering-record not only depends on the sample frequency of controller, is also the speed of data record.The speed of data record is low, the sampling efficiency of meeting restriction controller.
S102, by described data write high speed fifo buffer.
In the process that data store, need constantly to open and close file.If the operation of the box close file that once opens file to the data of each write, is very consuming time, significantly reduces the whole efficiency of data acquisition-record.
Therefore, the present invention is existing by the data write high speed fifo buffer that gathers, when to avoid each data directly stored in record cell, and the operation of box close file that each data are opened file.
S103, judge whether the data in described high speed fifo buffer reach pre-sizing.
Here, need to set the size of high speed fifo buffer.In order to improve storage efficiency, can according to the sample frequency design high speed fifo buffer of controller.
If S104 reaches pre-sizing, then the data in described high speed fifo buffer are write interim fifo buffer and open file.
High speed fifo buffer is write completely, and the data by buffer memory once import interim fifo buffer., open file meanwhile, prepare to store the data importing interim fifo buffer.
The data of S105, the interim fifo buffer of storage said write.
The data importing interim fifo buffer are stored.
S106, close described file.
Data store complete, close file.
In method provided by the invention, first to the data buffer storage sampled, by disposable for the data of buffer memory storage, the action of the box close file that so only once need open file to all data of disposable storage.When the solution of the present invention does not need to store data again, each data is once opened file and the operation of close file, therefore improve the efficiency storing data.
The invention also discloses a kind of method improving storage efficiency, see Fig. 2, this embodiment comprises following content:
S201, image data.
In a specific embodiment provided by the invention, according to the fixing sampling period, data are gathered during image data.
In other embodiments provided by the invention, after the data gathered, digital filtering can also be carried out to the data collected.Certainly, according to actual conditions, different filtering methods can be designed.
S202, data to be write in high speed fifo buffer.
Whether S203, the data judged in high speed fifo buffer reach pre-sizing.
Here, the design of the pre-sizing of high speed fifo buffer can adjust according to sample frequency.Under a concrete scene, following method for designing can be taked: the size of high speed fifo buffer is: A=(B+C) × E × F/D, wherein, A is the size of described high speed fifo buffer, B opens file and actuation time of close file, and C is the time of described data write, and D is the collection period of described image data, E is the size of data collected in described collection period, and F is the first predetermined coefficient.
Concrete, can F=1.5 be made, certainly, in a practical situation, also can adopt other numerical value.
If S204 reaches pre-sizing, then the data in described high speed fifo buffer are write interim fifo buffer and open file.
The size of interim fifo buffer be can be designed as: A=(B+C) × E × G/D, wherein, A is the size of described interim fifo buffer, B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and G is the second predetermined coefficient.Here, the concrete value of G can be 1.5.Certainly, in practice, can value as required.
Concrete, as F=G, size and the high speed fifo buffer of interim fifo buffer are in the same size, when data can be kept thus to import interim fifo buffer by high speed fifo buffer, once import complete.
If the data in S205 high speed fifo buffer do not reach pre-sizing, then continue write data.
Here, when high speed pushup storage is not write full, then continue to write the data collected, until write full.Under a concrete scene, also can be continue the data of write through digital filtering.
The data of S206, the interim fifo buffer of storage write.
S207, close file.
The invention also discloses a kind of system improving data storage efficiency, see Fig. 3, this system comprises:
Data acquisition module 10, for image data.
First Data write. module 20, for writing high speed fifo buffer 30 by data.
High speed fifo buffer 30, for data described in buffer memory.
Judging unit 40, for judging whether size of data reaches pre-sizing.
Described data if reach described pre-sizing for data, are then write interim fifo buffer 60, and open file by the second writing unit 50.
Interim fifo buffer 60, for storing the data that the second writing unit 50 writes.
Second judging unit 70, judges that described second writing unit 50 of storage writes described data and whether completes, if completed, and close file.
The present invention by a high speed fifo buffer and interim fifo buffer, after to data acquisition, first by it stored in high speed fifo buffer, then import interim fifo buffer.While data enter interim fifo buffer, writing unit opens file.After write work completes, close file.The present invention no longer opens file and close file to individual data, but the data for a certain size once open file and close file.Therefore improve the efficiency storing data.
The invention also discloses a kind of system improving data storage efficiency, see Fig. 4, this system comprises:
Data acquisition module 10, for image data.
In a specific embodiment, data acquisition module is the data acquisition module according to predetermined period image data.Certainly, the cycle of image data can be selected according to actual conditions.
Digital filtering unit 80, for carrying out digital filtering to the data gathered.
Digital filtering unit can process the data collected, and makes it meet the requirement of certain stored digital.It can meet the strict demand to amplitude and phase propetry, overcomes voltage and the problem such as temperature drift and noise.
First Data write. module 20, for writing high speed fifo buffer by filtered data.
In other embodiment, if the data in high speed fifo buffer 30 do not reach pre-sizing, then described first Data write. module 20 continues to write the data collected
High speed fifo buffer 30, for data described in buffer memory.
In one embodiment, the size of high speed fifo buffer is for can be designed as A=(B+C) × E × F/D;
Wherein, A is the size of high speed fifo buffer, and B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and F is the first predetermined coefficient.Concrete, F desirable 1.5.Certainly, also can value according to actual needs.
Judging unit 40, for judging whether described size of data reaches pre-sizing.
Described data if reach described pre-sizing for described data, are then write interim fifo buffer, and open file by the second writing unit 50.
Interim fifo buffer 60, for storing the described data of described second writing unit write.
In another embodiment, the size of interim fifo buffer be can be designed as: A=(B+C) × E × G/D, wherein, A is the size of described interim fifo buffer, B opens file and actuation time of close file, and C is the time of described data write, and D is the collection period of described image data, E is the size of data collected in described collection period, and G is the second predetermined coefficient.Here, the concrete value of G can be 1.5, also according to the size of high speed fifo buffer, can relate to the interim fifo buffer of size equal with it.Certainly, in practice, also can value as required.
Certainly, what the size of interim fifo buffer can also be designed to high speed fifo buffer is in the same size, to write full data once import interim fifo buffer from high speed fifo buffer.
Second judging unit 70, judges that storage second writing unit writes described data and whether completes, if completed, and close file.
Above for the system of raising storage efficiency provided by the invention, its concrete principle of work and implementation method please refer to aforesaid embodiment of the method, repeat no more herein.
Above the method and system of raising data storage efficiency provided by the present invention are described in detail.Apply specific case herein to set forth principle of the present invention and embodiment, the explanation of above embodiment just understands principle of the present invention and core concept thereof for helping.It should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention, can also carry out some improvement and modification to the present invention, these improve and modify and also fall in the protection domain of the claims in the present invention.
Claims (9)
1. improve a method for data storage efficiency, it is characterized in that, comprising:
Image data;
By in described data write high speed fifo buffer;
Judge whether the data in described high speed fifo buffer reach pre-sizing;
If reach pre-sizing, then the data in described high speed fifo buffer are write interim fifo buffer and open file;
Store the data of the interim fifo buffer of said write;
Close described file.
2. the method for claim 1, is characterized in that, also comprises: if the data in described high speed fifo buffer do not reach pre-sizing, then continue write data.
3. the method for claim 1, is characterized in that, described image data comprises: according to predetermined periodicity image data.
4. the method for claim 1, is characterized in that, after described image data, also comprises: carry out digital filtering to the data gathered.
5. improve a system for data storage efficiency, it is characterized in that, comprising:
Data acquisition module, for image data;
First Data write. module, for writing high speed fifo buffer by described data;
High speed fifo buffer, for data described in buffer memory;
Judging unit, for judging whether described size of data reaches pre-sizing;
Described data if reach described pre-sizing for described data, are then write interim fifo buffer, and open file by the second writing unit;
Interim fifo buffer, for storing the described data of described second writing unit write;
Second judging unit, judges that described second writing unit of described storage writes described data and whether completes, if completed, and close file.
6. system as claimed in claim 5, is characterized in that, also comprise digital filtering unit, for carrying out digital filtering to the data of described collection.
7. as claim 5 system, it is characterized in that, the size of described high speed fifo buffer is: A=(B+C) × E × F/D,
Wherein, A is the size of described high speed fifo buffer, and B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and F is the first predetermined coefficient.
8. system as claimed in claim 5, it is characterized in that, the size of described interim fifo buffer is: A=(B+C) × E × G/D,
Wherein, A is the size of described interim fifo buffer, and B opens file and actuation time of close file, C is the time of described data write, D is the collection period of described image data, and E is the size of data collected in described collection period, and G is the second predetermined coefficient.
9. system according to claim 5, is characterized in that, described data acquisition module is the data acquisition module according to predetermined period image data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640923.3A CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640923.3A CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104407988A true CN104407988A (en) | 2015-03-11 |
| CN104407988B CN104407988B (en) | 2018-09-25 |
Family
ID=52645621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410640923.3A Active CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104407988B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180143882A1 (en) * | 2016-11-23 | 2018-05-24 | 2236008 Ontario Inc. | File system framework |
| CN112181973A (en) * | 2019-07-01 | 2021-01-05 | 北京涛思数据科技有限公司 | Time sequence data storage method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN101794297A (en) * | 2010-01-15 | 2010-08-04 | 北京中创信测科技股份有限公司 | Method and system for processing information data |
| CN101882130A (en) * | 2009-05-07 | 2010-11-10 | 安凯(广州)微电子技术有限公司 | Method for accelerating to open/close files in fat formatted file system |
| CN103761227A (en) * | 2010-12-31 | 2014-04-30 | 新奥特（北京）视频技术有限公司 | Caching method and device based on files |
-
2014
- 2014-11-13 CN CN201410640923.3A patent/CN104407988B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN101882130A (en) * | 2009-05-07 | 2010-11-10 | 安凯(广州)微电子技术有限公司 | Method for accelerating to open/close files in fat formatted file system |
| CN101794297A (en) * | 2010-01-15 | 2010-08-04 | 北京中创信测科技股份有限公司 | Method and system for processing information data |
| CN103761227A (en) * | 2010-12-31 | 2014-04-30 | 新奥特（北京）视频技术有限公司 | Caching method and device based on files |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180143882A1 (en) * | 2016-11-23 | 2018-05-24 | 2236008 Ontario Inc. | File system framework |
| US11977456B2 (en) * | 2016-11-23 | 2024-05-07 | 2236008 Ontario Inc. | File system framework |
| CN112181973A (en) * | 2019-07-01 | 2021-01-05 | 北京涛思数据科技有限公司 | Time sequence data storage method |
| CN112181973B (en) * | 2019-07-01 | 2023-05-30 | 北京涛思数据科技有限公司 | Time sequence data storage method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104407988B (en) | 2018-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101923568B (en) | Method for increasing and canceling elements of Bloom filter and Bloom filter | |
| CN103049879B (en) | A kind of Infrared images pre-processing method based on FPGA | |
| CN103761056A (en) | Stream data storage method in video monitoring application | |
| CN104505049A (en) | Grid driving circuit | |
| CN102652313B (en) | Flash memory controller and operation method | |
| CN104407988A (en) | Method and system for improving data storage efficiency | |
| CN102420013A (en) | Semiconductor memory device | |
| CN103325426A (en) | Seamless acquisition method based on DDR2SDRAM array segmented storage | |
| CN104077492A (en) | Sample data interpolation method based on FPGA | |
| CN100535823C (en) | DDS signal generator and waveshape memory depth control method thereof | |
| CN106789697A (en) | A kind of method and device for improving high-volume network flow table ageing efficiency | |
| CN104102552A (en) | Message processing method and device | |
| CN104750621A (en) | Caching method and control system | |
| CN103488433B (en) | A kind of batch documents method of operating and system based on distributed file system | |
| CN103593148B (en) | The method and device that a kind of CDF sides offline bill data are quickly accessed | |
| CN102510286A (en) | Frequency conversion sampling method | |
| CN105117177B (en) | A kind of big data management method based on single hard disk multi partition | |
| CA2830868C (en) | A digital netlist partitioning system for faster circuit reverse-engineering | |
| CN104572903B (en) | A kind of method of the control data loading of Hbase database | |
| CN104391732A (en) | Software multi-timer dynamic state management method | |
| CN106484564A (en) | A kind of acquired data storage method | |
| CN103326700A (en) | Bootstrap sampling switch circuit | |
| CN104730307A (en) | Gray level digital oscilloscope and waveform processing module and method thereof | |
| CN104166585A (en) | Terminal timer control method and system | |
| CN104267908A (en) | Data storage and reading methods and devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Chengyang District of Shandong city of Qingdao province Jinhong road 266111 No. 88 Applicant after: CRRC QINGDAO SIFANG CO., LTD. Address before: Chengyang District of Shandong city of Qingdao province Jinhong road 266111 No. 88 Applicant before: CSR Qingdao Sifang Locomotive and Rolling Stock Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |