CN104407988B - A kind of method and system improving data storage efficiency - Google Patents
A kind of method and system improving data storage efficiency Download PDFInfo
- Publication number
- CN104407988B CN104407988B CN201410640923.3A CN201410640923A CN104407988B CN 104407988 B CN104407988 B CN 104407988B CN 201410640923 A CN201410640923 A CN 201410640923A CN 104407988 B CN104407988 B CN 104407988B
- Authority
- CN
- China
- Prior art keywords
- data
- fifo buffer
- file
- high speed
- size
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
Data by being first written a high speed fifo buffer by the present invention, then the mode of an interim fifo buffer is written, and the data stored to needs cache.Interim fifo buffer is written after data write full high speed pushup storage, then by the data of high speed pushup storage, opens simultaneously file, stores data.After storage, file is closed.It avoids that individual data is carried out opening file in this way and closes the operation of file, but batch of data is concentrated and carries out the primary operation opened file and close file, improve the efficiency of storage data.The efficiency for storing data improves, and no longer reduces the limitation to data sample rate, also to be integrally improved the speed of acquisition storing process.
Description
Technical field
The present invention relates to automation fields, more particularly to a kind of method and system improving data storage efficiency.
Background technology
Automated control technology has been suitable for the every field of social life at present.For automation control, number
According to acquisition and storage be all necessary and highly important.Working efficiency is especially improved, necessarily to the acquisition of data
Be stored with certain requirement.
Currently, in general acquisition-storing process, individual data will be acquired, open file, the single number of storage
According to, close file operation.In this process, because will carry out opening file to each data and close the behaviour of file
Make, thus even if improving data sampling frequency, the period of entire acquisition-storage in storing process as need constantly to beat
It open file and closes time for being consumed of file and increases.
Invention content
The object of the present invention is to provide a kind of method and system improving data storage efficiency, and data storage is improved to reach
The purpose of efficiency.
A method of data storage efficiency is improved, including：
Gathered data；
The data are written in high speed fifo buffer；
Judge whether the data in the high speed fifo buffer reach predefined size；
If reaching predefined size, interim first in, first out is written into the data in the high speed fifo buffer and is delayed
Storage simultaneously opens file；
Store the data of the interim fifo buffer of said write；
Close the file.
Preferably, further include：If the data in the high speed fifo buffer are not up to predefined size, after continuing
Enter data.
Preferably, the gathered data includes：According to predetermined periodicity gathered data.
Preferably, after the gathered data, further include：Digital filtering is carried out to the data of acquisition.
A kind of system improving data storage efficiency, including：
Data acquisition module is used for gathered data；
First Data write. module, for high speed fifo buffer to be written in the data；
High speed fifo buffer, for caching the data；
Judging unit, for judging whether the size of data reaches predefined size；
Second writing unit first enters data write-in if reaching the predefined size for the data temporarily
First go out buffer, and opens file；
Interim fifo buffer, the data for storing the second writing unit write-in；
Second judgment unit judges that storage second writing unit is written whether the data are completed, if complete
At then closing file.
Preferably, further include digital filtering unit, digital filtering is carried out for the data to the acquisition.
Preferably, the size of the high speed fifo buffer is：A=(B+C) × E × F/D,
Wherein, A is the size of the high speed fifo buffer, and B is the actuation time opened file and close file,
C is the time of data write-in, and D is the collection period of the gathered data, and E is collected data in the collection period
Size, F are the first predetermined coefficient.
Preferably, the size of the interim fifo buffer is：A=(B+C) × E × G/D,
Wherein, A is the size of the interim fifo buffer, and B is the actuation time opened file and close file,
C is the time of data write-in, and D is the collection period of the gathered data, and E is collected data in the collection period
Size, G are the second predetermined coefficient.
Preferably, if the data in the high speed fifo buffer are not up to predefined size, first data
Writing module continues to write to collected data.
Therefore, the present invention has the advantages that：
Data by being first written a high speed fifo buffer by the present invention, then that an interim first in, first out is written is slow
The mode of storage, the data stored to needs cache.After data write full high speed pushup storage, then will at a high speed first
Interim fifo buffer is written in the data for entering first to go out memory, opens simultaneously file, stores data.After storage, close
Closed file.It avoids that individual data is carried out opening file in this way and closes the operation of file, but batch of data is concentrated
The primary operation opened file and close file is carried out, the efficiency of storage data is improved.The efficiency for storing data improves, no longer
The limitation to data sample rate is reduced, also to be integrally improved the speed of acquisition-storing process.
Description of the drawings
Fig. 1 is a kind of method flow diagram improving data storage efficiency provided by the invention；
Fig. 2 is another method flow diagram for improving data storage efficiency provided by the invention；
Fig. 3 is a kind of system structure diagram improving data storage efficiency provided by the invention；
Fig. 4 is another system structure diagram for improving data storage efficiency provided by the invention.
Specific implementation mode
Core of the invention is to provide a kind of system and method improving data storage efficiency.
In order to make those skilled in the art more fully understand the solution of the present invention, below in conjunction with the accompanying drawings and specific embodiment party
The present invention is described in further detail for formula.
In the invention discloses a kind of method improving data storage efficiency, referring to Fig. 1, which includes following step
Suddenly：
S101, gathered data.
During data are acquired and are recorded, data acquisition is first consisted in.The process of acquisition depends on controller
Sample frequency.But the whole process required time of acquisition-record depends not only on the sample frequency of controller, also resides in
The speed of data record.The speed of data record is low, the sampling efficiency of meeting restriction controller.
S102, the data are written in high speed fifo buffer.
During data store, need constantly to open and close file.If to each write-in data into
The primary operation opened file and close file of row, is very time-consuming, significantly reduces the whole effect of data acquisition-record
Rate.
Therefore, it in the existing data write-in high speed fifo buffer by acquisition of the present invention, avoids direct to each data
When being stored in recording unit, each data are carried out opening file and close the operation of file.
S103, judge whether the data in the high speed fifo buffer reach predefined size.
Herein, the size to high speed fifo buffer is needed to set.In order to improve storage efficiency, Ke Yigen
High speed fifo buffer is designed according to the sample frequency of controller.
If S104, reaching predefined size, the data write-in in the high speed fifo buffer is first entered temporarily
First go out buffer and opens file.
After high speed fifo buffer is write completely, i.e., the data of caching are once imported into interim fifo buffer.Together
When, file is opened, the data that storage imports interim fifo buffer are prepared.
S105, the data for storing the interim fifo buffer of said write.
Data to importing interim fifo buffer store.
S106, the file is closed.
Data storage finishes, and closes file.
In method provided by the invention, first to the data buffer storage sampled, disposably stored by the data of caching, this
Sample need to only carry out all data disposably stored the primary action opened file and close file.The solution of the present invention is not required to
When storing data again, the primary operation opened file and close file is carried out to each data, therefore improve storage number
According to efficiency.
The invention also discloses a kind of methods improving storage efficiency, and referring to Fig. 2, which includes the following contents：
S201, gathered data.
In a specific embodiment provided by the invention, when gathered data according to the fixed sampling period to data into
Row acquisition.
It, can also be after the data of acquisition, to collected data into line number in other embodiment provided by the invention
Word filters.Certainly, according to actual conditions, different filtering methods can be designed.
S202, it writes data into high speed fifo buffer.
S203, judge whether the data in high speed fifo buffer reach predefined size.
Herein, the design of the predefined size of high speed fifo buffer can be adjusted according to sample frequency.
Under one specific scene, design method below can be taken：The size of high speed fifo buffer is：A=(B+C) ×
E × F/D, wherein A is the size of the high speed fifo buffer, and B is the actuation time opened file and close file, C
For the time of data write-in, D is the collection period of the gathered data, and E is collected data in the collection period
Size, F are the first predetermined coefficient.
Specifically, F=1.5 can be enabled, certainly, in a practical situation, other numerical value can also be used.
If S204, reaching predefined size, the data write-in in the high speed fifo buffer is first entered temporarily
First go out buffer and opens file.
The size of interim fifo buffer can be designed as：A=(B+C) × E × G/D, wherein A is described faces
When fifo buffer size, B be open file and close file actuation time, C be the data write-in time,
D is the collection period of the gathered data, and E is collected size of data in the collection period, and G is the second predetermined coefficient.
Herein, the specific value of G can be 1.5.It certainly, in practice, can value as needed.
Specifically, as F=G, the size and high speed fifo buffer of interim fifo buffer are in the same size,
It is possible thereby to when data being kept to import interim fifo buffer by high speed fifo buffer, primary import finishes.
If the data in S205, high speed fifo buffer are not up to predefined size, data are continued to write to.
Herein, when high speed pushup storage does not write full, then collected data are continued to write to, until being write
It is full.Can also be the data continued to write to by digital filtering under a specific scene.
The data of interim fifo buffer are written in S206, storage.
S207, file is closed.
The invention also discloses a kind of systems improving data storage efficiency, and referring to Fig. 3, which includes：
Data acquisition module 10 is used for gathered data.
First Data write. module 20, for writing data into high speed fifo buffer 30.
High speed fifo buffer 30, for caching the data.
Judging unit 40, for judging whether size of data reaches predefined size.
Data write-in is first entered elder generation by the second writing unit 50 temporarily if reaching the predefined size for data
Go out buffer 60, and opens file.
Interim fifo buffer 60, the data for storing the write-in of the second writing unit 50.
Second judgment unit 70 judges that storing second writing unit 50 is written whether the data are completed, if complete
At then closing file.
The present invention is by a high speed fifo buffer and interim fifo buffer, after being acquired to data,
It is first deposited into high speed fifo buffer, then imports interim fifo buffer.Enter interim first in, first out in data
While buffer, writing unit opens file.It is written after the completion of work, closes file.The present invention no longer to individual data into
Row opens file and closes file, but carries out primary opening file for a certain size data and close file.Therefore it carries
The high efficiency of storage data.
The invention also discloses a kind of systems improving data storage efficiency, and referring to Fig. 4, which includes：
Data acquisition module 10 is used for gathered data.
In a specific embodiment, data acquisition module is the data acquisition module according to predetermined period gathered data
Block.Of course, it is possible to select the period of gathered data according to actual conditions.
Digital filtering unit 80 carries out digital filtering for the data to acquisition.
Digital filtering unit can be handled collected data, comply with the requirement of certain stored digital.
It can meet the strict demand to amplitude and phase characteristic, the problems such as overcoming voltage and temperature drift and noise.
First Data write. module 20, for high speed fifo buffer to be written in filtered data.
In other one embodiment, if the data in high speed fifo buffer 30 are not up to predefined size,
First Data write. module 20 continues to write to collected data
High speed fifo buffer 30, for caching the data.
In one embodiment, the size of high speed fifo buffer is that can be designed as A=(B+C) × E × F/D；
Wherein, A is the size of high speed fifo buffer, and B is the actuation time opened file and close file, and C is
The time of the data write-in, D are the collection period of the gathered data, and E is that collected data are big in the collection period
Small, F is the first predetermined coefficient.Specifically, F desirable 1.5.It is of course also possible to value according to actual needs.
Judging unit 40, for judging whether the size of data reaches predefined size.
Second writing unit 50 data is written interim first if reaching the predefined size for the data
Enter and first go out buffer, and opens file.
Interim fifo buffer 60, the data for storing the second writing unit write-in.
In another embodiment, the size of interim fifo buffer can be designed as：A=(B+C) × E ×
G/D, wherein A is the size of the interim fifo buffer, and B is the actuation time opened file and close file, and C is
The time of the data write-in, D are the collection period of the gathered data, and E is that collected data are big in the collection period
Small, G is the second predetermined coefficient.Herein, the specific value of G can be 1.5, can also be according to high speed fifo buffer
Size, be related to the interim fifo buffer with its same size.It certainly, in practice, can also value as needed.
It is, of course, also possible to by the size of interim fifo buffer being sized so as to high speed fifo buffer
Unanimously, to write full data interim fifo buffer is once imported from high speed fifo buffer.
Second judgment unit 70 judges that storing the second writing unit is written whether the data are completed, and closes if completing
Closed file.
Above for the system provided by the invention for improving storage efficiency, specific operation principle and implementation method please join
According to embodiment of the method above-mentioned, details are not described herein again.
The method and system provided by the present invention for improving data storage efficiency are described in detail above.Herein
Applying specific case, principle and implementation of the present invention are described, and the explanation of above example is only intended to help
Understand the principle of the present invention and its core concept.It should be pointed out that for those skilled in the art, not taking off
, can be with several improvements and modifications are made to the present invention under the premise of from the principle of the invention, these improvement and modification also fall into this
In invention scope of the claims.
Claims (6)
1. a kind of method improving data storage efficiency, which is characterized in that including：
Gathered data；Wherein, the gathered data includes：According to predetermined periodicity gathered data；
The data are written in high speed fifo buffer；
Judge whether the data in the high speed fifo buffer reach predefined size；Wherein, the high speed first in, first out
The size of buffer is：A=(B+C) × E × F/D,
Wherein, A is the size of the high speed fifo buffer, and B is the actuation time opened file and close file, and C is
The time of the data write-in, D are the collection period of the gathered data, and E is that collected data are big in the collection period
Small, F is the first predetermined coefficient；
If reaching predefined size, interim fifo buffer is written into the data in the high speed fifo buffer
And open file；
Store the data of the interim fifo buffer of said write；
Close the file.
2. the method as described in claim 1, which is characterized in that further include：If the number in the high speed fifo buffer
According to not up to predefined size, then data are continued to write to.
3. the method as described in claim 1, which is characterized in that after the gathered data, further include：To the data of acquisition into
Row digital filtering.
4. a kind of system improving data storage efficiency, which is characterized in that including：
Data acquisition module is used for gathered data；Wherein, the data acquisition module is the number according to predetermined period gathered data
According to acquisition module；
First Data write. module, for high speed fifo buffer to be written in the data；
High speed fifo buffer, for caching the data；
Judging unit, for judging whether the size of data reaches predefined size；The high speed fifo buffer it is big
It is small to be：A=(B+C) × E × F/D,
Wherein, A is the size of the high speed fifo buffer, and B is the actuation time opened file and close file, and C is
The time of the data write-in, D are the collection period of the gathered data, and E is that collected data are big in the collection period
Small, F is the first predetermined coefficient；
If reaching the predefined size for the data interim first in, first out is written in the data by the second writing unit
Buffer, and open file；
Interim fifo buffer, the data for storing the second writing unit write-in；
Second judgment unit judges that storage second writing unit is written whether the data are completed, if completing
Close file.
5. system as claimed in claim 4, which is characterized in that further include digital filtering unit, for the number to the acquisition
According to progress digital filtering.
6. system as claimed in claim 4, which is characterized in that the size of the interim fifo buffer is：A=(B+
C) × E × G/D,
Wherein, A is the size of the interim fifo buffer, and B is the actuation time opened file and close file, and C is
The time of the data write-in, D are the collection period of the gathered data, and E is that collected data are big in the collection period
Small, G is the second predetermined coefficient.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640923.3A CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640923.3A CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104407988A CN104407988A (en) | 2015-03-11 |
| CN104407988B true CN104407988B (en) | 2018-09-25 |
Family
ID=52645621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410640923.3A Active CN104407988B (en) | 2014-11-13 | 2014-11-13 | A kind of method and system improving data storage efficiency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104407988B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180143882A1 (en) * | 2016-11-23 | 2018-05-24 | 2236008 Ontario Inc. | File system framework |
| CN112181973B (en) * | 2019-07-01 | 2023-05-30 | 北京涛思数据科技有限公司 | Time sequence data storage method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN101794297A (en) * | 2010-01-15 | 2010-08-04 | 北京中创信测科技股份有限公司 | Method and system for processing information data |
| CN101882130A (en) * | 2009-05-07 | 2010-11-10 | 安凯(广州)微电子技术有限公司 | Method for accelerating to open/close files in fat formatted file system |
| CN103761227A (en) * | 2010-12-31 | 2014-04-30 | 新奥特（北京）视频技术有限公司 | Caching method and device based on files |
-
2014
- 2014-11-13 CN CN201410640923.3A patent/CN104407988B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN101882130A (en) * | 2009-05-07 | 2010-11-10 | 安凯(广州)微电子技术有限公司 | Method for accelerating to open/close files in fat formatted file system |
| CN101794297A (en) * | 2010-01-15 | 2010-08-04 | 北京中创信测科技股份有限公司 | Method and system for processing information data |
| CN103761227A (en) * | 2010-12-31 | 2014-04-30 | 新奥特（北京）视频技术有限公司 | Caching method and device based on files |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104407988A (en) | 2015-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101923568B (en) | Method for increasing and canceling elements of Bloom filter and Bloom filter | |
| CN102541927B (en) | Method and device for realizing data cache | |
| CN109085997A (en) | Memory-efficient for nonvolatile memory continues key assignments storage | |
| CN105677742B (en) | A kind of method and device of storage file | |
| CN101673192B (en) | Method for time-sequence data processing, device and system therefor | |
| CN104935786B (en) | A kind of method of image signal source and its processing picture signal based on soft processor | |
| CN102222071A (en) | Method, device and system for data synchronous processing | |
| CN103164511B (en) | Method for storage event log to automatically filter repeated jitter data | |
| CN107368257B (en) | Data routing inspection method and device in solid-state memory | |
| CN104407988B (en) | A kind of method and system improving data storage efficiency | |
| CN103713972B (en) | A kind of incremental backup system and method | |
| CN102169460A (en) | Method and device for managing variable length data | |
| CN103064749B (en) | Inter-process communication method | |
| CN104391804B (en) | A kind of NAND flash data storage methods | |
| CN107870739A (en) | Disk file storage method and system | |
| CN102841894A (en) | Data storage method of file allocation table | |
| CN104750621A (en) | Caching method and control system | |
| CN105528302B (en) | A kind of method and system of the dynamic hyperdisk of logic-based volume | |
| CN102103643A (en) | Method for storing test vector during chip testing | |
| CN103020535A (en) | Data encryption and decryption system with comparing function | |
| CN103488433B (en) | A kind of batch documents method of operating and system based on distributed file system | |
| CN103076990A (en) | Data playback device based on FIFO (First In, First Out) caching structure | |
| CN106951193B (en) | Improve the method and its system of Nand Flash storage performance | |
| CN103593148A (en) | CDF side off-line call ticket data quick access method and device | |
| CN104572903B (en) | A kind of method of the control data loading of Hbase database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Chengyang District of Shandong city of Qingdao province Jinhong road 266111 No. 88 Applicant after: CRRC QINGDAO SIFANG CO., LTD. Address before: Chengyang District of Shandong city of Qingdao province Jinhong road 266111 No. 88 Applicant before: CSR Qingdao Sifang Locomotive and Rolling Stock Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |