CN103514112A - Data storage method and system - Google Patents
Data storage method and system Download PDFInfo
- Publication number
- CN103514112A CN103514112A CN201210223346.9A CN201210223346A CN103514112A CN 103514112 A CN103514112 A CN 103514112A CN 201210223346 A CN201210223346 A CN 201210223346A CN 103514112 A CN103514112 A CN 103514112A
- Authority
- CN
- China
- Prior art keywords
- buffer memory
- data
- event
- file system
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000013500 data storage Methods 0.000 title claims abstract description 9
- 230000015654 memory Effects 0.000 claims description 120
- 230000001143 conditioned effect Effects 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 abstract description 5
- 230000004044 response Effects 0.000 abstract description 4
- 230000004048 modification Effects 0.000 description 14
- 238000012986 modification Methods 0.000 description 14
- 238000012544 monitoring process Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000003607 modifier Substances 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000009885 systemic effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method and system in order to improve the response speed of file operation. The data storage method comprises the following steps that file system data are stored in a first cache, an event used for describing the operation is built and sent to a queue of a thread storing the file system data; the thread obtains an event from the queue and executes the event, and the execution result of the event is stored in a second cache; when a preset condition is satisfied, the file system data stored in the second cache are stored in a disc. The data storage method obviously improves the response speed of file operation, guarantees the completeness of the file system data when abnormal outage occurs, and loss of files in the abnormal outage is reduced to the maximum extent.
Description
Technical field
The present invention relates to technical field of data storage, particularly a kind of date storage method and system.
Background technology
The performance bottleneck of file system is the data interaction with storage medium, frequent visit storage medium has a strong impact on file system performance, for reducing the number of times of access storage media, improve the performance of file system, conventionally all can open up a memory headroom, allow most data interaction all occur in internal memory.
In embedded memory device, because memory headroom is limited, some critical datas of file system cannot accomplish to complete one-time write storage medium in internal memory, can only from buffer memory, write on storage medium in batches.If still critical data does not also write, equipment is with regard to abnormal power-down, and the critical data of the file system on storage medium is not just complete in this case, in the time of next carry, just may fall by a lot of File loses, and more severe patient file system is directly paralysed.
And existing another kind of technical scheme is when the critical data of memory file system, by buffer memory, directly do not write on storage medium.This scheme can effectively be avoided on the one hand because critical data loss file system is unavailable or the situation of large quantities of File loses, but owing to being quite consuming time, therefore can have a strong impact on the operating performance of file system again on the other hand in the situation that when storage do not have buffer memory.
Summary of the invention
For the problem existing in above-mentioned technology, the embodiment of the present invention provides a kind of date storage method and system, in order to improve the response speed of file operation.
For realizing technique scheme, the embodiment of the present invention provides a kind of date storage method on the one hand, comprises the following steps:
File system data is stored in the first buffer memory, and create for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
Described thread obtains an event and carries out this event from described queue, and the execution result of event is stored in the second buffer memory;
When meeting when pre-conditioned, the file system data of storing in the second buffer memory is stored in disk.
Preferably, described pre-conditioned comprising:
The data volume of storing in the second buffer memory reaches default data volume threshold value; Or,
The number of the execution result of the event of storing in the second buffer memory reaches default number threshold value; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold.
Preferably, described the first buffer memory comprises internal memory and/or disk.
Preferably, described the second buffer memory comprises internal memory and/or disk.
Preferably, in storing file system data into first buffer memory before, the method also comprises:
According to user, operate and determine the current file system data that needs storage.
The embodiment of the present invention also provides a kind of data-storage system on the other hand, and this system comprises: comprise the first buffer memory, the second buffer memory and disk, and:
The first control module, for file system data being stored into the first buffer memory, and creates for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
The second control module, obtains an event and carries out this event from described queue for controlling described thread, and the execution result of event is stored in the second buffer memory;
The 3rd control module, for when meeting when pre-conditioned, stores the file system data of storing in the second buffer memory in disk into.
Preferably, described pre-conditioned comprising:
The data volume of storing in the second buffer memory reaches default data volume threshold value; Or,
The number of the execution result of the event of storing in the second buffer memory reaches default number threshold value; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold.
Preferably, described the first buffer memory comprises internal memory and/or disk.
Preferably, described the second buffer memory comprises internal memory and/or disk.
Preferably, this system also comprises:
Pretreatment unit, before storing file system data into first buffer memory at described the first control module, operates and determines the current file system data that needs storage according to user, and this document system data is sent to described the first control module.
The technical scheme that the embodiment of the present invention adopts compared with prior art, has the following advantages:
The embodiment of the present invention guarantees data cached integrality by the mode that adopts memory cache, hard-disc storage to combine, and adopts the pattern of event to guarantee that critical data district synchronizes with buffer memory simultaneously.
On storage medium, set two block spaces and be respectively the first buffer memory and the second buffer memory as the buffer zone of critical data, the amendment record of storage critical data, and set up the corresponding relation of described amendment record and the information of modification; Store critical data into first buffer memory, and record the modification situation of critical data on the first buffer memory, obtain critical data and store described critical data into first buffer memory, and by the status modifier of described the first buffer memory for storing; The amendment record of the critical data in the data of storage is saved in during event lines up, and the described amendment record according to storage, obtains modification information corresponding to described critical data, and stores described modification information into second buffer memory; Judge the modification quantity of information of described critical data of described the second buffer memory and the relation of setting value; When reaching described setting value, the described critical chunk of described the second buffer memory is stored in harddisk memory.
The method that the present invention adopts can avoid file system to operate in a large number at short notice critical data district, as the critical data that in the short time, a large amount of deleted files cause is frequently rewritten, thereby due to the operation oversize performance that affects file system consuming time; Owing to having buffer area in file operation direct control, without pending file system-critical data, write successfully, therefore do not affect file operation performance, and when reaching setting value, once the critical data of the second buffer memory is stored in the storage critical data district of hard disk, therefore at every turn the data that are synchronized to critical data district from buffer memory are all complete, can guarantee the integrality of critical data, have avoided because of the caused file system collapse of abnormal power-down.
Accompanying drawing explanation
The flow process process flow diagram of the date storage method that Fig. 1 provides for the embodiment of the present invention;
The principle schematic of the date storage method that Fig. 2 provides for the embodiment of the present invention;
The structural representation of the data-storage system that Fig. 3 provides for the embodiment of the present invention.
Embodiment
The embodiment of the present invention provides a kind of date storage method and system, in order to improve the response speed of file operation.
Below in conjunction with embodiment and accompanying drawing, a kind of method and system provided by the invention are described.
As shown in Figure 1, a kind of date storage method process flow diagram for the embodiment of the present invention one provides, comprises the following steps:
Step 101: file system data is stored in the first buffer memory, and create for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
Step 102: described thread obtains an event and carries out this event from described queue, and the execution result of event is stored in the second buffer memory;
Wherein, the execution result of described event, carries out the resulting file system data of this event, and this document system data was stored in the first buffer memory originally.
Step 103: when meeting when pre-conditioned, the file system data of storing in the second buffer memory is stored in disk.
Preferably, described pre-conditioned comprising:
The data volume of storing in the second buffer memory reaches default data volume threshold value, and data volume threshold value is 4K for example, if internal memory is larger, and the File lose number that can tolerate during power-off is larger, can increase number, otherwise reduces number; Or,
The number of the execution result of the event of storing in the second buffer memory reaches default number threshold value, if for example 10 internal memories are larger for number threshold value, and the File lose number that can tolerate during power-off is larger, can increase number, otherwise reduces number; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold, and time threshold for example once, depending on real needs, frequently can reduce the time if file system data is revised 1 second, otherwise the increasing time.
Preferably, described the first buffer memory comprises internal memory and/or disk.
Preferably, described the second buffer memory comprises internal memory and/or disk.
The date storage method that the embodiment of the present invention provides, preferably, for data important for system (refer in particular to file system data, rather than video recording data), not for total data, the imperfect systemic breakdown equivalent risk that likely causes of above-mentioned data.
Therefore, preferably, in storing file system data into first buffer memory before, the method also comprises:
According to user, operate and determine the current file system data (hereinafter to be referred as critical data) that needs storage, for example, for example, while creating a file, need the index of revised file system, when open next time, just need to go for file by this index value.
That is to say, at the amendment record (so-called amendment record, for describing the event that critical data is stored into first this operation of buffer memory) of storing critical data, before, also comprise: obtain critical data, and store described critical data into first buffer memory; The amendment record of the critical data in the data of storage is saved in during event lines up, in described amendment record, at least comprises: the address information of described critical data, and set up the mapping relations of the amendment record of corresponding described address information and described critical data; According to the address information of described critical data, obtain the critical data amendment record information that described event is lined up the described address of middle correspondence; Described critical data amendment record is stored into the second buffer memory of internal memory.
Preferably, step 103 specifically comprises: judge the modification quantity of information of described critical data of described the second buffer memory and the relation of setting value, specifically comprise: the quantity of modification information of monitoring in real time or periodically the described critical data of described the second buffer memory; When judging the quantity of modification information of the described critical data of described the second buffer memory monitoring and reach setting value, described critical chunk is stored in harddisk memory.
Or, due to monitoring critical data with and modification information whether copied the second buffer memory completely to, can there is various ways, except the above-mentioned monitoring quantity of mentioning, can also monitor the time that critical data changes, if surpassed setting duration scope, the critical data of determining corresponding stored file has stored the second buffer memory completely into, therefore the critical data of the second buffer memory once can be stored into the critical chunk district of hard disk.
Or step 103 specifically comprises: the time that described in Real-Time Monitoring, the described critical data quantity of the second buffer memory changes; When judging time that the described critical data quantity of monitoring out changes while reaching setting value, described critical chunk is stored in harddisk memory.
Referring to Fig. 2, the principle schematic of the date storage method providing for the embodiment of the present invention.
According to storage described amendment record, obtain modification information corresponding to described critical data (comprising the file system datas such as file system index), and store described modification information into second buffer memory simultaneously, can also obtain critical data, and store described critical data into first buffer memory.That is to say, can when the first buffer memory receives data, can also store the revised context of critical data into second buffer memory, by least two threads, realize.
In addition, after storing described critical data amendment record into second buffer memory, also comprise: empty the critical data of storing on the second buffer memory, and by the status indication of described the second buffer memory for emptying.
After receiving critical data by the status modifier of the first buffer memory for using or storing, express the first current buffer memory and be the state that receives data using; After storing critical data into second buffer memory, also the status indicator of the second buffer memory is revised as and is stored.
After storing described critical data amendment record into second buffer memory, also comprise: empty the critical data of storing on the second buffer memory, and by the status indication of described the second buffer memory for emptying.
The embodiment of the present invention also provides a kind of data-storage system on the other hand, and as shown in Figure 3, this system comprises: the first buffer memory 11, the second buffer memory 12 and disk 13, and:
The first control module 14, for file system data being stored into the first buffer memory 11, and creates for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
The second control module 15, obtains an event and carries out this event from described queue for controlling described thread, and the execution result of event is stored in the second buffer memory 12;
The 3rd control module 16, for when meeting when pre-conditioned, stores the file system data of storage in the second buffer memory 12 in disk 13 into.
Preferably, described pre-conditioned comprising:
In the second buffer memory 12, the data volume of storage reaches default data volume threshold value; Or,
In the second buffer memory 12, the number of the execution result of the event of storage reaches default number threshold value; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold.
Preferably, described the first buffer memory 11 comprises internal memory and/or disk.
Preferably, described the second buffer memory 12 comprises internal memory and/or disk.
Preferably, this system also comprises:
In sum, the embodiment of the present invention has the following advantages:
Adopt the embodiment of the present invention to guarantee data cached integrality by the mode that adopts memory cache, hard-disc storage to combine, adopt the pattern of event to guarantee the method for synchronizeing with buffer memory in critical data district simultaneously, effectively avoided because of the caused file system collapse of abnormal power-down, and can be because of in the short time, heap file being operated and causes critical data frequently to be rewritten, thereby affect the stability of file system.
Obviously, those skilled in the art can carry out various changes and modification and not depart from the spirit and scope of the present invention the present invention.Like this, if within of the present invention these are revised and modification belongs to the scope of the claims in the present invention and equivalent technologies thereof, the present invention is also intended to comprise these changes and modification interior.
Claims (10)
1. a date storage method, is characterized in that, comprises the following steps:
File system data is stored in the first buffer memory, and create for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
Described thread obtains an event and carries out this event from described queue, and the execution result of event is stored in the second buffer memory;
When meeting when pre-conditioned, the file system data of storing in the second buffer memory is stored in disk.
2. method according to claim 1, is characterized in that, described pre-conditioned comprising:
The data volume of storing in the second buffer memory reaches default data volume threshold value; Or,
The number of the execution result of the event of storing in the second buffer memory reaches default number threshold value; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold.
3. method according to claim 1, is characterized in that, described the first buffer memory comprises internal memory and/or disk.
4. method according to claim 1, is characterized in that, described the second buffer memory comprises internal memory and/or disk.
5. method according to claim 1, is characterized in that, in storing file system data into first buffer memory before, the method also comprises:
According to user, operate and determine the current file system data that needs storage.
6. a data-storage system, is characterized in that, this system comprises the first buffer memory, the second buffer memory and disk, and:
The first control module, for file system data being stored into the first buffer memory, and creates for describing the event of this operation, this event is sent to the queue of the thread of store file sys-tem data;
The second control module, obtains an event and carries out this event from described queue for controlling described thread, and the execution result of event is stored in the second buffer memory;
The 3rd control module, for when meeting when pre-conditioned, stores the file system data of storing in the second buffer memory in disk into.
7. system according to claim 6, is characterized in that, described pre-conditioned comprising:
The data volume of storing in the second buffer memory reaches default data volume threshold value; Or,
The number of the execution result of the event of storing in the second buffer memory reaches default number threshold value; Or,
The processing time of the thread of store file sys-tem data surpasses default time threshold.
8. system according to claim 6, is characterized in that, described the first buffer memory comprises internal memory and/or disk.
9. system according to claim 6, is characterized in that, described the second buffer memory comprises internal memory and/or disk.
10. system according to claim 6, is characterized in that, this system also comprises:
Pretreatment unit, before storing file system data into first buffer memory at described the first control module, operates and determines the current file system data that needs storage according to user, and this document system data is sent to described the first control module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210223346.9A CN103514112B (en) | 2012-06-29 | 2012-06-29 | A kind of date storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210223346.9A CN103514112B (en) | 2012-06-29 | 2012-06-29 | A kind of date storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103514112A true CN103514112A (en) | 2014-01-15 |
| CN103514112B CN103514112B (en) | 2016-11-02 |
Family
ID=49896862
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210223346.9A Expired - Fee Related CN103514112B (en) | 2012-06-29 | 2012-06-29 | A kind of date storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103514112B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512232A (en) * | 2015-11-30 | 2016-04-20 | 北京金山安全软件有限公司 | Data storage method and device |
| CN109118609A (en) * | 2017-06-26 | 2019-01-01 | 深圳市道通智能航空技术有限公司 | Unmanned plane during flying log recording method, device, storage medium and computer equipment |
| CN109189654A (en) * | 2018-08-02 | 2019-01-11 | 山东浪潮通软信息科技有限公司 | A kind of mobile terminal user behavior track recording method |
| CN110221987A (en) * | 2019-05-05 | 2019-09-10 | 福建天泉教育科技有限公司 | Date storage method, storage medium in the case of system emergency |
| CN111428260A (en) * | 2020-06-15 | 2020-07-17 | 广东电网有限责任公司佛山供电局 | File security access control method and system based on chip protection device |
| CN111597364A (en) * | 2020-03-27 | 2020-08-28 | 广联达科技股份有限公司 | Engineering quantity calculation intermediate layer generation method and device, storage medium and electronic equipment |
| CN112052110A (en) * | 2020-09-02 | 2020-12-08 | 广州市百果园信息技术有限公司 | Storage method and device |
| CN114116634A (en) * | 2022-01-26 | 2022-03-01 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1517917A (en) * | 2003-01-27 | 2004-08-04 | ������������ʽ���� | Method for managing affairs processing correlation to enduring structure in multi-line program enviroment |
| US6865650B1 (en) * | 2000-09-29 | 2005-03-08 | Emc Corporation | System and method for hierarchical data storage |
| CN1749975A (en) * | 2004-01-17 | 2006-03-22 | 崔允太 | Quasi-system hard disc |
| CN101102281A (en) * | 2007-08-16 | 2008-01-09 | 中兴通讯股份有限公司 | Data processing method for reporting a large number of data in mobile communication system |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
-
2012
- 2012-06-29 CN CN201210223346.9A patent/CN103514112B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6865650B1 (en) * | 2000-09-29 | 2005-03-08 | Emc Corporation | System and method for hierarchical data storage |
| CN1517917A (en) * | 2003-01-27 | 2004-08-04 | ������������ʽ���� | Method for managing affairs processing correlation to enduring structure in multi-line program enviroment |
| CN1749975A (en) * | 2004-01-17 | 2006-03-22 | 崔允太 | Quasi-system hard disc |
| CN101102281A (en) * | 2007-08-16 | 2008-01-09 | 中兴通讯股份有限公司 | Data processing method for reporting a large number of data in mobile communication system |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512232A (en) * | 2015-11-30 | 2016-04-20 | 北京金山安全软件有限公司 | Data storage method and device |
| CN109118609A (en) * | 2017-06-26 | 2019-01-01 | 深圳市道通智能航空技术有限公司 | Unmanned plane during flying log recording method, device, storage medium and computer equipment |
| CN109189654A (en) * | 2018-08-02 | 2019-01-11 | 山东浪潮通软信息科技有限公司 | A kind of mobile terminal user behavior track recording method |
| CN110221987A (en) * | 2019-05-05 | 2019-09-10 | 福建天泉教育科技有限公司 | Date storage method, storage medium in the case of system emergency |
| CN111597364A (en) * | 2020-03-27 | 2020-08-28 | 广联达科技股份有限公司 | Engineering quantity calculation intermediate layer generation method and device, storage medium and electronic equipment |
| CN111597364B (en) * | 2020-03-27 | 2024-01-30 | 广联达科技股份有限公司 | Engineering quantity calculation middle layer generation method and device, storage medium and electronic equipment |
| CN111428260A (en) * | 2020-06-15 | 2020-07-17 | 广东电网有限责任公司佛山供电局 | File security access control method and system based on chip protection device |
| CN112052110A (en) * | 2020-09-02 | 2020-12-08 | 广州市百果园信息技术有限公司 | Storage method and device |
| CN112052110B (en) * | 2020-09-02 | 2024-04-05 | 广州市百果园信息技术有限公司 | Storage method and device |
| CN114116634A (en) * | 2022-01-26 | 2022-03-01 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
| CN114116634B (en) * | 2022-01-26 | 2022-04-22 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103514112B (en) | 2016-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103514112A (en) | Data storage method and system | |
| US11209989B2 (en) | Zoned namespaces in solid-state drives | |
| US8327076B2 (en) | Systems and methods of tiered caching | |
| US8463984B2 (en) | Dynamic data flow management in a multiple cache architecture | |
| US8225044B2 (en) | Storage system which utilizes two kinds of memory devices as its cache memory and method of controlling the storage system | |
| US9158700B2 (en) | Storing cached data in over-provisioned memory in response to power loss | |
| US20100325352A1 (en) | Hierarchically structured mass storage device and method | |
| JP5458568B2 (en) | Nonvolatile storage device, information recording system, and information recording method | |
| EP2733617A1 (en) | Data buffer device, data storage system and method | |
| US9069474B2 (en) | Retention based defecting in a hybrid memory system | |
| US11599304B2 (en) | Data aggregation in ZNS drive | |
| US9208101B2 (en) | Virtual NAND capacity extension in a hybrid drive | |
| US8225035B2 (en) | Systems and methods for operating a disk drive | |
| US20140089582A1 (en) | Disk array apparatus, disk array controller, and method for copying data between physical blocks | |
| US11500727B2 (en) | ZNS parity swapping to DRAM | |
| US8161251B2 (en) | Heterogeneous storage array optimization through eviction | |
| US7913029B2 (en) | Information recording apparatus and control method thereof | |
| JP6975202B2 (en) | Recovery process and equipment from momentary interruptions, and computer-readable storage media | |
| US20120047387A1 (en) | Cache control device and cache control method | |
| US20110138110A1 (en) | Method and control unit for performing storage management upon storage apparatus and related storage apparatus | |
| US20150067285A1 (en) | Storage control apparatus, control method, and computer-readable storage medium | |
| US20160259571A1 (en) | Storage subsystem | |
| CN110275678B (en) | STT-MRAM-based solid state memory device random access performance improvement method | |
| US11687263B2 (en) | Full die recovery in ZNS SSD | |
| US9009426B1 (en) | Methods, computer readable media, and systems for optimizing cache utilization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20161102 |