CN102184141A - Method and device for storing check point data - Google Patents
Method and device for storing check point data Download PDFInfo
- Publication number
- CN102184141A CN102184141A CN201110116063XA CN201110116063A CN102184141A CN 102184141 A CN102184141 A CN 102184141A CN 201110116063X A CN201110116063X A CN 201110116063XA CN 201110116063 A CN201110116063 A CN 201110116063A CN 102184141 A CN102184141 A CN 102184141A
- Authority
- CN
- China
- Prior art keywords
- memory
- data
- idle condition
- storage
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention discloses a method and device for storing check point data, wherein the method comprises the steps of: responding to a check point command, writing data needed to be stored in a first storage in a memory into a second storage, wherein the access speed of the second storage is higher than that of the first storage; and under the condition that the first storage and the second storage meet the preset idle condition, unloading the data written in the second storage into the first storage. In the invention, the data needed to be stored in the first storage are cached through setting a storage with a higher access speed, then the cached data are written in the first storage from the second storage, therefore, the efficiency of writing the check point data can be effectively increased, and the problem of serious loss of performances of devices such as a server and the like, caused by low writing speed, is avoided.
Description
Technical field
The present invention relates to the communications field, and especially, relate to the storage means and the device of a kind of checkpoint data.
Background technology
At present, as shown in Figure 1, it is interconnected that storage server and calculation server (also can be described as computing node) can pass through switch (for example, can be the gigabit networking switch or the IB network switch) usually.In framework shown in Figure 1, storage server can have 24 dish positions, and calculation server is then as long-range computing node unit.By framework mode shown in Figure 1, storage server can pass through network file system(NFS) (Network File System abbreviates NFS as) shares file system to calculation server.
When sharing, can be according to following flow process:
(1) all calculation server moves a parallel task;
(2) send consistency check point (Checkpoint) instruction by storage server;
(3) all calculation servers carry out local process storage under the Checkpoint instruction, and the memory content that needs are stored is placed in the NFS shared disk storage area;
(4) carry out common Checkpoint process;
(5) when system need restart a Checkpoint checkpoint, can issue a Restart order;
(6) all node reads Checkpoint file part separately and starts corresponding process by the NFS shared memory, can finish Restart work.
Check point file storage scheme speed based on hard disk is slower, be approximately the data that are merely able to store 30MB p.s., that is to say, for a calculation procedure that has the 30GB memory headroom, a Checkpoint job will be lost 16 minutes, and such Checkpoint process often needs per hour to carry out once, this globality loss of energy that just means server can reach about 25%, for the server bigger for memory headroom, that access speed is slower, the performance of loss is higher than regular meeting.
But,, effective solution is not proposed at present at writing speed slow problem in checkpoint in the correlation technique.
Summary of the invention
At writing speed slow problem in checkpoint in the correlation technique, the present invention proposes the storage means and the device of a kind of checkpoint data, can effectively improve the efficient that writes of checkpoint data, thereby server is effectively solved because of the checkpoint data write the problem that performance loss occurs.
Technical scheme of the present invention is achieved in that
The storage means of a kind of checkpoint data is provided according to an aspect of the present invention.
This method comprises: in response to the checkpoint order, the data that need in the internal memory to be stored in the first memory are written in the second memory, wherein, the access speed of second memory is higher than the access speed of first memory; Satisfy at first memory and second memory under the situation of predetermined idle condition, with the data conversion storage that writes in the second memory to first memory.
Wherein, in with internal memory, need to be stored to before data in the first memory are written in the second memory, this method can also comprise: judge whether second memory is in idle condition, if judged result is for being, then determine to allow the data that need in the internal memory to be stored in the first memory are written in the second memory; Otherwise wait for that first memory enters idle condition.
And above-mentioned predetermined idle condition can comprise: first memory is in idle condition, and second is in idle condition.
And above-mentioned predetermined idle condition can also comprise: the storage space of second memory is taken, or data quantity stored reaches predetermined threshold in the second memory.
Alternatively, first memory is a hard disk; Second memory is one of following: solid state hard disc, Flash hard disk.
The memory storage of a kind of checkpoint data is provided according to a further aspect in the invention.
This device comprises: second memory; First processing module is used in response to the checkpoint order, and the data that need in the internal memory to be stored in the first memory are written in the second memory, and wherein, the access speed of second memory is higher than the access speed of first memory; Second processing module is used for satisfying under the situation of predetermined idle condition at first memory and second memory, with the data conversion storage that writes in the second memory to first memory.
Wherein, second processing module is used for internal memory being needed to be stored to before data in the first memory are written in the second memory, further judge whether second memory is in idle condition, if judged result is for being, then determine to allow the data that need in the internal memory to be stored in the first memory are written in the second memory; Otherwise wait for that first memory enters idle condition.
And predetermined idle condition can comprise: first memory is in idle condition, and second is in idle condition.
Alternatively, predetermined idle condition can also comprise: the storage space of second memory is taken, or data quantity stored reaches predetermined threshold in the second memory.
Alternatively, first memory is a hard disk; Second memory is one of following: solid state hard disc, Flash hard disk.
The present invention comes buffer memory need store data in the first memory into by the higher storer of access speed (second memory) is set, again data in buffer is written in the first memory by second memory afterwards, thereby can effectively improve the efficient that the checkpoint data write, avoid causing slowly the problem of the performance heavy losses of equipment such as server owing to writing speed.
Description of drawings
Fig. 1 is the configuration diagram that storage server is connected with calculation server in the correlation technique;
Fig. 2 is according to the process flow diagram of the storage means of the checkpoint data of the embodiment of the invention;
Fig. 3 writes the fashionable process flow diagram that carries out condition judgment according to data in the storage means of the checkpoint data of the embodiment of the invention;
Fig. 4 is according to the block diagram of the memory storage of the checkpoint data of the embodiment of the invention;
Fig. 5 is a synoptic diagram of setting up the memory storage of checkpoint of the present invention data in existing system.
Embodiment
At writing speed slow problem in checkpoint in the correlation technique, the present invention proposes, come buffer memory need store data in the first memory into by means of the higher storer of access speed (second memory), again data in buffer is written in the first memory by second memory afterwards, thereby can effectively improve the efficient that the checkpoint data write, avoid causing slowly the problem of the performance heavy losses of equipment such as server owing to writing speed.
Below in conjunction with accompanying drawing, describe embodiments of the invention in detail.
As shown in Figure 2, the storage means according to the checkpoint data of the embodiment of the invention comprises:
Step S201 in response to the checkpoint order, is written to the data that need in the internal memory to be stored in the first memory in the second memory, and wherein, the access speed of second memory is higher than the access speed of first memory;
Step S203 satisfies at first memory and second memory under the situation of predetermined idle condition, with the data conversion storage that writes in the second memory to first memory.
By means of above-mentioned processing, by being set, the higher storer of access speed (second memory) come buffer memory need store data in the first memory into, again data in buffer is written in the first memory by second memory afterwards, can effectively improve the efficient that the checkpoint data write, avoid causing slowly the problem of the performance heavy losses of equipment such as server owing to writing speed.
Wherein, in with internal memory, need to be stored to before data in the first memory are written in the second memory, can judge at first whether second memory is in idle condition, if judged result is for being, then determine to allow the data that need in the internal memory to be stored in the first memory are written in the second memory; Otherwise wait for that first memory enters idle condition, thereby can guarantee further that data can normally be written in the second memory.
When the data with second memory write first memory, the predetermined idle condition of institute's reference can be in idle condition for first memory, and second be in idle condition, thereby guaranteed that data write accuracy.
And predetermined idle condition can also comprise: the storage space of second memory is taken, or data quantity stored reaches predetermined threshold in the second memory.
That is to say, when first memory and second memory are all idle, if being taken, can't continue when wherein writing the overabundance of data of data or second memory storage the storage space of second memory, data in the second memory can be write in the first memory, and empty second memory.So just can avoid second memory and first memory are carried out frequent read-write operation, reduce the processing complexity of device interior.
Alternatively, above-mentioned first memory can be hard disk, and second memory should be higher than the access speed of hard disk like this, for example, second memory can be solid state hard disc or Flash hard disk, also can be other speed storer faster, and concrete this paper enumerates no longer one by one.
As shown in Figure 3, write fashionablely carrying out data, can judge at first that high-speed component (promptly, above-mentioned second memory) whether idle, if the judgment is Yes, then data are write high-speed component, write (high-speed component is changed to lock-out state) to high-speed component otherwise suspend;
Afterwards, after data are write high-speed component, judge that high-speed component is whether idle, if the judgment is Yes, judge further then whether low speed parts (that is, above-mentioned first memory) are idle; Write low speed parts (high-speed component is changed to lock-out state) otherwise suspend; If it is idle judging high-speed component, judge further then whether the low speed parts are idle, if the judgment is Yes, then data are write the low speed parts from high-speed component, if judge that the low speed parts are not idle, then time-out writes low speed parts (the low speed parts are changed to lock-out state), waits for that the low speed parts write when being in idle condition again.
By means of above-mentioned processing, by being set, the higher storer of access speed (second memory) come buffer memory need store data in the first memory into, again data in buffer is written in the first memory by second memory afterwards, can effectively improve the efficient that the checkpoint data write, avoid causing slowly the problem of the performance heavy losses of equipment such as server owing to writing speed; In addition, be used for the predetermined idle condition that control data writes, can guarantee to carry out reading and writing of data, make the efficient of read-write operation and accuracy be guaranteed in the rational time by reasonable disposition.
The memory storage of a kind of checkpoint data also is provided according to another embodiment of the present invention.
As shown in Figure 4, the memory storage according to the checkpoint data of the embodiment of the invention comprises:
In addition, second processing module 43 also is used for internal memory being needed to be stored to before data in the first memory are written in the second memory 41, further judge whether second memory 41 is in idle condition, if judged result is for being, then determine to allow the data that need in the internal memory to be stored in the first memory are written in the second memory 41; Otherwise wait for that first memory enters idle condition.
And predetermined idle condition can be: first memory is in idle condition, and second is in idle condition.Preferably, predetermined idle condition can also comprise: the storage space of second memory is taken, or data quantity stored reaches predetermined threshold in the second memory.
Be used for the predetermined idle condition that control data writes by as above disposing, can guarantee that first and second storeies carry out data write under the situation of free time, thereby guarantee the accuracy that data write.
And above-mentioned first memory can be hard disk, and above-mentioned second memory high-speed memory for example, can be solid state hard disc or Flash hard disk etc.
When realizing the solution of the present invention, need change on a small quantity storage server and hardware structure and network, at the inner quick storage access component (second memory) that inserts of server, for example, can adopt high speed storing clamping part based on internal memory particle and FPGA.
The buffer memory of writing that needs in addition to be provided with based on this quick storage access component drives (that is, the first above-mentioned processing module and second processing module), and this driving can be the buffer memory of low speed storage device with the high speed storing equipment disposition, thereby realizes that disk writes fast.
As shown in Figure 5, under this drives, system can adopt original VFS file system operation fully, transparent fully to the application system, this is write the buffer memory driving-disc and can be connected with the VFS file system, and being connected to Journal File System simultaneously, for example, can be Ext3 (Third extended filesystem).
Describe before the processing of describing among the method embodiment before device shown in Figure 4 can be realized equally, detailed process, no longer repeat here.
In sum, by means of technique scheme of the present invention, by being set, the higher storer of access speed (second memory) come buffer memory need store data in the first memory into, again data in buffer is written in the first memory by second memory afterwards, can effectively improve the efficient that the checkpoint data write, avoid causing slowly the problem of the performance heavy losses of equipment such as server owing to writing speed; In addition, be used for the predetermined idle condition that control data writes, can guarantee to carry out reading and writing of data, make the efficient of read-write operation and accuracy be guaranteed in the rational time by reasonable disposition.
The above only is preferred embodiment of the present invention, and is in order to restriction the present invention, within the spirit and principles in the present invention not all, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. the storage means of checkpoint data is characterized in that, comprising:
In response to the checkpoint order, the data that need in the internal memory to be stored in the first memory are written in the second memory, wherein, the access speed of described second memory is higher than the access speed of described first memory;
Satisfy at described first memory and described second memory under the situation of predetermined idle condition, with the described data conversion storage that writes in the described second memory to described first memory.
2. method according to claim 1 is characterized in that, needs to be stored in internal memory before data in the first memory are written in the second memory, and described method also comprises:
Judge whether described second memory is in idle condition,, then determine to allow the data that need in the described internal memory to be stored in the described first memory are written in the described second memory if judged result is for being; Otherwise wait for that described first memory enters idle condition.
3. method according to claim 1 is characterized in that, described predetermined idle condition comprises: described first memory is in idle condition, and described second is in idle condition.
4. method according to claim 3 is characterized in that, described predetermined idle condition also comprises:
The storage space of described second memory is taken, or data quantity stored reaches predetermined threshold in the described second memory.
5. according to each described method in the claim 1 to 4, it is characterized in that described first memory is a hard disk; Described second memory is one of following: solid state hard disc, Flash hard disk.
6. the memory storage of checkpoint data is characterized in that, comprising:
Second memory;
First processing module is used in response to the checkpoint order, and the data that need in the internal memory to be stored in the first memory are written in the described second memory, and wherein, the access speed of described second memory is higher than the access speed of described first memory;
Second processing module is used for satisfying under the situation of predetermined idle condition at described first memory and described second memory, with the described data conversion storage that writes in the described second memory to described first memory.
7. device according to claim 6, it is characterized in that, described second processing module is used for internal memory being needed to be stored to before data in the first memory are written in the second memory, further judge whether described second memory is in idle condition, if judged result is for being, then determine to allow the data that need in the described internal memory to be stored in the described first memory are written in the described second memory; Otherwise wait for that described first memory enters idle condition.
8. device according to claim 6 is characterized in that, described predetermined idle condition comprises: described first memory is in idle condition, and described second is in idle condition.
9. device according to claim 8 is characterized in that, described predetermined idle condition also comprises:
The storage space of described second memory is taken, or data quantity stored reaches predetermined threshold in the described second memory.
10. according to each described device in the claim 6 to 9, it is characterized in that described first memory is a hard disk; Described second memory is one of following: solid state hard disc, Flash hard disk.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110116063XA CN102184141A (en) | 2011-05-05 | 2011-05-05 | Method and device for storing check point data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110116063XA CN102184141A (en) | 2011-05-05 | 2011-05-05 | Method and device for storing check point data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102184141A true CN102184141A (en) | 2011-09-14 |
Family
ID=44570321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110116063XA Pending CN102184141A (en) | 2011-05-05 | 2011-05-05 | Method and device for storing check point data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102184141A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294412A (en) * | 2012-03-01 | 2013-09-11 | Hgst荷兰公司 | Implementing large block random write hot spare SSD for SMR RAID |
| CN103678149A (en) * | 2013-12-19 | 2014-03-26 | 华为技术有限公司 | Data processing method and device |
| CN104081357A (en) * | 2012-04-27 | 2014-10-01 | 惠普发展公司，有限责任合伙企业 | Local checkpointing using a multi-level cell |
| CN105094985A (en) * | 2015-07-15 | 2015-11-25 | 上海新储集成电路有限公司 | Low-power-consumption data center for sharing memory pool and working method thereof |
| CN108984117A (en) * | 2018-06-15 | 2018-12-11 | 深圳市华傲数据技术有限公司 | A kind of data read-write method, medium and equipment |
| CN112540984A (en) * | 2020-11-23 | 2021-03-23 | 成都佳华物链云科技有限公司 | Data storage method, query method, device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101446924A (en) * | 2008-12-16 | 2009-06-03 | 成都市华为赛门铁克科技有限公司 | Method and system for storing and obtaining data |
| CN102016808A (en) * | 2008-05-01 | 2011-04-13 | 惠普发展公司，有限责任合伙企业 | Storing checkpoint data in non-volatile memory |
-
2011
- 2011-05-05 CN CN201110116063XA patent/CN102184141A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102016808A (en) * | 2008-05-01 | 2011-04-13 | 惠普发展公司，有限责任合伙企业 | Storing checkpoint data in non-volatile memory |
| CN101446924A (en) * | 2008-12-16 | 2009-06-03 | 成都市华为赛门铁克科技有限公司 | Method and system for storing and obtaining data |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294412A (en) * | 2012-03-01 | 2013-09-11 | Hgst荷兰公司 | Implementing large block random write hot spare SSD for SMR RAID |
| CN103294412B (en) * | 2012-03-01 | 2018-02-16 | Hgst荷兰公司 | Realize the bulk random writing Hot Spare solid-state drive for tiles magnet record |
| CN104081357A (en) * | 2012-04-27 | 2014-10-01 | 惠普发展公司，有限责任合伙企业 | Local checkpointing using a multi-level cell |
| US10025663B2 (en) | 2012-04-27 | 2018-07-17 | Hewlett Packard Enterprise Development Lp | Local checkpointing using a multi-level cell |
| CN103678149A (en) * | 2013-12-19 | 2014-03-26 | 华为技术有限公司 | Data processing method and device |
| CN103678149B (en) * | 2013-12-19 | 2017-01-18 | 华为技术有限公司 | Data processing method and device |
| CN105094985A (en) * | 2015-07-15 | 2015-11-25 | 上海新储集成电路有限公司 | Low-power-consumption data center for sharing memory pool and working method thereof |
| CN108984117A (en) * | 2018-06-15 | 2018-12-11 | 深圳市华傲数据技术有限公司 | A kind of data read-write method, medium and equipment |
| CN108984117B (en) * | 2018-06-15 | 2021-11-19 | 深圳市华傲数据技术有限公司 | Data reading and writing method, medium and equipment |
| CN112540984A (en) * | 2020-11-23 | 2021-03-23 | 成都佳华物链云科技有限公司 | Data storage method, query method, device, electronic equipment and storage medium |
| CN112540984B (en) * | 2020-11-23 | 2023-10-03 | 成都佳华物链云科技有限公司 | Data storage method, query method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11029853B2 (en) | Dynamic segment allocation for write requests by a storage system | |
| US9110669B2 (en) | Power management of a storage device including multiple processing cores | |
| US7596657B2 (en) | Increased storage capacity for solid state disks using data compression | |
| CN101866359B (en) | Small file storage and visit method in avicade file system | |
| CN102184141A (en) | Method and device for storing check point data | |
| US10558395B2 (en) | Memory system including a nonvolatile memory and a volatile memory, and processing method using the memory system | |
| CN105630700B (en) | A kind of storage system and reading/writing method with secondary cache structure | |
| US20180107601A1 (en) | Cache architecture and algorithms for hybrid object storage devices | |
| KR101200998B1 (en) | Hybrid raid controller having multi pci bus switching | |
| CN105630405B (en) | A kind of storage system and the reading/writing method using the storage system | |
| US9898413B2 (en) | Auto-adaptive system to implement partial write buffering for storage systems dynamic caching method and system for data storage system | |
| CN103473142A (en) | Virtual machine transferring method and device under cloud computing operating system | |
| CN104571955A (en) | Method and device for expanding storage capacity | |
| CN115794669A (en) | Method, device and related equipment for expanding memory | |
| US10459641B2 (en) | Efficient serialization of journal data | |
| CN109558335A (en) | A kind of file memory format of the embedded system based on Nor flash storage | |
| CN103927215A (en) | kvm virtual machine scheduling optimization method and system based on memory disk and SSD disk | |
| EP2642397A1 (en) | System for dynamically adaptive caching | |
| KR101200997B1 (en) | Raid controller having multi pci bus switching | |
| CN113220650B (en) | Data storage method, apparatus, device, storage medium, and program | |
| CN103500147A (en) | Embedded and layered storage method of PB-class cluster storage system | |
| CN104239231A (en) | Method and device for accelerating preheating of second-level cache | |
| CN103823634A (en) | Data processing method and system supporting non-random write mode | |
| CN103412727A (en) | Method for optimizing deletion command of solid state disk and solid state disk using same | |
| CN103092677A (en) | Internal storage energy-saving system and method suitable for virtualization platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 100193 Beijing, Haidian District, northeast Wang West Road, building 8, No. 36 Applicant after: Dawning Information Industry (Beijing) Co., Ltd. Address before: 100084 Beijing Haidian District City Mill Street No. 64 Applicant before: Dawning Information Industry (Beijing) Co., Ltd. |
|
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20110914 |