CN105094761A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN105094761A CN105094761A CN201410182608.0A CN201410182608A CN105094761A CN 105094761 A CN105094761 A CN 105094761A CN 201410182608 A CN201410182608 A CN 201410182608A CN 105094761 A CN105094761 A CN 105094761A
- Authority
- CN
- China
- Prior art keywords
- data
- duty
- storage
- day
- memory storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/16—Protection against loss of memory contents
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method and device, and the technical problem that the existing storage system in a data copy mode is not compatible with a read-write performance is solved to some extent. In some feasible embodiments of the invention, the data storage method comprises that a first copy of data to be stored is written in a plurality of main storage devices of a main storage subsystem in a first data organization manner, a second copy of the data to be stored is written in an on-duty auxiliary storage device of an auxiliary storage subsystem in a second data organization manner, data, stored in one main storage device corresponding to the on-duty auxiliary storage device, which is not consistent with the on-duty auxiliary storage device is read out, the read-out inconsistent data is written into the on-duty auxiliary storage device in a third data organization manner, the first data organization manner and the second data organization manner are write-optimized, the write-in speed of the second data organization manner is faster than that of the first data organization manner, and the third data organization manner is read-optimized.
Description
Technical field
The present invention relates to computing machine and technical field of memory, be specifically related to a kind of date storage method and equipment.
Background technology
The progress of digital technology and the development of memory technology have expedited the emergence of the data of magnanimity, and data need to store within the storage system with certain organizational form.Data, from producing in the whole life cycle of withering away, have different input and output (input/output, I/O) access characteristics in the different stages; Such as, produce the stage in data, data need with high speed writein storage system; At data analysis phase, data need with high speed readout or scanning thus participate in calculating.
In order to improve memory reliability, storing data grows in copy mode and getting the nod.Copy is the future trend of data redundancy mode.The typical scene of data trnascription is disk array (RedundantArraysofIndependentDisks, RAID, full name Redundant Array of Independent Disks (RAID) are called for short disk array).RAID10 is a kind of conventional disk array, comprises one group of master disk and one group of mirrored disk, and data copy is divided into multiple part by RAID10, is stored on multiple master disk respectively, to improve readwrite performance; Further, another copy of data is stored on corresponding mirrored disk, to improve reliability.
In the cost of storage system, the ratio shared by the disposable input cost such as software and hardware is limited, and energy consumption cost then becomes the major part in storage system total cost gradually.In a kind of prior art, adopt and rotate the energy consumption that daily record framework RoLo reduces disk array RAID10.Rotate the log space resource pool that the free space of multiple mirrored disk is integrated into a logic by daily record framework.Do synchronously distributed by excavated space timeslice, this log space resource pool can be recycled land productivity in order to improve performance and the efficiency of system.In other words, multiple mirrored disk is used as daily record dish on duty for the day in turn, and the dish of non-value day daily record is simultaneously switched to low energy consumption state, thus reduces energy consumption.
The disk array storage system of rotation daily record framework mode conventional at present has following defect: existing disk array storage system turns to a virtual disk by virtual for multiple physical disk, file system is based upon on virtual disk, namely, file system is not had on single disk, thus same Method of Data Organization can only be adopted in all disks to write data, this Method of Data Organization or write optimization, to improve write performance, read to optimize, read performance to improve, and readwrite performance can not be taken into account simultaneously.
Summary of the invention
The embodiment of the present invention provides a kind of date storage method and equipment, can not take into account the technical matters of readwrite performance with the storage system solving existing data trnascription mode to a certain extent.
First aspect present invention provides a kind of date storage method, and for storage system, described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system; Described method comprises: by first of data to be stored copy with the multiple main storage means in the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage; By that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization; Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
In the implementation that the first is possible, described method also comprises: judge whether the occupancy of the log space on duty for the day of standby memory storage on duty for the day reaches preset value; If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
In conjunction with the first possible implementation of first aspect or first aspect, in the implementation that the second is possible, the first Method of Data Organization described is log-structured merging LSM mode; Described the second Method of Data Organization is log-structured file system LFS mode; The third Method of Data Organization described is that B+ sets mode.
In conjunction with the implementation that the second of first aspect or first aspect is possible, in the implementation that the third is possible, described by first of data to be stored copy with the first Method of Data Organization write main storage subsystem before, also comprise: in each main storage means, create key assignments storage system LevelDB; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
In conjunction with the third possible implementation of first aspect or first aspect, in the 4th kind of possible implementation, described by first of data to be stored copy with the first Method of Data Organization write main storage subsystem in multiple main storage means comprise: first of data to be stored copy is divided into multiple part, described multiple part is write in the LevelDB of multiple main storage means respectively; Described write with the second Method of Data Organization by second of data to be stored copy comprises for the on duty for the day standby memory storage in storage subsystem: by the LFS of second of data to be stored described standby memory storage on duty for the day of copy write; Described the inconsistent data of reading to be comprised with the described standby memory storage on duty for the day of the third Method of Data Organization write: by the BDB of the described standby memory storage on duty for the day of inconsistent data write that reads.
In conjunction with the first of first aspect or first aspect to any one implementation in the 4th kind of mode, in the 5th kind of possible implementation, described storage system is disk array or node array.
Second aspect present invention provides a kind of data storage device, and for storage system, described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system; Described equipment comprises: the first module for reading and writing, for first of data to be stored copy to be write the multiple main storage means in main storage subsystem with the first Method of Data Organization; Second module for reading and writing, for second of data to be stored copy to be write the on duty for the day standby memory storage in standby storage subsystem with the second Method of Data Organization, described standby memory storage on duty for the day is uniquely in running order standby memory storage; Third reading writing module, for by that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and by the inconsistent data that reads with the described standby memory storage on duty for the day of the third Method of Data Organization write; Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
In the implementation that the first is possible, described equipment also comprises: memory storage monitoring module, and whether the occupancy for the log space on duty for the day judging standby memory storage on duty for the day reaches preset value; If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
In conjunction with the first possible implementation of first aspect or first aspect, in the implementation that the second is possible, described equipment also comprises: creation module, for creating key assignments storage system LevelDB in each main storage means; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
In conjunction with the implementation that the second of first aspect or first aspect is possible, in the implementation that the third is possible, described multiple part, specifically for first of data to be stored copy is divided into multiple part, writes in the LevelDB of multiple main storage means by described first module for reading and writing respectively; Described second module for reading and writing is specifically for writing in the LFS of described standby memory storage on duty for the day by second of data to be stored copy; Described third reading writing module is described on duty for the day in the BDB of memory storage specifically for the inconsistent data write that will read.
Third aspect present invention provides a kind of computer equipment, can comprise: processor, storer, communication interface, bus; Described processor, storer, communication interface is by the mutual communication of described bus; Described storer, comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system; Wherein, described processor is used for first of data to be stored copy with the multiple main storage means in the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage; By that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization; Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
Therefore, the embodiment of the present invention adopts and create file system in each main storage means and each standby memory storage, and, by first of data to be stored copy with the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, by what store in main storage means corresponding for described standby memory storage on duty for the day, the data reading inconsistent with described standby memory storage on duty for the day, and the inconsistent data read is write the technical scheme of described standby memory storage on duty for the day with the third Method of Data Organization, achieve following technique effect:
Each main storage means and each standby memory storage create and has file system, data with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype respectively; Wherein, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase; Thus make whole storage system take into account readwrite performance.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can avoid standby memory storage on duty for the day to produce write bottleneck.
Accompanying drawing explanation
In order to be illustrated more clearly in embodiment of the present invention technical scheme, be briefly described to the accompanying drawing used required in embodiment and description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the schematic diagram of a kind of date storage method provided by the invention;
Fig. 2 is the schematic diagram of another kind of date storage method provided by the invention;
Fig. 3 is the schematic diagram of disk array RAID10 in the present invention's scene embodiment;
Fig. 4 is that embodiment of the present invention method is for the data storage operations schematic diagram of RAID10;
Fig. 5 is the schematic diagram of a kind of data storage device provided by the invention;
Fig. 6 is the schematic diagram of another kind of data storage device provided by the invention;
Fig. 7 is the schematic diagram of a kind of computer equipment provided by the invention.
Embodiment
The disk array storage system of rotation daily record framework mode conventional at present has following defect:
One, in existing disk array storage system, a virtual level is had on disk, it is a virtual disk by multiple magnetic disc virtualization, file system is based upon on virtual disk, namely, do not have file system on single disk, each disk is block device, and whole disk array realizes in block level.This just causes: the data of write disk can not reflect the feature of data structure layer also can cause the reconstruction speed of block level disk array slow further.If one piece of disk failure, when recovering data from the mirrored disk of correspondence, need to recover each data block on mirrored disk, therefore speed is slower.
Two, in existing storage system, file system is set up on virtual disk, same Method of Data Organization is adopted to write data in all disks, this Method of Data Organization or write optimization, to improve write performance, be read to optimize, read performance to improve, and readwrite performance can not be taken into account simultaneously.Further, rotate in the disk array of framework log mode, a copy of data is divided into the multiple master disk of multiple partial write, and writing speed is very fast; But another copy of data is only written into a daily record dish on duty for the day, writing speed is comparatively slow, does not catch up with the writing speed of master disk, can form bottleneck, affect the performance of whole disk array.
The embodiment of the present invention provides a kind of date storage method and equipment, slow with the reconstruction speed that the storage system solving existing data trnascription mode causes because realizing in block level, the problems such as data structure layer feature can not be reflected, and to cause because adopting same Method of Data Organization to write data in all disks can not take into account readwrite performance, there will be the problems such as write bottleneck at daily record dish on duty for the day.
The present invention program is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the embodiment of a part of the present invention, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, should belong to the scope of protection of the invention.
Below by specific embodiment, be described in detail respectively.
Please refer to Fig. 1, the embodiment of the present invention provides a kind of date storage method.
The method is applied to storage system, and described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer.In the present embodiment, said storage system can be disk array or node array, and said active and standby memory storage can be disk or node etc.In the present embodiment, each main storage means and each standby memory storage create and has file system; The file system that different memory storages creates can be identical, also can be different.
As shown in Figure 1, embodiment of the present invention method can comprise:
110, first of data to be stored copy is write the multiple main storage means in main storage subsystem with the first Method of Data Organization, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage;
120, by that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization.
In the present embodiment, by every a data copy two copies to be stored, wherein first copy write main storage subsystem, second standby storage subsystem of copy write.In order to improve the readwrite performance of data, N number of main storage means of main storage subsystem all can be remained on duty, first copy is divided into N number of part, each partial write main storage means.In order to improve the reliability of data, at any time in section, in the N number of standby memory storage of standby storage subsystem can be remained on duty, as standby memory storage on duty for the day, second of data copy being write this standby memory storage on duty for the day; Other is then in dormancy or the low energy consumption state such as standby, to reduce energy consumption for memory storage.
In the embodiment of the present invention, in order to improve writing speed, the first Method of Data Organization above-mentioned and the second Method of Data Organization write optimization; And read performance in order to improve data in the analysis phase, the third Method of Data Organization is then read to optimize; Thus make whole storage system take into account readwrite performance.Further, in order to avoid forming write bottleneck on standby memory storage on duty for the day, the writing speed of writing speed faster than the first Method of Data Organization of the second Method of Data Organization can be made; So, although first of data to be stored copy adopts the first Method of Data Organization to be written simultaneously multiple main storage means, writing speed can be improved, but, Method of Data Organization speed the second Method of Data Organization faster during what second copy write standby memory storage on duty for the day of data adopted is than first, like this, so just can average out, as far as possible by the while of two of data copies or close to writing active and standby storage subsystem, to improve the write performance of whole storage system simultaneously.In addition, in the embodiment of the present invention, each memory storage is set up and has file system, the data of the memory storage of write exist with certain organizational form, can reflect the feature of data structure layer.
To sum up, embodiments provide a kind of date storage method, the method adopts above-mentioned technical characteristic, achieve following technique effect: each main storage means and each standby memory storage create file system, data with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype respectively; Wherein, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase; Thus make whole storage system take into account readwrite performance.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can avoid standby memory storage on duty for the day to produce write bottleneck.
In some embodiments of the invention, described method also comprises: judge whether the occupancy of the log space on duty for the day of standby memory storage on duty for the day reaches preset value.
In the embodiment of the present invention, each standby memory storage is the mirror storage device of corresponding main storage means, and the idle storage space on all standby memory storages is considered as log space, and the log space that standby memory storage on duty for the day provides is called log space on duty for the day.In the present embodiment, the service condition of the log space on duty for the day of standby memory storage on duty for the day can be monitored in real time, judgment value day the occupancy of log space whether reach preset value.
If when receiving data to be stored, judgment value day the occupancy of log space do not reach preset value, then normally perform step 110, respectively two of data copies write respectively main storage subsystem and standby memory storage on duty for the day.
If judgment value day, the occupancy of log space reached preset value, then in the present embodiment, described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.Further, the daily record of switching value day, for after memory storage, can trigger a synchronized process, and this synchronized process is used for the data syn-chronization in the main storage means of correspondence in log storage on duty for the day.Namely, in some embodiments, above-mentioned steps 120 can be on duty for the day for the execution that be triggered during memory storage in switching, by that store in main storage means corresponding for standby memory storage on duty for the day, inconsistent with standby memory storage on duty for the day data reading, and the inconsistent data read is write standby memory storage on duty for the day with the third Method of Data Organization.
This synchronous operation can perform on the backstage of computer equipment, utilizes idle bandwidth and the free space of memory storage, is synchronized to by the inconsistent data of reading in standby memory storage on duty for the day, thus does not consume additional energy.In the embodiment of the present invention, in order to improve write performance, above-mentioned the first and the second Method of Data Organization adopted when writing data can select to have the Method of Data Organization of higher write performance; In order to improve the performance of sense data, the third Method of Data Organization said can select to have the higher Method of Data Organization reading performance, so that follow-up standby memory storage can provide higher and read performance, and, because said synchronous operation is carried out on backstage, the write performance of the third Method of Data Organization therefore can not be considered.
In some embodiments of the invention, the first Method of Data Organization described can select sequential log structure, such as log-structured merging (LogStructuredMerge, LSM) mode; Described the second Method of Data Organization can select out-of-order log-structured, such as log-structured file system (LogStructuredFileSystem, LFS) mode; The third Method of Data Organization described can select local update index structure, and such as B+ sets mode.Wherein, LSM and LFS can provide good write performance, and B+ tree can provide reads performance preferably.And LFS writing rate is faster than the writing rate of LSM.
In order to realize above-mentioned several data organizational form, in the embodiment of the present invention, in the initialization procedure before step 110, each main storage means and each standby memory storage can be formatted as arbitrary file system respectively; Further, each main storage means creating LSM system, such as, can be specifically key assignments storage system LevelDB; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates B+ and sets system, such as, can be specifically key assignments storage system BDB.
Then in above-mentioned steps, described by first of data to be stored copy with the first Method of Data Organization write main storage subsystem can comprise: first of data to be stored copy is divided into multiple part, described multiple part is write in the LevelDB of multiple main storage means of described main storage subsystem respectively.Described write with the second Method of Data Organization by second of data to be stored copy can comprise for the on duty for the day standby memory storage in storage subsystem: by the LFS of second of data to be stored described standby memory storage on duty for the day of copy write.Described the inconsistent data of reading can be comprised with the described standby memory storage on duty for the day of the third Method of Data Organization write: by the BDB of the described standby memory storage on duty for the day of inconsistent data write that reads.
In the embodiment of the present invention, in two storage areas be divided out in each standby memory storage, first storage area creates LFS, and this storage area is used for using when data write; Second storage area creates BDB, and this storage area is used for using when data syn-chronization.Because multiple standby memory storage is used as standby memory storage on duty for the day in turn, in rotation process, the copy data write in LFS mode in first storage area is only ephemeral data, is ceaselessly written into, and be ceaselessly released, therefore first storage area does not need too large; Second storage area is used for preserving the data of synchronously coming from main storage means, be need long-term data of preserving, be ceaselessly written into, but be not generally released, therefore second storage area needs larger storage space, can take most of space of standby memory storage.
Finally, the data in all main storage means are all store with the first Method of Data Organization, not only reflect data structure layer feature, and have higher write performance; Standby memory storage needs the long-term data trnascription preserved to be all then store with the third Method of Data Organization, also can reflect data structure layer feature, and, higher reading performance is provided.Thus, when subsequent applications needs to read data, digital independent can be carried out with higher reading speed from standby memory storage.
Embodiment of the present invention method is preferably applicable to the storage system of facing cloud storage environment.
Above, the embodiment of the invention discloses a kind of date storage method, the method adopts establishment in each main storage means and each standby memory storage to have file system, and, by first of data to be stored copy with the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, by what store in main storage means corresponding for described standby memory storage on duty for the day, the data reading inconsistent with described standby memory storage on duty for the day, and the inconsistent data read is write the technical scheme of described standby memory storage on duty for the day with the third Method of Data Organization, achieve following technique effect:
1, owing to only having standby memory storage on duty for the day in running order, other is all in idle low energy consumption state for memory storage, therefore, can reduce energy consumption.
2, each main storage means and each standby memory storage create have file system, data store with certain Method of Data Organization in active and standby memory storage, can reflect data structure layer feature, and improve reconstruction speed.If one piece of main storage means lost efficacy, when recovering data from the standby memory storage of correspondence, directly can know which data needs to recover by Method of Data Organization, thus need not identify each data block, therefore rebuild speed block.
3, data are respectively with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype.Such as, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can be avoided producing write bottleneck on standby memory storage on duty for the day.
4, that store in the main storage means that standby memory storage on duty for the day is corresponding, inconsistent with described standby memory storage on duty for the day data, the idle bandwidth and the free space that utilize memory storage on backstage, be synchronized in standby memory storage on duty for the day, when not consuming additional energy, achieve the conversion of Method of Data Organization.
5, data are with certain Method of Data Organization write main storage means and standby memory storage, realize at data structure layer, realize comparatively flexible, both can be implemented on block device, and also can be implemented on node, such as, can be used for disk array, also can be used for node array, and, the many disk environment of single node can be implemented in, also can be implemented in multinode distributed storage environment.
For ease of better understanding the technical scheme that the embodiment of the present invention provides, be that example is introduced below by the embodiment under a concrete scene.
In this scene embodiment, be disk array RAID10 for said storage system.RAID10 comprises one group of master disk and one group for disk (or being called mirrored disk).One group of N number of master disk forms main storage subsystem (or being called master disk group), and one group of N number of mirrored disk forms standby storage subsystem (or being called standby disk group, mirrored disk group).
In the present embodiment, all mirrored disks are all considered as journal disk, the idle storage space on all mirrored disks are considered as the available log space of journal disk; Only a journal disk is remained on active state (i.e. duty) in section at any time, response write operation requests; The mirrored disk remaining on active state is called journal disk on duty for the day, and the log space that journal disk on duty for the day provides is called log space on duty for the day.
As shown in Figure 2, this scene embodiment comprises the following steps:
210, key assignments request step is received: the key assignments request receiving self-application, carry data (i.e. key assignments data) to be stored in key assignments request, this key assignments request can be specifically write (PUT), reads (GET) or delete (DELETE) etc.
220, key assignments data redirection step: be redirected on destination disc by the key assignments request received, namely in corresponding master disk and journal disk on duty for the day.
230, first of data to be stored copy is write master disk group in LSM mode;
240, judgment value day journal disk the occupancy of log space on duty for the day whether reach preset value;
If 250 occupancies reach preset value, switching value day journal disk, comprising: journal disk on duty for the day is switched to dormancy or holding state by the duty of activity, another journal disk on duty for the day waken up and is switched to duty; Simultaneously, the blocked operation of journal disk on duty for the day triggers a synchronized process, in synchronized process, by that store in master disk corresponding for journal disk on duty for the day, inconsistent with journal disk on duty for the day data reading, and the inconsistent data of reading is write journal disk on duty for the day in B+ tree mode;
If 260 occupancies do not reach preset value, second of data to be stored copy is write the journal disk on duty for the day of standby disk group in LFS mode.
Describe in further detail below.
As shown in Figure 3, suppose that disk array RAID10 comprises six pieces of disks, wherein, three pieces of master disks, represent with P0, P1 and P2 respectively, and the three piece mirrored disks corresponding with these three pieces of master disks represent with M0, M1 and M2 respectively.Two corresponding disks become mirrored disk pair, and this RAID10 comprises three mirrored disks pair, uses (P0, M0) respectively, and (P1, M1) and (P2, M2) represents.
In Fig. 3, right cylinder represents disk, and in right cylinder, black shaded area represents occupied storage space in disk, and white portion represents not yet occupied storage space in disk.Suppose M0, all each idle storage space of own 50% on these three mirrored disks of M1 and M2, the i.e. log space of 50%.By curve three mirrored disks M0, M1 and M2 coupling together of band arrow by as journal disk, the idle storage space in these three journal disk, puts and part that twill represents, as log space with loose respectively.The loose point coupled together with the curve of band arrow and diagonal section represent the log space that the idle storage space of all three mirrored disks is formed.The mirrored disk at loose some place is journal disk on duty for the day, and the disk at twill place is non-value day journal disk.M0, M1 and M2 are used as journal disk on duty for the day successively, that is, the 0th daily record cycle, M0 is journal disk on duty for the day; The 1st daily record cycle, M1 is journal disk on duty for the day; The 2nd daily record cycle, M2 is journal disk on duty for the day; The 3rd daily record cycle, M0 is journal disk on duty for the day again; The rest may be inferred.
As shown in Fig. 4 (a), the bond space of key assignments data is cut into into isometric key assignments segmentation (KeyRange, KR), be labeled as respectively KR1, KR2, KR3, KR4 ..., KRi, and be distributed in mirrored disk to (P0, M0), (P1 by the mode turned with wheel, M1) and on (P2, M2).
As shown in Fig. 4 (b), (c) He (d), in daily record cycle T 0, M0 is used as journal disk on duty for the day, first copy of the new write data in this daily record cycle T 0 divides work three part, i.e. D0T0, D1T0 and D2T0, writes master disk P0, P1 and P2 respectively; Second copy D0T0, D1T0 and D2T0, will be written to mirrored disk M0.Similarly, when entering daily record cycle T 1, first copy of new write data divides work three part, i.e. D0T1, D1T1 and D2T1, writes master disk P0, P1 and P2 respectively; Second copy D0T1, D1T1 and D2T1, will be written to mirrored disk M1.After, the like.The dotted line of the band arrow shown in Fig. 4 and data layout illustrate the ultimate principle of circular logging.
Each new daily record cycle, when starting, the switching of journal disk on duty for the day can trigger a synchronized process.As shown in Fig. 4 (b), in daily record cycle T 0, M0 is selected as journal disk on duty for the day, before T0, there are not inconsistent data between (P0, M0) in the 0th mirrored disk, therefore, in T0, mirrored disk between (P0, M0) without synchronous operation.In the 1st daily record cycle T 1, M1 is selected as journal disk on duty for the day, before T1,1st mirrored disk to there are inconsistent data between (P1, M1), i.e. D1T0, therefore, in T1 start time, mirrored disk is triggered to the synchronizing process between (P1, M1), data D1T0 is written into M1, meanwhile, the space storing D1T0 in M0 is released, and this synchronizing process just stops after inconsistent data all synchronously being completed.The rest may be inferred, in daily record cycle T 2, M2 is selected as journal disk on duty for the day, and in T2 start time, the 2nd mirrored disk is triggered to the synchronizing process between (P2, M2), data D2T0 and D2T1 is written into M2, meanwhile, the space storing D2T0 and D2T1 in M0 and M1 is released, and this synchronizing process just stops after inconsistent data all synchronously being completed.
Fig. 4 (b), (c) and (d) are illustrated respectively in the key assignments Data distribution8 situation in T0, T1 and T2 tri-daily record end cycle moment disk groups.Wherein, DmTn representative writes m mirrored disk to (Pm in the n-th daily record period Tn, Mm) all key assignments data, in the present embodiment, m is 0,1 or 2, n be more than or equal to 0 natural number, blank boxes represents not yet occupied storage space on master disk and mirror drive, grid with twill represents that the storage space on disk represented by this region is released, during the key assignments data being with the grid of nicking to represent that on master disk, this logic region is corresponding are set to the B+ in mirrored disk by synchronized update.
When a new journal disk is selected as journal disk on duty for the day time, a new synchronizing process is just triggered, and this new synchronizing process only have all inconsistent datas in day on duty journal disk be updated complete after just can be terminated.As shown in Fig. 4 (b), in daily record cycle T 0, M0 is selected as journal disk on duty for the day, key assignments write operation requests arrives mirrored disk to (P0, M0) time, key assignments write operation requests data are write in the LSM data structure of master disk P0, through judging, if the occupancy of log space on duty for the day does not exceed the threshold value T preset on the journal disk M0 on duty for the day in T0, now key assignments write operation requests data are write in journal disk M0 on duty for the day with the mode of log-structured file system order; If the occupancy of log space on duty for the day exceedes the threshold value T preset on the journal disk M0 on duty for the day in T0, now M0 is switched to the holding state of low energy consumption, select M1 as new journal disk on duty for the day, M1 is switched to the active state of high energy consumption, trigger mirrored disk to the synchronizing process between (P1, M1).The rest may be inferred, and the key assignments data do not upgraded in (P1, M1) and (P2, M2), in T1 and T2, are read by Fig. 4 (c) and (d) display respectively from the LSM of P1 or P2, and write in the B+ tree of M1 and M2.In T1 and T2 finish time, journal disk M2 and M0 is selected as new journal disk on duty for the day respectively.
The rectangular box of the solid line of arrow and band twill is with to show respectively distributing synchronizing process and log space release schematic diagram in Fig. 4 (c) and (d).When journal disk M1 is selected as journal disk on duty for the day, trigger mirrored disk to the synchronizing process in (P1, M1) between master disk P1 and mirrored disk M1.After mirrored disk terminates the synchronizing process between (P1, M1), the storage space shared by the D1T0 on M0 is released.Similarly, mirrored disk is triggered when M2 is chosen as journal disk on duty for the day to the synchronizing process between (P2, M2).After mirrored disk terminates the synchronizing process between (P2, M2), the storage space shared by the D2T1 on D2T0 and M1 on M0 is released.
Due to the occupied log space of major part on journal disk M0 respectively in daily record cycle T 1 and T2 along with mirrored disk is to (P1, M1) and mirrored disk to (P2, M2) synchronizing process between is released, and journal disk M0 can be selected as journal disk on duty for the day again.The rest may be inferred, on journal disk M1 and M2, the occupied log space of major part is respectively along with mirrored disk is to (P0, M0), (P2, M2) and mirrored disk to (P0, M0), (P1, M1) synchronizing process between is released, and therefore, M1 and M2 also can be selected as journal disk on duty for the day again.
In the embodiment of the present invention, in order to enable disk array support case method of the present invention, needing to carry out initialization process to disk array in advance, comprising:
In initialization procedure, by each master disk with eachly standby deposit disk and be formatted as arbitrary file system respectively; Further, each master disk creating LSM system, such as, can be specifically key assignments storage system LevelDB; In each standby disk, mark off two storage areas, wherein, create LFS at first storage area; Second storage area creates B+ and sets system, such as, can be specifically key assignments storage system BDB.Make follow-up can: first of data to be stored copy is write in the LevelDB of described main storage subsystem, by in the LFS of second of data to be stored copy write standby memory storage on duty for the day, and, in synchronizing process, by duty for the day in the BDB of memory storage for the inconsistent data write read.
Above, the present embodiment is illustrated for disk array, but needs to understand, in other embodiments, said storage system is not limited to disk array, and the disk in disk array can be naked disk, also can be formatted as specific file system after disk; Said storage system also can be node array, and node array goes for multinode distributed environment.
In addition, previously described the first, the second and the third Method of Data Organization can choose at random as required, such as, the first Method of Data Organization can select LSM, but B+ also can be selected to set, or other organizational form; The second Method of Data Organization can select LFS or B+ to set; The third Method of Data Organization can select B+ to set or LSM; No longer be described in detail herein.
Above, the embodiment of the invention discloses a kind of date storage method, achieve following technique effect:
1, owing to only having standby memory storage on duty for the day in running order, other is all in idle low energy consumption state for memory storage, therefore, can reduce energy consumption.
2, each main storage means and each standby memory storage create have file system, data all store with certain Method of Data Organization in active and standby memory storage, can reflect data structure layer feature, and improve reconstruction speed.If one piece of main storage means lost efficacy, when recovering data from the standby memory storage of correspondence, directly can know which data needs to recover by Method of Data Organization, thus need not identify each data block, therefore rebuild speed block.
3, data are respectively with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype.Such as, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can be avoided producing write bottleneck on standby memory storage on duty for the day.
4, that store in the main storage means that standby memory storage on duty for the day is corresponding, inconsistent with described standby memory storage on duty for the day data, the idle bandwidth and the free space that utilize memory storage on backstage, be synchronized in standby memory storage on duty for the day, when not consuming additional energy, achieve the conversion of Method of Data Organization.
5, data are with certain Method of Data Organization write main storage means and standby memory storage, realize at data structure layer, realize comparatively flexible, both can be implemented on block device, and also can be implemented on node, such as, can be used for disk array, also can be used for node array, and, the many disk environment of single node can be implemented in, also can be implemented in multinode distributed storage environment.
In order to better implement the such scheme of the embodiment of the present invention, be also provided for below coordinating the relevant apparatus implementing such scheme.
Please refer to Fig. 5, the embodiment of the present invention provides a kind of data storage device 500.This equipment is used for storage system, and described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system.Described equipment 500 can comprise:
First module for reading and writing 510, for writing the multiple main storage means in main storage subsystem with the first Method of Data Organization by first of data to be stored copy;
Second module for reading and writing 520, for second of data to be stored copy to be write the on duty for the day standby memory storage in standby storage subsystem with the second Method of Data Organization, described standby memory storage on duty for the day is uniquely in running order standby memory storage;
Third reading writing module 530, for by that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and by the inconsistent data that reads with the described standby memory storage on duty for the day of the third Method of Data Organization write;
Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
As shown in Figure 6, in some embodiments of the invention, described equipment can also comprise:
Memory storage monitoring module 540, whether the occupancy for the log space on duty for the day judging standby memory storage on duty for the day reaches preset value; If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
As shown in Figure 6, in other embodiments of the present invention, described equipment can also comprise:
Creation module 550, for creating key assignments storage system LevelDB in each main storage means; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
In other embodiments of the present invention, described first module for reading and writing specifically can be used for first of data to be stored copy to be divided into multiple part, writes in the LevelDB of multiple main storage means respectively by described multiple part; Described second module for reading and writing specifically can be used in the LFS of second of data to be stored described standby memory storage on duty for the day of copy write; The inconsistent data write that described third reading writing module specifically can be used for reading is described on duty for the day in the BDB of memory storage.
The data storage device of the embodiment of the present invention can be such as the computer equipment comprising disk array, or the network equipment of management node array.
Be appreciated that the function of each functional module of the data storage device of the embodiment of the present invention can according to the method specific implementation in said method embodiment, its specific implementation process can refer to the associated description in said method embodiment, repeats no more herein.
Therefore, in feasible embodiments more of the present invention, establishment in each main storage means and each standby memory storage is adopted to have file system, and, by first of data to be stored copy with the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, by what store in main storage means corresponding for described standby memory storage on duty for the day, the data reading inconsistent with described standby memory storage on duty for the day, and the inconsistent data read is write the technical scheme of described standby memory storage on duty for the day with the third Method of Data Organization, achieve following technique effect:
1, owing to only having standby memory storage on duty for the day in running order, other is all in idle low energy consumption state for memory storage, therefore, can reduce energy consumption.
2, each main storage means and each standby memory storage create have file system, data store with certain Method of Data Organization in active and standby memory storage, can reflect data structure layer feature, and improve reconstruction speed.If one piece of main storage means lost efficacy, when recovering data from the standby memory storage of correspondence, directly can know which data needs to recover by Method of Data Organization, thus need not identify each data block, therefore rebuild speed block.
3, data are respectively with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype.Such as, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can be avoided producing write bottleneck on standby memory storage on duty for the day.
4, that store in the main storage means that standby memory storage on duty for the day is corresponding, inconsistent with described standby memory storage on duty for the day data, the idle bandwidth and the free space that utilize memory storage on backstage, be synchronized in standby memory storage on duty for the day, when not consuming additional energy, achieve the conversion of Method of Data Organization.
5, data are with certain Method of Data Organization write main storage means and standby memory storage, realize at data structure layer, realize comparatively flexible, both can be implemented on block device, and also can be implemented on node, such as, can be used for disk array, also can be used for node array, and, the many disk environment of single node can be implemented in, also can be implemented in multinode distributed storage environment.
The embodiment of the present invention also provides a kind of computer-readable storage medium, and this computer-readable storage medium can have program stored therein, and comprises the part or all of step of the date storage method recorded in said method embodiment when this program performs.
Please refer to Fig. 7, the embodiment of the present invention also provides a kind of computer equipment 700, can comprise:
Processor 710, storer 720, communication interface 730, bus 740; Described processor 710, storer 720, communication interface 730 is by the mutual communication of described bus 740; Described communication interface 730, for receiving and sending data; Described storer 720 is for storage program; Described processor 710 is for performing the described program in described storer.Described storer 720, can comprise main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system.Said active and standby memory storage can be disk.
Wherein, processor 710 is for writing the multiple main storage means in main storage subsystem with the first Method of Data Organization by first of data to be stored copy, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage; By that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization; Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
In some embodiments of the invention, whether processor 710 also reaches preset value for the occupancy of the log space on duty for the day judging standby memory storage on duty for the day; If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
In some embodiments of the invention, the first Method of Data Organization described is log-structured merging LSM mode; Described the second Method of Data Organization is log-structured file system LFS mode; In described 3rd, Method of Data Organization is that B+ sets mode.
In some embodiments of the invention, processor 710 also for creating key assignments storage system LevelDB in each main storage means; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
In some embodiments of the invention, described multiple part, specifically for first of data to be stored copy is divided into multiple part, writes in the LevelDB of multiple main storage means by processor 710 respectively; By in the LFS of second of data to be stored described standby memory storage on duty for the day of copy write; By described on duty for the day in the BDB of memory storage for the inconsistent data write read.
Be appreciated that the function of each functional module of the computer equipment of the embodiment of the present invention can according to the method specific implementation in said method embodiment, its specific implementation process can refer to the associated description in said method embodiment, repeats no more herein.
Therefore, at feasible embodiments more of the present invention, achieve following technique effect:
1, owing to only having standby memory storage on duty for the day in running order, other is all in idle low energy consumption state for memory storage, therefore, can reduce energy consumption.
2, each main storage means and each standby memory storage create have file system, data store with certain Method of Data Organization in active and standby memory storage, can reflect data structure layer feature, and improve reconstruction speed.If one piece of main storage means lost efficacy, when recovering data from the standby memory storage of correspondence, directly can know which data needs to recover by Method of Data Organization, thus need not identify each data block, therefore rebuild speed block.
3, data are respectively with different Method of Data Organizations write main storage means and standby memory storage, can meet multiple application, multiple loadtype.Such as, the first and the second Method of Data Organization write optimization, can improve the writing speed of data; The third Method of Data Organization is read to optimize, and the data on final standby memory storage is mostly existed with the third organizational form, can have higher reading performance, to meet the demand of data analysis phase.Further, the writing speed of the second Method of Data Organization, faster than the writing speed of the first Method of Data Organization, can be avoided producing write bottleneck on standby memory storage on duty for the day.
4, that store in the main storage means that standby memory storage on duty for the day is corresponding, inconsistent with described standby memory storage on duty for the day data, the idle bandwidth and the free space that utilize memory storage on backstage, be synchronized in standby memory storage on duty for the day, when not consuming additional energy, achieve the conversion of Method of Data Organization.
5, data are with certain Method of Data Organization write main storage means and standby memory storage, realize at data structure layer, realize comparatively flexible, both can be implemented on block device, and also can be implemented on node, such as, can be used for disk array, also can be used for node array, and, the many disk environment of single node can be implemented in, also can be implemented in multinode distributed storage environment.
In the above-described embodiments, the description of each embodiment is all emphasized particularly on different fields, in certain embodiment, there is no the part described in detail, can see the associated description of other embodiment.
It should be noted that, for aforesaid each embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other order or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the present invention is necessary.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment is that the hardware that can carry out instruction relevant by program has come, this program can be stored in a computer-readable recording medium, and storage medium can comprise: ROM, RAM, disk or CD etc.
A kind of date storage method provided the embodiment of the present invention above and equipment are described in detail, apply specific case herein to set forth principle of the present invention and embodiment, the explanation of above embodiment just understands method of the present invention and core concept thereof for helping; Meanwhile, for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (11)
1. a date storage method, is characterized in that, for storage system, described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system;
Described method comprises:
By first of data to be stored copy with the multiple main storage means in the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage;
By that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization;
Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
2. method according to claim 1, is characterized in that, also comprises:
Judge whether the occupancy of the log space on duty for the day of standby memory storage on duty for the day reaches preset value;
If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
3. method according to claim 1, is characterized in that:
The first Method of Data Organization described is log-structured merging LSM mode;
Described the second Method of Data Organization is log-structured file system LFS mode;
The third Method of Data Organization described is that B+ sets mode.
4. method according to claim 3, is characterized in that, described by first of data to be stored copy with the first Method of Data Organization write main storage subsystem before, also comprise:
Each main storage means creates key assignments storage system LevelDB;
In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
5. method according to claim 4, is characterized in that:
Described by first of data to be stored copy with the first Method of Data Organization write main storage subsystem in multiple main storage means comprise: first of data to be stored copy is divided into multiple part, described multiple part is write in the LevelDB of multiple main storage means respectively;
Described write with the second Method of Data Organization by second of data to be stored copy comprises for the on duty for the day standby memory storage in storage subsystem: by the LFS of second of data to be stored described standby memory storage on duty for the day of copy write;
Described the inconsistent data of reading to be comprised with the described standby memory storage on duty for the day of the third Method of Data Organization write: by the BDB of the described standby memory storage on duty for the day of inconsistent data write that reads.
6., according to described method arbitrary in claim 1 to 5, it is characterized in that:
Described storage system is disk array or node array.
7. a data storage device, is characterized in that, for storage system, described storage system comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system; Described equipment comprises:
First module for reading and writing, for writing the multiple main storage means in main storage subsystem with the first Method of Data Organization by first of data to be stored copy;
Second module for reading and writing, for second of data to be stored copy to be write the on duty for the day standby memory storage in standby storage subsystem with the second Method of Data Organization, described standby memory storage on duty for the day is uniquely in running order standby memory storage;
Third reading writing module, for by that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and by the inconsistent data that reads with the described standby memory storage on duty for the day of the third Method of Data Organization write;
Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
8. equipment according to claim 7, is characterized in that, also comprises:
Memory storage monitoring module, whether the occupancy for the log space on duty for the day judging standby memory storage on duty for the day reaches preset value; If occupancy reaches preset value, then described standby memory storage on duty for the day is switched to dormant state, and another is switched to duty as standby memory storage on duty for the day for memory storage.
9. equipment according to claim 7, is characterized in that, also comprises:
Creation module, for creating key assignments storage system LevelDB in each main storage means; In each standby memory storage, mark off two storage areas, wherein, create LFS at first storage area, second storage area creates key assignments storage system BDB.
10. equipment according to claim 9, is characterized in that:
Described multiple part, specifically for first of data to be stored copy is divided into multiple part, writes in the LevelDB of multiple main storage means by described first module for reading and writing respectively;
Described second module for reading and writing is specifically for writing in the LFS of described standby memory storage on duty for the day by second of data to be stored copy;
Described third reading writing module is described on duty for the day in the BDB of memory storage specifically for the inconsistent data write that will read.
11. 1 kinds of computer equipments, is characterized in that, comprising:
Processor, storer, communication interface, bus; Described processor, storer, communication interface is by the mutual communication of described bus; Described storer, comprises main storage subsystem and standby storage subsystem; Described main storage subsystem comprises N number of main storage means, and described standby storage subsystem comprises the N number of standby memory storage corresponding to described N number of main storage means; N be greater than 1 positive integer; Wherein each main storage means and each standby memory storage create and have file system;
Wherein, described processor is used for first of data to be stored copy with the multiple main storage means in the first Method of Data Organization write main storage subsystem, by second of data to be stored copy with the on duty for the day standby memory storage in the standby storage subsystem of the second Method of Data Organization write, described standby memory storage on duty for the day is uniquely in running order standby memory storage; By that store in main storage means corresponding for described standby memory storage on duty for the day, inconsistent with described standby memory storage on duty for the day data reading, and the inconsistent data read is write described standby memory storage on duty for the day with the third Method of Data Organization; Wherein, the first Method of Data Organization and the second Method of Data Organization write optimization, and the writing speed of the second Method of Data Organization is faster than the writing speed of the first Method of Data Organization, and the third Method of Data Organization is then read to optimize.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410182608.0A CN105094761B (en) | 2014-04-30 | 2014-04-30 | A kind of date storage method and equipment |
| PCT/CN2015/077214 WO2015165351A1 (en) | 2014-04-30 | 2015-04-22 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410182608.0A CN105094761B (en) | 2014-04-30 | 2014-04-30 | A kind of date storage method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105094761A true CN105094761A (en) | 2015-11-25 |
| CN105094761B CN105094761B (en) | 2018-06-15 |
Family
ID=54358160
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410182608.0A Active CN105094761B (en) | 2014-04-30 | 2014-04-30 | A kind of date storage method and equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105094761B (en) |
| WO (1) | WO2015165351A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111198783A (en) * | 2018-11-16 | 2020-05-26 | 阿里巴巴集团控股有限公司 | Data access method, device, system, equipment and storage medium |
| CN111309267A (en) * | 2020-02-26 | 2020-06-19 | Oppo广东移动通信有限公司 | Storage space allocation method and device, storage equipment and storage medium |
| CN113704261A (en) * | 2021-08-26 | 2021-11-26 | 平凯星辰(北京)科技有限公司 | Key value storage system based on cloud storage |
| CN113704261B (en) * | 2021-08-26 | 2024-05-24 | 平凯星辰(北京)科技有限公司 | Key value storage system based on cloud storage |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090164718A1 (en) * | 2007-12-21 | 2009-06-25 | Fujitsu Limited | Disk array device control method |
| CN101840315A (en) * | 2010-06-17 | 2010-09-22 | 华中科技大学 | Data organization method of disk array |
| CN103645859A (en) * | 2013-11-19 | 2014-03-19 | 华中科技大学 | Disk array caching method for virtual SSD and SSD isomerous mirror image |
-
2014
- 2014-04-30 CN CN201410182608.0A patent/CN105094761B/en active Active
-
2015
- 2015-04-22 WO PCT/CN2015/077214 patent/WO2015165351A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090164718A1 (en) * | 2007-12-21 | 2009-06-25 | Fujitsu Limited | Disk array device control method |
| CN101840315A (en) * | 2010-06-17 | 2010-09-22 | 华中科技大学 | Data organization method of disk array |
| CN103645859A (en) * | 2013-11-19 | 2014-03-19 | 华中科技大学 | Disk array caching method for virtual SSD and SSD isomerous mirror image |
Non-Patent Citations (1)
| Title |
|---|
| YINLIANG YUE等: "RoLo: A Rotated Logging Storage Architecture for Enterprise Data Centers", 《IN PROCEEDINGS OF THE 30TH INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111198783A (en) * | 2018-11-16 | 2020-05-26 | 阿里巴巴集团控股有限公司 | Data access method, device, system, equipment and storage medium |
| CN111198783B (en) * | 2018-11-16 | 2023-04-07 | 阿里巴巴集团控股有限公司 | Data access method, device, system, equipment and storage medium |
| CN111309267A (en) * | 2020-02-26 | 2020-06-19 | Oppo广东移动通信有限公司 | Storage space allocation method and device, storage equipment and storage medium |
| CN111309267B (en) * | 2020-02-26 | 2023-10-03 | Oppo广东移动通信有限公司 | Storage space allocation method and device, storage equipment and storage medium |
| CN113704261A (en) * | 2021-08-26 | 2021-11-26 | 平凯星辰(北京)科技有限公司 | Key value storage system based on cloud storage |
| CN113704261B (en) * | 2021-08-26 | 2024-05-24 | 平凯星辰(北京)科技有限公司 | Key value storage system based on cloud storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105094761B (en) | 2018-06-15 |
| WO2015165351A1 (en) | 2015-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103503414B (en) | A kind of group system calculating storage and merge | |
| CN102467408B (en) | Method and device for accessing data of virtual machine | |
| CN102521072B (en) | Virtual tape library equipment and data recovery method | |
| WO2018098972A1 (en) | Log recovery method, storage device and storage node | |
| CN102682012A (en) | Method and device for reading and writing data in file system | |
| CN103049220B (en) | Storage controlling method, memory control device and solid-state memory system | |
| CN103823728B (en) | A kind of method of raid-array Intelligent Reconstruction | |
| CN103793291B (en) | Distributed data copying method and device | |
| CN102867035B (en) | A kind of distributed file system cluster high availability method and device | |
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| CN103516549B (en) | A kind of file system metadata log mechanism based on shared object storage | |
| CN102982182B (en) | Data storage planning method and device | |
| CN103617006A (en) | Storage resource management method and device | |
| CN104040512A (en) | Method and device for processing storage space and non-volatile computer readable storage medium | |
| CN102521114A (en) | File system log storage system under virtualization environment | |
| CN104424052A (en) | Automatic redundant distributed storage system and method | |
| CN105224256A (en) | A kind of storage system | |
| CN107632780A (en) | A kind of roll of strip implementation method and its storage architecture based on distributed memory system | |
| CN105657057A (en) | Calculation and storage fused cluster system | |
| CN107301021B (en) | Method and device for accelerating LUN (logical Unit number) by utilizing SSD (solid State disk) cache | |
| CN102725746B (en) | Method and apparatus for managing memories in distributed computer system | |
| US9015433B2 (en) | Resiliency with a destination volume in a replication environment | |
| CN101794246B (en) | Backup storage system, backup system, data backup method and recovery method | |
| CN105094761A (en) | Data storage method and device | |
| US11275518B2 (en) | System and method for implementing heterogeneous media types with raid |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |