CN105094761B

CN105094761B - A kind of date storage method and equipment

Info

Publication number: CN105094761B
Application number: CN201410182608.0A
Authority: CN
Inventors: 岳银亮; 熊劲
Original assignee: Huawei Technologies Co Ltd; Institute of Computing Technology of CAS
Current assignee: Huawei Technologies Co Ltd; Institute of Computing Technology of CAS
Priority date: 2014-04-30
Filing date: 2014-04-30
Publication date: 2018-06-15
Anticipated expiration: 2034-04-30
Also published as: CN105094761A; WO2015165351A1

Abstract

The invention discloses a kind of date storage method and equipment, to solve the technical issues of storage system in a manner of existing data copy cannot take into account readwrite performance to a certain extent.In some feasible embodiments of the invention, method includes：By first copy of data to be stored with multiple main storage means in the first Method of Data Organization write-in main storage subsystem, by second copy of data to be stored with the standby storage device on duty for the day in second of standby storage subsystem of Method of Data Organization write-in；Standby storage device on duty for the day is written with the third Method of Data Organization by data reading stored in the corresponding main storage means of standby storage device on duty for the day, inconsistent with standby storage device on duty for the day, and by the inconsistent data of reading；The first Method of Data Organization and second of Method of Data Organization write optimization, and the writing speed of second of Method of Data Organization is faster than the writing speed of the first Method of Data Organization, the third Method of Data Organization is then to read optimization.

Description

A kind of date storage method and equipment

Technical field

The present invention relates to computers and technical field of memory, and in particular to a kind of date storage method and equipment.

Background technology

The progress of digital technology and the development of memory technology have expedited the emergence of the data of magnanimity, and data are needed with certain tissue shape Formula stores within the storage system.Data have different inputs in the different stages from generating into the whole life cycle of extinction It exports (input/output, I/O) and accesses feature；For example, generating the stage in data, data, which need to store with high speed writein, is System；In data analysis phase, data are needed with high speed readout or scanning so as to participate in calculating.

In order to improve memory reliability, in a manner of copy storing data increasingly gets the nod.Copy is data redundancy side The future trend of formula.The typical scene of data copy is disk array (Redundant Arrays of Independent Disks, RAID, full name redundant array of independent disks, abbreviation disk array).RAID10 is a kind of common disk array, including One copy of data is divided into multiple portions, is respectively stored into multiple masters by one group of master disk and one group of mirrored disk, RAID10 On disk, to improve readwrite performance；Also, another copy of data is stored onto corresponding mirrored disk, it can with raising By property.

In the cost of storage system, the ratio shared by the disposable input cost such as software and hardware is limited, and energy consumption cost then by Gradually become the major part in storage system totle drilling cost.One kind in the prior art, magnetic is reduced using rotation daily record framework RoLo The energy consumption of disk array RAID10.The daily record that the free space of multiple mirrored disks is integrated into a logic by rotation daily record framework is empty Between resource pool.Distributed synchronization is done by excavated space timeslice, which can cyclically be utilized To improve the performance of system and efficiency.In other words, multiple mirrored disks are used as daily record disk on duty for the day, while non-value is everyday in turn Will disk is switched to low energy consumption state, so as to reduce energy consumption.

The disk array storage system of currently used rotation daily record framework mode has following defect：Existing disk battle array Multiple physical disks are virtually turned to a virtual disk by row storage system, and file system is established on virtual disk, that is, single There is no file system on a disk, thus data can only be written into all disks using same Method of Data Organization, Or this Method of Data Organization writes optimization, to improve write performance or be to read optimization, to improve reading performance, and cannot Take into account readwrite performance simultaneously.

Invention content

The embodiment of the present invention provides a kind of date storage method and equipment, to solve existing data pair to a certain extent The storage system of the method cannot take into account the technical issues of readwrite performance.

First aspect present invention provides a kind of date storage method, and for storage system, the storage system includes main memory Store up subsystem and standby storage subsystem；The main storage subsystem includes N number of main storage means, and the standby storage subsystem includes Corresponding to N number of standby storage device of N number of main storage means；N is the positive integer more than 1；Wherein each main storage means and Being created on each standby storage device has file system；The method includes：By first copy of data to be stored with first Multiple main storage means in kind of Method of Data Organization write-in main storage subsystem, by second copy of data to be stored with the Standby storage device on duty for the day in two kinds of standby storage subsystems of Method of Data Organization write-in, the standby storage device on duty for the day is uniquely to locate In the standby storage device of working condition；By stored in the corresponding main storage means of the standby storage device on duty for the day and described value Day reads for the inconsistent data of storage device, and by the inconsistent data of reading with described in the write-in of the third Method of Data Organization Standby storage device on duty for the day；Wherein, the first Method of Data Organization and second of Method of Data Organization write optimization, and second The writing speed of Method of Data Organization is faster than the writing speed of the first Method of Data Organization, the third Method of Data Organization is then Read optimization.

In the first possible implementation, the method further includes：Judge the daily record on duty for the day of standby storage device on duty for the day Whether the occupancy in space reaches preset value；If occupancy reaches preset value, the standby storage device on duty for the day is switched to not Dormancy state, and another is switched to working condition as standby storage device on duty for the day for storage device.

With reference to first aspect or the first possible realization method of first aspect, in second of possible realization method In, the first described Method of Data Organization is log-structured merging LSM modes；Second of Method of Data Organization is daily record knot Structure file system LFS modes；The third described Method of Data Organization is B+ tree modes.

With reference to first aspect or second of possible realization method of first aspect, in the third possible realization method In, before main storage subsystem is written with the first Method of Data Organization in first copy by data to be stored, also wrap It includes：Key assignments storage system LevelDB is created in each main storage means；Two storages are marked off in each standby storage device Region, wherein, LFS is created in first storage region, second storage region creates key assignments storage system BDB.

With reference to first aspect or the third possible realization method of first aspect, in the 4th kind of possible realization method In, multiple masters in main storage subsystem are written with the first Method of Data Organization for first copy by data to be stored Storage device includes：First copy of data to be stored is divided into multiple portions, the multiple part is respectively written into more In the LevelDB of a main storage means；It is described that second copy of data to be stored is written with second of Method of Data Organization Standby storage device on duty for the day in standby storage subsystem includes：By second copy write-in of data to be stored standby storage on duty for the day In the LFS of device；The standby storage device on duty for the day is written with the third Method of Data Organization in the inconsistent data by reading Including：The inconsistent data of reading is written in the BDB of the standby storage device on duty for the day.

With reference to first aspect or first aspect the first to any one of the 4th kind of mode realization method, the 5th In the possible realization method of kind, the storage system is disk array or node array.

Second aspect of the present invention provides a kind of data storage device, and for storage system, the storage system includes main memory Store up subsystem and standby storage subsystem；The main storage subsystem includes N number of main storage means, and the standby storage subsystem includes Corresponding to N number of standby storage device of N number of main storage means；N is the positive integer more than 1；Wherein each main storage means and Being created on each standby storage device has file system；The equipment includes：First module for reading and writing, for by data to be stored Multiple main storage means in main storage subsystem are written with the first Method of Data Organization for first copy；Second read-write mould Block, it is on duty for the day standby in standby storage subsystem for second copy of data to be stored to be written with second of Method of Data Organization Storage device, the standby storage device on duty for the day are unique in running order standby storage devices；Third module for reading and writing, for inciting somebody to action Data reading being stored in the corresponding main storage means of the standby storage device on duty for the day, inconsistent with the standby storage device on duty for the day Go out, and the standby storage device on duty for the day is written into the third Method of Data Organization in the inconsistent data of reading；Wherein, the first Method of Data Organization and second of Method of Data Organization write optimization, and the writing speed of second of Method of Data Organization is faster than The writing speed of the first Method of Data Organization, the third Method of Data Organization are then to read optimization.

In the first possible implementation, the equipment further includes：Storage device monitoring module, it is on duty for the day for judging Whether the occupancy of the log space on duty for the day of standby storage device reaches preset value；If occupancy reaches preset value, by described value Standby storage device was switched to dormant state, and another is switched to working condition as standby storage dress on duty for the day for storage device day It puts.

With reference to first aspect or the first possible realization method of first aspect, in second of possible realization method In, the equipment further includes：Creation module, for creating key assignments storage system LevelDB in each main storage means；Every Two storage regions are marked off in a standby storage device, wherein, create LFS, second storage region in first storage region Create key assignments storage system BDB.

With reference to first aspect or second of possible realization method of first aspect, in the third possible realization method In, first module for reading and writing is specifically used for first copy of data to be stored being divided into multiple portions, will be the multiple Part is respectively written into the LevelDB of multiple main storage means；Second module for reading and writing is specifically used for data to be stored Second copy is written in the LFS of the standby storage device on duty for the day；What the third module for reading and writing was specifically used for read differs Data is caused to be written in the BDB of the standby storage device on duty for the day.

Third aspect present invention provides a kind of computer equipment, it may include：Processor, memory, communication interface, bus； The processor, memory, communication interface pass through the mutual communication of the bus；The memory, including main storage subsystem With standby storage subsystem；The main storage subsystem includes N number of main storage means, and the standby storage subsystem includes corresponding to institute State N number of standby storage device of N number of main storage means；N is the positive integer more than 1；Wherein each main storage means are deposited with each standby All being created on storage device has file system；Wherein, the processor is used for first copy of data to be stored with the first Multiple main storage means in Method of Data Organization write-in main storage subsystem, by second copy of data to be stored with second Standby storage device on duty for the day in the standby storage subsystem of kind Method of Data Organization write-in, the standby storage device on duty for the day is uniquely to be in The standby storage device of working condition；By it is being stored in the corresponding main storage means of the standby storage device on duty for the day, with it is described on duty for the day The inconsistent data of standby storage device are read, and the inconsistent data of reading is written described value with the third Method of Data Organization Day is for storage device；Wherein, the first Method of Data Organization and second of Method of Data Organization write optimization, and second of number It is faster than the writing speed of the first Method of Data Organization according to the writing speed of organizational form, the third Method of Data Organization is then to read Optimization.

Therefore the embodiment of the present invention uses the establishment file in each main storage means and each standby storage device System and, by first copy of data to be stored with the first Method of Data Organization be written main storage subsystem, will wait to deposit Second copy of data is stored up with the standby storage device on duty for the day in second of standby storage subsystem of Method of Data Organization write-in, by institute State data reading being stored in the corresponding main storage means of standby storage device on duty for the day, inconsistent with the standby storage device on duty for the day Go out, and the inconsistent data of reading be written to the technical solution of the standby storage device on duty for the day with the third Method of Data Organization, Achieve following technique effect：

Being created in each main storage means and each standby storage device has file system, and data are respectively with different data Main storage means and standby storage device are written in organizational form, disclosure satisfy that a variety of applications, a variety of loadtypes；Wherein, the first Optimization is write with second of Method of Data Organization, the writing speed of data can be improved；The third Method of Data Organization is to read Optimization so that the data on final standby storage device mostly exist with the third organizational form, can have higher readability Can, to meet the needs of data analysis phase；So as to which entire storage system be made to take into account readwrite performance.Also, second of data group The writing speed for knitting mode is faster than the writing speed of the first Method of Data Organization, and standby storage device on duty for the day can be avoided to generate write-in Bottleneck.

Description of the drawings

Technical solution in order to illustrate the embodiments of the present invention more clearly, below will be to institute in embodiment and description of the prior art Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the present invention Example, for those of ordinary skill in the art, without creative efforts, can also obtain according to these attached drawings Obtain other attached drawings.

Fig. 1 is a kind of schematic diagram of date storage method provided by the invention；

Fig. 2 is the schematic diagram of another date storage method provided by the invention；

Fig. 3 is the schematic diagram of disk array RAID10 in a scene embodiment of the invention；

Fig. 4 is data storage operations schematic diagram of the present invention method by taking RAID10 as an example；

Fig. 5 is a kind of schematic diagram of data storage device provided by the invention；

Fig. 6 is the schematic diagram of another data storage device provided by the invention；

Fig. 7 is a kind of schematic diagram of computer equipment provided by the invention.

Specific embodiment

The disk array storage system of currently used rotation daily record framework mode has following defect：

First, in existing disk array storage system, there is a virtual level on disk, be one by multiple magnetic disc virtualizations A virtual disk, file system are established on virtual disk, that is, do not have file system, each disk on single disk It is block device, entire disk array is realized in block grade.This is resulted in：The data of write-in disk cannot reflect data structure layer Feature, the reconstruction speed that can also further result in block grade disk array is slow.If one piece of disk failure, from corresponding mirror image magnetic When disk restores data, need to restore, therefore speed is slower each data block on mirrored disk.

2nd, in existing storage system, file system is established on virtual disk, using same Method of Data Organization to Data are written in all disks, or this Method of Data Organization writes optimization, to improve write performance or be that reading is excellent Change, to improve reading performance, and readwrite performance cannot be taken into account simultaneously.Also, in the disk array for rotating framework log mode, data A copy be divided into multiple portions multiple master disks be written, writing speed is very fast；But another copy of data is only write Enter a daily record disk on duty for the day, writing speed is slower, does not catch up with the writing speed of master disk, can form bottleneck, influences entire disk The performance of array.

The embodiment of the present invention provides a kind of date storage method and equipment, to solve the storage in a manner of existing data copy It is slow that system rebuilds speed caused by being realized in block grade, it is impossible to the problems such as reflecting data structure layer feature and because using same A kind of Method of Data Organization cannot take into account readwrite performance caused by data are written into all disks, can go out in daily record disk on duty for the day The problems such as bottleneck is now written.

In order to which those skilled in the art is made to more fully understand the present invention program, below in conjunction in the embodiment of the present invention The technical solution in the embodiment of the present invention is clearly and completely described in attached drawing, it is clear that described embodiment is only The embodiment of a part of the invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people Member's all other embodiments obtained without making creative work should all belong to the model that the present invention protects It encloses.

Below by specific embodiment, it is described in detail respectively.

It please refers to Fig.1, the embodiment of the present invention provides a kind of date storage method.

This method is applied to storage system, and the storage system includes main storage subsystem and standby storage subsystem；It is described Main storage subsystem includes N number of main storage means, and the standby storage subsystem includes the N corresponding to N number of main storage means A standby storage device；N is the positive integer more than 1.In the present embodiment, described storage system can be disk array or node Array, described active and standby storage device can be disk or node etc..In the present embodiment, each main storage means and each All being created on standby storage device has file system；The file system created on different storage devices can be identical, can not also Together.

As shown in Figure 1, present invention method may include：

110th, first copy of data to be stored is written with the first Method of Data Organization in main storage subsystem Multiple main storage means, by second copy of data to be stored in second of standby storage subsystem of Method of Data Organization write-in Standby storage device on duty for the day, the standby storage device on duty for the day is unique in running order standby storage device；

120th, by stored in the corresponding main storage means of the standby storage device on duty for the day and standby storage device on duty for the day Inconsistent data are read, and by the inconsistent data of reading with the third Method of Data Organization write-in standby storage dress on duty for the day It puts.

In the present embodiment, by every portion two copies of data copy to be stored, wherein primary storage is written in first copy Subsystem, second standby storage subsystem of copy write-in.It, can be by main storage subsystem in order to improve the readwrite performance of data N number of main storage means are held at working condition, and first copy is divided into N number of part, each one primary storage of partial write Device.It, can will be in N number of standby storage device of standby storage subsystem at any time in section in order to improve the reliability of data One be maintained at working condition, as standby storage device on duty for the day, the standby storage dress on duty for the day is written into second copy of data It puts；Other standby storage devices are then in suspend mode or the low energy consumption states such as standby, to reduce energy consumption.

In the embodiment of the present invention, in order to improve writing speed, the first above-mentioned Method of Data Organization and second of data group The mode of knitting writes optimization；And in order to improve reading performance of the data in the analysis phase, the third Method of Data Organization is then to read Optimization；So as to which entire storage system be made to take into account readwrite performance.Also, in order to avoid forming write-in on standby storage device on duty for the day Bottleneck can make the writing speed of second of Method of Data Organization be faster than the writing speed of the first Method of Data Organization；Then, though First copy of right data to be stored is to be written simultaneously multiple main storage means using the first Method of Data Organization, can be carried Standby storage device on duty for the day is written using data organization side in than first in high writing speed, still, second copy of data Faster second of the Method of Data Organization of formula speed, in this way, balance can be thus obtained, it is as possible that two copies of data are same When or close to simultaneously active and standby storage subsystem is written, to improve the write performance of entire storage system.In addition, the present invention is implemented Being established in example, on each storage device has file system, and the data of the storage device of write-in exist with certain organizational form, It can reflect the feature of data structure layer.

To sum up, an embodiment of the present invention provides a kind of date storage method, this method uses above-mentioned technical characteristic, achieves Following technique effect：Being created in each main storage means and each standby storage device has file system, and data are respectively with difference Method of Data Organization write-in main storage means and standby storage device, disclosure satisfy that a variety of applications, a variety of loadtypes；Wherein, The first and second of Method of Data Organization write optimization, can improve the writing speed of data；The third data organization side Formula is to read optimization so that the data on final standby storage device mostly exist with the third organizational form, can have higher Performance is read, to meet the needs of data analysis phase；So as to which entire storage system be made to take into account readwrite performance.Also, second The writing speed of Method of Data Organization is faster than the writing speed of the first Method of Data Organization, can avoid standby storage device production on duty for the day Raw write-in bottleneck.

In some embodiments of the invention, the method further includes：Judge the log space on duty for the day of standby storage device on duty for the day Whether occupancy reaches preset value.

In the embodiment of the present invention, each standby storage device is the mirror storage device of corresponding main storage means, will be all Idle storage space on standby storage device is considered as log space, and the log space that standby storage device on duty for the day is provided is known as value Day log space.In the present embodiment, the service condition of the log space on duty for the day of standby storage device on duty for the day can be monitored in real time, judged Whether the occupancy of log space on duty for the day reaches preset value.

If receive data to be stored, judgment value day the occupancy of log space be not up to preset value, then normally hold Two copies of data are respectively written into main storage subsystem and standby storage device on duty for the day by row step 110 respectively.

It, will the standby storage dress on duty for the day in the present embodiment if the occupancy of judgment value day log space reaches preset value It puts and is switched to dormant state, and another is switched to working condition as standby storage device on duty for the day for storage device.Also, it cuts After daily record on duty for the day is changed for storage device, a synchronized process is can trigger, which is used for corresponding main storage means In data be synchronized in log storage on duty for the day.That is, in some embodiments, above-mentioned steps 120 can be on duty for the day standby in switching Be triggered execution during storage device, and standby storage being stored in the corresponding main storage means of standby storage device on duty for the day and on duty for the day is filled It puts inconsistent data to read, and by the inconsistent data of reading with the third Method of Data Organization write-in standby storage dress on duty for the day It puts.

The simultaneously operating can perform on the backstage of computer equipment, empty using the idle bandwidth of storage device and free time Between, the inconsistent data of reading is synchronized in standby storage device on duty for the day, so as to not consume additional energy.The embodiment of the present invention In, in order to improve write performance, above-mentioned the first used during data of writing can select to have with second of Method of Data Organization The Method of Data Organization of higher write performance；In order to improve the performance for reading data, the third described Method of Data Organization can be with Method of Data Organization of the selection with higher reading performance, so that follow-up standby storage device can provide higher reading performance, moreover, Since described simultaneously operating is carried out on backstage, the write performance of the third Method of Data Organization can not be considered.

In some embodiments of the invention, the first described Method of Data Organization can select sequential log structure, such as Log-structured merging (Log Structured Merge, LSM) mode；Second of Method of Data Organization can be selected without suitable Sequence is log-structured, such as log-structured file system (Log Structured File System, LFS) mode；The third Kind Method of Data Organization can select local update index structure, such as B+ tree modes.Wherein, LSM and LFS can be provided preferably Write performance, B+ trees can provide preferable reading performance.And LFS writing rates are faster than the writing rate of LSM.

In order to realize above-mentioned a variety of Method of Data Organization, in the embodiment of the present invention, the initialization before step 110 Each main storage means and each standby storage device can be formatted as arbitrary file system respectively by Cheng Zhong；Also, every LSM systems are created in a main storage means, such as can be specifically key assignments storage system LevelDB；In each standby storage device In mark off two storage regions, wherein, create LFS in first storage region, second storage region creates B+ tree systems, Such as specifically can be key assignments storage system BDB.

Then in above-mentioned steps, main memory is written with the first Method of Data Organization in first copy by data to be stored Storage subsystem may include：First copy of data to be stored is divided into multiple portions, the multiple part is respectively written into In the LevelDB of multiple main storage means of the main storage subsystem.Second copy by data to be stored is with Standby storage device on duty for the day in two kinds of standby storage subsystems of Method of Data Organization write-in may include：By second of data to be stored Copy is written in the LFS of the standby storage device on duty for the day.The inconsistent data by reading is with the third Method of Data Organization The standby storage device on duty for the day, which is written, may include：The inconsistent data of reading is written in the BDB of the standby storage device on duty for the day.

In the embodiment of the present invention, in two storage regions being divided into each standby storage device, first memory block Domain creates LFS, which is used to use when data are written；Second storage region creates BDB, which is used for It is used when data synchronize.Since multiple standby storage devices are used as standby storage device on duty for the day, during rotation, first in turn The copy data being written in a manner of LFS in storage region is only ephemeral data, is ceaselessly written into, and is ceaselessly released, because This first storage region does not need to be too big；Second storage region is used for preserving the data synchronized from main storage means, It is the data for needing to preserve for a long time, is ceaselessly written into, but be not released generally, therefore second storage region needs is larger Memory space can occupy most of space of standby storage device.

Finally, the data in all main storage means are stored with the first Method of Data Organization, not only reflect data Structure sheaf feature, and with higher write performance；The data copy preserved for a long time needed on standby storage device be all then with The third Method of Data Organization stores, and can also reflect data structure layer feature, moreover, providing higher reading performance.So as to, When subsequent applications need to read data, can digital independent be carried out from standby storage device with higher reading speed.

Present invention method is preferably applied to the storage system of facing cloud storage environment.

More than, the embodiment of the invention discloses a kind of date storage method, this method use in each main storage means and Created on each standby storage device have file system and, by first copy of data to be stored with the first data group Main storage subsystem is written in the mode of knitting, by second copy of data to be stored with second of standby storage of Method of Data Organization write-in Standby storage device on duty for the day in subsystem, by it is being stored in the corresponding main storage means of the standby storage device on duty for the day, with it is described The inconsistent data of standby storage device on duty for the day are read, and the inconsistent data of reading is written institute with the third Method of Data Organization The technical solution of standby storage device on duty for the day is stated, achieves following technique effect：

1st, due to only having standby storage device on duty for the day in running order, other standby storage devices are in idle low energy Therefore consumption state, can reduce energy consumption.

2nd, being created on each main storage means and each standby storage device has file system, and data are in active and standby storage dress It is stored in putting with certain Method of Data Organization, can reflect data structure layer feature, and improve reconstruction speed.If one piece Main storage means fail, and when restoring data from corresponding standby storage device, which can directly know by Method of Data Organization Data needs restore, and from without each data block is identified, therefore rebuild speed block.

3rd, data disclosure satisfy that a variety of respectively with different Method of Data Organization write-in main storage means and standby storage device Using, a variety of loadtypes.For example, the first and second of Method of Data Organization write optimization, writing for data can be improved Enter speed；The third Method of Data Organization is to read optimization so that the data on final standby storage device are mostly with the third group The mode of knitting exists, and can have higher reading performance, to meet the needs of data analysis phase.Also, second of data organization The writing speed of mode is faster than the writing speed of the first Method of Data Organization, can avoid generating on standby storage device on duty for the day and write Enter bottleneck.

4th, it is being stored in the corresponding main storage means of standby storage device on duty for the day, inconsistent with the standby storage device on duty for the day Data are to utilize the idle bandwidth and free space of storage device on backstage, are synchronized in standby storage device on duty for the day, are not consuming In the case of additional energy, the conversion of Method of Data Organization is realized.

5th, data are in data structure layer reality with certain Method of Data Organization write-in main storage means and standby storage device Existing, realization is more flexible, can both realize on block device, and can also realize on node, for example, available for disk battle array Row, it can also be used to node array, and, it can be achieved that in single node multiple disks environment, multinode distribution can also be implemented in and deposited Store up environment.

For ease of being better understood from technical solution provided in an embodiment of the present invention, below by the reality under a concrete scene It applies and is introduced for mode.

In this scene embodiment, by taking described storage system is disk array RAID10 as an example.RAID10 includes one group of master Disk and one group are for disk (or referred to as mirrored disk).One group of N number of master disk forms main storage subsystem (or referred to as main magnetic Disk group), one group of N number of mirrored disk forms standby storage subsystem (or being known as standby disk group, mirrored disk group).

In the present embodiment, all mirrored disks are accordingly to be regarded as journal disk, the free time on all mirrored disks is stored Space is considered as the available log space of journal disk；A journal disk is only maintained at active state in section at any time (i.e. working condition) responds write operation requests；The mirrored disk for being maintained at active state is known as journal disk on duty for the day, daily record on duty for the day The log space that disk is provided is known as log space on duty for the day.

As shown in Fig. 2, this scene embodiment includes the following steps：

210th, key assignments request step is received：The key assignments request for carrying out self-application is received, number to be stored is carried in key assignments request According to (i.e. key assignments data), key assignments request can be specifically write-in (PUT), read (GET) or delete (DELETE) etc..

220th, key assignments data redirection step：The key assignments received request is redirected on purpose disk, i.e., it is corresponding main On disk and journal disk on duty for the day.

230th, master disk group is written into first copy of data to be stored in a manner of LSM；

240th, whether the occupancy of the log space on duty for the day of judgment value day journal disk reaches preset value；

If the 250, occupancy reaches preset value, switching value day journal disk, including：By journal disk on duty for the day by activity Working condition is switched to suspend mode or standby mode, another journal disk wake-up on duty for the day is switched to working condition；It is meanwhile on duty for the day The handover operation of journal disk triggers a synchronized process, in synchronized process, will be deposited in the corresponding master disk of journal disk on duty for the day Storage, inconsistent with journal disk on duty for the day data reading, and day on duty for the day is written into the inconsistent data of reading in a manner of B+ trees Will disk；

If the 260, occupancy is not up to preset value, standby magnetic is written into second copy of data to be stored in a manner of LFS The journal disk on duty for the day of disk group.

It is detailed further below.

As shown in Figure 3, it is assumed that disk array RAID10 includes six pieces of disks, wherein, three pieces of master disks, respectively with P0, P1 It is represented with P2, three pieces of mirrored disks corresponding with this three pieces of master disks are represented respectively with M0, M1 and M2.Corresponding two disks As mirrored disk pair, which includes three mirrored disks pair, and respectively with (P0, M0), (P1, M1) and (P2, M2) is represented.

In Fig. 3, cylinder represents disk, and black shaded area represents that occupied storage is empty in disk in cylinder Between, white portion represents not yet occupied memory space in disk.Assuming that on these three mirrored disks of M0, M1 and M2 respectively There are 50% idle storage space, i.e., 50% log space.Three mirrored disk M0 being connected by curve with the arrow, M1 and M2 is by as journal disk, the idle storage space in three journal disks, the portion represented respectively with scatterplot and twill Point, as log space.The scatterplot and diagonal section connected with curve with the arrow represents all three mirrored disks The log space that idle storage space is formed.Mirrored disk where scatterplot is journal disk on duty for the day, and the disk where twill For non-value day journal disk.M0, M1 and M2 are used as journal disk on duty for the day successively, that is, the 0th daily record period, M0 is day on duty for the day Will disk；The 1st daily record period, M1 is journal disk on duty for the day；The 2nd daily record period, M2 is journal disk on duty for the day； 3 daily record periods, M0 are journal disk on duty for the day again；The rest may be inferred.

As shown in Fig. 4 (a), the bond space of key assignments data is cut into isometric key assignments segmentation (Key Range, KR), Be respectively labeled as KR1, KR2, KR3, KR4 ..., KRi, and be distributed in mirrored disk by the way of turning round to (P0, M0), (P1, M1) and on (P2, M2).

If Fig. 4 (b), (c) are with shown in (d), in daily record cycle T 0, M0 is used as journal disk on duty for the day, in daily record week First copy of the new write-in data in phase T0 is allocated as three parts, i.e. D0T0, D1T0 and D2T0, be respectively written into master disk P0, P1 and P2；Second copy D0T0, D1T0 and D2T0, will be written to mirrored disk M0.Similarly, when the entrance daily record period T1, new first copy that data are written are allocated as three parts, i.e. D0T1, D1T1 and D2T1, be respectively written into master disk P0, P1 and P2；Second copy D0T1, D1T1 and D2T1, will be written to mirrored disk M1.After, and so on.Band arrow shown in Fig. 4 The dotted line and data layout of head illustrate the basic principle of circular logging.

When each new daily record period starts, the switching of journal disk on duty for the day can trigger a synchronized process.Such as Fig. 4 (b) Shown, in daily record cycle T 0, M0 is selected as journal disk on duty for the day, before T0, the 0th mirrored disk to (P0, M0 there is no inconsistent data between), and therefore, in T0, mirrored disk is between (P0, M0) without simultaneously operating.At the 1st In daily record cycle T 1, M1 is selected as journal disk on duty for the day, and before T1, the 1st mirrored disk is deposited between (P1, M1) In inconsistent data, i.e. D1T0, therefore, in T1 start times, synchronizing process of the mirrored disk between (P1, M1) is touched Hair, data D1T0 is written into M1, meanwhile, the space that D1T0 is stored in M0 is released, and the synchronizing process is will be inconsistent It is just terminated after the fully synchronized completion of data.The rest may be inferred, and in daily record cycle T 2, M2 is selected as journal disk on duty for the day, And in T2 start times, the 2nd synchronizing process of the mirrored disk between (P2, M2) is triggered, data D2T0 and D2T1 quilt M2 is written, meanwhile, the space of storage D2T0 and D2T1 is released in M0 and M1, and the synchronizing process is by inconsistent data It is just terminated after fully synchronized completion.

Fig. 4 (b), (c) are illustrated respectively in the key assignments in tri- daily record end cycle moment disk groups of T0, T1 and T2 with (d) Data distribution situation.Wherein, m-th of mirrored disk is written to all of (Pm, Mm) in DmTn representatives in n-th of daily record period Tn Key assignments data, in the present embodiment, m 0,1 or 2, n are the natural number more than or equal to 0, and blank boxes represent master disk and mirror drive On not yet occupied memory space, grid with twill represents that the memory space on disk represented by the region is released It puts, the grid with nicking represents that the corresponding key assignments data of the logic region are by synchronized update to mirrored disk on master disk In B+ trees in.

As soon as when a new journal disk is selected as journal disk on duty for the day, a new synchronizing process is touched Hair, and the new synchronizing process only just can be by end after inconsistent datas all in journal disk on duty for the day are updated and finish Only.As shown in Fig. 4 (b), in daily record cycle T 0, M0 is selected as journal disk on duty for the day, and key assignments write operation requests reach mirror During as disk to (P0, M0), key assignments write operation requests data are write in the LSM data structures of master disk P0, through judging, if The occupancy of log space on duty for the day is less than preset threshold value T on journal disk M0 on duty for the day in T0, at this time writes key assignments Operation requests data sequence in a manner of log-structured file system is write in journal disk M0 on duty for the day；If the day on duty for the day in T0 The occupancy of log space on duty for the day is more than preset threshold value T on will disk M0, and M0 is switched to the standby of low energy consumption at this time State selects M1 as new journal disk on duty for the day, and M1 is switched to the active state of high energy consumption, triggering mirrored disk to (P1, M1 the synchronizing process between).The rest may be inferred, and Fig. 4 (c) and (d) are shown, in T1 and T2, respectively by (P1, M1) and (P2, M2) In not newer key assignments data read from the LSM of P1 or P2, and be written in the B+ trees of M1 and M2.In T1 and T2 finish times, Journal disk M2 and M0 are selected as new journal disk on duty for the day.

In Fig. 4 (c) and (d) solid line with the arrow and the rectangular box with twill show respectively distributing synchronizing process and Log space discharges schematic diagram.When journal disk M1 is selected as journal disk on duty for the day, triggering mirrored disk is to (P1, M1) Synchronizing process between middle master disk P1 and mirrored disk M1.When synchronizing process of the mirrored disk between (P1, M1) terminates it Afterwards, the occupied memory spaces of D1T0 on M0 are released.Similarly, synchronizing process of the mirrored disk between (P2, M2) exists M2 is chosen as being triggered during journal disk on duty for the day.After synchronizing process of the mirrored disk between (P2, M2) terminates, on M0 The occupied memory spaces of D2T1 on D2T0 and M1 are released.

Since occupied log space most of on journal disk M0 is respectively in daily record cycle T 1 and T2 with mirror Picture disk is released the synchronizing process of (P1, M1) and mirrored disk between (P2, M2), and journal disk M0 can be chosen again It selects as journal disk on duty for the day.The rest may be inferred, on journal disk M1 and M2 most of occupied log space respectively with Mirrored disk is released the synchronizing process of (P0, M0), (P2, M2) and mirrored disk between (P0, M0), (P1, M1), because This, M1 and M2 can also be selected as journal disk on duty for the day again.

In the embodiment of the present invention, in order to which disk array is enable to support present example method, need in advance to disk battle array Row carry out initialization process, including：

In initialization procedure, by each master disk and each standby deposit disk and be formatted as arbitrary file system respectively；And And LSM systems are created on each master disk, such as can be specifically key assignments storage system LevelDB；In each standby disk Two storage regions are marked off, wherein, create LFS in first storage region；Second storage region creates B+ tree systems, example As that can be specifically key assignments storage system BDB.So that subsequently can：The main memory is written into first copy of data to be stored In the LevelDB for storing up subsystem, second copy of data to be stored is written in the LFS of standby storage device on duty for the day and, In synchronizing process, the inconsistent data of reading is written in the BDB of standby storage device on duty for the day.

More than, the present embodiment is illustrated by taking disk array as an example, but it is to be understood that in other embodiments, described Storage system be not limited to disk array, the disk in disk array can be naked disk or be formatted as specific text Disk after part system；Described storage system can also be node array, and node array can be adapted for multinode point Cloth environment.

In addition, it is previously described the first, second and the third Method of Data Organization can choose at random as needed, For example, the first Method of Data Organization can select LSM, but B+ trees or other organizational forms can also be selected；Second Method of Data Organization can select LFS or B+ trees；The third Method of Data Organization can select B+ trees or LSM；Herein no longer in detail Carefully repeat.

More than, the embodiment of the invention discloses a kind of date storage methods, achieve following technique effect：

2nd, being created on each main storage means and each standby storage device has file system, and data are in active and standby storage dress It is all stored in putting with certain Method of Data Organization, can reflect data structure layer feature, and improve reconstruction speed.If one Block main storage means fail, and when restoring data from corresponding standby storage device, which can directly know by Method of Data Organization A little data needs restore, and from without each data block is identified, therefore rebuild speed block.

In order to preferably implement the said program of the embodiment of the present invention, it is also provided below and implements said program for coordinating Relevant apparatus.

Fig. 5 is please referred to, the embodiment of the present invention provides a kind of data storage device 500.The equipment is used for storage system, described Storage system includes main storage subsystem and standby storage subsystem；The main storage subsystem includes N number of main storage means, described Standby storage subsystem includes N number of standby storage device corresponding to N number of main storage means；N is the positive integer more than 1；Wherein Being created in each main storage means and each standby storage device has file system.The equipment 500 may include：

First module for reading and writing 510, for first copy of data to be stored to be written with the first Method of Data Organization Multiple main storage means in main storage subsystem；

Second module for reading and writing 520, for second copy of data to be stored to be written with second of Method of Data Organization Standby storage device on duty for the day in standby storage subsystem, the standby storage device on duty for the day are unique in running order standby storage dresses It puts；

Third module for reading and writing 530, for by stored in the corresponding main storage means of the standby storage device on duty for the day and institute It states the inconsistent data of standby storage device on duty for the day to read, and the inconsistent data of reading is written with the third Method of Data Organization The standby storage device on duty for the day；

Wherein, the first Method of Data Organization and second of Method of Data Organization write optimization, and second of data group The writing speed for knitting mode is faster than the writing speed of the first Method of Data Organization, the third Method of Data Organization is then to read optimization 's.

As shown in fig. 6, in some embodiments of the invention, the equipment can also include：

Storage device monitoring module 540, for whether judging the occupancy of the log space on duty for the day of standby storage device on duty for the day Reach preset value；If occupancy reaches preset value, the standby storage device on duty for the day is switched to dormant state, and by another Standby storage device is switched to working condition as standby storage device on duty for the day.

As shown in fig. 6, in other embodiments of the invention, the equipment can also include：

Creation module 550, for creating key assignments storage system LevelDB in each main storage means；It standby is deposited each Two storage regions are marked off in storage device, wherein, LFS is created in first storage region, second storage region creates key It is worth storage system BDB.

In other embodiments of the invention, first module for reading and writing is particularly used in first pair of data to be stored Originally multiple portions are divided into, the multiple part are respectively written into the LevelDB of multiple main storage means；Second read-write Module is particularly used in, and second copy of data to be stored is written in the LFS of the standby storage device on duty for the day；The third Module for reading and writing is particularly used in, and the inconsistent data of reading is written in the BDB of the standby storage device on duty for the day.

The data storage device of the embodiment of the present invention for example can be computer equipment or the management for including disk array The network equipment of node array.

It is appreciated that the function of each function module of the data storage device of the embodiment of the present invention can be according to the above method Method specific implementation in embodiment, specific implementation process can refer to the associated description in above method embodiment, herein not It repeats again.

Therefore in some feasible embodiments of the present invention, using in each main storage means and each standby On storage device all create have file system and, by first copy of data to be stored with the first Method of Data Organization Main storage subsystem is written, by second copy of data to be stored with second of standby storage subsystem of Method of Data Organization write-in In standby storage device on duty for the day, by it is being stored in the corresponding main storage means of the standby storage device on duty for the day, with it is described on duty for the day standby The inconsistent data of storage device are read, and the inconsistent data of reading is described on duty for the day with the write-in of the third Method of Data Organization The technical solution of standby storage device, achieves following technique effect：

The embodiment of the present invention also provides a kind of computer storage media, which can have program stored therein, should Part or all of step including the date storage method described in above method embodiment when program performs.

Fig. 7 is please referred to, the embodiment of the present invention also provides a kind of computer equipment 700, it may include：

Processor 710, memory 720, communication interface 730, bus 740；The processor 710, memory 720, communication Interface 730 passes through the mutual communication of the bus 740；The communication interface 730, for sending and receiving data；The storage Device 720 is used to store program；The processor 710 is used to perform the described program in the memory.The memory 720, It may include main storage subsystem and standby storage subsystem；The main storage subsystem includes N number of main storage means, the standby storage Subsystem includes N number of standby storage device corresponding to N number of main storage means；N is the positive integer more than 1；It is wherein each main Being created on storage device and each standby storage device has file system.Described active and standby storage device can be disk.

Wherein, processor 710 is used to be written first copy of data to be stored with the first Method of Data Organization and be led Second copy of data to be stored is written multiple main storage means in storage subsystem with second of Method of Data Organization Standby storage device on duty for the day in standby storage subsystem, the standby storage device on duty for the day are unique in running order standby storage dresses It puts；It will be being stored in the corresponding main storage means of the standby storage device on duty for the day, inconsistent with the standby storage device on duty for the day Data are read, and the inconsistent data of reading is written the standby storage device on duty for the day with the third Method of Data Organization；Wherein, The first Method of Data Organization and second of Method of Data Organization write optimization, and the write-in speed of second of Method of Data Organization Degree is faster than the writing speed of the first Method of Data Organization, the third Method of Data Organization is then to read optimization.

In some embodiments of the invention, processor 710 is additionally operable to judge that the daily record on duty for the day of standby storage device on duty for the day is empty Between occupancy whether reach preset value；If occupancy reaches preset value, the standby storage device on duty for the day is switched to suspend mode State, and another is switched to working condition as standby storage device on duty for the day for storage device.

In some embodiments of the invention, the first described Method of Data Organization is log-structured merging LSM modes；Institute It is log-structured file system LFS modes to state second of Method of Data Organization；Method of Data Organization is B+ tree sides in the third Formula.

In some embodiments of the invention, processor 710 is additionally operable to create key assignments storage in each main storage means System LevelDB；Two storage regions are marked off in each standby storage device, wherein, it is created in first storage region LFS, second storage region create key assignments storage system BDB.

In some embodiments of the invention, processor 710 is specifically used for dividing in first copy of data to be stored For multiple portions, the multiple part is respectively written into the LevelDB of multiple main storage means；By the second of data to be stored A copy is written in the LFS of the standby storage device on duty for the day；The standby storage device on duty for the day is written into the inconsistent data of reading BDB in.

It is appreciated that the function of each function module of the computer equipment of the embodiment of the present invention can be according to above method reality The method specific implementation in example is applied, specific implementation process can refer to the associated description in above method embodiment, herein no longer It repeats.

Therefore the present invention some feasible embodiments, achieve following technique effect：

In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, is not described in some embodiment Part, may refer to the associated description of other embodiments.

It should be noted that for aforementioned each method embodiment, in order to be briefly described, therefore it is all expressed as a series of Combination of actions, but those skilled in the art should know, the present invention is not limited by described sequence of movement because according to According to the present invention, certain steps may be used other sequences or be carried out at the same time.Secondly, those skilled in the art should also know, Embodiment described in this description belongs to preferred embodiment, and not necessarily the present invention must for involved action and module Must.

One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment is can It is completed with instructing relevant hardware by program, which can be stored in a computer readable storage medium, storage Medium can include：ROM, RAM, disk or CD etc..

A kind of date storage method and equipment provided above to the embodiment of the present invention is described in detail, herein It applies specific case to be expounded the principle of the present invention and embodiment, the explanation of above example is only intended to help Understand the method and its core concept of the present invention；Meanwhile for those of ordinary skill in the art, thought according to the present invention, There will be changes in specific embodiments and applications, in conclusion the content of the present specification should not be construed as to this The limitation of invention.

Claims

1. a kind of date storage method, which is characterized in that for storage system, the storage system include main storage subsystem and Standby storage subsystem；The main storage subsystem includes N number of main storage means, and the standby storage subsystem includes corresponding to described N number of standby storage device of N number of main storage means；N is the positive integer more than 1；Wherein each main storage means and each standby storage All being created on device has file system；

The method includes：

By first copy of data to be stored with multiple main memories in the first Method of Data Organization write-in main storage subsystem Storage device, wherein, first copy is divided into N number of part, each one main storage means of partial write；

Second copy of data to be stored standby is deposited on duty for the day in the standby storage subsystem of second Method of Data Organization write-in Storage device, the standby storage device on duty for the day are unique in running order standby storage devices；

It will be being stored in the corresponding main storage means of the standby storage device on duty for the day, inconsistent with the standby storage device on duty for the day Data are read, and the inconsistent data of reading is written the standby storage device on duty for the day with the third Method of Data Organization；

Wherein, the first Method of Data Organization and second of Method of Data Organization write optimization, and second of data organization side The writing speed of formula is faster than the writing speed of the first Method of Data Organization, the third Method of Data Organization is then to read optimization.

2. it according to the method described in claim 1, it is characterized in that, further includes：

Judge whether the occupancy of the log space on duty for the day of standby storage device on duty for the day reaches preset value；

If occupancy reaches preset value, the standby storage device on duty for the day is switched to dormant state, and by another for storage Device is switched to working condition as standby storage device on duty for the day.

3. according to the method described in claim 1, it is characterized in that：

The first described Method of Data Organization is log-structured merging LSM modes；

Second of Method of Data Organization is log-structured file system LFS modes；

The third described Method of Data Organization is B+ tree modes.

4. according to the method described in claim 3, it is characterized in that, first copy by data to be stored is with the first Before Method of Data Organization write-in main storage subsystem, further include：

Key assignments storage system LevelDB is created in each main storage means；

Two storage regions are marked off in each standby storage device, wherein, create LFS in first storage region, second Storage region creates key assignments storage system BDB.

5. according to the method described in claim 4, it is characterized in that：

First copy by data to be stored is written multiple in main storage subsystem with the first Method of Data Organization Main storage means include：First copy of data to be stored is divided into multiple portions, the multiple part is respectively written into In the LevelDB of multiple main storage means；

Second copy by data to be stored is on duty for the day in second of standby storage subsystem of Method of Data Organization write-in Standby storage device includes：Second copy of data to be stored is written in the LFS of the standby storage device on duty for the day；

The inconsistent data by reading is written the standby storage device on duty for the day with the third Method of Data Organization and includes：It will read The inconsistent data gone out is written in the BDB of the standby storage device on duty for the day.

6. according to the method any in claim 1 to 5, it is characterised in that：

The storage system is disk array or node array.

7. a kind of data storage device, which is characterized in that for storage system, the storage system include main storage subsystem and Standby storage subsystem；The main storage subsystem includes N number of main storage means, and the standby storage subsystem includes corresponding to described N number of standby storage device of N number of main storage means；N is the positive integer more than 1；Wherein each main storage means and each standby storage All being created on device has file system；The equipment includes：

First module for reading and writing, for first copy of data to be stored is sub with the first Method of Data Organization write-in primary storage Multiple main storage means in system, wherein, first copy is divided into N number of part, each one main memory of partial write Storage device；

Second module for reading and writing, for second copy of data to be stored is sub with second of standby storage of Method of Data Organization write-in Standby storage device on duty for the day in system, the standby storage device on duty for the day are unique in running order standby storage devices；

Third module for reading and writing, for by it is being stored in the corresponding main storage means of the standby storage device on duty for the day, with it is described on duty for the day The inconsistent data of standby storage device are read, and the inconsistent data of reading is written described value with the third Method of Data Organization Day is for storage device；

8. equipment according to claim 7, which is characterized in that further include：

Whether storage device monitoring module, the occupancy for judging the log space on duty for the day of standby storage device on duty for the day reach default Value；If occupancy reaches preset value, the standby storage device on duty for the day is switched to dormant state, and another standby storage is filled It puts and is switched to working condition as standby storage device on duty for the day.

9. equipment according to claim 7, which is characterized in that further include：

Creation module, for creating key assignments storage system LevelDB in each main storage means；In each standby storage device Two storage regions are marked off, wherein, LFS is created in first storage region, second storage region creates key assignments storage system Unite BDB.

10. equipment according to claim 9, it is characterised in that：

First module for reading and writing is specifically used for first copy of data to be stored being divided into multiple portions, will be the multiple Part is respectively written into the LevelDB of multiple main storage means；

Second module for reading and writing is specifically used for second copy of the data to be stored standby storage device on duty for the day is written In LFS；

The third module for reading and writing is specifically used for the inconsistent data read being written in the BDB of the standby storage device on duty for the day.

11. a kind of computer equipment, which is characterized in that including：

Processor, memory, communication interface, bus；The processor, memory, communication interface are by the bus mutual Communication；The memory, including main storage subsystem and standby storage subsystem；The main storage subsystem includes N number of primary storage Device, the standby storage subsystem include N number of standby storage device corresponding to N number of main storage means；N is more than 1 just Integer；Being created in wherein each main storage means and each standby storage device has file system；

Wherein, the processor is used to first copy of data to be stored primary storage is written with the first Method of Data Organization Multiple main storage means in subsystem, wherein, first copy is divided into N number of part, each one master of partial write Storage device；By second copy of data to be stored on duty for the day in second of standby storage subsystem of Method of Data Organization write-in Standby storage device, the standby storage device on duty for the day are unique in running order standby storage devices；By the standby storage on duty for the day Data reading being stored in the corresponding main storage means of device, inconsistent with the standby storage device on duty for the day, and by reading The standby storage device on duty for the day is written with the third Method of Data Organization in inconsistent data；Wherein, the first Method of Data Organization Optimization is write, and the writing speed of second of Method of Data Organization is faster than the first data group with second of Method of Data Organization The writing speed of mode is knitted, the third Method of Data Organization is then to read optimization.