CN107729536A - A kind of date storage method and device - Google Patents
A kind of date storage method and device Download PDFInfo
- Publication number
- CN107729536A CN107729536A CN201711043732.9A CN201711043732A CN107729536A CN 107729536 A CN107729536 A CN 107729536A CN 201711043732 A CN201711043732 A CN 201711043732A CN 107729536 A CN107729536 A CN 107729536A
- Authority
- CN
- China
- Prior art keywords
- data
- osd
- copy
- main
- chunk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application, which provides a kind of date storage method and device, this method, to be included：Obtain data file to be stored, LUN identification corresponding to the data file；Multiple object storage device OSD according to corresponding to cluster view information determines the LUN identification, the multiple OSD quantity are not less than data trnascription number M corresponding to the LUN identification；The data file is split into multiple data blocks, and multiple data chunks are generated according to the multiple data block；For each data chunk, determined from the multiple OSD for store the master data in the data chunk main OSD and the data chunk in copy data from OSD, and by the primary data store main OSD corresponding to main partition, by copy data storage to it is corresponding from OSD from point；The monitoring server is arrived into metadata storage corresponding to the data file.By the technical scheme of the application, higher data reliability can be provided.
Description
Technical field
The application is related to communication technical field, more particularly to a kind of date storage method and device.
Background technology
With mobile network, big data, cloud computing extensive application, memory capacity and storage performance are proposed increasingly
High requirement, Ceph (distributed file system based on Linux PB), GFS (Google File System, based on Linux
Proprietary distributed file system), Gluster, HDFS (Hadoop Distributed File System, distributed document
System) distributed file system obtained quick development.
Wherein, distributed file system can (Object Storage Device, object storage be set including multiple OSD
It is standby), OSD is data storage service unit, and an OSD can correspond to a physical disk.
In order to realize the storage of data file, data file is split into multiple data objects, obtains each data object
Metadata (metadata), and by the data object storage arrive OSD, by the metadata storage arrive distributed file system pipe
Manage node.Wherein, the metadata of data object, can include storage location of the data object in OSD, data object it is big
Small, temporal information (such as creation time, modification time, access time), attribute (such as read-only, hiding), file type, unfolding mode
Deng.
But due to being required to store metadata for each data object, when the quantity of data object is a lot, own
The metadata of data object can take a large amount of storage resources, and need the management section of frequent visit distributed file system
Point, to read metadata from management node, occupy a large amount of reading resources.
The content of the invention
The application provides a kind of date storage method, and applied to client, methods described includes：
Obtain data file to be stored, LUN identification corresponding to the data file；Institute is determined according to cluster view information
Multiple OSD corresponding to LUN identification are stated, the multiple OSD quantity is not less than data trnascription number M corresponding to the LUN identification；Institute
Stating cluster view information is obtained from monitoring server, and cluster view information includes LUN identification and OSD corresponding relation, and M is
Positive integer more than 1；
The data file is split into multiple data blocks, and multiple data chunks are generated according to the multiple data block；
Wherein, the data number of blocks in each data chunk is M, and each data chunk includes a master data and M-1 copy data；
For each data chunk, determined from the multiple OSD for storing the master data in the data chunk
Main OSD and the copy data in the data chunk from OSD, and by the primary data store main OSD corresponding to master point
Area, by copy data storage to corresponding from OSD from subregion；Wherein, each OSD includes a main partition and M-1
From subregion；
The monitoring server is arrived into metadata storage corresponding to the data file.
The application provides a kind of data storage device, and applied to client, described device includes：
Acquisition module, for obtaining data file to be stored, LUN identification corresponding to the data file；Regarded according to cluster
Figure information determines multiple OSD corresponding to the LUN identification, and the multiple OSD quantity is not less than number corresponding to the LUN identification
According to number of copies M；The cluster view information is obtained from monitoring server, and cluster view information includes LUN identification and OSD
Corresponding relation, M are the positive integer more than 1；
Generation module, for the data file to be split into multiple data blocks, and generated according to the multiple data block
Multiple data chunks；Wherein, the data number of blocks in each data chunk is M, each data chunk include a master data and
M-1 copy data；
Processing module, for for each data chunk, determining to be used to store the data chunk from the multiple OSD
In master data main OSD and the copy data in the data chunk from OSD, and by the primary data store to corresponding to
Main OSD main partition, by copy data storage to corresponding from OSD from subregion；Wherein, each OSD includes a master
Subregion and M-1 are individual from subregion；
Sending module, for metadata storage corresponding to the data file to be arrived into the monitoring server.
Based on above-mentioned technical proposal, in the embodiment of the present application, the metadata of whole data file is only stored, greatly reduces member
Data bulk, save the storage resource that metadata takes.Due to only reading the metadata of whole data file from monitoring server,
Without reading a large amount of metadata, so as to mitigate the processing pressure of client, mitigate the access frequency of metadata, improve first number
According to acquisition efficiency, mitigate the complexity of metadata management.
In addition, by generating multiple data chunks, each data chunk includes a master data and multiple copy datas, from
And the redundancy backup (data distribution strategy that redundancy is realized by more copies) of data can be realized, there is provided higher data can
By property.In addition, by agreement for storing the main OSD of master data, for stored copies data from OSD, so as to quick
Position inter-related master data and copy data；Master data is directed to, copy data corresponding to the master data can be inquired
Storage location, for copy data, the storage location of master data corresponding to the copy data can also be inquired, realize data
Mapping addressing.
Brief description of the drawings
, below will be to the application in order to clearly illustrate the embodiment of the present application or technical scheme of the prior art
The required accompanying drawing used is briefly described in embodiment or description of the prior art, it should be apparent that, in describing below
Accompanying drawing is only some embodiments described in the application, for those of ordinary skill in the art, can also be according to this Shen
Please these accompanying drawings of embodiment obtain other accompanying drawings.
Fig. 1 is the application scenarios schematic diagram in a kind of embodiment of the application；
Fig. 2 is the schematic diagram of the corresponding relation of the LUN and OSD in a kind of embodiment of the application；
Fig. 3 is the flow chart of the date storage method in a kind of embodiment of the application；
Fig. 4 A- Fig. 4 H are the storage schematic diagrames of the data block in a kind of embodiment of the application；
Fig. 5 is the structure chart of the data storage device in a kind of embodiment of the application；
Fig. 6 is the hardware structure diagram of the client in a kind of embodiment of the application.
Embodiment
Purpose in the term that the embodiment of the present application uses merely for the sake of description specific embodiment, and this unrestricted Shen
Please." one kind " of singulative used in the application and claims, " described " and "the" are also intended to including more number forms
Formula, unless context clearly shows that other implications.It is also understood that term "and/or" used herein refers to include one
Individual or multiple associated list items purposes any or all may combine.
It will be appreciated that though various letters may be described using term first, second, third, etc. in the embodiment of the present application
Breath, but these information should not necessarily be limited by these terms.These terms are only used for same type of information being distinguished from each other out.For example,
In the case where not departing from the application scope, the first information can also be referred to as the second information, and similarly, the second information can also
It is referred to as the first information.Depending on linguistic context, in addition, used word " if " can be construed to " ... when " or
" when ... " or " in response to determining ".
A kind of date storage method is proposed in the embodiment of the present application, this method can apply to include client and server
System, such as distributed file system, server can include data server (Data server) and monitoring server
(Monitor server), it is shown in Figure 1, it is the structure chart of distributed file system.
Wherein, client can be the interface that user carries out data write-in/digital independent, and user accesses number by client
According to server, so as to write data in data server, read data from data server.
Wherein, monitoring server be used for monitoring data server health status, manage data server storage resource,
The metadata and cluster view information of data storage server.If the health status of data server is abnormal, it is used for quick
Recover the failure of data server.User can be connected to monitoring server by management interface, so as to manage data server
Storage resource, inquire about the metadata and cluster view information of the storage of data server.Monitoring server is typically High Availabitity
Pattern, to avoid monitoring server failure.
Wherein, data server can include multiple OSD, and each OSD is used to data storage.An in addition, LUN
(Logical Unit Number, LUN) can correspond to multiple OSD, equivalent to one physical disk of each OSD, LUN
It can be the logic storage medium of these physical disks composition.In addition, independent OSD service processes are provided for each OSD, should
OSD service processes are used for the OSD processing.For example, OSD service processes 1 can be provided for OSD1, and provided for OSD2
OSD service processes 2, so, if the processing for OSD1 for receiving client transmission is asked, carried out by OSD service processes 1
Processing, in processing procedure, OSD service processes 1 can write data or the physics from OSD1 in OSD1 physical disk
Data are read in disk；If the processing for OSD2 for receiving client transmission is asked, carried out by OSD service processes 2
Reason, by that analogy., can be quick extensive in OSD service processes exceptions by providing independent OSD service processes for each OSD
Multiple OSD service processes, and other OSD service processes will not be impacted.For example, when OSD service processes 1 are abnormal, then OSD
Service processes 2 can work on, because OSD service processes 1 only provide OSD1 service, the resource that OSD service processes 1 manage
Seldom, therefore can quick-recovery OSD service processes 1 soon.
For example, LUN1 corresponds to OSD1, OSD2, OSD3, OSD4, LUN2 corresponds to OSD5, OSD6, OSD7, OSD8, if user needs
Writing data into LUN1, then client is write data into OSD1, OSD2, OSD3, OSD4 corresponding to LUN1, with
This analogizes.
In one example, it is shown in Figure 2, be LUN and OSD corresponding relation example, LUN1 correspond to OSD1 with
OSD2, LUN2 correspond to OSD2 and OSD3, and LUN3 corresponds to OSD3 and OSD4.
Assuming that OSD1 is 10G memory spaces, OSD2 is 20G memory spaces, and OSD3 is 20G memory spaces, and OSD4 is that 8G is deposited
Space is stored up, then：LUN1 can take OSD1 10G memory spaces, take OSD2 10G memory spaces (such as preceding 10G storages sky
Between)；LUN2 takes OSD2 10G memory spaces (such as rear 10G memory spaces), takes OSD3 10G memory spaces (such as preceding 10G
Memory space)；LUN3 takes OSD3 10G memory spaces (such as rear 10G memory spaces), takes OSD4 8G memory spaces.On
The corresponding relation example for being merely given as LUN and OSD is stated, it is without limitation.
Wherein, a LUN may belong to a storage pool, and multiple OSD corresponding to a LUN can be located at main frame (HOST)
On, these main frames can be located in frame (RACK).For example, distributed file system can include frame 1 and frame 2, frame 1
Can include main frame 11 and main frame 12, frame 2 can include main frame 21 and main frame 22, main frame 11 can include OSD111,
OSD112, OSD113, main frame 12 can include OSD121, OSD122, OSD123, OSD124, main frame 21 can include OSD211,
OSD212, main frame 22 can include OSD221, above-mentioned to be merely given as an example, without limitation.
Based on above-mentioned framework, multiple OSD corresponding to a LUN can be distributed on the different main frames of different frames as far as possible.
For example, 4 OSD corresponding to a LUN, can be OSD111, OSD121, OSD211, OSD221 respectively, OSD111 is located at main frame
On 11, OSD121 is located on main frame 12, and OSD211 is located on main frame 21, and OSD221 is located on main frame 22.So, when certain is led
When machine breaks down, only an OSD can not be used, and other OSD can be continuing with corresponding to LUN, avoid institute corresponding to LUN
There is OSD can not all use.When certain frame breaks down, only two OSD can not be used, and other OSD corresponding to LUN can be after
It is continuous to use, avoid all OSD corresponding to LUN from can not all using.
It is shown in Figure 3 under above-mentioned application scenarios, it is the flow of the date storage method proposed in the embodiment of the present application
Figure, this method can apply to client, and this method may comprise steps of：
Step 301, data file to be stored is obtained, LUN identification corresponding to the data file.
Wherein, when user needs data storage file (to describe, can claim the data for needing to store to OSD for convenience
For data file, such as 400M data file) when, then client can get data file to be stored and the data
LUN identification corresponding to file (LUN identification that i.e. user specifies).
Step 302, multiple OSD according to corresponding to cluster view information determines the LUN identification.
In one example, the cluster view information is that client obtains from monitoring server, and the cluster view is believed
Breath can include but is not limited to LUN identification and OSD corresponding relation.Therefore, client is after cluster view information is obtained, can be with
Multiple OSD according to corresponding to cluster view information determines the LUN identification.
In one example, before step 302 or step 301, client can be established with monitoring server and connected,
And send view information to monitoring server and obtain message, monitoring server, can be with after receiving view information and obtaining message
Cluster view information (also referred to as Data distribution information) is sent to client.
Wherein, the cluster view information can include but is not limited to the corresponding magnetic of the corresponding relation of LUN identification and OSD, OSD
The starting free address etc. of disk capacity, OSD, is not limited to this content.
For example, LUN1 corresponds to OSD1, OSD3, OSD5, OSD7；LUN2 corresponds to OSD2, OSD4, OSD6, OSD8；LUN1 belongs to
Storage pool 1；LUN2 belongs to storage pool 2；OSD1 disk size is that 10G, OSD2 disk size are 10G, OSD3 disk appearance
Amount is that 10G, OSD4 disk size are that 10G, OSD4 disk size is that 12G, OSD6 disk size are 12G, OSD7 magnetic
Disk capacity is that 11G, OSD8 disk size are 12G；The starting free address of each OSD in OSD1-OSD8.
Based on above-mentioned cluster view information, client is after LUN identification corresponding to data file is got, it is assumed that LUN is marked
Know for LUN1, it is determined that OSD corresponding to LUN1 is OSD1, OSD3, OSD5, OSD7.
Step 303, data file is split into multiple data blocks, and multiple data chunks is generated according to multiple data blocks；
Wherein, the data number of blocks in each data chunk is data trnascription number M corresponding to LUN identification, and each data chunk includes one
Individual master data and M-1 copy data.For data trnascription number corresponding to LUN identification, in subsequent process, by data trnascription number
M is abbreviated as, the M is the positive integer more than 1.
Wherein it is possible to the size of data block is preassigned, if the size of data block is 4M, it is assumed that the size of data file is
400M, then data file can be split into 100 4M data block.Here it is that data file is split into data block, data
Block is exactly the data in data file, not the other information such as including metadata.
Wherein, for the process of " generating multiple data chunks according to multiple data blocks ", can include but is not limited to as follows
Mode：Mode one, it is that each data block generates a data chunk；In the data chunk, master data is the data block, and secondary
Notebook data is the M-1 data block replicated for the data block.Mode two, N number of data block is chosen from multiple data blocks, determine N
Verification data corresponding to individual data block, and N number of data block and the verification data are formed into a data chunk；In the data
In block group, master data is any data in the data chunk, and other data outside master data are copy data, and N is big
In 1 positive integer.
Wherein it is possible to be pre-configured with M corresponding to LUN identification, this M is preassigned.
RAID1 (Redundant Arrays of Independent Disks, independent redundancy disk battle array are used for example, working as
Row) mode data storage block when, then handled using aforesaid way one, and M can be 2,3,4 etc., M value is not limited.
When using RAID5 modes or RAID6 mode data storage blocks, then handled using aforesaid way two, and M can be 4,5,6
Deng not being limited to M value.
For mode one, it is assumed that data file is split into data block 1- data blocks 100, M 3, then：For data block 1
Data chunk 1 is generated, in data chunk 1, master data can be data block 1, and copy data is 2 replicated for data block 1
Data block, this 2 data blocks are respectively data block 1A and data block 1B, and data block 1A and data block 1B with the phase of data block 1
Together.Data chunk 2 is generated for data block 2, in data chunk 2, master data can be data block 2, and copy data is for number
2 data blocks replicated according to block 2, this 2 data blocks are respectively data block 2A and data block 2B, by that analogy.
For mode two, it is assumed that data file is split into data block 1- data blocks 100, M 4, deposited using RAID5 modes
Data block is stored up, then：Data block 1, data block 2 and data block 3 are chosen from data block 1- data blocks 100, determines this 3 data blocks
Corresponding verification data A1, generates data chunk 1, and data chunk 1 includes data block 1, data block 2, data block 3 and verification data
A1；In data chunk 1, master data can be any one in data block 1, data block 2, data block 3 and verification data A1
Data, if master data is data block 1, then copy data is data block 2, data block 3 and verification data A1.Then, from data block
Data block 4, data block 5 and data block 6 are chosen in 4- data blocks 100, determines verification data A2 corresponding to this 3 data blocks, it is raw
Into data chunk 2, data chunk 2 includes data block 4, data block 5, data block 6 and verification data A2；It is main in data chunk 2
Data can be verification data A2, and copy data is data block 4, data block 5, data block 6.By that analogy, to other data blocks
Repeat no more.
For mode two, it is assumed that data file is split into data block 1- data blocks 100, M 6, deposited using RAID6 modes
Data block is stored up, then：Data block 1, data block 2, data block 3 and data block 4 can be chosen from data block 1- data blocks 100, and
Verification data B1 and verification data C1 corresponding to this 4 data blocks is determined, then generates data chunk 1, data chunk 1 includes number
According to block 1, data block 2, data block 3, data block 4, verification data B1 and verification data C1；In data chunk 1, master data can be with
It is any one data in data block 1, data block 2, data block 3, data block 4, verification data B1 and verification data C1, it is such as main
Data are data blocks 1, then copy data is data block 2, data block 3, data block 4, verification data B1 and verification data C1.With this
Analogize, other data blocks are repeated no more.
Step 304, for each data chunk, determined from multiple OSD for storing the main number in the data chunk
According to main OSD and the data chunk in copy data from OSD.
In one example, can be different from OSD corresponding to main OSD corresponding to master data and copy data, it is more when existing
, can also be different from OSD corresponding to different copy datas during individual copy data.That is, for each in data chunk
Data, unique OSD can be corresponded to.For example, the data block 1 included for data chunk 1, data block 2, data block 3 and verification
Data A1, then data block 1 can correspond to OSD1,2 corresponding OSD3 of data block, data block 3 corresponding OSD5, verification data A1 are corresponding
OSD7。
In order to realize this process, because the data bulk in data chunk is M, therefore, OSD corresponding to LUN identification
Quantity can be not less than M.For example, when M is 4, then OSD quantity is not less than 4.
In one example, for " determining from multiple OSD the master for storing the master data in the data chunk
OSD " process, it can include：Arbitrarily one OSD of selection from multiple OSD, the OSD of selection is defined as being used to store main number
According to main OSD.Or based on balance policy, the main OSD for storing master data is determined from multiple OSD, balance policy is used
It is less than predetermined threshold value (such as 1,2) in making differences of the different OSD as main OSD number；For example, it is assumed that exist OSD1, OSD2,
OSD3 and OSD4, for data chunk 1, it is determined that be OSD1 for the main OSD for storing master data, for data chunk 2, it is determined that with
In the main OSD of storage master data be OSD2, for data chunk 3, it is determined that be OSD3 for the main OSD for storing master data, for
Data chunk 4, it is determined that being OSD4 for the main OSD for storing master data, for data chunk 5, it is determined that for storing master data
Main OSD is OSD1, and by that analogy, it is main OSD that can determine OSD1, OSD2, OSD3 and OSD4 successively.So, difference can be made
OSD is identical as main OSD number, or approximately the same (i.e. the difference of number is up to 1).
In one example, for " determine from multiple OSD for store the copy data in the data chunk from
OSD " process, following manner can be included but is not limited to：Multiple OSD corresponding to LUN identification are ranked up；Based on multiple
OSD ranking results, since main OSD corresponding to above-mentioned master data, map using mapping policy counterclockwise or clockwise plan
Slightly determine corresponding to each copy data from OSD.
Wherein, mapping policy counterclockwise be used for since the previous OSD of main OSD corresponding to above-mentioned master data, forward according to
It is secondary to determine corresponding to each copy data from OSD, and different copy datas correspond to it is different from OSD；For in ranking results
First OSD, its previous OSD are last OSD in ranking results.Mapping policy clockwise is used for from above-mentioned master data
Corresponding main OSD the latter OSD starts, and determines successively backward corresponding to each copy data from OSD, and different copy datas
It is corresponding different from OSD；For last OSD in ranking results, its latter OSD is first in ranking results
OSD。
For example, it is assumed that OSD ranking results are OSD1, OSD2, OSD3 and OSD4, main OSD is OSD2, using counterclockwise
From OSD it is OSD1 (OSD2 previous OSD) corresponding to first copy data during mapping policy, second copy data pair
Answer from OSD be OSD4 (OSD1 previous OSD, when not having OSD before OSD1, using last OSD4 as OSD1's
Previous OSD), from OSD it is OSD3 (OSD4 previous OSD) corresponding to the 3rd copy data, by that analogy.Using suitable
From OSD it is OSD3 (OSD2 the latter OSD) corresponding to first copy data during hour hands mapping policy, second number of copies
According to it is corresponding from OSD be OSD4 (OSD3 the latter OSD), from OSD be OSD1 (after OSD4 corresponding to the 3rd copy data
One OSD, when there is no OSD before OSD4, the latter OSD using first OSD1 as OSD4), by that analogy.
In one example, each OSD also includes M subregion corresponding to LUN identification, in these subregions, a subregion
Main partition, and other subregions be from subregion, and from number of partitions be M-1.Master data is the main partition that OSD is arrived in storage, and
Copy data is storage to OSD from subregion.
Below in conjunction with several concrete application scenes, the processing procedure of above-mentioned steps 304 is described in detail.
Application scenarios 1：It is the schematic diagram of this application scene referring to shown in Fig. 4 A, is 2, LUN marks using M under this application scene
The quantity of OSD corresponding to knowledge illustrates exemplified by being 2.In addition, each OSD can include two subregions, P1 subregions are main partitions,
P2 subregions are from subregion.
Assuming that data chunk 1 includes master data (data block 1) and copy data (data block 1A), the correspondence of data block 1 is determined
Main OSD be OSD1, from OSD be OSD2 corresponding to data block 1A, and data block 1 stores OSD1 P1 subregions, data block 1A
Store OSD2 P2 subregions.Assuming that data chunk 2 includes master data (data block 2) and copy data (data block 2A), it is determined that
Main OSD corresponding to data block 2 is OSD2, from OSD is OSD1 corresponding to data block 2A, and data block 2 store OSD2 P1 and divided
Area, the P2 subregions of data block 2A storages to OSD1.Main OSD corresponding to the master data of data chunk 3 is OSD1, and copy data is corresponding
From OSD be OSD2, by that analogy.
Application scenarios 2：It is the schematic diagram of this application scene referring to shown in Fig. 4 B, is 2, LUN marks using M under this application scene
The quantity of OSD corresponding to knowledge illustrates exemplified by being 3.In addition, each OSD can include two subregions, P1 subregions are main partitions,
P2 subregions are from subregion.Under this application scene, to be determined using mapping policy clockwise corresponding to each copy data from OSD.
For example, it is assumed that data chunk 1 includes data block 1 and data block 1A, it is determined that main OSD is corresponding to data block 1
From OSD it is OSD2 corresponding to OSD1, data block 1A, and data block 1 store OSD1 P1 subregions, data block 1A is stored and is arrived OSD2
P2 subregions.Assuming that data chunk 2 includes data block 2 and data block 2A, it is determined that main OSD is OSD2 corresponding to data block 2, number
According to from OSD being OSD3 corresponding to block 2A, and data block 2 stores OSD2 P1 subregions, P2 points of data block 2A storages to OSD3
Area.Assuming that data chunk 3 includes data block 3 and data block 3A, it is determined that main OSD corresponding to data block 3 is OSD3, data block 3A
Corresponding from OSD is OSD1, and data block 3 stores OSD3 P1 subregions, the P2 subregions of data block 3A storages to OSD1.With this
Analogize.
Application scenarios 3：It is the schematic diagram of this application scene referring to shown in Fig. 4 C, is 2, LUN marks using M under this application scene
The quantity of OSD corresponding to knowledge illustrates exemplified by being 3.In addition, each OSD can include two subregions, P1 subregions are main partitions,
P2 subregions are from subregion.Under this application scene, to be determined using mapping policy counterclockwise corresponding to each copy data from OSD.
From OSD it is OSD3 corresponding to data block 1A, the storage of data block 1 is arrived for example, main OSD corresponding to data block 1 is OSD1
OSD1 P1 subregions, the P2 subregions of data block 1A storages to OSD3.Main OSD corresponding to data block 2 is OSD2, and data block 2A is corresponding
Be OSD1 from OSD, and data block 2 stores OSD2 P1 subregions, the P2 subregions of data block 2A storages to OSD1.Data block 3
Corresponding main OSD is OSD3, from OSD is OSD2 corresponding to data block 3A, and data block 3 stores OSD3 P1 subregions, data
P2 subregion of the block 3A storages to OSD2.By that analogy.
Application scenarios 4：It is the schematic diagram of this application scene referring to shown in Fig. 4 D, is 3, LUN marks using M under this application scene
Exemplified by the quantity of OSD corresponding to knowledge is 3.Each OSD can include three subregions, and P1 subregions are main partitions, and P2 subregions are first
From subregion, P3 subregions are second from subregion.To be determined using mapping policy counterclockwise corresponding to each copy data from OSD.
Assuming that data chunk 1 includes data block 1, data block 1A and data block 1B, determine that main OSD is corresponding to data block 1
From OSD it is OSD3 corresponding to OSD1, data block 1A, from OSD is OSD2 corresponding to data block 1B, the storage of data block 1 arrives OSD1's
P1 subregions, the P2 subregions of data block 1A storages to OSD3, the P3 subregions of data block 1B storages to OSD2.Assuming that data chunk 2 is wrapped
Data block 2, data block 2A and data block 2B are included, it is OSD2 to determine main OSD corresponding to data block 2, and data block 2A is corresponding from OSD
It is OSD1, from OSD is OSD3 corresponding to data block 2B, data block 2 store OSD2 P1 subregions, and data block 2A is stored and arrived OSD1
P2 subregions, data block 2B storage to OSD3 P3 subregions.Assuming that data chunk 3 includes data block 3, data block 3A and data block
3B, determine that main OSD corresponding to data block 3 is OSD3, from OSD be OSD2 corresponding to data block 3A, from OSD corresponding to data block 3B
It is OSD1, data block 3 stores OSD3 P1 subregions, the P2 subregions of data block 3A storages to OSD2, and data block 3B storages are arrived
OSD1 P3 subregions.By that analogy.
Application scenarios 5：It is the schematic diagram of this application scene, it is assumed that OSD1, OSD2, OSD3 storage referring to shown in Fig. 4 E
Space is 2 times of OSD4 memory space, and if OSD1, OSD2, OSD3 memory space are 20G, OSD4 memory space is
10G, therefore, OSD1, OSD2, OSD3, OSD4 can be divided into same storage pool 1, storage pool 1 take OSD1, OSD2,
OSD3, OSD4 preceding 10G memory spaces, and LUN1 belongs to the storage pool 1, so, LUN1 cans occupancy OSD1, OSD2,
OSD3, OSD4 preceding 10G memory spaces, i.e. OSD quantity corresponding to LUN1 are 4.
In order to make full use of OSD storage resource, then OSD1, OSD2, OSD3 can also be divided into same storage pool
2, storage pool 2 takes OSD1, OSD2, OSD3 rear 10G memory spaces, and LUN2 belongs to storage pool 2, and so, LUN2 just takes
OSD1, OSD2, OSD3 rear 10G memory spaces, compared with LUN1, LUN2 does not correspond to OSD4, i.e. OSD quantity corresponding to LUN2
For 3.
M corresponding to LUN1 is that M corresponding to 3, LUN2 is 3, and each OSD includes three subregions, and P1 subregions are main partitions, and P2 divides
Area is first from subregion, and P3 subregions are second from subregion.
For LUN1, exemplified by determining corresponding to each copy data from OSD using mapping policy counterclockwise, then data block
Main OSD corresponding to 1 is OSD1, from OSD is OSD4 corresponding to data block 1A, from OSD is OSD3 corresponding to data block 1B, data block
1 storage stores the P3 subregions to OSD3 to OSD1 P1 subregions, the P2 subregions of data block 1A storages to OSD4, data block 1B.Number
It is OSD2 according to main OSD corresponding to block 2, from OSD is OSD1 corresponding to data block 2A, from OSD is OSD4 corresponding to data block 2B, with
This analogizes, and the processing for other data chunks repeats no more.
For LUN2, exemplified by determining corresponding to each copy data from OSD using mapping policy counterclockwise, then data block
Main OSD corresponding to 1 is OSD1, from OSD is OSD3 corresponding to data block 1A, from OSD is OSD2 corresponding to data block 1B, data block
1 storage stores the P3 subregions to OSD2 to OSD1 P1 subregions, the P2 subregions of data block 1A storages to OSD3, data block 1B, with
This analogizes.
Application scenarios 6：It is the schematic diagram of this application scene, it is assumed that OSD1, OSD2, OSD3 storage referring to shown in Fig. 4 F
Space is 2 times of OSD4 memory space, and if OSD1, OSD2, OSD3 memory space are 20G, OSD4 memory space is
10G.Under original state, OSD1, OSD2, OSD3 are divided into same storage pool 1, storage pool 1 takes OSD1, OSD2, OSD3
Preceding 10G memory spaces, and LUN1 belongs to storage pool 1, and so, LUN1 can take OSD1, OSD2, OSD3 preceding 10G storages
Space, OSD quantity corresponding to LUN1 are 3, and M corresponding to LUN1 is 3.
For LUN1, exemplified by determining corresponding to each copy data from OSD using mapping policy counterclockwise, then data block
Main OSD corresponding to 1 is OSD1, from OSD is OSD3 corresponding to data block 1A, from OSD is OSD2 corresponding to data block 1B, data block
1 storage stores the P3 subregions to OSD2 to OSD1 P1 subregions, the P2 subregions of data block 1A storages to OSD3, data block 1B, with
This analogizes.
In use, dilatation can be carried out as needed, i.e., on the basis of OSD1, OSD2, OSD3, dilatation one
OSD4, so, OSD1, OSD2, OSD3, OSD4 are divided into same storage pool 2, storage pool 2 take OSD1, OSD2, OSD3,
OSD4 rear 10G memory spaces, LUN2 belong to storage pool 2, so, after LUN2 can take OSD1, OSD2, OSD3, OSD4
OSD quantity corresponding to 10G memory spaces, i.e. LUN2 is 4, and M corresponding to LUN2 is 3.
For LUN2, exemplified by determining corresponding to each copy data from OSD using mapping policy counterclockwise, then data block
Main OSD corresponding to 1 is OSD1, from OSD is OSD4 corresponding to data block 1A, from OSD is OSD3 corresponding to data block 1B, data block
1 storage stores the P3 subregions to OSD3 to OSD1 P1 subregions, the P2 subregions of data block 1A storages to OSD4, data block 1B, with
This analogizes.
Application scenarios 7：It is the schematic diagram of this application scene referring to shown in Fig. 4 G, under original state, M 2, LUN identification pair
The OSD answered quantity is 3, exemplified by determining corresponding to each copy data from OSD using mapping policy counterclockwise, each data block
The storage of group is referring to application scenarios 3.In use, it is assumed that OSD2 is damaged, then the quantity of OSD corresponding to LUN identification
It is changed into 2, because data block 2A is located at OSD1 P2 subregions, therefore, in order to be stored separately with data block 2A, then by OSD2 P1 points
Data block 2 in area moves to OSD3 P1 subregions.Similarly, because data block 3 is located at OSD3 P1 subregions, therefore, by OSD2
P2 subregions in data block 3A move to OSD1 P2 subregions.It is the storage for two OSD in subsequent process, referring to
Application scenarios 1.
Application scenarios 8：It is the schematic diagram of this application scene referring to shown in Fig. 4 H, is 6, LUN marks using M under this application scene
The quantity of OSD corresponding to knowledge illustrates exemplified by being 6.Each OSD can include six subregions, and P1 subregions are main partitions, and P2 divides
Area's-P6 subregions are from subregion.Under this application scene, with using mapping policy counterclockwise determine corresponding to each copy data from
OSD。
For example, it is assumed that data chunk 1 includes data block 1, data block 2, data block 3, data block 4, verification data B1 and school
Test data C1, it is determined that main OSD corresponding to data block 1 (assuming that data block 1 is master data) is OSD1, corresponding to data block 2 from
OSD is OSD6, from OSD is OSD5 corresponding to data block 3, from OSD is OSD4 corresponding to data block 4, corresponding to verification data B1
It is OSD3 from OSD, from OSD is OSD2 corresponding to verification data C1.Further, data block 1 stores OSD1 P1 subregions, number
OSD6 P2 subregions are stored according to block 2, data block 3 stores OSD5 P3 subregions, and data block 4 stores OSD4 P4 subregions,
OSD3 P5 subregions, the P6 subregions of verification data C1 storages to OSD2 are arrived in verification data B1 storages.Assume further that data chunk 2 is wrapped
Include data block 5, data block 6, data block 7, data block 8, verification data B2 and verification data C2, it is determined that data block 5 is (assuming that number
Be master data according to block 5) corresponding to main OSD be OSD2, from OSD be OSD1 corresponding to data block 6, be from OSD corresponding to data block 7
OSD6, from OSD be OSD5 corresponding to data block 8, from OSD be OSD4 corresponding to verification data B2, corresponding to verification data C2 from
OSD is OSD3, by that analogy.
Step 305, by primary data store main OSD corresponding to main partition, and by copy data storage to it is corresponding from
OSD from subregion.In above process, it has been determined that go out for store master data main OSD, for stored copies data from
OSD, and each OSD includes main partition and from subregion, therefore, can be by primary data store to main OSD main partition, and by pair
Notebook data storage is arrived from OSD from subregion.
In above-mentioned 7 application scenarios, the P1 subregions of by the agency of primary data store to main OSD, each copy data store
To the P2 subregions from OSD, P3 subregions etc., it is no longer repeated herein.
Wherein, for the process of " by primary data store main OSD corresponding to main partition ", can include：By master data
Main OSD corresponding to being sent to, so that corresponding main OSD stores master data in the main partition of itself.
Wherein, for the process of " by copy data storage to corresponding from OSD from subregion ", can include but is not limited to
Following manner：Mode one, client by the copy data be sent to corresponding to from OSD so that it is corresponding from OSD itself from
The partitioned storage copy data.Or the copy data can be sent to master corresponding to the master data by mode two, client
OSD, so that corresponding main OSD stores the copy data to corresponding from OSD from subregion, for example, main OSD can be by the pair
Notebook data be sent to corresponding to from OSD so that it is corresponding from OSD itself from the partitioned storage copy data.
For " client by the copy data be sent to corresponding to from OSD so that it is corresponding from OSD itself from point
Area stores the copy data " mode, then client master data can be sent to corresponding to main OSD, and copy data is sent out
Completed corresponding to giving from OSD, the i.e. storage of master data and copy data by client, higher transmission effect can be provided
Rate, main OSD transmission pressure will not be allowed excessive, reduce I/O time delays.In correcting and eleting codes process (application scenarios 8), sequential write generation
Redundancy check block it is more efficient.
For " client by the copy data be sent to corresponding to main OSD so that corresponding main OSD is by the copy data
Store corresponding from OSD from subregion " mode, master data and copy data can be sent to corresponding master by client
OSD, i.e. master data storage are completed by client, and the storage of copy data is completed by main OSD, so, independent of client
Ensure I/O integrality, in client power down, the processing of data consistency is simpler, even if data are not the feelings of sequential write
Condition, the time delay that data are write are small.
In one example, for the master data (such as data block or verification data) and number of copies in all data chunks
According to (such as data block or verification data), when by these data Cun Chudao OSD, each data take an OSD storage
Space, the size of sub- memory space is identical with the size of data block, is such as 4M.Moreover, each occupied sub- storages of OSD are empty
Between be it is continuous, the sub- memory space of each of which be used for store a data.Wherein, OSD can provide a full storage space,
Such as 10G memory spaces.In order to store 4M data block, then it is empty can be divided into small storage one by one for complete memory space
Between, each small memory space can be the sub- memory space of 4M sizes, corresponding with data block size.It is convenient to distinguish, by the 4M
The memory space of size is referred to as sub- memory space, and every sub- memory space stores a data block.
For example, it is assumed that the quantity of the occupied sub- memory spaces of OSD1 is the number of the occupied sub- memory spaces of 25, OSD2
The quantity measured as the occupied sub- memory spaces of 25, OSD3 is that the quantity of the occupied sub- memory spaces of 25, OSD4 is 25, then
OSD1, can (such as OSD1 preceding 100 sub- memory spaces be occupied since being originated free address after data block is received
When, then it can be the 101st sub- memory space to originate free address), use continuous 25 sub- memory space (such as the 101st sons
Memory space is to 125 sub- memory spaces) master data or copy data in data storage block group.Similarly, OSD2 is being received
To after data block, can be deposited since OSD2 starting free address (such as the 101st sub- memory space) using continuous 25 sons
Store up the master data or copy data in the data storage block group of space；By that analogy.
Step 306, monitoring server is arrived into metadata storage corresponding to data file.
In one example, after using multiple data blocks corresponding to aforesaid way data storage file, can also obtain
Metadata corresponding to data file, and monitoring server is arrived into metadata storage corresponding to data file.
Wherein, the metadata can include but is not limited to：OSD, multiple data blocks exist corresponding to LUN identification, LUN identification
OSD initial address, end address etc., are not limited to this metadata.
Wherein, metadata corresponding to data file can be sent to monitoring server by client, be existed by monitoring server
The metadata is stored in local database.Moreover, database can be separately positioned in a high-performance disk so that data
Storehouse will not turn into performance bottleneck.Therefore, the metadata of whole data file is only stored, greatly reduces metadata quantity, saves member
The storage resource that data take.Being additionally, since only needs the metadata that whole data file is read from monitoring server, without reading
A large amount of metadata are taken, so as to mitigate the processing pressure of client, mitigates the access frequency of metadata, improves obtaining for metadata
Efficiency is taken, mitigates the complexity of metadata management.
When needing to read data file, client can also first obtain metadata corresponding to data file, and be based on institute
State metadata and read data file from each OSD.For example, the initial address based on OSD1 is (such as the 101st sub- memory space
Initial address) and end address (end address of such as the 125th sub- memory space), client can read out 25 from OSD1
Individual data block, by that analogy, this reading process is not limited.Then, client can by all data chunks of reading into
Above-mentioned data file.
In the above-described embodiments, it is to use RAID modes (such as RAID1, RAID5, RAID6) data storage block, and uses
The reason for RAID mode data storage blocks is：If not using RAID mode data storage blocks, when some OSD breaks down, deposit
The data block stored up in the OSD can lose, and cause data block unavailable.By using RAID mode data storage blocks, when certain
When individual OSD breaks down, data block can be avoided to lose, data block can be recovered.For example, when the son where data block is deposited
, can be with the data block of Recover from damaging when storing up space damage.For example, as shown in Figure 4 A, when the number stored in OSD1 P1 subregions
When being damaged according to block 1, data block 1A corresponding to the data block 1 can be read from OSD2 P2 subregions, due to data block 1A with
Data block 1 is identical, data block 1A can be used to recover data block 1.
In summary, aforesaid way can realize that the redundancy backup of data (realizes the data point of redundancy by more copies
Cloth strategy), there is provided higher data reliability, avoid unavailable problem caused by loss of data.Moreover, for using RAID1
Mirror-image fashion, the correcting and eleting codes mode using RAID5/RAID6, the storage side of the data redundancy of direct addressin can be provided
Method, while data efficient access is provided, there is provided data high reliability.Due to not using file system as flexible data
The support of storage, but directly use address of cache, i.e. the address of data storage be it is fixed in advance, therefore, can be according to solid
Master data and copy data are mapped directly to specific subregion, so as to complete by OSD subregions by fixed strategy according to mapping techniques
Data redundancy is distributed, and realizes the metadata of discharge data object, and the purpose of addressing is directly mapped using data.
In one example, when data storage block 1 in the P1 subregions in OSD1, in OSD2 P2 partitioned storage data blocks 1A
When, if the data block 1 in OSD1 P1 subregions is damaged, then, how from OSD2 P2 subregions to inquire the correspondence of data block 1
Data block 1A, then following means are also proposed in the embodiment of the present application：In each OSD corresponding to LUN identification, main partition is deposited
Storage capacity is identical with the memory capacity from subregion, in addition, being directed to each data chunk, the master data in the data chunk is in master
The offset in OSD main partition is identical in the offset from subregion from OSD with the copy data in the data chunk.
Referring to shown in Fig. 4 A, memory capacity, the memory capacity of P2 subregions of OSD1 P1 subregions, OSD2 P1 subregions are deposited
Capacity, the memory capacity of P2 subregions are stored up, can be with all same.If data block 1 stores the 100th son storage of OSD1 P1 subregions
Space, then data block 1A is 100th sub- memory space of the storage to OSD2 P2 subregions.If the storage of data block 2 arrives OSD2's
100th sub- memory space of P1 subregions, then data block 2A is 100th sub- memory space of the storage to OSD1 P2 subregions.
If data block 3 stores the 101st sub- memory space of OSD1 P1 subregions, data block 3A is P2 subregion of the storage to OSD2
The 101st sub- memory space.If data block 4 stores the 101st sub- memory space of OSD2 P1 subregions, data block 4A
It is 101st sub- memory space of the storage to OSD1 P2 subregions.By that analogy.For application scenarios such as Fig. 4 B- Fig. 4 H, realize
Process is similar, repeats no more.
In summary, when some data block breaks down, can also quick search to it is corresponding with the data block association
Data, then recovered using associated data.For example, when the data block 1 in OSD1 P1 subregions is damaged, due to number
The 100th sub- memory space of OSD1 P1 subregions is stored according to block 1, therefore, can directly arrive the 100th of OSD2 P2 subregions
Individual sub- memory space inquiry and 1 corresponding associated data of data block, i.e. data block 1A.In another example when in OSD2 P1 subregions
, therefore, can be with because data block 4 stores the 101st sub- memory space of OSD2 P1 subregions when data block 4 is damaged
Directly arrive the 101st of OSD1 P2 subregions memory space inquiry and 4 corresponding associated data of data block, i.e. data block 4A,
By that analogy.I.e. by make " master data in data chunk main OSD main partition offset, and in the data chunk
Copy data is identical in the offset from subregion from OSD ", then can be with incidence number corresponding to quick search to each data block
According to.
Therefore, by arrange the main OSD of master data, copy data from OSD, master data is in main OSD subregion, number of copies
According in the subregion from OSD, for master data in the offset of main OSD subregion, copy data, can in the offset of the subregion from OSD
With the inter-related master data of fast positioning and copy data；For master data, copy corresponding to the master data can be inquired
The storage location of data, for copy data, the storage location of master data corresponding to the copy data can also be inquired, is realized
The mapping addressing of data.
In one example, the direct mapping addressing system in this programme, compared with SAN storage modes, its difference is：
When SAN storage modes use RAID stripe, data and verification data cross occurrence in data file, order read-write property is influenceed
Can, and it is relatively low to pre-read operating characteristics.In this programme, the more excellent subregion mapping addressing system of use, master data, each copy data
All it is the subregion for being continually stored in same offset, so, can greatly promotes and user's order is read, user's sequential write and pre-
The performance of read operation.
Conceived based on the application same with the above method, a kind of data storage device also proposed in the embodiment of the present application,
Client is can apply to, as shown in figure 5, being the structural representation of the device, the device includes：
Acquisition module 501, for obtaining data file to be stored, LUN identification corresponding to the data file；According to cluster
View information determines multiple OSD corresponding to the LUN identification, and the multiple OSD quantity is not less than corresponding to the LUN identification
Data trnascription number M；Cluster view information is obtained from monitoring server, and cluster view information includes LUN identification and OSD pair
It should be related to, M is the positive integer more than 1；
Generation module 502, for the data file to be split into multiple data blocks, and given birth to according to the multiple data block
Into multiple data chunks；Wherein, the data number of blocks in each data chunk is M, and each data chunk includes a master data
With M-1 copy data；
Processing module 503, for for each data chunk, determining to be used to store the data from the multiple OSD
The main OSD of master data in block group and the copy data in the data chunk from OSD, and by the primary data store to pair
The main OSD answered main partition, by copy data storage to corresponding from OSD from subregion；Each OSD includes one main point
Area and M-1 are individual from subregion；
Sending module 504, for metadata storage corresponding to the data file to be arrived into monitoring server.
The generation module 502, specifically for during multiple data chunks are generated according to the multiple data block,
A data chunk is generated for each data block；In the data chunk, master data is the data block, copy data be for
The M-1 data block that the data block replicates；Or
N number of data block is chosen from the multiple data block, determines verification data corresponding to N number of data block, by institute
State N number of data block and the verification data forms a data chunk；In the data chunk, master data is the data block
Any data in group, other data outside master data are copy data, and N is the positive integer more than 1, and the N is less than institute
State M.
The processing module 503, specifically for being determined from the multiple OSD for storing in the data chunk
Copy data during the OSD, multiple OSD corresponding to the LUN identification are ranked up；Sequence based on multiple OSD
As a result, since main OSD corresponding to the master data, determined using mapping policy counterclockwise or mapping policy clockwise each
From OSD corresponding to copy data；
The mapping policy counterclockwise is used for since the previous OSD of main OSD corresponding to the master data, forward successively
It is determined that from OSD corresponding to each copy data, different copy datas correspond to different from OSD；For first in ranking results
Individual OSD, its previous OSD are last OSD in ranking results；
The mapping policy clockwise is used for since the latter OSD of main OSD corresponding to the master data, backward successively
It is determined that from OSD corresponding to each copy data, different copy datas correspond to different from OSD；For last in ranking results
One OSD, its latter OSD are first OSD in ranking results.
In each OSD corresponding to the LUN identification, the memory capacity in main partition is identical with the memory capacity from subregion；
For each data chunk, master data in the data chunk main OSD main partition offset, with the data chunk
In copy data it is identical in the offset from subregion from OSD.
In one example, the processing module 503, specifically for by the primary data store to corresponding to main OSD
Main partition during, by the master data be sent to corresponding to main OSD so that corresponding main OSD is in the main partition of itself
Store the master data；
During corresponding to arriving copy data storage from OSD from subregion, the copy data is sent
To corresponding from OSD so that it is corresponding from OSD itself from copy data described in partitioned storage；Or by the number of copies
According to main OSD corresponding to the master data is sent to, so that corresponding main OSD stores the copy data to corresponding from OSD
From subregion.
The client provided in the embodiment of the present application, for hardware view, its hardware structure schematic diagram can specifically join
As shown in Figure 6.It can include：Machinable medium and processor, wherein：
Machinable medium：Store instruction code.
Processor：Communicate, stored in reading and execution machinable medium described with machinable medium
Instruction code, realize data storage operations disclosed in the application above-mentioned example.
Here, machinable medium can be any electronics, magnetic, optics or other physical storage devices, can be with
Comprising or storage information, such as executable instruction, data, etc..For example, machinable medium can be：RAM(Radom
Access Memory, random access memory), volatile memory, nonvolatile memory, flash memory, memory driver is (as hard
Disk drive), solid state hard disc, any kind of storage dish (such as CD, dvd), either similar storage medium or they
Combination.
System, device, module or the unit that above-described embodiment illustrates, it can specifically be realized by computer chip or entity,
Or realized by the product with certain function.One kind typically realizes that equipment is computer, and the concrete form of computer can
To be personal computer, laptop computer, cell phone, camera phone, smart phone, personal digital assistant, media play
In device, navigation equipment, E-mail receiver/send equipment, game console, tablet PC, wearable device or these equipment
The combination of any several equipment.
For convenience of description, it is divided into various units during description apparatus above with function to describe respectively.Certainly, this is being implemented
The function of each unit can be realized in same or multiple softwares and/or hardware during application.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the embodiment of the present application can use wherein includes computer usable program code in one or more
The computer that computer-usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of program product.
The application is with reference to the flow according to the method for the embodiment of the present application, equipment (system) and computer program product
Figure and/or block diagram describe.It is generally understood that by each in computer program instructions implementation process figure and/or block diagram
Flow and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer journeys can be provided
Sequence instruction to all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices processor with
Produce a machine so that produce and be used for by the instruction of computer or the computing device of other programmable data processing devices
Realize the dress for the function of being specified in one flow of flow chart or multiple flows and/or one square frame of block diagram or multiple square frames
Put.
Moreover, these computer program instructions can also be stored in computer or other programmable datas can be guided to handle and set
In the standby computer-readable memory to work in a specific way so that the instruction being stored in the computer-readable memory produces
Manufacture including command device, the command device are realized in one flow of flow chart or multiple flows and/or block diagram one
The function of being specified in individual square frame or multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, so as in computer
Or the instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram
The step of function of being specified in one square frame or multiple square frames.
Embodiments herein is the foregoing is only, is not limited to the application.For those skilled in the art
For, the application can have various modifications and variations.All any modifications made within spirit herein and principle, it is equal
Replace, improve etc., it should be included within the scope of claims hereof.
Claims (10)
1. a kind of date storage method, it is characterised in that applied to client, methods described includes：
Obtain data file to be stored, LUN LUN identification corresponding to the data file；
Multiple object storage device OSD, the multiple OSD number according to corresponding to cluster view information determines the LUN identification
Amount is not less than data trnascription number M corresponding to the LUN identification；The cluster view information is obtained from monitoring server, collection
Group's view information includes LUN identification and OSD corresponding relation, and M is the positive integer more than 1；
The data file is split into multiple data blocks, and multiple data chunks are generated according to the multiple data block；Wherein,
Data number of blocks in each data chunk is M, and each data chunk includes a master data and M-1 copy data；
For each data chunk, the main OSD for storing the master data in the data chunk is determined from the multiple OSD
, will with the copy data in the data chunk from OSD, and by the primary data store to corresponding main OSD main partition
Copy data storage is to corresponding from OSD from subregion；Wherein, each OSD includes a main partition and M-1 from points
Area；
The monitoring server is arrived into metadata storage corresponding to the data file.
2. according to the method for claim 1, it is characterised in that
The process that multiple data chunks are generated according to the multiple data block, is specifically included：
A data chunk is generated for each data block；In the data chunk, master data is the data block, copy data
It is the M-1 data block replicated for the data block；Or
N number of data block is chosen from the multiple data block, determines verification data corresponding to N number of data block, will be described N number of
Data block and the verification data form a data chunk；In the data chunk, master data is in the data chunk
Any data, other data outside master data are copy data, and N is the positive integer more than 1, and the N is less than the M.
3. method according to claim 1 or 2, it is characterised in that described to determine to be used to store institute from the multiple OSD
The process from OSD of the copy data in data chunk is stated, is specifically included：
Multiple OSD corresponding to the LUN identification are ranked up；
Based on multiple OSD ranking results, since main OSD corresponding to the master data, using mapping policy counterclockwise or
Mapping policy clockwise is determined corresponding to each copy data from OSD；
The mapping policy counterclockwise is used for since the previous OSD of main OSD corresponding to the master data, determines successively forward
From OSD corresponding to each copy data, different copy datas correspond to different from OSD；For first in ranking results
OSD, its previous OSD are last OSD in ranking results；
The mapping policy clockwise is used for since the latter OSD of main OSD corresponding to the master data, determines successively backward
From OSD corresponding to each copy data, different copy datas correspond to different from OSD；For last in ranking results
OSD, its latter OSD are first OSD in ranking results.
4. according to the method for claim 1, it is characterised in that
In each OSD corresponding to the LUN identification, the memory capacity in main partition is identical with the memory capacity from subregion；For
Each data chunk, master data in the data chunk main OSD main partition offset, and in the data chunk
Copy data is identical in the offset from subregion from OSD.
5. according to the method for claim 1, it is characterised in that
By the process in the primary data store main OSD main partition corresponding to, specifically include：
The master data is sent to corresponding main OSD, so that corresponding main OSD stores the main number in the main partition of itself
According to；
By copy data storage to the corresponding process from subregion from OSD, specifically include：
By the copy data be sent to corresponding to from OSD so that it is corresponding from OSD itself from copy described in partitioned storage
Data；Or the copy data is sent to main OSD corresponding to the master data, so that corresponding main OSD is by the copy
From OSD from subregion corresponding to data Cun Chudao.
6. a kind of data storage device, it is characterised in that applied to client, described device includes：
Acquisition module, for obtaining data file to be stored, LUN LUN identification corresponding to the data file；Root
Determine that multiple object storage device OSD, the multiple OSD quantity is not small corresponding to the LUN identification according to cluster view information
In data trnascription number M corresponding to the LUN identification；The cluster view information is obtained from monitoring server, the cluster
View information includes LUN identification and OSD corresponding relation, and M is the positive integer more than 1；
Generation module, for the data file to be split into multiple data blocks, and it is multiple according to the generation of the multiple data block
Data chunk；Wherein, the data number of blocks in each data chunk is M, and each data chunk includes a master data and M-1
Copy data；
Processing module, for for each data chunk, being determined from the multiple OSD for storing in the data chunk
The main OSD of master data and being led from OSD, and by the primary data store corresponding to for the copy data in the data chunk
OSD main partition, by copy data storage to corresponding from OSD from subregion；Wherein, each OSD includes one main point
Area and M-1 are individual from subregion；
Sending module, for metadata storage corresponding to the data file to be arrived into the monitoring server.
7. device according to claim 6, it is characterised in that
The generation module, specifically for being each during multiple data chunks are generated according to the multiple data block
Data block generates a data chunk；In the data chunk, master data is the data block, and copy data is for the number
The M-1 data block replicated according to block；Or
N number of data block is chosen from the multiple data block, determines verification data corresponding to N number of data block, will be described N number of
Data block and the verification data form a data chunk；In the data chunk, master data is in the data chunk
Any data, other data outside master data are copy data, and N is the positive integer more than 1, and the N is less than the M.
8. the device according to claim 6 or 7, it is characterised in that
The processing module, specifically for being determined from the multiple OSD for storing the number of copies in the data chunk
According to during the OSD, multiple OSD corresponding to the LUN identification are ranked up；Based on multiple OSD ranking results, from
Main OSD starts corresponding to the master data, and each number of copies is determined using mapping policy counterclockwise or mapping policy clockwise
According to corresponding from OSD；
The mapping policy counterclockwise is used for since the previous OSD of main OSD corresponding to the master data, determines successively forward
From OSD corresponding to each copy data, different copy datas correspond to different from OSD；For first in ranking results
OSD, its previous OSD are last OSD in ranking results；
The mapping policy clockwise is used for since the latter OSD of main OSD corresponding to the master data, determines successively backward
From OSD corresponding to each copy data, different copy datas correspond to different from OSD；For last in ranking results
OSD, its latter OSD are first OSD in ranking results.
9. device according to claim 6, it is characterised in that
In each OSD corresponding to the LUN identification, the memory capacity in main partition is identical with the memory capacity from subregion；For
Each data chunk, master data in the data chunk main OSD main partition offset, and in the data chunk
Copy data is identical in the offset from subregion from OSD.
10. device according to claim 6, it is characterised in that the processing module, specifically for by the master data
During the main partition for storing corresponding main OSD, the master data is sent to corresponding main OSD, so that corresponding master
OSD stores the master data in the main partition of itself；
By copy data storage to it is corresponding from OSD from subregion during, the copy data is sent to pair
Answer from OSD so that it is corresponding from OSD itself from copy data described in partitioned storage；Or the copy data is sent out
Give main OSD corresponding to the master data so that corresponding main OSD by copy data storage to it is corresponding from OSD from
Subregion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711043732.9A CN107729536B (en) | 2017-10-31 | 2017-10-31 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711043732.9A CN107729536B (en) | 2017-10-31 | 2017-10-31 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107729536A true CN107729536A (en) | 2018-02-23 |
| CN107729536B CN107729536B (en) | 2020-09-08 |
Family
ID=61202434
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711043732.9A Active CN107729536B (en) | 2017-10-31 | 2017-10-31 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107729536B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710456A (en) * | 2018-12-10 | 2019-05-03 | 新华三技术有限公司 | A kind of data reconstruction method and device |
| CN112115001A (en) * | 2020-09-18 | 2020-12-22 | 深圳市欢太科技有限公司 | Data backup method and device, computer storage medium and electronic equipment |
| CN112486411A (en) * | 2020-11-25 | 2021-03-12 | 湖南麒麟信安科技股份有限公司 | Method, device and medium for distributing disks of distributed file system in super-fusion environment |
| CN112711376A (en) * | 2019-10-25 | 2021-04-27 | 北京金山云网络技术有限公司 | Method and device for determining object master copy file in object storage system |
| CN114780987A (en) * | 2021-12-29 | 2022-07-22 | 张海滨 | Data distribution, storage, reading and transmission method and distributed system |
| CN116820286A (en) * | 2023-08-24 | 2023-09-29 | 深圳市嘉利达专显科技有限公司 | Control method and system of multi-window display |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032186A (en) * | 2021-05-24 | 2021-06-25 | 广东睿江云计算股份有限公司 | Data storage method and system based on raid and ceph |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103778255A (en) * | 2014-02-25 | 2014-05-07 | 深圳市中博科创信息技术有限公司 | Distributed file system and data distribution method thereof |
| CN103797770A (en) * | 2012-12-31 | 2014-05-14 | 华为技术有限公司 | Method and system for sharing storage resources |
| US20150301759A1 (en) * | 2012-12-31 | 2015-10-22 | Huawei Technologies Co., Ltd. | Method and system for sharing storage resource |
| CN106339181A (en) * | 2016-08-19 | 2017-01-18 | 华为技术有限公司 | Method and system for processing data in storage system |
-
2017
- 2017-10-31 CN CN201711043732.9A patent/CN107729536B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103797770A (en) * | 2012-12-31 | 2014-05-14 | 华为技术有限公司 | Method and system for sharing storage resources |
| US20150301759A1 (en) * | 2012-12-31 | 2015-10-22 | Huawei Technologies Co., Ltd. | Method and system for sharing storage resource |
| CN103778255A (en) * | 2014-02-25 | 2014-05-07 | 深圳市中博科创信息技术有限公司 | Distributed file system and data distribution method thereof |
| CN106339181A (en) * | 2016-08-19 | 2017-01-18 | 华为技术有限公司 | Method and system for processing data in storage system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710456A (en) * | 2018-12-10 | 2019-05-03 | 新华三技术有限公司 | A kind of data reconstruction method and device |
| CN109710456B (en) * | 2018-12-10 | 2021-03-23 | 新华三技术有限公司 | Data recovery method and device |
| CN112711376A (en) * | 2019-10-25 | 2021-04-27 | 北京金山云网络技术有限公司 | Method and device for determining object master copy file in object storage system |
| CN112711376B (en) * | 2019-10-25 | 2022-12-23 | 北京金山云网络技术有限公司 | Method and device for determining object master copy file in object storage system |
| CN112115001A (en) * | 2020-09-18 | 2020-12-22 | 深圳市欢太科技有限公司 | Data backup method and device, computer storage medium and electronic equipment |
| CN112486411A (en) * | 2020-11-25 | 2021-03-12 | 湖南麒麟信安科技股份有限公司 | Method, device and medium for distributing disks of distributed file system in super-fusion environment |
| CN114780987A (en) * | 2021-12-29 | 2022-07-22 | 张海滨 | Data distribution, storage, reading and transmission method and distributed system |
| CN114780987B (en) * | 2021-12-29 | 2023-08-29 | 张海滨 | Data distribution, storage, reading and transmission method and distributed system |
| CN116820286A (en) * | 2023-08-24 | 2023-09-29 | 深圳市嘉利达专显科技有限公司 | Control method and system of multi-window display |
| CN116820286B (en) * | 2023-08-24 | 2024-01-19 | 深圳市嘉利达专显科技有限公司 | Control method and system of multi-window display |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107729536B (en) | 2020-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10983860B2 (en) | Automatic prefill of a storage system with conditioning of raid stripes | |
| US11687423B2 (en) | Prioritizing highly performant storage systems for servicing a synchronously replicated dataset | |
| US11581943B2 (en) | Queues reserved for direct access via a user application | |
| CN107729536A (en) | A kind of date storage method and device | |
| CN107807794A (en) | A kind of date storage method and device | |
| US10372541B2 (en) | Storage device storing data using raid | |
| US11481121B2 (en) | Physical media aware spacially coupled journaling and replay | |
| US20190065093A1 (en) | Automatic load rebalancing of a write group | |
| US20200034231A1 (en) | Efficient computation of parity data in storage system implementing data striping | |
| US10852966B1 (en) | System and method for creating mapped RAID group during expansion of extent pool | |
| CN106687911B (en) | Online data movement without compromising data integrity | |
| US10915813B2 (en) | Search acceleration for artificial intelligence | |
| CN111158587B (en) | Distributed storage system based on storage pool virtualization management and data read-write method | |
| US10521152B2 (en) | Storage device storing data in raid manner | |
| US8639898B2 (en) | Storage apparatus and data copy method | |
| CN108064374B (en) | Data access method, device and system | |
| CN103763383A (en) | Integrated cloud storage system and storage method thereof | |
| CN102981927A (en) | Distribution type independent redundant disk array storage method and distribution type cluster storage system | |
| US20210034267A1 (en) | Memory-efficient processing of raid metadata bitmaps | |
| CN103534688A (en) | Data recovery method, storage equipment and storage system | |
| KR20160081851A (en) | Computing system with distributed compute-enabled storage group and operating method thereof | |
| CN108228085A (en) | For managing the method and apparatus of storage system | |
| CN106528338A (en) | Remote data replication method, storage equipment and storage system | |
| CN103617010A (en) | Data storage method and storage device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |